PPO (proximal policy optimization)
Beginner
A popular way to train AI by trial and error that takes only small, careful steps each time.
Novice
A reinforcement-learning algorithm that improves the model while keeping each update close to the previous version. For language models it also trains a second network, the value model, to judge partial answers.
Expert
Policy gradient with a clipped probability ratio, so the objective stops rewarding moves outside . In LLM training, advantages usually come from a learned value model via GAE, typically about as large as the policy.