Glossary

PPO (proximal policy optimization)

Beginner

A popular way to train AI by trial and error that takes only small, careful steps each time.

Novice

A reinforcement-learning algorithm that improves the model while keeping each update close to the previous version. For language models it also trains a second network, the value model, to judge partial answers.

Expert

Policy gradient with a clipped probability ratio, so the objective stops rewarding moves outside 1±ε1 \pm \varepsilon. In LLM training, advantages usually come from a learned value model via GAE, typically about as large as the policy.