Glossary

GRPO (group relative policy optimization)

Beginner

A way to train a thinking AI: it tries one question many times, and each try is graded against the group’s average.

Novice

A reinforcement-learning method that samples a group of answers to each question and rewards each one by how much better it did than the group average. It needs no separate value model.

Expert

PPO-style clipped updates with the advantage of each sample set to its reward normalized by the group’s mean and standard deviation, plus a KL penalty to a reference model in the loss. Introduced in DeepSeekMath, used for DeepSeek-R1.