Beginner

Training a model with people’s votes on which of its answers is better.

Novice

Reinforcement learning from human feedback: people rank model answers, a reward model learns to predict their rankings, and the language model is trained to earn a high reward.

Expert

SFT → reward model on pairwise preferences → policy optimization (PPO) with a KL penalty to the SFT model. Effective but complex and sensitive to reward hacking.