RLHF
Beginner
Training a model with people’s votes on which of its answers is better.
Novice
Reinforcement learning from human feedback: people rank model answers, a reward model learns to predict their rankings, and the language model is trained to earn a high reward.
Expert
SFT → reward model on pairwise preferences → policy optimization (PPO) with a KL penalty to the SFT model. Effective but complex and sensitive to reward hacking.