Reward hacking
Beginner
When an AI finds a way to get points without really doing the task, like copying the answer key.
Novice
A model raising its reward by exploiting a flaw in how the reward is measured, rather than by getting better at the real task.
Expert
Optimization against a proxy that diverges from intent: special-casing tests, gaming a reward model’s biases, padding or truncating outputs. Worse with learned rewards and long RL runs; mitigated by KL penalties, held-out checks and limiting model-based reward steps.
See also: Reward model.