DPO (direct preference optimization)
Beginner
A simpler way to learn from people’s votes, without a separate scoring model.
Novice
A way to tune a model on pairs of better and worse answers directly, with an ordinary training loss, instead of training a reward model and running reinforcement learning.
Expert
A logistic loss on the difference of policy-to-reference log-ratios for chosen and rejected responses; the implicit reward is .
See also: RLHF.