Glossary

DPO (direct preference optimization)

Beginner

A simpler way to learn from people’s votes, without a separate scoring model.

Novice

A way to tune a model on pairs of better and worse answers directly, with an ordinary training loss, instead of training a reward model and running reinforcement learning.

Expert

A logistic loss on the difference of policy-to-reference log-ratios for chosen and rejected responses; the implicit reward is βlog⁡(π/πref)\beta \log(\pi / \pi_{\mathrm{ref}}).