Glossary

Cross-entropy loss

Beginner

A score for how surprised the model was by the real next word. Lower is better.

Novice

The training score for a language model: minus the log of the probability it gave the correct next token, averaged over all positions. A perfect guess scores 0; a confident wrong guess scores high.

Expert

L=−1T∑tlog⁡p(xt+1∣x≤t)\mathcal{L} = -\tfrac{1}{T}\sum_t \log p(x_{t+1} \mid x_{\le t}); its gradient with respect to the logits is p−yp - y. eLe^{\mathcal{L}} is the perplexity; a uniform guess over V tokens gives ln⁡V\ln V.