Cross-entropy loss
Beginner
A score for how surprised the model was by the real next word. Lower is better.
Novice
The training score for a language model: minus the log of the probability it gave the correct next token, averaged over all positions. A perfect guess scores 0; a confident wrong guess scores high.
Expert
; its gradient with respect to the logits is . is the perplexity; a uniform guess over V tokens gives .
See also: Logits, Backpropagation.