Glossary

Softmax

Beginner

A math step that turns a list of scores into chances that add up to 100%.

Novice

A function that turns a list of scores into probabilities: it exponentiates each score and divides by the total, so bigger scores get a bigger share and everything sums to 1.

Expert

softmax(z)i=ezi/∑jezj\mathrm{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}. Invariant to adding a constant, so implementations subtract the max for stability. Used for attention weights and for the next-token distribution.