Glossary

Data contamination

Beginner

When the test questions, or their answers, were already in what the AI studied, so a high score may mean it remembered instead of reasoned.

Novice

Test problems, or their answers, ending up in a model’s training data. The model may then score well by recall, which inflates the result.

Expert

Hard to rule out when the training data is undisclosed and the benchmark came from a public website. Detection uses statistical signals such as Min-K% Prob or how repetitive the model’s samples are; mitigations include held-out or freshly written problems.

Explained in AI in EDA (Extras).

See also: pass@k.

All 896 terms →