Glossary

pass@k

Beginner

A score for code-writing AI: the chance that at least one of several tries passes the tests.

Novice

A score for code-writing models: the chance that at least one of kk attempts passes a problem’s tests, averaged over problems. pass@1 is the first-try success rate.

Expert

Estimated without bias by drawing n≥kn \ge k samples, counting the cc that pass, and averaging 1−(n−ck)/(nk)1 - \binom{n-c}{k} \big/ \binom{n}{k} over problems. It assumes something can pick out the passing attempt, and it is only as strong as the tests behind it.

Explained in AI in EDA (Extras).

See also: Data contamination.

All 896 terms →