Glossary

LoRA (low-rank adaptation)

Beginner

A cheap way to retrain a big model: freeze it and train two thin add-on pieces.

Novice

A fine-tuning method that freezes the original weights and learns a small correction for some of them, written as the product of two thin matrices. Far fewer numbers are trained and stored.

Expert

W+BAW + BA with B∈Rd×rB \in \mathbb{R}^{d \times r}, A∈Rr×kA \in \mathbb{R}^{r \times k}, r≪dr \ll d; B starts at zero. Merged into W after training, so inference costs nothing extra.