QLoRA
Beginner
A trick to train a big model on one computer: squeeze its numbers small, then train a small add-on.
Novice
LoRA on a model whose frozen weights are stored in 4 bits instead of 16. The model takes about a quarter of the memory, and only the small LoRA add-on is trained, at normal precision.
Expert
4-bit NF4 storage of the frozen base with double quantization of the scale constants, BF16 compute (weights dequantized on the fly), LoRA adapters on all linear layers, and paged optimizer state. Took 65B fine-tuning from over 780 GB to under 48 GB.