What is LoRA fine-tuning?

LoRA was introduced in the paper "LoRA: Low-Rank Adaptation of Large Language Models" by Edward Hu, Yelong Shen, and collaborators at Microsoft Research (arXiv:2106.09685, June 2021). The core insight: when you adapt a large pretrained model to a downstream task, the change in weights has low "intrinsic rank" — meaning the update matrix can be approximated by the product of two much smaller matrices. Instead of updating the full weight matrix W (which for GPT-3 175B is enormous), LoRA freezes W and learns two small matrices A and B such that the effective weight becomes W + BA, where BA is the low-rank update.

Concretely, if W is a d × k matrix, LoRA decomposes the update ΔW = BA where B is d × r and A is r × k, with rank r much smaller than min(d, k). Typical rank values in practice are 4, 8, 16, or 64. For GPT-3 175B, the authors report reducing trainable parameters by 10,000x (from 175B to 17.5M when applying LoRA only to attention query and value projections at rank 4) and reducing the optimizer-state memory footprint by 3x, all while matching or beating full fine-tuning on GLUE, WikiSQL, and SAMSum benchmarks.

LoRA has become the default fine-tuning method for open-weight LLMs because of three properties. First, no inference latency penalty: at deployment, you can merge BA back into W (W' = W + BA), so the served model has identical FLOPs to the base. Second, task-switching is cheap: you keep one frozen base model in GPU memory and swap small LoRA adapters (often 10-200 MB) per task. Third, it composes with quantization. The QLoRA paper (Dettmers et al., arXiv:2305.14314, May 2023) showed you can fine-tune a 65B-parameter model on a single 48GB GPU by quantizing the frozen base to 4-bit NF4 while keeping LoRA adapters in 16-bit, with no measurable quality loss versus 16-bit full fine-tuning on Vicuna evaluation.

The Hugging Face PEFT library (github.com/huggingface/peft) provides the reference open-source implementation, and the method is supported natively in major training stacks including Hugging Face Transformers, PyTorch Lightning, NVIDIA NeMo, and DeepSpeed. Apple's MLX framework, Meta's torchtune, and Microsoft's DeepSpeed-Chat all ship LoRA as a first-class fine-tuning path. As of 2024-2025, follow-on variants include DoRA (Weight-Decomposed Low-Rank Adaptation, arXiv:2402.09353), LoRA+ (arXiv:2402.12354) which uses different learning rates for A and B, and VeRA (arXiv:2310.11454) which shares the random projection matrices across layers to shrink adapter size further.

Practical hyperparameters matter. The two main LoRA knobs are rank r (capacity of the adapter) and alpha (scaling factor; the effective update is (alpha/r) × BA). Common configurations: r=8, alpha=16 for general instruction tuning; r=16-64 for domain adaptation; r=4 for style transfer. Target modules typically include attention query and value projections (the original paper's setting); adding key, output, and MLP projections increases capacity at the cost of more parameters. Learning rates are typically 1e-4 to 5e-4, roughly 10x higher than full fine-tuning.

LoRA is not a free lunch. It underperforms full fine-tuning when the target task requires substantial behavioral shift from the base model (e.g., teaching new languages from scratch), and the choice of rank, alpha, and target modules requires tuning. For continued pretraining on large new corpora, full fine-tuning or higher-rank LoRA (r=128+) is generally preferred.