LoRA: Low-Rank Adaptation of Large Language Models

Full fine-tuning of a large language model means storing a second copy of every weight for each downstream task — prohibitive at GPT-3 scale. Adapter layers help, but many insert extra depth and add inference latency.

LoRA freezes the pretrained matrix $W_0$ and learns a low-rank update $\Delta W = BA$ with rank $r \ll \min(d, k)$. Only $A$ and $B$ train. At inference you can merge $W = W_0 + BA$ so deployed latency matches the base model, or keep adapters swappable per task.

What this paper explains

For a dense multiply $h = W_0 x$ with $W_0 \in \mathbb{R}^{d \times k}$, LoRA uses:

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$. Typical init: $A$ Gaussian, $B$ zero, so $\Delta W$ starts at zero. A scalar $\alpha / r$ can scale the update for tuning stability.

They inject LoRA primarily into attention projection matrices (e.g. $W_q$, $W_v$ in Transformer blocks), keeping MLPs frozen in many experiments — a practical sweet spot between quality and parameter count.

Prior limits

• Full fine-tuning — best flexibility, worst storage/multi-tenant cost.
• Adapter modules (Houlsby et al.) — small bottleneck layers; extra depth → some latency unless carefully fused.
• Prefix / prompt tuning — very few parameters, but can lag full FT on harder tasks and still add sequence overhead.
• Sparse / difference editing — related goals, different inductive biases.

The mechanism

The hypothesis: adaptation updates to big models have low intrinsic rank. Instead of estimating a full $\Delta W$, constrain it to rank $r$. Gradient updates flow only into $A$ and $B$; $W_0$ stays fixed (and can sit in pure FP16/INT8 storage shared across tasks).

Task switching: load a small $(A,B)$ pair. Serving many customers becomes a memory problem about adapter sets, not full model replicas.

Algorithm / figure walkthrough

1. Freeze $W_0$.
2. Choose which matrices get LoRA (often $q$ and $v$ projections).
3. Pick rank $r$ and scaling $\alpha$.
4. Train on downstream data; optimize only $A,B$ (and any task head).
5. Deploy: either merge $BA$ into $W_0$ or keep separate for hot-swapping.

The opening figure shows the frozen highway $W_0$ beside a thin $A\to B$ bypass that carries the learned delta.

What to notice when reading

• Rank $r$ as the knob: quality vs storage (paper shows small $r$ often suffices).
• Mergeability — no inference tax when fused.
• Where they apply LoRA (attention vs MLP) changes the parameter/quality curve.

Results and evidence

On RoBERTa, DeBERTa, GPT-2, and GPT-3 175B adaptations, LoRA matches or nearly matches full fine-tuning quality on the paper’s benchmarks while using far fewer trainable parameters (e.g. adapting GPT-3 with ranks like $r=4$ or $r=8$ on selected matrices). They report large reductions in checkpoint size versus full FT and no added inference latency when weights are merged. Use the paper’s tables for task-by-task numbers rather than inventing aggregates.

Limitations

• Low-rank constraint can underfit if $r$ is too small for a hard domain shift.
• Choosing which layers/matrices to adapt is still a hyperparameter search.
• Merged models lose easy task-switching unless you keep separate deltas.
• Orthogonal to quantization and pruning — often combined in later systems (QLoRA, etc.).

How to read the paper

1. Abstract + §1 — motivation vs adapters / full FT.
2. §3 — reparameterization $\Delta W=BA$.
3. §4 — which weights to adapt; rank study.
4. §5 — GPT-2 / GPT-3 / RoBERTa experiments.
5. Related work — parameter-efficient fine-tuning landscape.

Knowledge check

Keep reading

1. Original paper (arXiv:2106.09685)
2. Related Fanout Daily: BERT, Attention Is All You Need.

Sources

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 — arXiv:2106.09685