Adapters and LoRA

Full fine-tuning rewrites every weight in the model, which means storing a gradient and two optimizer states for each one. LoRA replaces that with a small, low-rank correction, so a 7B model can be adapted on a single GPU while training well under 1% of its parameters.

The idea

Fine-tuning tends to change a weight matrix in a way that is simpler than the matrix itself — the update has low intrinsic rank. LoRA freezes the original weight W and learns a correction ΔW = BA, where B is d × r, A is r × k, and r is small (often 4 to 64). The layer computes:

A is initialized randomly and B is initialized to zero, so ΔW = 0 at step 0 and training starts exactly at the base model. The α/r factor keeps the effective size of the update roughly constant when you change r.

Two consequences matter:

• Memory. Only A and B need gradients and optimizer state. Everything else stays frozen.
• Latency. The update is linear, so after training it can be merged into the base weight, W' = W + (α/r)BA. The served model has the same shape and speed as the original.

The broader family is adapters: small modules inserted into the network. Bottleneck adapters use a down-projection, a nonlinearity, and an up-projection; they work but add latency because they cannot be merged. LoRA became the default because a rank correction is mergeable and easy to target.

Worked example

Take one attention projection of size 4096 × 4096, which has 16,777,216 weights. With r = 8:

• A: 8 × 4096 = 32,768
• B: 4096 × 8 = 32,768
• total: 65,536 trainable weights, about 0.39% of the layer

For a 32-layer model with LoRA on the query and value projections, that is roughly 4.2 million trainable parameters instead of 7 billion. Their Adam state is on the order of tens of megabytes rather than hundreds of gigabytes.

In code

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """Freeze a Linear layer and learn a rank-r correction."""
    def __init__(self, base: nn.Linear, r: int = 8, alpha: int = 16):
        super().__init__()
        self.base = base
        self.merged = False
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.empty(r, base.in_features).normal_(0, 0.01))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r

    def forward(self, x):
        if self.merged:
            return self.base(x)
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

    @torch.no_grad()
    def merge(self):
        self.base.weight += self.scale * (self.B @ self.A)
        self.merged = True  # serving now uses the base layer alone

layer = LoRALinear(nn.Linear(4096, 4096, bias=False))
print(sum(p.numel() for p in layer.parameters() if p.requires_grad))  # 65536

Check yourself

1. Why is B initialized to zero instead of A?
2. You double r from 8 to 16 but leave α fixed. What happens to the effective scale of the update, and why do people also double α?
3. After merging an adapter into W, how does inference cost compare to the base model?

Key takeaways

• LoRA learns a low-rank correction while the base weights stay frozen.
• Trainable parameters drop to r(d_in + d_out) per layer instead of d_in · d_out.
• Because the correction is linear, it can be merged, so serving adds no latency.