QLoRA and Quantized Training

QLoRA makes fine-tuning memory-bound rather than compute-bound. It stores the frozen base model in 4-bit precision and trains LoRA adapters in bf16 on top, so the largest consumer — the weights — shrinks by roughly 4x while the trainable state stays small.

The idea

The base model is the expensive part. A 7B model in bf16 needs about 14 GB just for weights. Stored at 4 bits, the same weights need about 3.5 GB. QLoRA makes that work by combining three tricks:

• NF4 (normal-float 4-bit). A 4-bit type whose 16 levels are spaced to match the quantiles of a standard normal distribution. The QLoRA paper argues this is optimal for zero-mean normally distributed weights, which is what trained weights roughly look like. Weights are stored in blocks, and each block carries one scale factor.
• Double quantization. Those per-block scale factors are themselves quantized to 8 bits, removing most of the overhead the scales add.
• Paged optimizers. Optimizer state can be paged to CPU memory, turning a memory spike into a slowdown instead of an out-of-memory crash.

Compute still happens in bf16. Each block is dequantized, used in a matrix multiply, and discarded. The LoRA adapters stay in bf16 because they hold the gradients and the optimizer state; quantizing them would hurt learning for no memory win.

The trade-off is honest: QLoRA saves memory, not time. Dequantization happens on every forward pass, so steps are usually slower than 16-bit LoRA. The QLoRA paper reports matching 16-bit fine-tuning quality on its benchmarks, which is what makes the memory saving worth accepting.

Worked example

For a 7B model:

• bf16 weights: 7e9 × 2 bytes = 14.0 GB
• NF4 weights: 7e9 × 0.5 bytes = 3.5 GB
• block scales at block size 64 in fp32: 7e9 / 64 × 4 bytes ≈ 0.44 GB
• the same scales after double quantization: ≈ 0.11 GB

So the frozen base lands near 3.6 GB, and the remaining memory goes to the bf16 adapters, activations, and gradients. That is the difference between fitting on one consumer GPU and not fitting at all.

In code

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # normal-float 4-bit
    bnb_4bit_use_double_quant=True,    # quantize the block scales too
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "v_proj"]
))
model.print_trainable_parameters()

Check yourself

1. Why are the LoRA adapters kept in bf16 instead of 4-bit?
2. A 7B model in NF4 uses about 3.5 GB for weights. Name two other memory consumers that still limit batch size.
3. QLoRA saves memory but often costs throughput. Why?

Key takeaways

• QLoRA is a 4-bit frozen base plus bf16 LoRA adapters and a paged optimizer.
• NF4 and double quantization bring 7B weights to roughly 3.6 GB without changing the architecture.
• It reduces memory, not step time, because dequantization runs on every forward pass.