DeepSeek V3 From Scratch

DeepSeek-V3 is a 671B-parameter mixture-of-experts model that activates roughly 37B parameters per token, and nearly every design choice exists to make that asymmetry pay off. Three parts define it: Multi-head Latent Attention for a small KV cache, fine-grained expert routing balanced by a bias control loop instead of an auxiliary loss, and multi-token prediction during training.

The idea

Multi-head Latent Attention (MLA). Instead of caching per-head keys and values, the model caches one compressed latent vector $c^{KV}_t$ per token and reconstructs K and V with up-projections at attention time. Only the cached representation shrinks; the attention mathematics is unchanged in the forward pass. A small decoupled vector carries the rotary part of the keys so RoPE survives the compression.

Fine-grained MoE with auxiliary-loss-free balancing. The feed-forward is split into many small experts — 256 routed, plus one shared expert every token uses — and each token routes to the top 8. Balancing matters, because an unbalanced router wastes experts. Instead of adding an auxiliary loss that fights the training objective, DeepSeek-V3 tracks a per-expert bias $b_i$ added to the router affinity. Overloaded experts get their bias nudged down, idle experts up. Two details matter: the bias acts only on which experts are selected, while the weight used to combine their outputs comes from the unbiased affinity; and this is a control loop on the router, not a term in the loss.

Multi-token prediction (MTP). Extra heads predict the token $k$ steps ahead, giving a denser training signal at every position and a draft model for speculative decoding at inference.

Training also runs in FP8 mixed precision with fine-grained quantization tiles.

Worked example

KV cache per token, per layer, for a configuration with 128 heads of dimension 128:

• Standard MHA caches $2 \times 128 \times 128 = 32{,}768$ values.
• MLA caches a 512-dimensional latent plus a 64-dimensional decoupled RoPE key: $576$ values.
• That is $32{,}768 / 576 \approx 57$ times smaller, per token, per layer.

Parameter activation: 671B total with ~37B active means about 5.5% of the weights participate in any single token's forward pass. Compute tracks the active set; memory must hold the whole model.

Routing, concretely: a token produces 256 affinities. The router adds the bias vector, takes the top 8 indices, and normalizes the unbiased affinities of those 8 to sum to 1. If expert 12 sits at rank 9 with bias $+0.05$, it joins the route; the batch statistics then push its bias back down.

In code

import torch

def update_bias(bias, load, target, step=1e-3):
    """Aux-loss-free balancing: overloaded experts down, idle experts up."""
    return bias + step * torch.sign(target - load)   # load: fraction of tokens per expert

def route(x, gate_w, bias, k=8):
    affinity = torch.sigmoid(x @ gate_w.T)                     # (..., E)
    idx = (affinity + bias).topk(k, dim=-1).indices            # bias shifts selection only
    weights = affinity.gather(-1, idx)
    return weights / weights.sum(-1, keepdim=True), idx        # unbiased weights

Check yourself

1. MLA leaves the attention computation unchanged. What exactly does it change, and why does that shrink inference memory?
2. Why balance experts with a selection bias rather than an auxiliary loss?
3. If only 37B of 671B parameters are active per token, which resource still has to scale with the full 671B?

Key takeaways

• MLA shrinks the KV cache by caching a compressed latent instead of full per-head keys and values.
• Auxiliary-loss-free balancing turns expert load into a bias control loop rather than a competing loss term.
• MTP adds training signal everywhere and hands you a speculative-decoding draft for free.