Qwen 3 From Scratch

Qwen 3 is a family rather than one model: dense variants from 0.6B to 32B parameters, and mixture-of-experts variants like 30B-A3B and 235B-A22B, where the number after the A is what runs per token. The backbone is a modern Llama-style decoder — pre-norm RMSNorm, RoPE, causal attention, GQA, SwiGLU, no linear biases — with one architectural addition worth naming: an RMSNorm applied to each query and key head before the dot product. The same weights answer both with and without a reasoning trace.

The idea

QK-Norm. Apply a per-head RMSNorm of dimension $d_{head}$ to $q$ and $k$ before computing attention scores. RoPE is a rotation, so it changes the direction of a head vector without changing its norm; nothing in the architecture bounds $\|q\|$ or $\|k\|$, so the softmax temperature can drift as training proceeds. Normalizing each head fixes that scale and removes the need for the QKV bias earlier Qwen models added for the same reason.

Dense and MoE from one recipe. The MoE variants split the feed-forward into a fine-grained expert pool and route each token to a small active subset — 128 experts with 8 active per token in the larger models. You get the capacity of a much bigger network while compute stays that of a small one, and one training recipe serves both.

Thinking mode is post-training, not architecture. Qwen 3 is trained with a reasoning-heavy reinforcement-learning stage followed by general alignment, and the chat template controls whether the model emits a long reasoning trace before answering. One checkpoint serves both modes; only the template and the length budget differ.

Long context comes from native training at 32K tokens, with rotary scaling (YaRN) in the config to extend it.

Worked example

Take a representative Qwen-3-style block: $d_{model} = 4096$, 64 query heads and 4 KV heads of $d_{head} = 128$.

• Attention projection sizes: queries $4096 \times 8192$; keys and values $4096 \times (4 \times 128) = 4096 \times 512$ each.
• KV cache per token per layer: $2 \times 4 \times 128 = 1024$ values — a sixteenth of the full multi-head cache.
• QK-Norm arithmetic: RMSNorm gives each head vector unit RMS, so the raw dot product over 128 dimensions has RMS about $\sqrt{128} \approx 11.3$. Dividing by $\sqrt{d_{head}}$ brings the logits back to order 1 however the learned gains drift.
• Expert capacity: with 128 experts and 8 active, the feed-forward weight memory is 16× the weight count of the experts actually used on any given token.

In code

import torch

class RMSNorm(torch.nn.Module):
    def __init__(self, d, eps=1e-6):
        super().__init__()
        self.g = torch.nn.Parameter(torch.ones(d))
        self.eps = eps

    def forward(self, x):
        return self.g * x / torch.sqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

q_norm, k_norm = RMSNorm(head_dim), RMSNorm(head_dim)   # learned per-head gains

def qk_norm_attention(q, k, v, head_dim):
    """q, k, v: (B, H, T, head_dim). Normalize per head, then scale the dot product."""
    q, k = q_norm(q), k_norm(k)
    scores = (q @ k.transpose(-1, -2)) / head_dim**0.5   # logits now of order 1
    return scores.softmax(-1) @ v

Check yourself

1. What problem does QK-Norm solve that a QKV bias also addressed in earlier Qwen models?
2. In a 128-expert, top-8 MoE, how do parameter count and per-token compute diverge?
3. Why can one Qwen 3 checkpoint serve both a thinking and a non-thinking mode?

Key takeaways

• Qwen 3 keeps the modern decoder recipe and adds per-head QK-Norm to keep attention logits stable.
• Dense and MoE variants of one recipe let the family span very different compute budgets.
• Thinking mode is a post-training and prompting property, not an architectural one.