SwiGLU — Better Neural Networks

SwiGLU is the activation used in the feed-forward block of most modern open language models, including LLaMA, PaLM, and the transformer implementations in this course. It replaces a plain ReLU MLP with a gated one and buys better quality at the same parameter count.

The idea

A standard transformer MLP is W₂ · ReLU(W₁x). SwiGLU splits the first projection into two and multiplies them:

Swish(z) = z · σ(z), also called silu, and ⊙ is the elementwise product. The important part is the gate: it lets the network scale each hidden feature up or down depending on the input, a smooth learned switch rather than a fixed threshold at zero.

The gate costs a third weight matrix, so the hidden size is shrunk to keep the block comparable to a plain 4d MLP. The usual choice is about 8/3 · d, rounded up to a multiple of 256. Modern implementations also drop all bias terms, matching the rest of the transformer block.

Worked example

Take d = 4096, the width of LLaMA-2's hidden states. A plain ReLU MLP with a 4d hidden size has:

• W₁: 4096 × 16384 = 67.1M parameters
• W₂: 16384 × 4096 = 67.1M parameters
• total: 134.2M

A SwiGLU block has three matrices, so its hidden size h must satisfy 3 × 4096 × h ≈ 134.2M, which gives h ≈ 10,922. That is exactly 8/3 × 4096 = 10,922.7. Rounded to a multiple of 256, h = 11,008, and the block holds 3 × 4096 × 11,008 ≈ 135.3M parameters — within 1% of the ReLU block for the same number of parameters and a better-behaved gate.

In code

import torch
import torch.nn as nn
import torch.nn.functional as F

class SwiGLU(nn.Module):
    def __init__(self, d_model: int, hidden: int | None = None):
        super().__init__()
        hidden = hidden or int(8 * d_model / 3)
        hidden = 256 * ((hidden + 255) // 256)   # round up to a multiple of 256
        self.gate = nn.Linear(d_model, hidden, bias=False)
        self.up = nn.Linear(d_model, hidden, bias=False)
        self.down = nn.Linear(hidden, d_model, bias=False)

    def forward(self, x):
        return self.down(F.silu(self.gate(x)) * self.up(x))

block = SwiGLU(4096)
print(sum(p.numel() for p in block.parameters()))  # 135,266,304

F.silu is Swish with the default β = 1, so the gate and the value path are two separate projections of the same input.

Check yourself

1. What does the gate do that a plain ReLU cannot?
2. Why is the hidden size of a SwiGLU MLP set to roughly 8/3 · d instead of 4d?
3. SwiGLU uses three weight matrices instead of two. Where does the extra cost appear at inference?

Key takeaways

• SwiGLU multiplies a Swish gate by a value projection, then projects back down.
• Use a hidden size near 8/3 · d to match a 4d ReLU MLP in parameter count.
• It is the default MLP block in modern open language models.