BitNet b1.58: Ternary Weights for 1.58-bit LLMs

Modern Transformers store most of their parameters as FP16/BF16 floats. Inference cost is then two coupled taxes: floating multiply–adds inside every nn.Linear, and DRAM → SRAM weight traffic every time you decode a token. Post-training quantization (GPTQ, AWQ, …) squeezes weights after the fact. BitNet b1.58 asks a sharper question: what if the model is born with only three weight values?

The answer in this tech report is a LLaMA-alike Transformer where every weight is ternary $\{-1,0,1\}$ — about $1.58$ bits in the binary alphabet ($\log_2 3 \approx 1.585$). Activations stay 8-bit. The replacement block is BitLinear. On the authors’ RedPajama 100B-token runs, that recipe matches their FP16 LLaMA baseline in perplexity and zero-shot accuracy starting at the 3B scale, while cutting measured memory and latency.

What this paper explains

BitNet b1.58 is a from-scratch training recipe for Transformer LLMs:

1. Ternary weights. Each parameter is constrained to $\{-1,0,1\}$ with an absmean quantizer (Eqs. 1–3 in the paper). The extra $0$ versus pure $\pm 1$ BitNet is explicit feature filtering — and the reason the bit-width is quoted as $1.58$.
2. BitLinear. Swap nn.Linear for a layer whose forward pass uses those ternary weights (and 8-bit activations), so matrix multiplies are dominated by integer addition / subtraction, not FP16 multiply–add.
3. LLaMA-alike stack. RMSNorm, SwiGLU, RoPE, no biases — so the architecture plugs into open-source serving stacks with minimal glue.
4. Pareto claim (authors’ runs). Same model size and token budget versus a reproduced FP16 LLaMA: competitive quality from 3B upward, with substantially lower memory / latency / energy estimates (Tables 1–3, Figures 2–3).

This is not another post-training INT4 recipe. AWQ (already in Daily) calibrates salience scales then quantizes a finished FP16 checkpoint. BitNet b1.58 never keeps an FP16 weight matrix at inference — the ternary grid is the trained object.

Prior limits

• FP16/BF16 Transformers are expensive to serve. Weight traffic and FP multiply–adds dominate decoding energy and DRAM bandwidth.
• Post-training quantization is widely deployed but sub-optimal in the authors’ framing: you compress a model that was never asked to live on a low-bit grid.
• Prior 1-bit BitNet (WMD+23 in the paper’s citation) already showed integer-add matmuls with $\pm 1$ weights, but modeling capacity without an explicit $0$ was weaker.
• Industry drift toward 4-bit PTQ still leaves floating multiplies (or dequant GEMMs) in the hot path and keeps the “train full precision, squash later” recipe.

The mechanism

Absmean weight quantization. For a weight matrix $W \in \mathbb{R}^{n \times m}$, define the mean absolute scale

Then

Every entry of $\widetilde{W}$ lands in $\{-1,0,1\}$. Divide-by-$\gamma$ sets the typical magnitude near $1$; round + clip does the rest.

Activations. Follow BitNet’s 8-bit path, with one reported change: scale each token’s activations to $[-Q_b,Q_b]$ (symmetric, no zero-point) rather than forcing a $[0,Q_b]$ range before nonlinearities. The paper calls the quality impact negligible in their runs.

Why $0$ matters. Pure $\pm 1$ weights cannot zero a feature. Ternary weights can — the authors call this feature filtering, and treat it as the modeling upgrade over original BitNet.

BitLinear + LLaMA parts. Architecturally you still have attention and MLP blocks; the linear maps inside are BitLinear. RMSNorm / SwiGLU / RoPE / bias-free design match the open LLaMA lineage so Hugging Face / vLLM / llama.cpp integration stays thin.

A one-line mental model for a single output coordinate: if weights are ternary, a floating MAC collapses toward signed accumulation of activations

i.e. add $x_j$, subtract $x_j$, or skip — plus the activation quantization the runtime actually uses. Treat that as pedagogy for the compute story; kernels still need careful packing (the paper measures with a 2-bit Ladder kernel inside FasterTransformer).

Algorithm / figure walkthrough

1. Start from a LLaMA-alike Transformer skeleton (RMSNorm, SwiGLU, RoPE, no biases).
2. Replace every nn.Linear with BitLinear.
3. During training, keep latent full-precision weights for optimization, but quantize with absmean RoundClip so the forward (and the inference artifact) sees only $\{-1,0,1\}$.
4. Quantize activations to 8-bit per token on the symmetric $[-Q_b,Q_b]$ grid.
5. At inference, store ternary (packed) weights; matmuls are integer-add heavy; embeddings may remain higher precision (the paper notes embedding memory share shrinks as models grow).

The opening visual is the same arc in three beats: ENTER with dense FP16 weights and floating MACs; RUN absmean $\gamma$ + RoundClip into the ternary grid; LEAVE with packed $\{-1,0,1\}$ weights whose matmul is mostly add/sub.

What to notice when reading

• Train-from-scratch vs PTQ. If you mentally substitute “run AWQ on a finished LLaMA,” you will misread every table. These models are trained under the ternary constraint.
• Fairness protocol. Comparisons are against the authors’ reproduced FP16 LLaMA on RedPajama for 100B tokens, not against every public LLaMA checkpoint.
• Match point at 3B. Table 1 / Table 2 are where they claim perplexity and zero-shot averages meet or beat the FP16 baseline; smaller sizes still trail slightly.
• Kernel caveat. Latency / memory curves in Figure 2 use a 2-bit kernel — so reported speedups are not yet a perfect ternary ISA.
• Energy numbers are modeled. Figure 3 cites arithmetic-energy models ([Hor14], [ZZL22] in the paper), not a wall-meter on a shipping accelerator.
• “New scaling law” language is the authors’ framing of Pareto tradeoffs (e.g. “13B BitNet more efficient than 3B FP16” as reference equivalences) — read it as cost/quality recipe talk, not a Chinchilla refit.

Results and evidence

All numbers below are from the paper’s Tables 1–4 and Figures 2–3 — not re-measured here.

Perplexity + cost (Table 1). On the authors’ setup, BitNet b1.58 tracks FP16 LLaMA closely by 1.3B–3B:

Zero-shot averages (Table 2). Tasks: ARC-E/C, HellaSwag, BoolQ, OpenBookQA, PIQA, WinoGrande. Gap narrows with scale; 3B BitNet avg 50.2 vs FP16 49.7; 3.9B BitNet avg 51.2 while still cheaper than FP16 3B.

Scaled cost (Figure 2). Latency speedups vs FP16 grow with size (paper annotates 4.10× at 70B); memory reduction similarly climbs (7.16× at 70B) as the full-precision embedding becomes a smaller fraction.

Throughput (Table 3). On two 80GB A100s with pipeline parallelism, sequence length 512: FP16 70B max batch 16 / 333 tok/s vs BitNet 70B max batch 176 (11.0×) / 2977 tok/s (8.9×).

Energy (Figure 3). Modeled 7nm arithmetic energy: matrix-multiply path dominated by INT8 adds for BitNet vs FP16 add+mul for LLaMA; paper quotes 71.4× arithmetic-ops energy savings for that matmul slice, and end-to-end energy ratios rising with size (annotates up to 41.2× at 70B on their plot).

2T-token run (Table 4). BitNet b1.58 3B trained for 2T tokens under a StableLM-3B data recipe reports higher zero-shot averages than StableLM-3B at 2T on the authors’ Winogrande / PIQA / SciQ / LAMBADA / ARC-Easy mix (74.34 vs 73.22 avg).

Limitations

• Short tech report. Architecture ablations beyond “add zero / LLaMA-alike parts” are thin; lots of systems detail lives in citations (BitNet, Ladder, FasterTransformer).
• Author-reproduced baselines. Results are not a bake-off against every public FP16 checkpoint or every PTQ method at matched bits.
• Kernel / hardware gap. Measured latency uses a 2-bit kernel; the “new hardware for 1-bit LLMs” call-to-action is aspirational relative to shipping silicon.
• Energy is modeled, not a full datacenter PUE study.
• Training recipe cost. You still pay to train under quantization — this is not a free lunch on an existing FP16 checkpoint (contrast AWQ/GPTQ).
• Work-in-progress banner. The arXiv comment flags the note as WIP; treat numbers as the authors’ reported snapshot.

How to read the paper

1. Figure 1 — the Pareto cartoon: FP16 MAC vs ternary add/sub.
2. §2 Quantization Function — Eqs. (1)–(3); this is the whole mechanism.
3. Table 1 + Table 2 — where quality meets cost at 3B.
4. Figure 2 + Table 3 — latency, memory, throughput scaling.
5. Figure 3 — energy model; read assumptions before quoting the 71.4× figure.
6. Table 4 — longer 2T training sanity check vs StableLM-3B.
7. §4 Discussion — MoE / long-context / native 1-bit hardware speculation (future work, not results).

Knowledge check

Keep reading / Sources

• Paper: The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (arXiv:2402.17764)
• Prior BitNet: BitNet: Scaling 1-bit Transformers for Large Language Models
• Contrast in Daily: AWQ: Activation-aware Weight Quantization
• Serving context: Sarathi-Serve, FlexGen
• Open-source lineage named in-paper: LLaMA, RMSNorm, SwiGLU, RoPE, vLLM, llama.cpp, FasterTransformer, Ladder