Skip to main content
fanout
Archive
issue 2026-08-14Inference45 minMLSys 2024interactive

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

This research paper explains why LLM weights are not equally important under quantization, shows that salience comes from activation magnitudes (not weight norms), and derives a hardware-friendly per-channel scaling transform that protects those channels while keeping a uniform low-bit format.

Lin et al. · Original paper

AWQ protects activation-salient weight channels by scaling before INT quantizationLeft: calibration activations light a few high-magnitude channels. Middle: those weight columns scale up by s greater than 1. Right: uniform low-bit quantization snaps the whole matrix while protected channels keep lower relative error.ENTER · ACTIVATERANK BY |x|~1% CHANNELS LIGHT UPRUN · SCALEw ← s·w · x ← x/sWCOLs·WPROTECTRELATIVE ERR SHRINKS ~1/sLEAVE · QUANTIZEUNIFORM INT GRIDINT4GROUPGPUEDGENO MIXED-PRECISION KERNELRANK BY ACTIVATIONS · SCALE TO PROTECT · QUANTIZE UNIFORMLYAWQ activate → scale → quantizeStacked flow: rank channels by activation magnitude, scale salient weights, apply uniform INT quantization.ENTER · SCALE · LEAVEENTER · RANK BY |x|~1% activation-salient channelsRUN · SCALE s·Wfuse x/s into previous opLEAVE · UNIFORM INThardware-friendly W4A16 packPROTECT WITHOUT MIXED DTYPESRANK → SCALE → QUANTIZE

On-device LLM serving is memory-bound: generation spends most of its time dragging weights from DRAM, not multiplying them. Weight-only INT4 cuts that traffic by about 4×4\times in theory — if you can quantize without wrecking perplexity.

Naive round-to-nearest (RTN) often cannot. A few weight channels matter disproportionately; quantizing them carelessly blows up WikiText perplexity. AWQ’s punchline is precise: find those channels from the activation distribution, then protect them by scaling — not by keeping a mixed-precision format.

What this paper explains

Activation-aware Weight Quantization (AWQ) is a post-training, weight-only method for LLMs (typically W4A16 / W3A16 with group size 128). It does three things:

  1. Show that protecting a tiny fraction of weights (on the order of 0.1%0.1\%–1%1\%) nearly restores FP16-quality perplexity — but only if those channels are chosen from activations, not weight magnitudes.
  2. Replace awkward mixed FP16+INT storage with an equivalent scaled transform so every weight stays in the same low-bit grid.
  3. Search per-channel scales on a small calibration cache to minimize layer output error after quantization, without backprop through QQ.

Alongside the algorithm, the authors ship TinyChat, an on-device runtime that turns the ∼4×\sim 4\times smaller weight footprint into measured generation speedups via fused dequant GEMM and packed INT4 layouts.

Activation-aware salience mapBar chart of per-channel activation magnitudes. Channels above the salience cutoff highlight the corresponding weight columns that AWQ will protect.calibration |x|weight columnstop ~1% by activationprotect orange columns (not large |W|)
Salience comes from activation magnitudes. Weight-norm ranking fails the same ablation (paper Table 1).

Prior limits

  • RTN is brittle at 3–4 bits. Group-wise round-to-nearest on LLM weights can spike perplexity even when average weight error looks modest.
  • Magnitude pruning intuition fails. Keeping large-∥W∥\lVert W\rVert channels in FP16 barely helps; randomly kept channels also fail. The “important” weights are not the heaviest ones.
  • Mixed precision is accurate but awkward. Literally storing 1%1\% of channels in FP16 restores quality (Table 1), yet breaks uniform kernels, packing, and memory layouts on edge GPUs.
  • Reconstruction / GPTQ-style methods can overfit the calibration set and need heavier solvers; AWQ aims for a calibration-light, backprop-free alternative that still generalizes to instruction-tuned and multimodal models.

The mechanism

Weight-only quantization (paper Eq. 1). For a group of weights w\mathbf{w} and bit-width NN:

Q(w)=Δ⋅Round⁡ ⁣(wΔ),Δ=max⁡(∣w∣)2N−1.Q(\mathbf{w})=\Delta\cdot\operatorname{Round}\!\left(\frac{\mathbf{w}}{\Delta}\right),\qquad \Delta=\frac{\max(|\mathbf{w}|)}{2^{N-1}}.

Δ\Delta is shared across the group — that shared scale is why a few bad channels can dominate rounding error for everyone else.

Salience from activations. A weight channel that multiplies a consistently large activation feature has outsized impact on y=wxy=\mathbf{w}\mathbf{x}. Collect calibration activations; rank input channels by magnitude statistics; the top ∼1%\sim 1\% are “salient.”

Why scaling protects. Instead of leaving salient ww in FP16, multiply that channel by s>1s>1 and divide the corresponding activation by ss:

Q(w⋅s)⋅xs=Δ′⋅Round⁡ ⁣(wsΔ′)⋅x⋅1s.Q(w\cdot s)\cdot\frac{x}{s} =\Delta'\cdot\operatorname{Round}\!\left(\frac{ws}{\Delta'}\right)\cdot x\cdot\frac{1}{s}.

Empirically, Round⁡\operatorname{Round} error stays ∼0.25\sim 0.25 on average, and a single scaled channel rarely moves the group max, so Δ′≈Δ\Delta'\approx\Delta. The relative error on the protected channel therefore shrinks roughly with 1/s1/s, while the forward math stays equivalent in the unquantized limit.

Search, don’t guess ss. Scaling forever hurts non-salient channels (their share of Δ\Delta grows). AWQ searches per-input-channel scales:

s∗=arg⁡min⁡s∥Q(Wdiag⁡(s)) (diag⁡(s)−1X)−WX∥\mathbf{s}^{*}=\arg\min_{\mathbf{s}}\left\lVert Q(\mathbf{W}\operatorname{diag}(\mathbf{s}))\, (\operatorname{diag}(\mathbf{s})^{-1}\mathbf{X}) -\mathbf{W}\mathbf{X} \right\rVert

with X\mathbf{X} cached from a small pretraining calibration set. The inverse scales fuse into the previous operator (LayerNorm / residual path), so inference sees ordinary low-bit weights plus a cheap fused rescale.

Interactive

Which channels deserve protection?

Numbers are OPT-6.7B WikiText PPL from AWQ Table 1 under INT3-g128 while keeping a fraction of channels in FP16 (the motivating ablation). Deployed AWQ replaces that mixed format with scaling — but the ranking lesson is the same.

Activation ranking: protecting 1% FP16 channels nearly restores FP16 (paper Table 1).

Hardware angle. Generation is memory-bound: weight bytes dwarfs activation bytes (paper Fig. 3). W4A16 raises arithmetic intensity about 4×4\times versus FP16 weights. AWQ keeps a uniform INT grid so TinyChat can pack, fuse, and dequant on the fly — unlike a true mixed FP16/INT layout.

Activation-aware scaling before quantizationA salient weight is multiplied by s greater than 1 while the matching activation is divided by s. Uniform quantization Q then sees a larger weight with smaller relative rounding error.w · xFP16 pairbefore(s·w) · (x/s)equivalent if exactprotect · s>1Q(s·w)·(x/s)uniform INT gridafter AWQ
Paper Eq. 2: scale the salient channel, invert on the activation, then quantize — relative rounding error on the protected weight shrinks with ~1/s when Δ stays put.

Algorithm walkthrough

  1. Pick target bit-width and group size (paper default: group size 128128; experiments often use INT3/INT4).
  2. Run a small calibration batch through the FP16 model; cache per-layer input activations X\mathbf{X}.
  3. For each linear layer weight W\mathbf{W}:

- Rank input channels by activation magnitude → mark the salient set (order 1%1\%). - Search scales s\mathbf{s} (salience-biased grid) minimizing quantized output error vs WX\mathbf{WX}. - Form W′=Wdiag⁡(s)\mathbf{W}'=\mathbf{W}\operatorname{diag}(\mathbf{s}) and fuse diag⁡(s)−1\operatorname{diag}(\mathbf{s})^{-1} into the preceding op. - Quantize W′\mathbf{W}' with group-wise QQ to INT3/INT4; store scales/zeros for dequant.

  1. At inference: load packed low-bit weights; TinyChat dequantizes on chip into the GEMM path.

The opening visual is the same story in space: an activation bar chart lights a few channels; those weight columns inflate; a uniform INT grid snaps everything down; a compact edge GPU emits tokens without a mixed-precision exception path.

What to notice when reading

  • AWQ is activation-aware weight quantization — activations stay higher precision (W4A16), unlike SmoothQuant’s W8A8 migration story.
  • The 1%1\% FP16 ablation is the motivation, not the deployed format. Deployed AWQ is scaled + fully quantized.
  • No STE / QAT loop: scales come from a grid search on calibration activations.
  • Calibration is intentionally small and from pretraining data so instruction / VLM heads do not overfit a downstream task.
  • Group size 128128 is the practical knob pairing accuracy with kernel friendliness.

Results and evidence

Numbers below are as reported by Lin et al. — not universal constants.

Table 1 (WikiText PPL ↓\downarrow, INT3-g128). Keeping a few channels in FP16 only helps when chosen by activations. For OPT-6.7B: FP16 =10.86=10.86, RTN =23.54=23.54, act-based 1%1\% FP16 =11.39=11.39, weight-magnitude 1%1\% FP16 ≈22.37\approx 22.37, random 1%1\% ≈24.23\approx 24.23. Same pattern on OPT-1.3B / 13B: activation-based protection collapses the RTN blow-up; weight- or random-based does not.

Table 2 (scaling intuition on OPT-6.7B). Multiplying the 1%1\% salient channels by ss improves RTN’s 23.5423.54 PPL to 11.9211.92 at s=2s=2 (best among {1.25,1.5,2,4}\{1.25,1.5,2,4\}). Larger ss protects salient channels more but eventually hurts the rest via Δ\Delta drift.

Mixed-precision vs scaled AWQ (Table 3). Scaling-based protection matches the accuracy idea of “1%1\% FP16” while remaining hardware-friendly under INT3-g128.

Systems. TinyChat reports more than 3×3\times generation speedup vs HuggingFace FP16 across several LLM families on desktop/mobile GPUs, and enables large models on constrained devices (e.g. Llama-2-70B on Jetson Orin 64GB; interactive 13B on an 8GB laptop GPU — paper claims). AWQ also reports strong retention on instruction-tuned and multimodal models relative to prior PTQ baselines in their eval suite.

Limitations

  • Still a calibration method: bad or tiny caches can mis-rank channels (though the paper emphasizes robustness vs reconstruction overfitting).
  • Weight-only focus: does not by itself solve activation quantization / W8A8 deployment (pair with SmoothQuant-style ideas if you need that).
  • Search adds a one-time offline cost per layer; not free compared to pure RTN.
  • Extreme bit-widths and pathological groups can still force quality/speed tradeoffs; group size and bit-width remain knobs.
  • Mixed-precision oracle (0.1%0.1\% FP16) can look slightly better in ablations but is intentionally abandoned for systems reasons.

How to read the paper

  1. Fig. 2 — RTN damage → 1%1\% FP16 by activation → AWQ scaled protection. Read this before the equations.
  2. §3.1 / Table 1 — prove to yourself that activation ranking beats weight ranking.
  3. §3.2 / Eqs. 1–2 / Table 2 — why s>1s>1 shrinks relative error without mixed dtypes.
  4. §3.3 search objective (Eq. 4) — how scales are chosen without differentiating through QQ.
  5. Fig. 3 + TinyChat — why weight-only INT4 maps to real tokens/sec on edge GPUs.
  6. Skim related work against GPTQ / SmoothQuant to place AWQ on the PTQ map.

Knowledge check

Why does AWQ rank channels using activations instead of weight magnitudes?

Deployed AWQ protects salient weights primarily by…

On-device generation benefits from W4A16 mainly because…

Keep reading

  • SmoothQuant (Xiao et al.) — migrate quantization difficulty from activations into weights for W8A8.
  • GPTQ (Frantar et al.) — second-order, reconstruction-based post-training weight quantization.
  • DistServe / Orca Daily issues — complementary serving angles once weights already fit.

Sources

  • Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” MLSys 2024. https://arxiv.org/abs/2306.00978
  • Project / community implementations referenced in the paper ecosystem (Hugging Face, TensorRT-LLM, vLLM, etc.) — verify current docs for production kernels.