Efficient Streaming Language Models with Attention Sinks

Chatbots and other streaming apps want models that keep generating for hours. Two walls show up immediately: the KV cache grows with every past token, and most pretrained Transformers were only trained inside a finite attention window (for example 4K on Llama-2). A tempting fix is window attention — keep only the most recent $W$ key/value states. It is cheap. It also fails hard once the first tokens fall out of the window.

Xiao et al. name the missing piece attention sinks: Softmax attention piles large mass on the earliest positions even when those tokens are not semantically special. Evict their KV states and you rip out a large slice of the Softmax denominator. StreamingLLM’s recipe is almost embarrassingly small — keep a handful of sink KVs (often four) together with a rolling recent window — and pretrained Llama-2, MPT, Falcon, and Pythia models stream stably for millions of tokens without fine-tuning.

What this paper explains

Autoregressive decoding caches key and value vectors for every past token so each new step can attend without recomputing the whole prompt. For a stream of length $T$, that cache is $\Theta(T)$ in memory and decode bandwidth. Length-extrapolation tricks expand the training window, but the accepted context stays finite.

The authors ask a systems-shaped question: can we deploy an already-trained LLM on an infinite input stream without paying full-history KV cost and without collapsing language-modeling quality?

Their answer has three layers:

1. Diagnose why window attention’s perplexity explodes the moment initial tokens leave the cache.
2. Show that restoring a few initial KV states (even linebreak placeholders) recovers stability — the attention sink phenomenon.
3. Ship StreamingLLM: sink tokens + sliding recent window, optionally with a dedicated sink token in pretraining.

Prior limits

• Dense attention stores every past KV. Memory and latency grow with stream length; once $T$ exceeds the pretraining window, position encodings and attention patterns leave the training distribution and quality collapses.
• Window attention caps cache size at $W$ recent tokens. Efficient — until eviction removes the first positions, after which the paper shows a sharp perplexity surge.
• Sliding window with recomputation can restore quality by rebuilding attention over a larger span, but the paper’s streaming comparison treats that path as an expensive oracle baseline (up to $22.2\times$ slower than StreamingLLM in their reported streaming setting).
• Expanding context via fine-tuning or position interpolation helps finite longer windows; it does not by itself give a constant-size cache for endless dialogue.

The mechanism

Softmax cannot be all zeros. For logits $x_1,\ldots,x_N$,

If early-layer heads learn $x_1 \gg x_j$ for later $j$, a large fraction of the denominator lives in the sink. Remove token 1’s KV and every remaining weight renormalizes — not a small local tweak, a global redistribution.

Attention sinks vs semantics. The paper visualizes attention maps (Llama-2-7B and relatives): beyond the bottom layers, heads consistently focus on initial tokens. Replacing the first four tokens with "\n" still attracts that mass; putting those linebreak KVs back restores perplexity. So the sink is largely about being first in the autoregressive prefix, not about carrying unique meaning.

StreamingLLM cache. At decode time keep:

• a small sink set of size $S$ (default experiments use $S = 4$ initial tokens), and
• a rolling window of the most recent $W$ tokens’ KV states.

Total cache size is $S + W$, constant as the stream grows. Positions for RoPE/ALiBi are handled so the rolling window stays consistent with how the model expects relative offsets inside the visible set (see paper §3 for the position-assignment details used with each encoding family).

Optional pretraining sink. Adding a dedicated placeholder sink token at the start of every pretraining sample concentrates sink behavior into one stable slot. Their 160M Pythia-style ablation keeps normal benchmark accuracy while improving how few sink tokens streaming needs at inference.

Algorithm / figure walkthrough

1. Prefill the prompt as usual; materialize KV for the prefix.
2. Designate the first $S$ positions as sinks that will never be evicted.
3. For each new generated token, append its KV into the recent window.
4. When the recent window would exceed $W$, drop the oldest non-sink KV.
5. Attend only over the concatenated sink + recent KV set (size $S+W$).
6. Repeat — cache size stays flat while logical stream length grows.

Contrast the three modes in the first figure: dense keeps everything; window drops sinks and breaks Softmax; StreamingLLM pins sinks and rolls only the working set.

What to notice when reading

• The failure mode is not “the model forgot the topic.” It is a Softmax denominator fracture when sinks vanish.
• Four sinks are an empirical default in their main plots, not a magical constant — the pretraining-sink ablation shows you can push toward a single dedicated sink.
• StreamingLLM is an inference cache policy for already-trained models; the sink-token pretraining idea is a complementary training-time improvement, not required to use the framework.
• They evaluate both RoPE models (Llama-2, Falcon, Pythia) and ALiBi (MPT), so the sink story is not tied to one position encoding.

Results and evidence

Numbers below are from the paper’s reported experiments (arXiv:2309.17453), not independent re-runs.

Long-book language modeling (PG19). On concatenated PG19 books (~20K-token traces in Figure 3), dense attention fails past the pretraining window; window attention collapses once the cache fills and initial tokens leave; StreamingLLM tracks the sliding-window-with-recomputation oracle’s perplexity closely for Llama-2, MPT, Falcon, and Pythia (cache set to half the pretraining window for visualization: 2048 for Llama-2, 1024 for the others).

Extreme streams. Figure 5 reports stable perplexity while streaming more than 4 million tokens across Llama-2-[7,13,70]B, Falcon-[7,40]B, Pythia-[2.8,6.9,12]B, and MPT-[7,30]B.

Speed. In streaming settings, StreamingLLM outperforms the sliding-window recomputation baseline by up to $22.2\times$ wall-clock speedup (abstract / §1 claim).

Pretraining with a sink token (160M ablation). Two Pythia-160M runs on the Pile (143k steps, batch 256 on 8×A6000): loss curves stay aligned; zero-shot averages across ARC-c/e, HellaSwag, LAMBADA, OpenBookQA, PIQA, Winogrande stay comparable (Table 4: e.g. ARC-e 45.2 → 45.6, HellaSwag 29.4 → 29.8, PIQA 62.2 → 62.6). Streaming perplexity improves so fewer sink tokens are required at inference (Table 3 narrative).

Limitations

Compressed from the paper’s scope and what it does not claim:

• StreamingLLM preserves local recent context plus sinks; it is not a full long-range retrieval system. Facts that lived only in the middle of an evicted span are gone unless you add retrieval or other memory.
• Quality matches the recompute sliding-window oracle on their LM perplexity traces; that is different from claiming parity with ideal infinite dense attention on every downstream task.
• Default $S=4$ is empirical; pathological prompts or models with different sink structure may need retuning.
• Position bookkeeping for RoPE/ALiBi must be implemented carefully — a naive “keep absolute indices of evicted tokens” approach is exactly what breaks.
• The work targets cache content selection; it complements rather than replaces paged KV managers, continuous batching, or speculative decoding in a full serving stack.

How to read the paper

1. Abstract + §1 — streaming deployment problem and the $22.2\times$ teaser vs recompute.
2. §2 / Figure 1 — dense vs window vs StreamingLLM cache cartoons.
3. §3.1 + Equation (1) + Figure 2 — Softmax sinks and the linebreak substitution check.
4. §3.2 — StreamingLLM algorithm and position handling.
5. §4.1 Figures 3 & 5 — PG19 perplexity and 4M-token stability across families.
6. §4.2 Tables 3–4 + Figure 6 — dedicated sink-token pretraining ablation.

Knowledge check

Keep reading

1. Original paper (arXiv:2309.17453) — §3 mechanism, Figures 1–5, Tables 3–4.
2. Related Fanout Daily: PagedAttention, KV cache management survey, RoPE, FlashAttention.
3. Press et al., ALiBi — arXiv:2108.12409.
4. Code: mit-han-lab/streaming-llm.

Sources

• Xiao, Tian, Chen, Han, Lewis, Efficient Streaming Language Models with Attention Sinks, ICLR 2024 — arXiv:2309.17453
• Touvron et al., Llama 2, 2023 — arXiv:2307.09288
• Su et al., RoFormer (RoPE) — arXiv:2104.09864
• Press, Smith, Lewis, ALiBi — arXiv:2108.12409