DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Autoregressive LLM serving has two phases that look similar on a whiteboard and fight each other on a GPU. Prefill reads the whole prompt and must finish before the first token streams — its clock is time-to-first-token (TTFT). Decode then emits tokens one at a time — its clock is time-per-output-token (TPOT). Most engines colocate both phases on the same devices and batch them together. DistServe’s claim is that this colocation is not a free lunch: it creates prefill–decode interference and couples resource and parallelism choices that should be independent.

The fix is architectural. Put prefill and decode on separate instances, specialize each side for its latency goal, and move the KV cache across when a request graduates from prompt to generation. Then choose a placement (how many GPUs / what parallelism each side gets) that maximizes per-GPU goodput: the highest request rate you can sustain while still meeting both SLOs.

What this paper explains

An LLM request is not one homogeneous job. After the prompt arrives:

1. Prefill runs a compute-heavy pass over all prompt tokens and materializes the KV cache for that prefix.
2. Decode repeatedly runs a lighter step that appends one token (and one KV column) at a time.

Applications care about both ends of the latency story. Chat wants a snappy first token and a smooth stream. Summarization may tolerate a slower start if tokens keep flowing. DistServe defines success as maximizing the rate of requests that meet a joint SLO attainment target (e.g. 90% of requests within TTFT and TPOT budgets) per GPU provisioned — goodput, not raw tokens/s with silent SLO violations.

The paper argues two colocated failure modes:

• Interference. A large prefill in the batch spikes decode latency; decode batches starve TTFT when they hog the device.
• Coupling. Tensor / pipeline parallelism and replica counts chosen for one phase are rarely optimal for the other, so colocated systems over-provision or miss one SLO.

Prior limits

Systems in the Orca / vLLM lineage already improved iteration-level batching and paged KV memory. Those wins are real — and DistServe builds on that world — but they still typically run prefill and decode on the same GPU pool.

Under tight dual SLOs, colocation forces ugly compromises:

• Prioritize decode batching → TTFT blows up when prompts are long.
• Prioritize prompt latency → decode batches shrink and TPOT / cost suffer.
• Throw more GPUs at a coupled plan → cost-per-query rises even when utilization looks “busy.”

DistServe’s prior-art move is not “batch better inside one box.” It is “stop putting both clocks on the same box.”

The mechanism

Phase disaggregation. Maintain two kinds of instances:

• Prefill instances — optimized for TTFT (prompt-parallel compute, parallelism tailored to prefill).
• Decode instances — optimized for TPOT (large decode batches, parallelism tailored to memory-bandwidth-bound steps).

When prefill finishes, the system transfers the KV cache for that request to a decode instance and continues generation there. High-bandwidth GPU fabrics make that handoff practical when managed carefully; the paper’s latency breakdowns treat transmission as a first-class cost, not an afterthought.

Uncoupled planning. Because phases no longer share a device schedule, each side can pick its own degree of parallelism and replica count. DistServe’s placement algorithm searches that expanded design space for the schema that maximizes cluster-wide per-GPU goodput under the stated TTFT/TPOT targets.

Conceptually, goodput is:

Higher goodput means lower cost per successful query at the same service quality.

Algorithm / figure walkthrough

1. Admit a request to a prefill instance; run prompt compute; build KV for the prefix.
2. Handoff KV (and control metadata) to a decode instance — do not keep running decode on the prefill GPU just because the tensors are already there.
3. Decode on the decode pool with batching suited to TPOT.
4. Place the cluster: choose how many devices (and which parallelism) go to prefill vs decode so the bottlenecked phase is not over-served while the other misses SLOs.
5. Measure success as SLO-compliant rate per GPU, not peak tokens/s with hidden latency debt.

The opening visual shows the same request as a token that must leave the prefill lane and enter the decode lane — with KV riding along — instead of fighting for one shared lane.

What to notice when reading

• DistServe’s unit of optimization is goodput under dual SLOs, not single-metric throughput.
• Disaggregation is a systems change: scheduling, placement, and networking — the Transformer math is unchanged.
• KV transfer is the tax you pay to buy independent TTFT/TPOT control; the paper spends pages showing when that tax is worth it.
• Placement is not “half and half GPUs.” Prefill-heavy vs decode-heavy apps want different splits.
• Related work like chunked-prefill schedulers (e.g. Sarathi-Serve) attack interference inside a colocated engine; DistServe attacks it by separating the engines. Both are teachable; they are not the same lever.

Results and evidence

From the authors’ OSDI’24 evaluation (their workloads, models, and SLO settings — not universal constants):

• On popular LLMs and application-like traces (chat / assistant / summarization patterns), DistServe reports serving up to $7.4\times$ higher request rate, or supporting up to $12.6\times$ more stringent SLOs, while still meeting latency requirements for over 90% of requests versus strong colocated baselines.
• On ShareGPT-style traffic, they report sustaining about $2.0\times$–$4.6\times$ higher request rate than vLLM in the cited comparison setting, attributing the gap to removing prefill–decode interference and specializing parallelism per phase.
• Under a tighter 99% attainment goal, they report still higher relative rate / SLO headroom versus vLLM and competitive gains versus DeepSpeed-MII in the paper’s tables.
• Latency breakdowns (e.g. OPT-175B on ShareGPT in their figures) show KV transmission as a visible but manageable slice when the fabric and placement are sane.

Limitations

Compressed from the paper’s setting and common follow-on caveats:

• Benefits depend on network / interconnect quality for KV handoff; a weak fabric re-introduces latency.
• Placement search assumes you can measure or model prefill vs decode load; highly non-stationary mixes need re-planning.
• Disaggregation adds operational complexity (two pools, routing, failure domains) versus a single colocated engine.
• Complementary tricks (chunked prefill, priority scheduling, speculation) can shrink interference even when phases stay colocated — DistServe is one strong point in a design space, not the only lever.
• Reported multipliers are workload- and SLO-specific; copy the paper’s setup before citing a factor in a capacity plan.

How to read the paper

1. Abstract + §1 — dual SLOs, goodput, interference vs coupling.
2. Background / motivation — why TTFT and TPOT pull the scheduler in opposite directions.
3. Disaggregation design — instance types, KV transfer, control plane.
4. Tradeoff analysis + placement — how they pick prefill vs decode resources.
5. Evaluation — ShareGPT and application traces, comparisons to vLLM / DeepSpeed-MII, breakdown figures.
6. Discussion / related work — situate next to continuous batching and chunked-prefill schedulers.

Knowledge check

Keep reading / Sources

• Paper (arXiv): https://arxiv.org/abs/2401.09670
• OSDI 2024 proceedings entry via the arXiv HTML / USENIX pages linked from the paper.
• Code pointer cited by the authors: https://github.com/LLMServe/DistServe
• Related Fanout Daily: Orca continuous batching, PagedAttention, StreamingLLM attention sinks
• Next mechanisms to compare: Sarathi-Serve chunked prefill (colocated interference control) and Ring Attention (sequence-parallel long context)