S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Production LLM stacks rarely ship one frozen model. Teams fine-tune many small LoRA adapters on a shared base — one per customer, tool, or style — then need to serve them concurrently. Naively merging each adapter into its own weight copy, or packing separate vLLM processes, burns GPU memory and loses the chance to batch requests that share the same base $W$.

S-LoRA treats multi-adapter serving as a systems problem: keep the adapter zoo in host memory, fetch only the adapters needed by the currently running batch onto the GPU, and manage those dynamic $A,B$ tiles together with the KV cache inside one Unified Paging pool. The base forward stays batched; the LoRA delta $xAB$ is computed on the fly with custom kernels that tolerate heterogeneous ranks and non-contiguous pages.

What this paper explains

Low-Rank Adaptation (Hu et al., 2021) replaces a fine-tuned update $\Delta W$ with a product of thin factors. If the base forward is $h = xW$, LoRA uses

with $A \in \mathbb{R}^{d_{\mathrm{in}} \times r}$, $B \in \mathbb{R}^{r \times d_{\mathrm{out}}}$, and rank $r \ll \min(d_{\mathrm{in}}, d_{\mathrm{out}})$. Training updates only $A,B$; inference can either merge $W' = W + AB$ or keep the delta separate.

When many adapters share one base, merging each into its own $W'$ duplicates the giant base. HuggingFace PEFT-style serving that loops adapters serially also throws away iteration-level batching across requests. S-LoRA’s claim: store all adapters in main memory, page the active set into GPU memory with the KV cache, and batch base compute while running heterogeneous LoRA kernels.

Prior limits

• Merge-per-adapter. Materialize $W+AB$ for each adapter. Simple for one model; catastrophic when you want hundreds of personalized copies on one GPU.
• vLLM-packed. The paper’s baseline merges LoRA into the base and packs multiple vLLM model instances. Few adapters fit; requests for different adapters cannot share a batch.
• PEFT multi-adapter loops. Parameter-efficient libraries can load adapters, but without unified paging and continuous batching the throughput collapses as $n$ (adapter count) grows — Table 3 shows PEFT already at $0.25$ req/s for $n=100$ on setup S1 while S-LoRA stays near $8$ req/s.
• KV-only paging. PagedAttention (vLLM) pages the KV cache. Adapter weights with varying ranks add a second dynamic tensor family that also fragments GPU memory if allocated naively.

The mechanism

Separate base and delta. Keep one base $W$ resident. For each request’s adapter, compute $xAB$ on the fly (Eq. 2 in the paper) instead of merging. Base matmuls batch across adapters; LoRA work is adapter-specific but cheap relative to $xW$.

Host-resident adapter store. All adapters live in CPU / main memory. Only adapters touched by the current running batch are fetched to the GPU — so the number of serveable adapters is bounded by host RAM, not by packing full bases on the device.

Unified Paging. Extend the PagedAttention idea: one GPU memory pool allocates pages for (1) KV blocks of varying sequence length and (2) LoRA weight tiles of varying rank. A block table maps logical tensors → physical pages, cutting fragmentation when ranks and lengths mix inside one batch.

Heterogeneous batching kernels. Custom CUDA paths gather non-contiguous adapter pages and run batched LoRA GEMMs even when ranks differ across the batch (with clustering / padding strategies discussed in the paper).

Tensor parallelism. For multi-GPU, they shard the base and coordinate LoRA communication so scaling out does not reintroduce a merged-weight tax.

Algorithm / figure walkthrough

1. Admit online requests (with optional admission control / adapter clustering so a batch does not thrash the working set).
2. Prefetch the LoRA factors for adapters in the next iteration into the unified pool, overlapping I/O with compute when possible.
3. Run an iteration-level schedule (Orca-style): shared base forward for the batched tokens, then LoRA $xAB$ for each request’s adapter via the custom kernels.
4. Page KV growth and adapter presence through the same allocator; unused adapters can leave GPU memory while remaining on the host.
5. Decode continues; new adapters join when their requests are scheduled — throughput stays high even as $n$ grows into the thousands.

What to notice when reading

• Unified really means adapters and KV share the allocator — not that every adapter stays on GPU forever.
• Rank heterogeneity is a first-class systems issue: a batch with $r\in\{8,16,64\}$ needs paging + kernels that do not assume one contiguous packing.
• Throughput vs PEFT ($30\times$) and vs vLLM-packed ($4\times$) are different baselines measuring different failures (serial PEFT vs packed merged models).
• S-LoRA is complementary to continuous batching and PagedAttention; it specializes the stack for the many-adapter deployment pattern that plain LoRA training papers leave unspecified.

Results and evidence

Numbers below are from the paper’s reported experiments (arXiv:2311.03285), not independent re-runs.

Abstract / §1 headline. Versus HuggingFace PEFT, S-LoRA improves throughput by up to $30\times$. Versus vLLM with naive multi-LoRA support (vLLM-packed), up to $4\times$ throughput and several orders of magnitude more adapters served.

Table 3 (single A100 80GB). On synthetic setup S1, S-LoRA reports $8.05$ / $7.99$ / $7.64$ / $7.61$ req/s for $n \in \{5,100,1000,2000\}$ adapters, while vLLM-packed is $2.04$ at $n=5$ and OOM thereafter, and PEFT falls from $0.88$ ($n=5$) to $0.25$ ($n=100$). S-LoRA still serves $2000$ adapters in that table.

Ablations (Figure 5 narrative). Removing unified memory or replacing custom kernels with a slower BMM path hurts throughput and latency — the pool + kernels are not cosmetic.

Multi-GPU (Figure 8). Tensor-parallel scaling shows small overhead from LoRA communication; moving from 2→4 A100s more than doubles throughput in their plot under a compute-bound regime.

Limitations

Compressed from the paper’s scope and what it does not claim:

• Gains assume many adapters share one base; unrelated full fine-tunes still need separate weight sets.
• Host memory and PCIe/NVLink bandwidth bound how fast adapters can be swapped; pathological adapter churn can still thrash.
• Custom kernels and tensor-parallel LoRA paths add engineering surface area versus “merge and call vLLM.”
• Quality equals serving the intended adapter — S-LoRA does not invent a new training objective; bad adapters stay bad.
• Complementary to — not a replacement for — speculative decoding, KV eviction policies, or weight quantization.

How to read the paper

1. Abstract — $30\times$ / $4\times$ / “thousands of adapters” framing.
2. §2–3 — LoRA forward $h=xW+xAB$; why merging fails for multi-adapter serving.
3. Figure 1–2 — batching sketch and host→GPU adapter fetch.
4. §5 Unified Paging — pool layout for ranks + KV (the systems core).
5. §6 Tensor parallelism — multi-GPU communication cost.
6. §7 Tables 3– / Figures 5–8 — synthetic + real traces, ablations.

Knowledge check

Keep reading / Sources

• Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, arXiv:2311.03285 — https://arxiv.org/abs/2311.03285
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models — https://arxiv.org/abs/2106.09685
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM) — Fanout Daily 2026-07-21-pagedattention
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models — Fanout Daily 2026-08-04-orca-continuous-batching
• Fanout Daily LoRA explainer — 2026-07-12-lora