Recursive Language Models

Frontier models still choke on long prompts in two ways. First, a hard context window — past ~272K tokens for GPT-5 in this paper, the input simply does not fit. Second, context rot: quality falls as prompts get longer and tasks get denser, even inside the window. Compaction (summarize when full) and coding agents help some tasks, but they either throw away early detail or still stuff the prompt into the model history.

Recursive Language Models (RLMs) flip the interface. The user prompt $P$ is not fed into the Transformer as tokens. It is loaded as a variable in a persistent REPL environment. The root model sees only constant-size metadata (length, a short prefix, how to access slices), then writes code to peek, chunk, and — crucially — recursively call an LLM or nested RLM over programmatic snippets. The scaffold still looks like a language model from the outside: string in, string out.

What this paper explains

RLMs are an inference-time paradigm for scaling effective input (and, by stitching sub-calls, effective output) far past the base model's window. The authors evaluate with GPT-5 and Qwen3-Coder-480B-A35B on tasks whose difficulty scales differently with length:

They also show a small post-trained model, RLM-Qwen3-8B, distilled from RLM trajectories of the large coder — evidence that “acting as an RLM” can be trained, not only prompted.

Prior limits

• Bigger windows alone do not fix context rot. Figure 1 in the paper shows GPT-5 degrading as length and task complexity rise; past the red region the input does not fit at all.
• Compaction / condensation repeatedly summarizes once a length threshold is hit. That assumes early detail can be forgotten — fatal for dense aggregation where almost every line matters.
• Retrieval / CodeAct agents can search or execute code, but if the user prompt still lives in the model history, you inherit the window and fall back to compaction.
• Verbalized sub-agents (delegate a few subtasks in natural language) cannot express loops that launch $\Omega(|P|)$ or $\Omega(|P|^2)$ sub-calls over slices; they are also capped by the model's output length when answers must be spoken token-by-token.

The tension: you want dense access to an unbounded prompt without paying unbounded tokens in the root window.

The mechanism

Algorithm (intuition)

1. Init REPL with prompt = P and a sub_RLM / llm_query function.
2. Seed the root history with metadata only (not $P$).
3. Loop: model emits code → REPL executes → only truncated stdout metadata returns to history.
4. Model builds buffers in variables; when it sets Final (via FINAL(...) or FINAL_VAR(...)), return that string.

Keeping stdout metadata small is intentional: it forces the model to store long strings in REPL variables and sub-calls instead of polluting the root window.

Three design choices (vs a “similar-looking” scaffold)

The paper’s Algorithm 2 looks adjacent — tools, code, sub-LLM — but fails expressively:

1. Symbolic handle to $P$ — RLM puts the prompt in the environment. Naive scaffolds put $P$ in hist and immediately inherit the window.
2. Output via environment — returning a REPL variable allows answers longer than one context window by stitching sub-call results.
3. Symbolic recursion — code can loop over slices and launch unbounded sub-calls. Separate “Exec” and “sub_LLM” actions that cannot be composed programmatically only support a few verbalized delegates.

Recursion depth

• Depth 0 — REPL offload only; no sub-calls. Already unlocks beyond-window inputs (and beats many baselines on CodeQA / BrowseComp+).
• Depth 1 — sub-LLM calls from code (authors’ default for main GPT-5 runs often use GPT-5 root + GPT-5-mini leaves).
• Depth ≥ 2 — sub-calls can themselves be RLMs; helps most on information-dense tasks like OOLONG-Pairs.

Algorithm walkthrough

A typical successful trajectory on a dense task:

1. Probe — print length / structure of context; peek at prefixes and delimiters.
2. Decompose — choose a chunking strategy (by document, line, header, or pair blocks).
3. Map — loop: answers.append(llm_query(f"... {chunk}")) into a buffer variable.
4. Reduce — one more sub-call (or local code) aggregates buffers into the answer.
5. Finalize — FINAL_VAR(final_answer) so the response can be longer than a single generation.

On BrowseComp-Plus, models often use priors to narrow which documents to query. On OOLONG-Pairs, stitching many pairwise sub-calls inside the REPL is essentially required — base GPT-5 scores ~0.1 F1 while RLM(depth=1) reaches 58.0 in the authors’ table.

What to notice when reading

• Figure 1 is the whole motivation: same model, same tasks, length on the x-axis — vanilla collapses; RLM holds.
• Table 1’s gray cost bands: RLMs are often comparable or cheaper than compaction / coding agents, not just more accurate.
• Depth is not free: Qwen3-Coder’s higher depths can hurt when syntax errors propagate into nested RLMs (Figure 4b).
• First decomposition matters (Figure 4a): in-context RLM examples in the system prompt improve both the initial plan and final score on OOLONG — even when the example domain differs.
• Training insight: leaf sub-calls look like ordinary LLM requests; the scarce skill is being a good root — REPL hygiene, when to recurse, how to chunk.

Results and evidence

Authors’ reported numbers (Table 1 / observations — not timeless absolutes):

GPT-5 family (RLM root GPT-5, recursive calls GPT-5-mini where noted):

∗ hit input context limits on some runs.

Median gains cited for GPT-5 RLM vs alternatives: +26% vs compaction, +130% vs CodeAct with sub-calls, +13% vs Claude Code across the evaluated suite. On BrowseComp+ (6–11M tokens), RLM(depth=1) averages about $0.99 vs a linear extrapolation of $1.50–$2.75 for stuffing GPT-5-mini with the full input.

Beyond long context: on LongCoT-mini, RLM(GPT-5.2, depth=1) rises from a 38.7 base overall to 50.6, and to 65.6 with explicit decomposition hints — the REPL builds a reasoning graph and solves nodes via sub-calls.

Training: RLM-Qwen3-8B, fine-tuned on ~1,000 filtered RLM trajectories from Qwen3-Coder on LongBenchPro (unrelated to the eval suite), improves median performance by ~28% as an RLM vs base Qwen3-8B and runs more than 3× faster on trajectories by making fewer mistakes. A separate RLVR run on short MRCRv2 splits generalizes to longer, harder needle settings.

Limitations

From the paper (and their Appendix B negative results), compressed:

• Guardrails for exploding sub-call cost / runaway recursion are under-explored.
• Blocking sequential LM calls make wall-clock slow; async sub-calls are left to future systems work.
• Models need solid coding ability; small models struggle as RLM roots without training.
• FINAL / FINAL_VAR tagging is brittle — models sometimes emit plans as final answers.
• One system prompt does not transfer cleanly across model families (Qwen needed an extra “don’t spam sub-calls” line).
• Harder natural long-context workloads and production sandboxing remain open.

How to read the paper

1. Abstract + §1 — context rot framing and Figures 1–2.
2. §2 — Algorithm 1 vs Algorithm 2 (the three design choices).
3. §3 — task taxonomy by complexity scaling; baselines.
4. §4 / Table 1 — main results; Observations 1–6.
5. §5 — trajectory analysis (decomposition, syntax errors).
6. Appendix A — RLM-Qwen3-8B distillation recipe; Appendix C — system prompts.

Knowledge check

Keep reading

1. Original paper (arXiv:2512.24601) — Figures 1–2, Table 1, §2 algorithms.
2. Code release — github.com/alexzhang13/rlm.
3. Related Fanout Daily: PagedAttention, speculative decoding, Orca continuous batching, KV-cache survey.
4. Context rot discussion cited by the authors — Hong et al., 2025.

Sources

• Zhang, Kraska, Khattab, Recursive Language Models, 2025 — arXiv:2512.24601
• Bertsch et al., OOLONG, 2025 — arXiv:2511.02817
• Chen et al., BrowseComp-Plus, 2025 — arXiv:2508.06600
• Bai et al., LongBench v2, 2025 — arXiv:2412.15204
• Wang et al., CodeAct, 2024 — arXiv:2402.01030