100x AI Reasoning — Tiny Recursive Model

ARC-AGI shows a model a few input–output grid examples and asks it to produce the output for a new grid. The task is easy for people and hard for language models. The Tiny Recursive Model (TRM) attacks it with a single 2-layer network of about 7 million parameters, trained to refine its own answer over and over.

The idea

TRM is a supervised reasoning model, not a generator. It predicts a whole answer grid at once, so one bad cell cannot derail an autoregressive chain. It carries two states between steps:

• y, the current proposed answer (an embedding of the predicted grid)
• z, a latent reasoning feature that cannot be decoded on its own

A single network net(x, y, z) is applied recursively. Each latent recursion updates z six times in a row, then updates y once. Each deep recursion runs the full latent recursion several times; early passes run without gradients to move the state, and the last pass carries gradients. The answer is supervised on every supervision step, up to 16 of them, and the improved y and z are carried forward.

That carrying is the trick. The model learns to improve an answer rather than emit one in a single shot, so it can correct its own earlier mistakes. This is called deep supervision, and it is what lets a 2-layer network behave like a much deeper one.

Worked example

Take T = 3 recursions and n = 6 latent updates. One supervision step touches 3 × 7 × 2 = 42 layer applications of forward compute, yet backpropagation flows through only the final recursion: 7 × 2 = 14 layers. The state that arrives at the next supervision step is the product of countless passes, so the network can iterate far beyond its literal depth.

The paper's "less is more" findings are equally notable. Two layers generalized better than four, a single network beat two separate networks, and an exponential moving average of the weights (0.999) was needed to keep tiny-dataset runs from collapsing. The reported result is 44.6% on ARC-AGI-1 and 7.8% on ARC-AGI-2, using about 7M parameters, compared with 40.3% and 5.0% for the 27M-parameter Hierarchical Reasoning Model. Training uses roughly 1,000 examples per task with heavy augmentation.

In code

import torch

def latent_recursion(net, x, y, z, n=6):
    for _ in range(n):        # refine the latent reasoning state
        z = net(x, y, z)
    y = net(y, z)             # update the proposed answer
    return y, z

def deep_recursion(net, x, y, z, n=6, T=3):
    with torch.no_grad():
        for _ in range(T - 1):        # move the state, no gradients
            y, z = latent_recursion(net, x, y, z, n)
    y, z = latent_recursion(net, x, y, z, n)   # one pass with gradients
    return y.detach(), z.detach()

The detach is deliberate: each supervision step starts from the previous state but does not backpropagate through it.

Check yourself

1. What two states does the network carry between supervision steps, and why does it need both?
2. Why does deep supervision help even though only the final recursion pass carries gradients?
3. TRM has about 7M parameters and HRM about 27M. Why can the smaller network generalize better on these tasks?

Key takeaways

• TRM frames reasoning as iterative refinement of an answer plus a latent state.
• Deep supervision with carried state emulates depth without adding layers.
• On tiny datasets, recursion and weight averaging can beat raw capacity; the paper reports about 45% on ARC-AGI-1 with 7M parameters.