Self-Study LLM

Reading papers is not the same skill as building the thing, and a self-study plan that only reads produces notes nobody can check. The alternative is a loop: pick a small target with a pass/fail test, implement the minimal version, train it on real data, measure honestly, and log what changed. The loop matters more than any single run, because the loop is what turns an evening into a skill.

The idea

Five steps, and the whole point is keeping each one small enough to finish.

1. Pick a target with a measurable test. "Reimplement GPT-2 small and reproduce its loss curve on a fixed corpus" is a target. "Understand transformers" is a mood.
2. Shrink before you scale. Get the shape right at 1M parameters, then move one axis at a time. A run that finishes in minutes lets you iterate; one that takes a week teaches nothing when it fails at hour 100.
3. Reverse-engineer from the reference. Read the config, then write your model so its parameter count matches the reference to the digit; a mismatch means the wrong architecture, not a rounding difference.
4. Budget in tokens per parameter. Pretraining compute is roughly proportional to parameters × tokens. The compute-optimal ratio found by the Chinchilla study is near 20 tokens per parameter; small runs usually want more, because step count is limited by time rather than data.
5. Keep a lab notebook. Config, commit hash, seed, loss curve, and one sentence on what changed. An ablation without a log entry is a rumor in a week.

The order matters: steps 2 and 3 are cheap and catch most mistakes; steps 4 and 5 are what make the result mean anything.

Worked example

Suppose your 12M-parameter model trains at 12,000 tokens per second. In one GPU-hour you process

At the 20-tokens-per-parameter ratio, 12M parameters want about $2.4 \times 10^8$ tokens — roughly 5.6 hours for a single pass. So the honest plan is a six-hour run, not a one-hour one.

With only one hour, cut the model instead of the ratio: a 2M-parameter model wants $4 \times 10^7$ tokens, about one hour at the same throughput. Training the 12M model on one hour's data is not a smaller experiment — it is a different and much worse one.

A cheap check: a freshly initialized model should produce a loss near the entropy of the uniform distribution over the vocabulary, $\ln V$. For $V = 50{,}257$ that is about 10.82 nats. If your first logged loss is 2.0, the labels are leaking into the input.

In code

import math

def budget(params, tokens_per_param=20, tokens_per_sec=1.2e4):
    tokens = params * tokens_per_param
    return tokens, tokens / tokens_per_sec / 3600   # (tokens, GPU-hours)

print(budget(12_000_000))   # (240000000, 5.555...)
print(math.log(50257))      # 10.8249... -> expected loss at step 0

Run this sanity check before every long run.

Check yourself

1. Why check the initial loss against $\ln V$ before looking at any part of the training curve?
2. Why is a 2M-parameter model a better first target than a 2B-parameter one, when the goal is to learn?
3. What should a notebook entry contain for an ablation to still be usable a month later?

Key takeaways

• Shrink the target until data, training, evaluation, and logging all run end to end in minutes.
• Match parameter counts against a reference implementation to prove the architecture is right.
• Budget by tokens per parameter, and treat a missing log entry as a lost experiment.