Code, Write & Publish AI Research Paper

A paper is not a description of a model. It is a falsifiable claim with evidence attached, plus enough detail that someone else can reproduce the evidence. The workflow that produces one is ordinary engineering with an unusual amount of bookkeeping — and the bookkeeping is what reviewers actually judge.

The idea

Four phases, each ending in an artifact you can point at.

1. Frame the claim. Write one sentence: "X improves Y on Z, because W." If you cannot state what observation would refute it, it is not a claim yet. Write the abstract first — the paper you are aiming at — and expect it to be wrong; it is a target, not a summary.
2. Build the comparison. The baseline must be tuned at least as hard as your method, and every arm should get the same compute and parameter budget. A win bought with ten times the compute is a systems result, not a methods result, and should be reported as one.
3. Make the evidence statistical. Multiple seeds, mean and spread, and the number of runs stated in the caption. A single run cannot separate an effect from initialization noise. Fix seeds, and record every hyperparameter, including the ones you changed while debugging.
4. Write for the person who will reproduce you. Section order: abstract, introduction, related work, method, experiments, limitations, conclusion. Limitations is not a formality — it is the section reviewers read to decide whether you understand your own result.

Artifacts finish the job: code, configs, a pinned data version, and one script that regenerates the main table. A preprint on arXiv establishes priority; a conference or workshop submission subjects the claim to review, which is a rewrite cycle rather than a verdict.

Worked example

An ablation table with illustrative numbers; the point is the reporting format, not the values. Three seeds per arm, same data split:

Baseline: mean 71.5, sample standard deviation ≈ 0.79. Method: mean 73.3, sample standard deviation ≈ 0.62. The gap is 1.8 points, roughly 2.3× the baseline spread.

With three seeds, that is encouraging but far from conclusive — the honest write-up is "73.3 ± 0.6 versus 71.5 ± 0.8 over 3 seeds," with per-seed values in an appendix. Quoting a p-value from three samples would overstate what the data supports, and quoting a bare 73.3 versus 71.5 with no spread is the mistake that gets a paper rejected on its own table.

In code

import statistics

def report(name, accs):
    m = statistics.mean(accs)
    s = statistics.stdev(accs)          # sample std, n - 1 denominator
    return f"{name}: {m:.1f} ± {s:.1f} (n={len(accs)})"

print(report("baseline", [71.2, 72.4, 70.9]))    # baseline: 71.5 ± 0.8 (n=3)
print(report("long-ctx", [73.1, 74.0, 72.8]))    # long-ctx: 73.3 ± 0.6 (n=3)

Generate the table from this function so text and table can never disagree.

Check yourself

1. Why must the baseline be tuned at least as hard as the proposed method?
2. What does a 1.8-point gain over three seeds tell you, and what does it not tell you?
3. Which part of the paper most often decides whether a reviewer trusts the experimental setup?

Key takeaways

• A paper is a falsifiable claim plus evidence another lab could reproduce.
• Report seeds, spread, and compute parity; a single number is not a result.
• Write the abstract and the limitations section first — they constrain everything else.