Simulation run on this device. Scenario values are not sent anywhere — the engine is fully local.
research game · free
Reproduction Sprint
Sequence environment, data, training, and eval against the clock.
Seven days to reproduce the headline number — plan it.
Engine 1.0.0 · game
7
- Days
- 7
- Paper
- not named
- Failure modes
- 3
| Day | Focus | Task |
|---|---|---|
| Day 1 | Environment + data | Pin versions, download dataset, reproduce the reported split counts |
| Day 2 | Baseline first | Run the paper's simplest baseline; record metric to compare later |
| Day 3 | Core method | Implement the headline component; unit-test one forward pass |
| Day 4 | Train / scale up | Launch a small run, check the loss curve against the paper's figure |
| Day 5 | Full eval | Re-run the eval protocol; capture variance across seeds |
| Day 6 | Diagnose gaps | Compare deltas to the paper; isolate the first divergence |
| Day 7 | Write it up | Repro notes, what matched, what did not, and the next experiment |
- mediumSilent data mismatch
the split or preprocessing differs from the paper → checksum + count the exact files on day 1
- mediumUndocumented hyperparameters
a missing LR/optimizer detail moves the result → ablate the missing knob before scaling
- mediumEval protocol drift
metric definition differs (micro vs macro, decoding) → re-read the eval section before comparing numbers
One week is enough to reach a defensible first comparison, not a full replication.
Method
- Distributes a seven-stage reproduction plan (environment → baseline → method → train → eval → diagnose → write-up) across the days available.
- Ships the three failure modes that kill most repro attempts, each with a pre-emptive move.