Experiment Tracking

Experiment tracking is a database of what you tried and what happened. Its job is to make a run reproducible and comparable months later, when the notebook is gone and only a run ID and a metrics table survive. Without it, choosing a model is a memory test rather than an engineering decision.

The idea

A tracked run records five things:

• Code — the Git commit, plus whether the working tree was dirty.
• Params — hyperparameters, data version, split seed.
• Metrics — scalar series over steps, not just the final number.
• Artifacts — checkpoints, plots, and the model itself.
• Environment — lockfile or image digest.

Tools such as MLflow, Weights & Biases, and Neptune differ mainly in hosting and UI. Three disciplines matter more than the tool:

• Log from the script. A metric you must remember to type in is a metric you will lose. Wrap the training entry point so logging is not optional.
• One run, one config. A sweep creates one run per configuration, including failed ones. Overwriting a row destroys the comparison.
• Metrics are time series. A final validation loss can look fine while the curve spiked when you changed the augmentation mid-run.

Tracking is not a model registry. Tracking answers "what did I try and what did it score"; a registry answers "what is approved to serve right now." The run is the evidence; the registry entry is the decision.

Worked example

A sweep over lr ∈ {1e-3, 3e-4, 1e-4} and dropout ∈ {0, 0.1} produces six runs. The lowest final validation loss is 0.41 at lr=3e-4, dropout=0.1. Logging only finals, you ship that run.

Logging per-step loss plus the git diff tells a different story: that run's loss curve jumps upward at step 4,000, exactly where the augmentation code was edited. The run is contaminated, not best. It is now excluded, and the honest winner is the 0.43 run that trained cleanly. Only step-level metrics made the spike visible at all.

In code

import mlflow

with mlflow.start_run(run_name="baseline-lr3e4") as run:
    mlflow.log_params({"lr": 3e-4, "dropout": 0.1, "seed": 0, "data_hash": dh})
    for step, loss in enumerate(train_loop()):
        mlflow.log_metric("train_loss", loss, step=step)
    mlflow.log_metric("val_loss", val_loss)
    mlflow.log_artifact("config.yaml")
    mlflow.pytorch.log_model(model, "model")
    print(run.info.run_id)

The printed run ID is what you paste into a ticket, a registry entry, or a slide.

Check yourself

1. Why log per-step metrics instead of only the final value?
2. What does a registry tell you that a tracking run does not?
3. Two runs are only directly comparable if they share which recorded fields?

Key takeaways

• Track automatically from code; one run per configuration, failures included.
• Store the commit and data hash with every run.
• Final scalars hide failures that curves reveal.