Reproducible Environments

An environment is reproducible when the same commit plus the same lockfile produces the same numbers on a clean machine. "The same libraries are installed" is not enough — version resolution, the CUDA/driver pairing, and unseeded randomness all break bit-exactness. A second person must get either identical results or an explicit error, never a silently different model.

The idea

Pinning happens at three levels, and only the last one is trustworthy:

• Loose — pip install torch resolves to whatever is current today.
• Pinned — torch==2.4.0 fixes the top-level version but leaves transitive dependencies free.
• Locked and hashed — every transitive package with a hash, plus a base image pinned by digest.

What this buys: no environment resolves itself at install time. Tools such as uv pip compile, pip-tools, and conda-lock write the full graph once; CI installs with --require-hashes.

Know the boundary. A Python lockfile does not pin the system libraries, the CUDA runtime, or the GPU driver. Only a container image digest covers those. That is why the lockfile and the image are complementary, not redundant.

Determinism needs more than versions:

• Seeds for Python, NumPy, and each framework.
• torch.use_deterministic_algorithms(True) to refuse nondeterministic kernels.
• Awareness that some cuDNN convolutions and atomic GPU ops are nondeterministic, and that cuDNN bench-marking selects algorithms run to run.
• Fixed DataLoader worker counts and ordering, since worker processes consume the RNG independently.

Worked example

Environment A resolved numpy==1.26.4; environment B, created a month later from the same loose spec, resolved numpy==2.0.1. The code calls np.float_ — a removed alias in NumPy 2.0 — so B fails at import while A trains fine. Both environments satisfy the written requirements. The difference is only visible in the lockfile.

The same class of problem appears on GPU: torch==2.4.0 ships as several CUDA-tagged wheels (12.1, 12.4). They are the same version number but different compiled kernels, so small numeric differences appear across machines with identical code.

In code

import os, random
import numpy as np
import torch

def seed_everything(seed: int = 0) -> None:
    os.environ["PYTHONHASHSEED"] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.use_deterministic_algorithms(True)

Call this before any data loading. Seeds make runs comparable; they do not make them identical across different kernels or library versions.

Check yourself

1. Why can pinning top-level versions still leave two "identical" environments disagreeing?
2. What sources of nondeterminism does seeding fail to fix?
3. Why should CI install the same lockfile the trainer uses instead of resolving fresh?

Key takeaways

• Pin transitively with hashes; resolve once, install everywhere.
• Lockfiles cover Python; container digests cover the OS and CUDA stack.
• Seeding is necessary but not sufficient for exact reproduction.