Data Versioning

Git versions code well and data badly: a 40 GB CSV does not belong in a commit, and data/final_v3.csv is a filename, not a version. Data versioning gives large immutable datasets the content-addressed identity Git gives text, so an experiment from six months ago still resolves to the exact bytes it trained on.

The idea

The mechanism has three parts:

• Content addressing. A dataset's identity is the hash of its content (DVC uses MD5; Git LFS uses a SHA-256 object ID). Change one byte and the hash changes, so old versions never silently mutate.
• Pointer in Git, bytes in a remote. A small tracked file records the hash; the data lives in S3, GCS, or a shared filesystem.
• A cache with deduplication. Identical content is stored once, so two near-identical dataset versions share most objects.

What this buys: checking out an older commit and running dvc checkout reproduces the training input exactly, and dvc push / dvc pull move bytes without touching history.

Bytes are not the whole contract. A data version also includes the label definition, the time window, and the filters applied. Those live in code, so version the split-generation script alongside the data. If splits are drawn from an unseeded shuffle, evaluation leaks across reruns even though the raw file is pinned.

Worked example

Take data/raw.csv at 100 MB. DVC hashes it to 9d3f7c1b... and copies it into .dvc/cache/9d/3f7c1b.... Git tracks only data/raw.csv.dvc, a few lines. The real CSV is git-ignored.

Now edit one row. The new hash is different, a second cache object appears, and the old object stays. The commit that used the old version still resolves. Two experiments that used byte-identical data point at the same object, so storage is not double-counted — deduplication is a consequence of content addressing, not a separate feature.

In code

outs:
  - md5: 9d3f7c1b2a4e5f60718293a4b5c6d7e8
    size: 104857600
    hash: md5
    path: raw.csv

Commit this pointer. Run dvc push to upload the bytes to the configured remote and dvc pull to restore them on another machine.

Check yourself

1. Why is a content hash a better version identifier than a name like final_v3?
2. Why split the pointer (Git) from the bytes (a remote) instead of storing both together?
3. Why must the split-generation code be versioned along with the dataset?

Key takeaways

• Content addressing makes datasets immutable and deduplicated.
• Git stores pointers; a remote stores bytes; a cache makes restores cheap.
• Splits are part of the data contract, not a detail of one script.