ML Pipeline with DVC & AWS S3

DVC turns a folder of scripts into a pipeline whose outputs are hash-addressed and cacheable, and an S3 bucket into the shared cache for a team. dvc repro reruns only the stages whose inputs actually changed; dvc push and dvc pull move the bytes. The result is a training run that another machine can reproduce from a commit.

The idea

Three pieces make it work:

• dvc.yaml — the stage graph. Each stage declares cmd, deps, outs, and params.
• dvc.lock — the resolved hashes of every dependency and output after the last successful run.
• A remote — an S3 prefix plus credentials, configured once per repo.

The key behavior: DVC compares hashes, never timestamps. If nothing a stage depends on changed, the stage is skipped and its outputs are restored from local cache. Touch an unrelated file and nothing reruns.

S3 setup essentials:

• dvc remote add -d s3remote s3://bucket/dvc-store
• Credentials from AWS_PROFILE, environment variables, or an instance role — never committed to .dvc/config.
• Bucket versioning and a lifecycle rule that does not expire cache objects. A deleted cache object is a permanently unreproducible experiment.

Keep the remote and the Git remote decoupled. git push does not push data; dvc push does. CI needs both.

Worked example

A three-stage DAG: prepare → train → evaluate.

1. Edit train.max_epochs in params.yaml.
2. Run dvc repro. DVC hashes params.yaml and the stage deps, sees that prepare depends only on params.prepare.seed and data/raw, and skips it.
3. train reruns; evaluate reruns because its dependency models/model.pt changed.
4. dvc.lock now holds new hashes — commit it together with the metrics.

With a 20-minute prepare step and a 2-minute train, skipping the unchanged stage is the difference between a 2-minute iteration and a 22-minute one. Inspect the graph with dvc dag; force a full rerun with dvc repro --force when you suspect stale cache.

In code

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw
    params:
      - prepare.seed
    outs:
      - data/processed
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/processed
    params:
      - train.lr
      - train.max_epochs
    outs:
      - models/model.pt
      - metrics.json

After a successful dvc repro, run dvc push so the new outputs land in S3 for the next collaborator or CI job.

Check yourself

1. What exactly does dvc repro compare when deciding to skip a stage?
2. Where should S3 credentials come from, and why not from .dvc/config?
3. What is lost forever if a cache object is deleted from the bucket?

Key takeaways

• A hash-based DAG means only stale stages rerun.
• Commit dvc.lock; never commit the data bytes.
• The remote cache is the shared source of truth — protect it like a database.