CI for Machine Learning

CI for machine learning is CI for code plus CI for data and model behavior. The tests that catch real production failures are rarely full-dataset accuracy runs — they are schema checks, leakage checks, and a fast smoke train on a tiny sample. If a pull request has to train the whole model before it can be judged, the checks will be skipped under deadline.

The idea

Layer the checks cheapest first:

• Lint and types — ruff, mypy. Seconds, no data.
• Unit tests on pure functions: metric math, windowing, feature transforms.
• Data tests — schema, dtype, allowed ranges, null rate ceilings, duplicate IDs, label distribution, and time ordering so no future rows leak into training.
• Pipeline smoke test — dvc repro or the orchestrator on a fixed fixture, asserting the DAG runs end to end and produces an artifact.
• Model contract tests — a golden input with a recorded expected output, serialization round-trip, shape and dtype guarantees, and a latency sanity bound.
• Quality gate — compare the candidate metric against the incumbent on the same held-out set and fail on regression beyond a tolerance.

Three rules keep the suite trustworthy:

• Deterministic and fast. The heavy job runs on a schedule or behind a label, not on every push.
• No live external services. Use fixtures and recorded responses so outages do not fail honest code.
• The gate must not read the tuning set. A gate evaluated on data used for model selection is leakage with extra ceremony.

Worked example

A refactor rewrites the feature function. Full-dataset accuracy is unchanged — the change was a no-op for the mean — but tenure_days now has a 4% null rate, because the new code drops rows where an upstream join misses.

A data test asserting null_rate("tenure_days") < 0.01 fails in seconds with a clear message. Without it, the pipeline would have spent an hour retraining and shipped a model with a 4% null rate that quietly shifted predictions for a subset of customers. The cheap test caught what the expensive one could not see.

In code

name: ci
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12", cache: pip}
      - run: pip install --require-hashes -r requirements.lock.txt
      - run: ruff check . && mypy src
      - run: pytest -q tests/unit
      - run: pytest -q tests/data
      - run: python -m src.train --sample-fraction 0.02 --max-epochs 1
      - run: python -m src.gate --candidate metrics.json --tol 0.005

The smoke train uses a sample fraction, so the gate step compares on a small but identical slice for both candidate and incumbent.

Check yourself

1. Why is a data schema test often more valuable than a full-dataset accuracy test?
2. What makes a model quality gate meaningless even when it reports PASS?
3. Which checks belong on every pull request, and which belong on a schedule?

Key takeaways

• Cheapest deterministic checks first: lint, unit, data, smoke, gate.
• Assert contracts — schema, golden output, latency — not just accuracy.
• Gate against an incumbent on a held-out set the tuning loop never touched.