Incident Response for ML Systems

ML incidents look like ordinary outages until they are not: the service is healthy, latency is fine, and the predictions are wrong. Response needs the standard discipline of detect, mitigate, and learn, plus the ML-specific evidence required to tell which failure you are facing. The most common mistake is spending the first hour diagnosing while users are still being served bad answers.

The idea

Classify before you diagnose, because the classes have different mitigations:

• Service failure — the endpoint is down, timing out, or erroring.
• Data and feature failure — stale features, an upstream schema change, silent nulls falling back to defaults.
• Model failure — degradation, drift, or a bad rollout.
• Business and label failure — the metric moved because the label pipeline changed, not the model.

Then follow the rules that make response effective:

• Mitigate before you diagnose. Roll back the model version, disable the endpoint, or fall back to a previous or heuristic scorer. Rollback must be rehearsed and one command, not a meeting.
• Preserve evidence. The request logs with feature values, the exact model version, and the deployment timeline. A restart or a redeploy can destroy the only copy of what actually happened.
• Assign roles. An incident commander decides, a communications lead writes updates, and a scribe keeps the timeline. The commander is one person even when several teams are involved.
• Review without blame, and end with dated action items plus a test or alert that would have caught the failure earlier.

Common anti-patterns: paging the model's author instead of the service on-call who owns the pager; "retrain and see" as a mitigation; and bundling a data migration with a model release, which removes the rollback path.

Worked example

Walk through a constructed scenario. Recommendation click-through drops by half and the service metrics are green. The commander confirms features are fresh and latency is normal, then checks the deployment timeline: a new model version went live six hours earlier. The first action is to pin the previous version and confirm recovery — mitigation, not investigation. Evidence is preserved by snapshotting the prediction records for the affected window before any restart. Only afterward does the team establish the cause: the training window for the new version overlapped a promotional period, so the model over-weighted promotion-specific behavior. The lasting fixes are a training-window check in the pipeline and a guardrail that halts a rollout when click-through on the canary falls below the control.

Check yourself

1. Why is rollback attempted before the root cause is known?
2. Which ML-specific evidence must survive an incident for the review to be possible?
3. What failure class does "everything is healthy but the numbers are wrong" describe?

Key takeaways

• Mitigate first: roll back or fall back, then diagnose.
• Request logs with feature values and the exact model version make the review possible.
• Every incident should end with a lasting detection or test, not only a fix.