Alerting and On-Call for ML

An alert is a promise that a human will stop what they are doing and act. Most ML alerting breaks that promise by firing on noise until the team ignores it. Good alerting starts from user-visible symptoms, adds a small number of model-specific signals, and gives every page an owner and a runbook.

The idea

• Page on symptoms, not causes. "Checkout error rate above the SLO burn rate" is actionable; "GPU utilization 92 percent" is trivia. The person on call should be woken by something a user would notice.
• Prefer burn-rate alerts to static thresholds. A multi-window burn rate on an error budget fires fast for severe outages and slowly for slow burns, which avoids both false alarms and late detection.
• Add a short list of ML-specific pages: the prediction endpoint is unhealthy, the feature store lookup error rate is elevated, no predictions have been served for N minutes (staleness), and score distribution has moved outside a pre-agreed band.
• Tier everything. A page wakes someone now; a ticket is handled next business day; a dashboard informs without demanding action. Most drift and quality signals belong in the latter two, not in the pager.
• Every page needs four things: an owner, a runbook link, a rollback or fallback path, and a stated criterion for acting versus observing.

Alert fatigue is measurable. Track pages per week per person and the fraction that led to a real action. Delete or downgrade any alert that repeatedly fires without producing one, and treat a noisy alert as a bug in the alert itself.

Worked example

A team configures data drift as "PSI greater than 0.1 on any feature." With 200 features evaluated daily, the alert fires almost every day, and within a month nobody reads it. The signal was real but the rule was useless.

The replacement combines three conditions: the feature must be in the top importance tier for the model, PSI must exceed 0.25, and the condition must hold for three consecutive days. That rule fires rarely, and when it does the on-call engineer has a specific next step: check the upstream pipeline, confirm the shift is real, and decide whether to retrain. Separately, the outcome metric gets its own burn-rate alert, so a real quality regression pages through the path that users actually feel.

In code

alert: feature_drift_sustained
expr: drift_psi{feature="income", importance="high"} > 0.25
for: 3d
labels:
  severity: ticket
  team: risk-ml
annotations:
  runbook: https://runbooks.internal/ml/drift
  summary: "income drifted beyond the guardrail for three days"

Check yourself

1. Why page on user-visible symptoms instead of model internals?
2. What distinguishes a page from a ticket, and where does most drift alerting belong?
3. What makes an alert actionable?

Key takeaways

• Every page must require action; everything else is a ticket or a dashboard.
• Sustained, importance-weighted drift beats a threshold on every feature.
• No owner and no runbook means no page.