Monitoring Concept Drift

Concept drift is a change in the relationship between inputs and the target, $P(Y \mid X)$. The features can look exactly the same while the correct answer changes, because the world moved and the rule the model learned no longer holds. Data drift and concept drift both degrade a model, but only concept drift guarantees the old mapping is wrong.

The idea

• Data drift changes $P(X)$. Concept drift changes $P(Y \mid X)$. A model can survive a large shift in inputs if the decision boundary still separates the classes.
• Shapes matter: sudden (a rule or policy change), gradual (slow consumer behavior change), recurring (a weekly or seasonal cycle), and blip (an outage that reverts).
• Recurring patterns are usually features, not drift. If weekends behave differently every week, the fix is a day-of-week feature, not a retrain.
• Detection requires labels or a proxy. Ground truth may arrive days later, so teams use reviewed samples, dispute or chargeback outcomes, or a correlated early label such as a click standing in for a purchase.
• Under covariate shift alone, the best possible predictor given the true conditional is unchanged; reweighting training data by the density ratio can help calibration, but the underlying rule still holds.

Worked example

Consider a fraud model after a payment provider rolls out stronger authentication on some transactions. The same features now describe transactions with a lower fraud rate, so the mapping from features to labels has moved: a pattern that once predicted fraud now mostly predicts a verified purchase. This is concept drift, and no amount of input monitoring will show it, because P(X) is unchanged.

Labels arrive only when chargebacks settle, roughly thirty days later. Waiting a month to detect the problem is unacceptable, so the team builds a leading indicator: a matched sample of flagged transactions reviewed by analysts within 24 hours. Agreement between the review labels and the model's predictions becomes the early signal, monitored weekly, with the settled chargebacks as the eventual ground truth that validates it.

Check yourself

1. What changes under concept drift that does not change under pure covariate shift?
2. Why is a recurring weekly pattern usually not concept drift?
3. When labels arrive weeks late, what can you monitor in the meantime?

Key takeaways

• Concept drift is a change in $P(Y \mid X)$; input monitoring alone cannot see it.
• Detecting it requires labels or a proxy that tracks the real outcome closely.
• Treat recurring patterns as features, and reserve retraining for genuine changes in the rule.