Monitoring Data Drift

Inputs change even when the model does not. Data drift is a shift in the distribution of features, $P(X)$, between the reference period you trained on and live traffic. It is the earliest warning you get, because features arrive immediately while labels lag by hours, weeks, or never.

The idea

Monitoring drift means comparing the live distribution of each feature against a reference and deciding whether the gap is meaningful.

• Univariate tests: population stability index (PSI), KL divergence, or a Kolmogorov–Smirnov test on numeric features, plus chi-square on categoricals. They are cheap and explainable, and they miss interactions.
• Multivariate drift: train a classifier to separate reference rows from live rows. An area under the curve near 0.5 means the two sets are indistinguishable; a high value means something separates them, and feature importances name the culprit.
• Binning matters: PSI depends on the histogram edges. Freeze the edges from the reference data and reuse them, or two runs are not comparable.

where $p$ is the reference bin proportion and $q$ is the live proportion.

A common rule of thumb treats PSI below 0.1 as stable, 0.1–0.25 as worth watching, and above 0.25 as an investigation trigger. Treat those numbers as conventions, not verdicts. Drift in a feature the model barely uses often changes nothing, while a small shift in a high-importance feature can move every score.

Worked example

A model uses income, binned into four ranges. The reference proportions are p = [0.40, 0.30, 0.20, 0.10] and this week's live proportions are q = [0.25, 0.30, 0.25, 0.20].

• Bin 1: $(0.25-0.40)\ln(0.25/0.40) = 0.071$
• Bin 2: $0.00 \times \ln(1.0) = 0$
• Bin 3: $(0.25-0.20)\ln(0.25/0.20) = 0.011$
• Bin 4: $(0.20-0.10)\ln(0.20/0.10) = 0.069$

The total is about 0.151 — moderate. That alone does not justify retraining. Check whether income is important in the model, whether the shift is one day of unusual data or a sustained trend, and whether the score distribution moved with it. Drift is a symptom to investigate, not an action to take blindly.

In code

import numpy as np

def psi(reference, live, bins=10):
    edges = np.histogram_bin_edges(reference, bins=bins)  # freeze from reference
    p = np.histogram(reference, edges)[0] / len(reference)
    q = np.histogram(live, edges)[0] / len(live)
    eps = 1e-6
    p, q = np.clip(p, eps, None), np.clip(q, eps, None)
    return float(np.sum((q - p) * np.log(q / p)))

Check yourself

1. Why does data drift give you a signal before label-based accuracy does?
2. What does PSI compare, and why does changing the bin edges change the result?
3. Why is drift in a low-importance feature less alarming than drift in a top feature?

Key takeaways

• Data drift watches $P(X)$ and is measurable immediately, without labels.
• Freeze the reference distribution and the bin edges, or the number is meaningless.
• Combine sustained drift with feature importance before you spend compute on a retrain.