Logging and Observability

Observability for ML means being able to answer "why did this request get this score?" without redeploying or guessing. A plain service needs logs, metrics, and traces; a model needs all three plus the feature values, the model version, and the score, because a prediction is only explainable next to its inputs. The goal is not maximum data collected but the smallest set of records that reconstructs any decision after the fact.

The idea

The three pillars, applied to a model:

• Logs are discrete events. Emit one structured record per prediction, not free-text lines.
• Metrics are aggregates over time: request rate, latency percentiles, error counts, score quantiles. They are cheap to store and alert on.
• Traces follow one request across services, so a slow prediction can be attributed to the feature lookup, the model, or the post-processing step.

What an ML record must add:

• Identity: request_id or trace_id, so the record joins the trace.
• Version: the model name and exact version or artifact hash.
• Inputs: the feature values (or a hash plus the raw values in a linked blob). Without them, debugging is guesswork.
• Output: the score, the threshold applied, and the decision.

Two constraints shape the design. Cardinality: metric labels with unbounded values such as user_id explode storage and cost, so keep high-cardinality keys in logs and traces and use metric labels only for bounded sets. Privacy: prompts, documents, and user attributes are sensitive, so redact or hash at write time and set a retention window, because deleting from logs later is far harder than never writing the field.

Worked example

A request returns a score of 0.91 when the operator expected 0.12. With only the score and the version in the logs, the investigation has no next step. With a full record you can see that the feature lookup missed in the online store and silently substituted a default of zero for days_active, which turned a long-tenure customer into an apparently brand-new account. That is a serving bug, not a model bug, and the log proved it in minutes. The same record also tells you how many other requests hit the same fallback, which turns one complaint into a measurable incident size.

In code

record = {
    "request_id": request_id,
    "model": "churn-scorer",
    "model_version": "v42",
    "route": "online",
    "features": {"days_active": 12, "plan": "pro"},
    "score": 0.91,
    "decision": "contact",
    "latency_ms": 43,
    "feature_fetch": {"store_hit": False, "fallback": "default"},
}
logger.info("prediction", extra=record)

Check yourself

1. Which fields must a prediction record include to make a bad score explainable after the fact?
2. Why do per-user metric labels cause trouble at scale, and where should those keys live instead?
3. Why redact sensitive fields at write time rather than at read time?

Key takeaways

• Log the inputs, the version, and the output together; a score without its features cannot be debugged.
• Keep unbounded cardinality in logs and traces, not in metric labels.
• Decide retention and redaction before the first request, not after an incident.