Feature Stores

A feature store is a shared system that computes features once and serves them consistently to training and inference. Its real product is not storage but a guarantee: the value a model was trained on is the value it sees at serving time. Teams often buy the complexity before they have the problem, so the first question is whether reuse and latency actually require it.

The idea

A feature store has two sides with opposite requirements:

• Offline store — columnar files or a warehouse table with full history, optimized for large scans that build training sets.
• Online store — a low-latency key-value store holding the latest value per entity, optimized for single-row lookups inside a request budget.

The core guarantee is point-in-time correctness. When you build a training set for a label observed at time $T$, each feature must be joined as of $T$, not as of today. Joining the latest value leaks the future into training: the model sees information that did not exist when the label was decided, so offline metrics look better than production forever. The leak is silent, which is why it survives review.

Supporting pieces:

• Feature definition — one transformation, versioned, ideally shared by both paths or verified equivalent.
• Freshness SLA — how stale an online value may be before it is a bug.
• TTL and defaults — an explicit policy for missing values at serving time. A silent default of zero is how bugs get into production.
• Entity key and timestamp — the join contract. Without a timestamp on every feature row, point-in-time joins are impossible.

Worked example

A churn model labels a customer at time $T$ and joins days_active from the current table. If the customer's account was closed and days_active later grew to a large value, the naive join teaches the model that a huge days_active predicts churn — the opposite of the truth. Point-in-time joins fetch the value recorded at or before $T$, which is what the model could have known at prediction time. The offline score with the leak can be excellent while production is useless, and no amount of drift monitoring connects the two.

In code

training = (  # point-in-time correct feature join
    labels.select(["customer_id", "label_ts", "label"])
    .join_asof(
        features.sort("feature_ts"),
        left_on="label_ts",
        right_on="feature_ts",
        by="customer_id",
        strategy="backward",  # latest value at or before the label
    )
)

Check yourself

1. Why does joining the latest feature value leak label information into training?
2. What does the online store trade away compared with the offline store?
3. When is a feature store not worth its operational cost?

Key takeaways

• Point-in-time correctness is the core guarantee; the rest is plumbing around it.
• One definition, two stores: history for training, latest values at low latency for serving.
• Reuse across many models justifies the complexity; a single pipeline rarely does.