Batch vs Online Inference

The same model can be asked for predictions two very different ways: all at once on a schedule, or one request at a time under a latency budget. Batch and online are separate engineering problems, and choosing wrong either wastes money or fails the product requirement. Freshness and latency decide, not model quality.

The idea

Batch inference:

• Scores a whole table on a schedule — nightly, hourly, or on a trigger.
• Optimizes throughput; latency is irrelevant.
• Runs on cheap, interruptible hardware and tolerates restarts.
• Writes results to a warehouse, table, or cache the app reads.

Online inference:

• Maps one request to one response, with a tail latency budget such as p95.
• Optimizes latency; throughput is secondary.
• Needs warm models, autoscaling, health checks, timeouts, and a fallback path.
• Needs exactly the same feature computation used in training.

The complication either way is train/serve skew. Batch is safer here because the offline feature pipeline can be reused verbatim against an offline table. Online often forces features to be rewritten against a low-latency store, and that rewrite is where skew enters.

A common hybrid: precompute scores in batch, serve them from a cache, and fall back to online scoring for entities the batch has never seen.

Worked example

Churn scoring for 2 million customers is a batch problem. Each customer needs one score per day, so day-old freshness is acceptable, and the whole table can be scored on spot instances and restarted freely.

Fraud detection at checkout cannot be batched. The input is a transaction that does not exist until the user clicks, so there is no row to precompute, and the answer must arrive inside the checkout latency budget. Same modeling skills, entirely different infrastructure: batch needs a scheduler and a partitioned table, online needs a warm service and a tail-latency SLO.

In code

def score_batch(path: str, out: str, chunk_size: int = 50_000) -> None:
    done = set(load_completed_parts(out))          # resume after an interrupt
    for i, rows in enumerate(read_chunks(path, chunk_size)):
        if i in done:
            continue
        preds = model.predict(transform(rows))
        write_part(f"{out}/part-{i:05d}", preds)   # atomic write per part
        mark_completed(out, i)

Per-part checkpointing is what makes spot instances practical: a crash costs one chunk, not the whole run.

Check yourself

1. Which requirement forces online inference even when batch would be cheaper?
2. Why does online serving tend to create train/serve skew that batch avoids?
3. What does a batch job need so a mid-run crash does not restart from zero?

Key takeaways

• Batch optimizes throughput on cheap hardware; online optimizes tail latency on warm capacity.
• The freshness requirement usually decides the architecture, not the model.
• Precomputed batch scores served from a cache are a valid hybrid, with an online fallback.