Cost Management for Training and Serving

ML cost is unusual because it scales with experimentation as much as with users. Training spend is bursty, interruptible, and easy to reason about; serving spend is continuous, latency-bound, and usually the larger line item over a year. Managing both means knowing which driver you are paying for before optimizing anything.

The idea

Training cost is accelerator hours plus data movement. High-leverage levers:

• Right-size the accelerator and batch size; a large GPU at low utilization is the most expensive option available.
• Use preemptible or spot capacity with frequent checkpointing, so an interruption costs minutes.
• Mixed precision and fused kernels for throughput, early stopping and smaller sweeps to avoid wasted runs.
• Cache tokenized or preprocessed data; re-running preprocessing every epoch can dominate cheap training jobs.

Serving cost is the long-run problem. Drivers are always-on replicas, over-provisioned latency targets, model size, context length, batch size, and cache hit rate. Scale to zero where the latency budget allows; keep a warm pool where it does not. Continuous batching and prompt caching change the cost per request far more than a smaller model does when quality is equal.

Two habits keep spend honest:

• Cost per useful unit. Compare dollars per thousand predictions or per million tokens, not dollars per GPU-hour. Utilization and price are inputs; the deliverable is the output.
• Tagging and showback. Cost attributed to a team, project, and experiment is managed; untagged spend is invisible until the invoice. Storage and egress — checkpoints, feature tables, logs, embeddings — belong in the same view.

Worked example

Suppose one always-on replica costs USD 2.50 per hour, and the service answers 40 million predictions per day. The daily cost is $2.50 \times 24 = 60$ USD, about 0.0015 USD per thousand predictions. Now suppose a smaller, faster model could handle the same 40 million predictions from a single replica at USD 0.60 per hour but loses two points of accuracy. The infrastructure saving is real, but the decision depends on what those two points cost in business terms, so the comparison must be expressed as cost per prediction at fixed quality — otherwise the cheaper option always wins by accident.

In code

cost_per_1k = (hourly_usd * hours) / (predictions / 1000)

cost_per_run = (hourly_usd * gpu_hours) + (storage_gb * storage_usd)  # per accepted run
cost_per_result = sum(cost_per_run for run in runs) / accepted_experiments

Check yourself

1. Why is cost per million predictions a better decision metric than dollars per GPU-hour?
2. Which serving choices raise cost without improving quality?
3. How does checkpointing change the economics of preemptible training?

Key takeaways

• Measure cost per useful unit, at fixed quality, rather than per resource-hour.
• Serving spend is dominated by utilization, model size, and latency targets; training spend by interruptions and wasted sweeps.
• Tag spend by team, project, and experiment, or you cannot manage it.