Model Serving with FastAPI

FastAPI is a thin, well-typed way to expose a model over HTTP: it validates the request body with pydantic, runs your predict function, and returns JSON. The model loads once when the process starts, not once per request. The endpoint should be boring, because all the interesting risk lives in input validation and concurrency.

The idea

The anatomy of a serving app:

• Load once in a lifespan hook, storing the model on app.state. A per-request reload pays disk or network cost on every call.
• Validate at the edge with a pydantic request model. Malformed input becomes a 422 response instead of an exception inside inference.
• Keep endpoints boring — /healthz for liveness (process is up), /readyz for readiness (model is loaded and warm), /predict for work. Load balancers need the split to avoid routing traffic to a starting process.
• Do not block the event loop. FastAPI runs a plain def endpoint in a thread pool; an async def endpoint runs on the event loop itself. A blocking predict in an async def stalls every other request.
• Cap concurrency. A predictor that can exhaust memory under load needs a queue and a limit, not more retries.
• Batch requests. Accept a list of inputs. This amortizes HTTP and Python overhead and lets a GPU do real work per call.

Scaling has a sharp edge: uvicorn --workers N runs N processes, each with its own copy of the model. Memory multiplies by N. GPU serving usually means one worker per GPU, not many.

Worked example

Suppose per-request framing, JSON parsing, and validation cost about 5 ms, and the model takes about 2 ms. Serving one row per request spends 5 ms of overhead for 2 ms of compute, so inference is a minority of wall time. Sending 32 rows in one request pays the 5 ms once and attempts 64 ms of compute, so inefficiency from overhead falls by roughly an order of magnitude. That is why the response schema is a list.

Validation is the other half: a request with 31 features instead of 32 is rejected before predict is called, so the model never has to defend itself against bad shapes.

In code

from contextlib import asynccontextmanager
from fastapi import FastAPI, Request
from pydantic import BaseModel, Field

class PredictRequest(BaseModel):
    x: list[float] = Field(min_length=32, max_length=32)

class PredictResponse(BaseModel):
    scores: list[float]
    version: str

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.model = load_model("artifacts/model.pt")  # once, at startup
    yield

app = FastAPI(lifespan=lifespan)

@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest, request: Request) -> PredictResponse:  # def, not async
    model = request.app.state.model
    return PredictResponse(scores=model.predict([req.x]).tolist(), version=model.version)

Check yourself

1. Why load the model during startup instead of on the first request?
2. Why should a blocking predict endpoint be declared def rather than async def?
3. What is the memory cost of raising uvicorn --workers from 1 to 4?

Key takeaways

• Validate with pydantic at the edge; keep inference free of input shaping code.
• One model per process, and every worker process multiplies memory.
• Batch requests, split liveness from readiness, and never block the event loop.