Containerizing a Model Service

A container makes the serving environment part of the artifact: the same image that passed CI runs in staging and in production. For machine learning that means pinning the Python dependencies and the CPU or CUDA stack, and keeping the image small enough that rollouts and cold starts stay cheap.

The idea

The Dockerfile rules that matter:

• Pin the base image by digest, not tag: FROM python:3.12-slim@sha256:.... A tag is a moving pointer to a newer patch release.
• Build in stages. A builder stage installs compilers and wheels; the runtime stage copies only the installed packages. Build tools never ship.
• Copy the lockfile before the source. Copying requirements.lock.txt and installing, then copying src/, means a code edit rebuilds only the final layers instead of reinstalling everything.
• Run as non-root and avoid --no-cache-dir mistakes: install with it, run without it.
• Exec the server directly in CMD so SIGTERM reaches the process and shutdown is graceful.

Two weight strategies, chosen deliberately:

• Bake the model into the image. Immutable and self-contained; a new model means a new image and a full rollout.
• Pull weights at startup from a registry or S3 using a URI plus checksum. Smaller image, faster promotion, but startup now depends on the network and needs a readiness gate.

GPU images add one hard constraint: the CUDA runtime inside the image must be compatible with the host driver. Mixing a pip-installed CUDA library with an unrelated base image is a common source of "works on my laptop, fails on the node" failures.

Worked example

Two builds of the same service. With FROM python:3.12-slim, a rebuild months later can silently pick up a new patch with different system libraries, and the image hash changes even though the code never did. Pin the digest and the base is byte-identical forever.

The layer order matters just as much. Since dependency installation is the expensive step, a source-only change should invalidate only the COPY src/ layer and everything after it. If the source is copied before pip install, every commit reinstalls the full dependency set. That ordering difference is the main reason CI builds are slow.

In code

services:
  api:
    build: .
    image: fanout/model-api:1.4.0
    ports: ["8000:8000"]
    environment:
      MODEL_URI: s3://fanout-models/churn/8/model.pt
      MODEL_SHA256: "9d3f7c1b2a4e5f60718293a4b5c6d7e8"
    read_only: true
    tmpfs: ["/tmp"]
    deploy:
      resources:
        limits: {cpus: "2", memory: 4G}
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://localhost:8000/readyz')"]
      interval: 10s

The checksum lets the process refuse to serve weights that were silently replaced at the URI.

Check yourself

1. Why pin a base image by digest instead of by tag?
2. What belongs in the builder stage but never in the runtime stage?
3. What fails when the container's CUDA version does not match the host driver?

Key takeaways

• The image is the environment contract; pin the OS and CUDA stack too.
• Multi-stage builds plus lockfile-first layer ordering keep builds small and fast.
• Decide explicitly between baking weights and pulling them at startup.