Canary and Shadow Deployments

Two release patterns let you test a new model on real traffic without betting the product on it. A canary exposes a slice of users to the candidate and compares outcomes. A shadow sends real requests to the candidate and discards the responses, and the two patterns answer different questions and are usually run in sequence.

The idea

• Canary: route a small percentage of live requests to the candidate and keep the incumbent for the rest. Users are affected, so decide the rollback threshold before sending a single request.
• Shadow: duplicate live requests to the candidate, throw the response away, and log it. No user risk, but no outcome signal either — if the candidate never acts, you cannot see how users would have reacted.
• Sticky assignment: hash a stable key such as account_id into buckets so a user stays in one arm for the whole test. Per-request assignment gives users a mix of both experiences and muddies the comparison.
• Guardrails: error rate, p99 latency, and a floor on a business metric. Wire them to an automatic rollback, not an alert that waits for a human.

A shadow deployment still tests a lot: crashes, latency, memory, feature-lookup failures, output schema violations, and score-distribution shift against the incumbent. It cannot test anything downstream of the decision — click-through, conversion, or whether a blocked transaction was really fraud.

Worked example

A fraud model is moving from v1 to v2. Run v2 in shadow for two days on the full request stream and check three things: p99 latency inside the budget, zero feature-lookup failures, and a score distribution close to v1 on identical inputs. Those checks carry no user risk.

Then canary at 5 percent of accounts, sticky by account, and compare blocked-transaction precision against the control while watching the false-decline rate. If precision holds and declines do not rise, ramp 5 → 25 → 50 → 100 percent, pausing long enough at each step to collect labels. Because a shadow model never blocks anything, the canary is the only stage that can tell you whether the new thresholds are correct.

In code

canary:
  candidate: fraud-scorer:v2
  stable_key: account_id
  steps: [5, 25, 50, 100] # percent of traffic
  guardrails:
    error_rate: "> 0.02 -> rollback"
    p99_latency_ms: "> 250 -> rollback"
    precision_at_budget: "< control - 0.02 -> halt"
  shadow:
    enabled: true # duplicate requests before the canary starts
    log_fields: [model_version, score, latency_ms, features_hash]

Check yourself

1. Why can a shadow deployment not tell you whether users prefer the new model?
2. What must be true about traffic assignment for a canary comparison to be valid?
3. Which class of defect does shadow catch that an offline evaluation on last month's data misses?

Key takeaways

• Shadow is risk-free but blind to outcomes; canary measures outcomes and carries real risk.
• Sticky buckets and pre-committed guardrails turn a rollout into a measurement.
• Ramp in steps so a rollback costs one step, not the whole launch.