Shadow Deployment in ML: How to Test Models on Live Traffic
Shadow deployment, also called shadow mode or shadow testing, runs a new machine-learning model beside the one in production. The new model receives copies of real requests and produces predictions, but those predictions are logged and never returned to users or downstream systems. The current model keeps making every decision that counts.
For an enterprise, the appeal is simple. An offline test set tells you how a model behaved on yesterday’s data. Shadow mode tells you how it behaves on today’s traffic, with today’s messy inputs, at today’s volume, before a single customer or process depends on it. This guide explains how it works, how to set it up safely, what to measure, and how to decide whether to promote, revise or reject the model.
TL;DR
Shadow deployment mirrors live requests to a candidate model, logs its outputs and discards them, so users never see the candidate’s answers.
It answers “is this model safe to serve?” It cannot answer “is it better for users?”, which needs a canary release or an A/B test afterwards.
Mirroring all traffic roughly doubles inference compute. Sample the stream, or size the shadow stack smaller, and control side effects.
Define promotion criteria before you start: agreement rate with the incumbent, tail latency, error rate and reviewed disagreements.
In Malaysia, logged inputs that contain personal data are still personal data. Plan retention and access for the shadow store from day one.
Table of Contents
What shadow deployment means in practice
The incumbent model is often called the champion and the candidate the challenger. In a shadow setup, a router, gateway or service mesh duplicates each incoming request. The champion answers the caller as usual. The challenger processes the same input on a separate path, and its prediction goes into a log or table for later analysis.
Three properties define the pattern. The first is zero user impact: nothing the challenger produces reaches a customer, a case officer or another system. The second is identical inputs: both models must see the same features for the same request, otherwise any difference you measure may come from the data pipeline rather than the model. The third is bounded cost: the shadow path runs on its own resources so it cannot slow the champion down.
Amazon’s own description of shadow testing in SageMaker follows this shape. A new serving stack is deployed alongside the current one, a portion of requests is routed to it, and only the production variant’s responses are returned while shadow responses are discarded or logged for offline comparison. Snowflake’s model gateway works similarly: the primary service handles production responses, while the gateway mirrors requests to a challenger and does not return the shadow response to clients.
How shadow deployment differs from other release strategies
Teams often confuse shadowing with canary releases and A/B tests because all three involve running two models at once. The difference is who sees what.
Approach | Data it uses | User exposure | What it tells you |
Offline evaluation | A fixed, held-out test set | None | How the model scores on a historical snapshot |
Shadow deployment | Live production requests | None, outputs discarded | Behaviour, latency, cost and errors on the real input distribution |
Canary release | Live requests | A small share of users | Real outcome signals at limited risk |
A/B test | Live requests split between variants | Split across variants | Causal evidence of which variant performs better |
Shadow mode sits between offline testing and exposure. It closes the gap that a static test set leaves open: production data drifts, edge cases appear that nobody labelled, and traffic patterns differ by hour, day and season. If you want to see how that drift shows up in practice, our guide to data drift detection covers what to monitor.
The limitation is built in. Because no user ever sees the challenger’s output, you learn nothing about user response, conversion or business impact. Shadowing establishes that a model is safe to serve. Establishing that it is better is the job of the next rung, a canary or an A/B test.
When an enterprise should use shadow deployment
Not every model update needs a shadow phase. A small refresh of a low-stakes recommendation widget can go straight to a canary. The case for shadowing grows with the cost of a wrong prediction.
AWS’s Well-Architected machine learning guidance frames the choice as a trade-off between availability requirements, how critical the predictions are, and monitoring capability. It points to shadow deployment for higher-risk scenarios, to reduce rollout risk and limit the blast radius.
In our work with finance, insurance, manufacturing and logistics teams, the situations where shadow mode earns its cost tend to look alike:
A credit, fraud or underwriting model whose outputs feed decisions that carry regulatory or financial consequences.
A document-processing model that extracts fields from invoices, claims or bills of lading, where one wrong value can travel through several downstream systems.
A conversational agent or LLM-based assistant whose new prompt, retrieval layer or base model must be checked on real questions before customers meet it.
A predictive quality or maintenance model on a production line, where a bad call means scrap, downtime or a missed defect. Our article on hybrid quality prediction in manufacturing shows how such models are built to be dependable.
Shadowing also suits you when you can judge the challenger without waiting for user behaviour: either you have labels that arrive later (a loan outcome, a confirmed fraud flag, a reviewed document), or you can compare against the incumbent and review the cases where they disagree.
How to set up shadow testing without touching production decisions
The engineering goal is one sentence: the challenger must be unable to affect the champion’s response or any real-world action. These steps keep that promise.
Mirror requests at the routing layer
Put the duplication in a router, load balancer, service mesh or model gateway rather than inside the application code. The caller’s request goes to the champion and a copy goes to the challenger. Managed platforms offer this natively, as the SageMaker and Snowflake examples above show, and self-hosted stacks can do the same with proxy-level mirroring.
Make the shadow call asynchronous
The champion should never wait for the challenger. Treat the shadow call as fire-and-forget: if it times out or throws an error, log the failure and carry on. Otherwise a slow challenger becomes a latency problem for live users, which defeats the purpose.
Feed both models identical inputs
Compute features once where you can, or make sure the shadow path uses the same feature definitions and the same point-in-time values. A mismatch here is the most common reason a shadow comparison produces confusing results. It is also an argument for a disciplined feature pipeline and for experiment tracking that records exactly which model version, data and configuration each run used.
Block side effects
A model that only returns a score is easy to shadow. A model or agent that sends an email, writes to a ledger, calls a payment API or triggers a work order is not. Stub or sandbox every external call on the shadow path, and write outputs to a separate store that no production process reads. For agentic systems, this is the single most important control.
Give the shadow stack its own resources
Run the challenger on a separate resource pool so that a heavy model cannot starve the champion. This is also where cost shows up. If you mirror every request to a model of similar size, you roughly double inference compute for the duration of the test. Two sensible responses are to shadow a sampled share of traffic, and to provision fewer replicas for the challenger. For GPU-heavy or LLM workloads, our piece on cutting GenAI inference costs is a useful companion, and the trade-offs of sizing infrastructure are covered in this note on how to scale compute resources without migration.
Log what you will need to compare
For every shadowed request, store a request identifier, the model versions, both predictions, confidence scores, latency for each path, any error, and a timestamp. Add the ground-truth label later when it becomes available, joined on the request identifier. If you cannot join afterwards, you cannot evaluate.
What to compare during the shadow period
Shadow comparison works on signals that do not depend on user behaviour. Snowflake’s gateway monitoring lists three families of model metrics: drift metrics, performance metrics such as accuracy when ground truth is configured, and stat metrics such as counts and nulls. Gateway-level measures such as request count, error rate, latency and resource use sit alongside them.
In practice we group the comparison into four questions.
Does the challenger agree with the champion? Compute the agreement rate: the share of requests where both models give the same prediction. For continuous outputs, compare score correlation and the shape of the output distributions. A challenger that agrees 99% of the time may not be worth shipping if it is meant to be an improvement. One that agrees 70% of the time needs an explanation.
Where they disagree, who is right? Sample the disagreements and review them, with a subject-matter expert if the domain demands it. In underwriting or claims, a case officer reviewing fifty divergent cases will teach you more than an aggregate metric. This is also where a well-designed human review step matters; see our phases for making human-in-the-loop ML genuinely effective rather than ceremonial.
Is it operationally safe? Compare tail latency (p95 and p99, not the average) against your service-level objective, error and timeout rates, memory and compute consumption, and the cost per request. A model that is more accurate but breaches the latency budget is not ready.
Is it accurate where labels exist? When ground truth arrives, evaluate both models on the same shadow-period requests. Because the data is live, you will see the cases your offline set never contained.
Interpreting results: roll out, revise or reject
Decide the criteria before you switch shadowing on. Teams that look at the results first tend to rationalise whatever they see.
A workable rule has four parts. Run the shadow for long enough to cover your traffic cycles: at least several days, and a full month-end if your business has one. Require the challenger to meet the latency objective and the error budget. Set a minimum on agreement or on labelled accuracy, with reviewed disagreements judged in the challenger’s favour or neutral. And confirm that no segment (a product line, a branch, a document type, a language) behaves markedly worse than the rest, since an overall average can hide a weak slice.
There are three outcomes. If the challenger meets the criteria, promote it to a canary release. If it shows problems, revise and shadow again. If it fails in ways that point to a fundamental flaw, reject it. All three happen without a single user being affected, which is the entire value of the technique.
The promotion path is a ladder: offline evaluation first, shadow deployment for higher-fidelity evidence, canary for real outcome signals, then a full rollout. Treat the shadow phase as evidence of safety, not proof of business improvement. If you skip the canary and go from shadow to 100%, you are assuming user outcomes will follow.
A caution on interpretation. If the champion itself is degrading because of drift, a high agreement rate may simply mean both models are wrong in the same way. Compare against labels where you can, and review your model retraining strategy so that silent failures are caught in the incumbent too.
Privacy, security and cost for Malaysian organisations
Shadow mode creates a second copy of your inference traffic. If requests contain customer names, identity numbers, account details or health information, that copy is a store of personal data, and it should be treated as one. Malaysian enterprises should check with their data protection officer or legal counsel how the Personal Data Protection Act 2010 applies to the shadow log, including purpose, security, retention and who may access it. Where the challenger is hosted with a cloud provider in another country, cross-border transfer obligations deserve a specific review.
Practical controls follow from that. Mask or tokenise identifiers before they reach the shadow store. Set a short retention window, since the log is only needed until the decision is made. Restrict access to the people doing the evaluation. Record the purpose and the approval, which helps with audit trails in regulated sectors such as banking and insurance.
On reliability, keep the shadow path isolated and the call asynchronous, as described above. On cost, budget the doubled compute before you begin, and set an end date for the test. A shadow deployment that nobody turns off becomes a permanent line in the cloud bill.
For LLM-based systems, add observability to the plan. Shadowing a new prompt or model is far more useful when you can trace, score and review responses; see what to implement first in LLM observability, and the wider set of LLMOps practices for production.
Shadow testing also sits inside a governance picture. Defining who approves promotion, what evidence they need and how it is recorded is the same discipline covered in these AI governance basics for small teams, and larger enterprises need the same clarity at greater scale.
Our perspective: shadowing is a contract, not a feature
Most teams treat shadow deployment as a technical switch. We treat it as a contract between the people who build a model and the people who answer for its decisions. That contract has three clauses. Before the test, agree what evidence will justify promotion. During the test, guarantee that nothing the challenger does can reach a customer. After the test, keep the record, because an auditor or regulator may ask why a model went live. This is why we do not split strategy, build and operations across different teams. When one senior team owns the use case, the engineering and the running system, the shadow phase is designed with production in mind from day one. Criteria are tied to the business result, and governance and auditability are built in rather than added at the end. Shadowing is the first rung of a ladder, not the whole ladder. We pair it with canary releases, monitoring and managed operations so that a model which passes in shadow keeps performing after it ships. Thomas Samuel
Putting shadow deployment into your MLOps practice
Shadow mode works best when it is a standard stage in your release process rather than a one-off project. That means a reusable mirroring mechanism, a standard logging schema, agreed promotion thresholds per risk tier, and an owner for each decision. Our summary of MLOps practices for a 6 to 12 month rollout shows where it fits alongside CI/CD, monitoring and retraining.
If you are earlier in the journey, start small. Pick one model whose errors are expensive, mirror a sample of its traffic to a candidate, and compare agreement and latency for a week. The lessons about feature parity, side effects and logging will pay back on every model that follows.
Work with a team that owns the whole lifecycle
Sentient Concepts designs, builds and runs production AI for enterprises in finance, insurance, manufacturing and logistics. Our Deployment & MLOps service covers release patterns such as shadow testing, canary rollout, monitoring and evaluation, and ongoing optimisation, with one accountable team from strategy to operation.
If you are planning a model release and want shadow testing designed into it, talk to us about your AI roadmap. We will help you define the promotion criteria, the infrastructure and the governance before the first request is mirrored.
FAQ
Is shadow deployment the same as shadow testing?
Yes. Shadow deployment, shadow mode, shadow testing and dark launch all describe the same idea: a new model processes real traffic in the background while its outputs are logged and never returned to users. Teams use the terms interchangeably, though “shadow deployment” usually refers to the infrastructure and “shadow testing” to the evaluation exercise.
How long should a shadow deployment run?
Long enough to cover your normal traffic cycles, which is usually several days at a minimum and longer if your volumes spike at month-end or by season. Set the duration and the promotion criteria in advance. Extend the test if rare but important case types have not yet appeared in the log.
Does shadow deployment replace a canary release or A/B test?
No. Shadowing shows whether a model is safe to serve on real inputs, but because users never see its outputs, it cannot show user response or business impact. Most teams follow a successful shadow phase with a canary release on a small share of traffic, and then an A/B test where they need causal evidence.
Does shadow mode double my infrastructure cost?
If you mirror all traffic to a challenger of similar size, inference compute roughly doubles while the test runs. You can reduce that by shadowing only a sample of requests, running fewer challenger replicas, and ending the test as soon as you have enough evidence.
Sources
Recommended