How to Evaluate a New LLM in 48 Hours: The Shadow-Mode Playbook

Two frontier models launched within 24 hours of each other this month. Before that it was a coding-agent release, and before that a voice API. The launch cadence has outrun the ability of any team to evaluate carefully, which produces two equally bad behaviors: switching models on the strength of a benchmark table, or refusing to evaluate anything and quietly falling three generations behind.

We've settled into a 48-hour process that decides the question properly without turning into a project. It's not sophisticated. Its whole value is that it's cheap enough to actually run every time, which is the only property that matters when launches arrive weekly.

Hour 0: the precondition

None of this works without one architectural decision made in advance: your model calls go through an interface, not a vendor SDK sprinkled through the codebase.

Tool execution, retries, transcript logging, and prompt assembly live behind that interface. Adding a candidate model becomes a config entry. If you can't add a new provider in under an hour, stop reading and fix that first — it's the single highest-leverage thing in your stack, and it's the difference between a launch being an experiment and a launch being a project.

Most newly launched models are OpenAI-compatible, which makes this cheaper than it used to be. Some ship Anthropic-compatible endpoints too. Either way, the swap should be a base URL and a model name.

Hours 1–4: shadow mode

Route a copy of real production traffic to the candidate. Do not serve its output. Log everything.

Shadow mode is the whole trick, and it's underused because it feels like it doesn't count. It counts more than any eval set you've built, for a simple reason: your eval set encodes the distribution you thought you had when you wrote it, while shadow traffic is the distribution you actually have. Every team I've worked with has a meaningful gap between the two, usually in the messy tail where models fail.

Budget-wise this is cheap. A day of shadowed traffic on a current cheap-tier model costs single-digit dollars for most workloads. Some providers hand out enough free credits to cover the whole exercise.

Hours 4–24: the four metrics

Benchmark tables report averages across other people's distributions. These four are about yours.

Cost per completed task, not per token. The metric that matters, and the one sticker prices mislead on constantly. A model with 30% cheaper tokens that needs an extra attempt on 15% of requests is not cheaper. Measure the whole loop through to a result you'd ship.

Retry and validation-failure rate. How often does output fail your schema, your validator, or your review step? This is where literal instruction-following shows up, and it varies more between models than any benchmark suggests. It's also the metric most predictive of ongoing engineering cost.

Latency at p95, not average. Averages hide the tail, and the tail is what users experience as "sometimes it hangs." For agent loops, measure per-turn latency and turn count separately — a model that's slower per turn but needs fewer turns can be faster overall.

Failure mode, not just failure rate. Two models failing 5% of the time are not equivalent if one fails loudly with an error and the other silently returns confident nonsense. Read the failures. Categorize them. This is manual and worth it.

Hours 24–36: the adversarial pass

Take your worst 20 shadow-mode cases and try to make the candidate fail on purpose. Ambiguous instructions. Conflicting constraints. Missing context it should ask about rather than invent. Inputs that are subtly malformed.

What you're testing is escalation behavior. A model that says "the credential referenced in step three is missing, stopping here" is dramatically more valuable in production than one that improvises around it, even if their aggregate accuracy is identical. The improviser's failures are invisible until something downstream breaks, which is the expensive kind.

Hours 36–48: the decision

Three possible outcomes, and the middle one is the most common and most underrated.

Route it in, scoped. Not "we've switched." A specific route where the evidence supports it, with the incumbent still handling everything else. Routing tables should be per-workload, and a model earning one slot is a normal and good result.

Park it with notes. The candidate lost, but you now know where it lost, which is exactly what makes the next version's evaluation take four hours instead of forty-eight. Most launches end here. That's fine.

Reject on operational grounds. Sometimes the model is good and the surrounding product isn't ready: thin documentation, region restrictions, no rate-limit guarantees, a preview label on a critical path. Capability isn't the only axis, and a model you can't operate is not a model you can use.

What not to do

Don't switch wholesale on a benchmark table. Especially not one published by the vendor or an author affiliated with them. Benchmarks are a signal for what to test, never a substitute for testing.

Don't evaluate with your evals alone. They're stale by construction and biased toward the cases you already understood.

Don't skip the price-per-outcome calculation. The sticker-price comparison is the single most common way teams talk themselves into a migration that costs more.

Don't run this on more than two candidates at once. The analysis quality collapses, and you end up making a vibes decision with extra steps.

Why the cadence matters more than the outcome

The specific models change every few weeks. What compounds is having a process cheap enough to run every time one launches, because the alternative is a stack that ossifies around whatever was best the last time someone had a free week.

Forty-eight hours, mostly unattended, against traffic you're already serving. Run it on the next launch, then the one after. Within a quarter you'll have a routing table built from your own measurements instead of someone else's marketing, and that's the actual asset — not any individual model in it.