Three Tiers, One Loop: Routing Between Model Sizes Without Breaking Your Agent

OpenAI shipped the GPT-5.6 family on 9 July 2026 in three tiers on the same day: Sol at the top, Terra balanced, Luna for speed and cost. Anthropic's positioning of Sonnet 5 followed the same logic from the other direction, offering Opus-class coding at roughly a third of the price. The tiered family is now the default shape of a frontier release, and it makes an obvious suggestion to anyone running an agent loop: use the cheap model for the easy steps and the expensive one for the hard steps.

The suggestion is correct. It is also considerably more expensive to act on than it looks, and the costs land in places that do not show up on the pricing page. This post is about those costs, and about the two situations that justify paying them.

Why routing looks free

A tiered family makes the switch trivial at the code level. Same provider, same SDK, same tool-calling format, same message schema. Changing which tier answers a given step is a string change in a config file. There is no integration work, no new authentication, no second vendor relationship.

That frictionlessness is the trap. The engineering cost of routing was never the integration. It is everything downstream of having two models in one system, and none of that becomes visible until the second model is already in production.

Cost one: your eval surface doubles per routed step

If a step in your loop can be served by two models, that step now has two behaviours, and both need to be regression tested. This is not the same as testing twice as much. It is worse than that, because the failure modes are not related.

A cheaper tier does not fail like the expensive one, only more often. It fails differently. It is more likely to take an ambiguous instruction literally, more likely to drop a constraint that appeared early in a long context, more likely to produce a structurally valid answer that misses the point of the request. Those are distinct failure classes, and a test set built while you were running one model will systematically underrepresent them, because you wrote the tests against the failures you had seen.

The practical consequence is that adding a cheap tier to a step means building new evals for that step, targeting the failure modes of the cheap tier specifically. If you route six steps, that is six new eval sets. Teams that skip this do not find out they skipped it until a customer reports something odd, and then they get to debug a system where the answer depends on which model happened to serve a step.

Cost two: prompt caching, which nobody budgets for

This is the one that surprises people, and it is the one most likely to eat the savings entirely.

Agent loops are input-heavy. A representative fifty-turn coding session runs on the order of a million input tokens against forty thousand output tokens, a ratio near twenty-five to one, with input accounting for roughly 85% of the total cost. That shape exists because an agent re-sends its accumulated context on every tool call. The conversation grows, and you pay for the whole thing again each turn.

Prompt caching is what makes that survivable. The stable prefix of your context, system prompt, tool definitions, and the earlier turns, gets cached, and repeated reads of it cost a fraction of the uncached price.

Caches are per-model. Switching tiers mid-conversation means the new model has never seen this prefix, so the entire accumulated context is billed cold. Switch back and you may well have lost the original cache too, depending on the TTL and what happened in between.

Work the arithmetic on your own numbers before you commit. If routing a step to a cheaper tier saves you some fraction of that step's cost, but the switch forces a cold read of a context that represents 85% of the request's cost, the trade can invert. It does not always invert. On a short loop with a small context, routing wins comfortably. On a long agent session with a large accumulated prefix, a handful of tier switches can cost more than they saved, and the effect is entirely invisible unless you are tracking cache hit rates alongside spend.

The mitigation is architectural, not clever: batch your routing decisions so that a given model handles a contiguous run of steps rather than alternating. If Luna is going to handle the mechanical extraction steps, let it handle all of them in sequence, then hand off once. Alternating per step is the pattern that destroys caches.

Cost three: debugging across two models

When a task fails in a single-model agent, the question is what went wrong. When it fails in a routed agent, the first question is which model was serving the step, and the second is whether it would have failed on the other one.

This sounds minor and it is not. It changes what a reproduction means. A bug report that says "the agent gave the wrong answer for this input" is no longer sufficient, because rerunning it may route differently if your routing logic depends on anything dynamic. Every trace needs to record the tier that served each step, and every reproduction needs to pin it.

Budget for that in your observability work before you route, not after. Retrofitting model identity into traces after you have a production incident is a bad afternoon.

Do not route on a runtime difficulty score

There is a tempting design where you classify each incoming step by difficulty, then route based on the classification. It is tempting because it sounds adaptive and it demos well.

It has two problems. First, classifying with a model call adds a full round trip of latency to every step, and if you classify with the cheap model you have added a component whose failure mode is misrouting the hard requests to the model least able to handle them. The exact inputs that are hardest to classify correctly are the ambiguous ones, which are also the ones that most need the capable tier. Your classifier's errors are correlated with the cases where the error is most expensive.

Second, it makes behaviour non-deterministic in a way that interacts badly with the debugging cost above. The same input can route differently across runs, which means "works on my machine" acquires a new and more annoying meaning.

Route on step class instead, decided statically. The extraction step always goes to the cheap tier. The sufficiency check always goes to the capable one. This is legible, cacheable, reproducible, and testable, and you can change the assignment deliberately after measuring rather than having a classifier change it for you at runtime.

The two cases where routing clearly pays

Having spent most of this post on costs, here is when I think it pays.

High-volume mechanical steps in a long loop. If your agent has a step that runs on every iteration, does something fully determined by its input, and runs thousands of times a day, routing that step to the cheap tier is close to free money. The behaviour is easy to test because the correct output is well defined, the volume makes the savings material, and if you group these steps contiguously you keep your cache intact. This is the case that justifies the whole pattern.

A capability cliff you have actually measured. If you have evidence that one specific step fails on the mid tier and succeeds on the top tier, routing that step up is obviously correct, and the cost is bounded because it is one step. The word doing the work in that sentence is measured. A belief that a step is hard is not evidence. A trace-backed comparison across tiers on your own inputs is.

Everything else, the middle ground where routing feels like it should help but you have not measured a difference, is where the costs above quietly exceed the benefits.

Token efficiency changes the baseline

One more consideration that has emerged in the 2026 releases and cuts against aggressive routing.

The headline claim on GPT-5.6 was not a benchmark number, it was efficiency: Sam Altman put Sol at 54% more token efficient on agentic coding tasks. That framing is now common across launches, and it matters here because a more capable model that solves a task in fewer steps and fewer tokens can be cheaper per task than a less capable one that takes more turns to get there, even at a higher per-token price.

The number to compare is not the one on the pricing page. It is cost per task on your workload, with your loop, measured end to end. Gartner's March 2026 analysis put agentic workflows at five to thirty times the tokens per task of a simple chatbot query, which is exactly the regime where step count dominates and per-token price stops being a useful proxy for anything.

I have seen enough cases now where the expensive model turned out cheaper per completed task that I would treat it as a live hypothesis rather than a curiosity. Before you build a router, run the boring experiment: the whole loop on the top tier, the whole loop on the mid tier, cost and completion rate measured per task. Sometimes the answer is that you do not need a router at all, and that is the cheapest outcome available, because a router you did not build costs nothing to test, cache, debug, or maintain.

The order of operations

If you are going to do this, do it in this order.

Measure single-tier baselines first, on cost per completed task rather than cost per token. Tag your steps by class, as a static property of your loop rather than a runtime judgment. Pick the one step class with the highest volume and the clearest correct answer, route only that, and group it contiguously so the cache survives. Build evals for the cheap tier's failure modes on that step before you ship it. Record the serving tier in every trace.

Then measure again. If the number did not move, revert. A routing layer that is not paying for itself is not neutral, it is a permanent tax on every future change you make to that loop.