Million Token Windows Did Not Kill Retrieval. They Changed What It Is For.

Every time context windows get bigger, the same argument comes back: retrieval was a workaround for small windows, the windows are not small anymore, so put everything in the prompt and delete the vector database.

Kimi K3 shipped in July with a one million token window. Google has signalled a two million token window for Gemini 3.5 Pro, though as of early August that model remains in limited preview on Vertex AI and has not launched publicly, so treat the specifics as unconfirmed. Either way, million-token-plus is now a normal number rather than a headline, and the argument is back.

It is wrong, but not for the reason it is usually rebutted. The usual rebuttal is that models get worse at finding things in long contexts, partly true and mostly beside the point. The real reason is economic, and it is specific to how agents work.

You re-send the window on every turn

A chat application sends context once and gets an answer. An agent does not. It sends the accumulated context, gets a tool call, executes it, appends the result, and sends the whole thing again. Thirty steps of that means thirty sends of a context that grows the entire time.

It is why agent workloads run input-to-output ratios around twenty-five to one, with input somewhere near 85% of the bill. It is also why filling a large window is a different decision in an agent than it is in a chat interface.

Put a million tokens of documents into the context at step one, and you have not made a one-time purchase. You have set the floor for every subsequent step. Step two carries it, step twelve carries it, step thirty carries it. You bought the loading dose once and pay the maintenance dose forever.

Prompt caching changes the magnitude of this and not its shape. A stable prefix cached across turns is dramatically cheaper than re-sending it cold, which is exactly why cache hit rate belongs on the same dashboard as spend. But cache writes are not free, cache entries expire, and the moment your agent's context stops being a stable prefix, because a tool result got inserted in the middle or the retrieved set changed between turns, you are paying uncached rates on a very large payload.

The discipline this implies is simple to state and easy to skip: everything you put in context should earn its place against the cost of carrying it for the rest of the loop.

Two long-context capabilities that are not the same

The technical objection to giant contexts is usually phrased as "models lose things in the middle," and it gets waved away by pointing at near-perfect needle-in-a-haystack results. Both sides of that exchange are talking past each other, because there are two distinct capabilities involved.

Retrieval from context. Given a large document set and a question whose answer sits in one specific place, find it. Models have become very good at this, and it is what most long-context benchmarks measure.

Instruction adherence under a full window. Given an instruction at the start of a session, keep honouring it thirty tool calls later when the window is packed with file contents, API responses, and prior reasoning. This is harder, it is what agents actually need, and it is much less well measured because it is expensive to test properly.

In the first task, the model is told what it is looking for. The second requires the model to keep applying a constraint that nothing in the recent context is reminding it about. An agent that was told to always cite the source document, and that has since processed four hundred thousand tokens of tool output, is in the second regime.

Synthetic long-context tests mostly probe the first. Your agent lives in the second. A model advertising a million-token window is making a claim about what fits, not a promise about what it will still be doing at the far end of it.

What retrieval is for now

The job changed. Stating the new job precisely matters, because it argues for different techniques than the old one.

Retrieval used to be a capacity workaround. The window was small, the corpus was large, so you found the few chunks most likely to matter and sent those. Every design decision followed from scarcity: small chunks, tight top-k, aggressive filtering, because you had maybe a few thousand tokens to spend.

Now the window is not the constraint. What retrieval does is decide what deserves to occupy an expensive resource that gets re-sent on every turn of the loop. That is a precision problem, and precision problems favour different choices than capacity problems.

Larger chunks become reasonable, because the reason chunks were small was budget, and preserving enough surrounding context that a passage is interpretable is worth more than fitting more passages. Higher top-k becomes reasonable for the same reason: retrieving twenty candidates instead of five costs little when the window is large, and it substantially reduces the chance that the one document that mattered fell just below the cut.

What becomes more important, not less, is knowing what you put in. A curated set of twenty relevant documents and a dump of the entire corpus can occupy the same number of tokens, and they do not produce the same results. The dump contains contradictions, superseded versions, and irrelevant material that the model has to reason past on every turn. Providing that is not neutral. It actively competes with the instruction you are trying to get followed.

The architecture that actually works

Retrieval feeding a large window, rather than either alone.

Retrieve generously, because you have the room. Do not retrieve everything, because you do not want the noise or the recurring bill. Structure what you send so the model can navigate it: sources labelled, versions marked, ordering meaningful rather than by cosine similarity alone. Keep the retrieved set stable across the turns of a single task where you can, because a stable prefix is a cacheable prefix, and a set that changes every turn is a cache miss every turn.

Then put the instruction that must survive the whole loop somewhere the model will keep seeing it. If your framework supports re-stating critical constraints near the end of the context rather than only at the start, use it. This feels inelegant. It also works, and elegance is not the metric.

When to skip retrieval entirely

There is a real case where the "just put it all in" argument is correct, and I want to name it so this does not read as reflexive defence of the vector database.

If your corpus fits, is stable, and is small relative to the window, retrieval is machinery you do not need. A product's full documentation set, a single codebase of moderate size, one contract and its amendments, a company handbook. In those cases retrieval adds an indexing pipeline, an embedding model, a similarity threshold that will need tuning, and a class of failure where the right document simply did not come back. Skipping all of it in favour of a stable, cached prefix containing the whole corpus is a legitimately good architecture, and it is cheaper to operate than people expect precisely because a fully stable prefix caches beautifully.

The test is whether the corpus is stable. A fixed set that changes weekly caches well and can go straight in. A set that changes per request, per user, or per turn does not, and that is where retrieval earns its keep regardless of window size.

What to measure

If you are deciding between these architectures, three measurements settle it faster than any argument.

Cost per completed task, with cache hit rate alongside it, for both approaches on the same workload. This is where the re-send effect becomes visible, and it is frequently the whole decision.

Instruction adherence at depth. Take a constraint that must hold for the entire task, run twenty real tasks to completion, and check whether the constraint held on the last step as often as the first. If it degrades, more context is making your agent worse and you should be sending less of it, not more.

Time to first token, at your realistic context size. Large contexts cost latency as well as money, and an agent that is thirty turns deep has paid that latency thirty times. Users notice the sum, not the per-call number.

None of those are exotic. All three are things you can measure this week, and they will tell you more about whether to keep your retrieval layer than another round of leaderboard reading.

The window got big. That is good, and it removed a real constraint that shaped a lot of awkward engineering. What it did not do is make the question of what belongs in context go away. It made that question more expensive to get wrong.