Home › Guides › Context and limits

Context windows: limits, costs, and why bigger is not better

Million-token context windows changed what is possible, but filling one is usually both expensive and counterproductive. The failure mode is quiet: the model does not error, it just gets worse.

Context and limits · 10 min read · Last reviewed 2026-07-28

Context windows have grown from a few thousand tokens to millions in a few years, and the marketing has moved faster than the practice. A million-token window genuinely enables things that were impossible before. It also creates a new and expensive way to make an application worse while believing you are making it better.

What the limit actually is

The context window is the maximum number of tokens the model can process in a single request, counting both your input and its generated output. If a model has a 200,000-token window and you send 199,000 tokens, you have roughly 1,000 tokens of response available.

This is a hard architectural limit, not a soft cost boundary. Exceed it and most providers reject the request with an error. Some truncate instead, which is worse, because the model silently answers based on incomplete information and you have no signal that it happened.

The limit comes from how attention works. Every token attends to every other token, so the attention computation grows quadratically with sequence length, and the KV cache holding the internal state grows linearly and consumes GPU memory for the whole request. Long contexts are expensive to serve for structural reasons — the same reasons discussed in why output tokens cost more. Modern implementations use various optimisations to make long contexts tractable, but the underlying pressure remains.

The gap between advertised and useful

Here is the part that is easy to miss: a model's advertised window is the point at which it errors, not the point at which it stops working well. Those are different, often by a lot.

Lost in the middle

The best-documented long-context failure is positional. Models retrieve information from the beginning and end of a long context substantially more reliably than from the middle. A fact placed halfway through a long document may be effectively invisible to the model even though it is unambiguously present in the input.

This was characterised in research on long-context retrieval and has been reproduced repeatedly across model families. The performance curve as a function of position is U-shaped: high at both ends, sagging in the middle.

The practical instruction is direct. If something matters, put it at the start or the end of the prompt. Never bury a critical instruction in the middle of a long context and assume it will be followed.

Retrieval versus reasoning

"Needle in a haystack" tests — hiding one fact in a long document and asking for it — are the standard long-context benchmark, and models now score very well on them. This is genuinely good, and it is also a much easier task than what applications actually need.

Finding one stated fact is retrieval. Synthesising across many passages, noticing that two sections contradict each other, tracking an entity through a long narrative, or reasoning over the whole input at once are all considerably harder, and performance on them degrades noticeably earlier than needle tests suggest.

So strong needle-test scores should not be read as "this model handles a million tokens well". They mean it can find a specific string in a million tokens.

Distraction

Irrelevant context does not merely fail to help; it actively hurts. Adding plausible-looking but unrelated material to a prompt measurably reduces accuracy, because the model attends to it. A retrieval system that returns twenty chunks where five are relevant can perform worse than one returning just the five.

Precision beats recall in context construction. Sending more, on the assumption that the model will sort it out, is usually a mistake.

The cost dimension

Every token of context is billed on every request, at the input rate. That sounds cheap — input is the inexpensive side — and at scale it is not.

Consider a document Q&A application that loads a 100,000-token document into context for each question. At input rates of a few dollars per million tokens, each question costs on the order of tens of cents in input alone, before the answer is generated. Ten thousand questions a month is a meaningful bill for what the user experiences as a simple lookup.

Two ways out. Prompt caching is the obvious one: if the document is the stable prefix and questions go at the end, cached reads can cut that by 50–90%. This is the single highest-value caching scenario there is — see the caching guide.

The other is retrieval: index the document, and send only the relevant few thousand tokens per question. Usually cheaper still, and often more accurate for narrow questions because it eliminates the distraction problem. It costs you an indexing pipeline and the risk of retrieving the wrong passages.

The choice is not obvious and depends on question breadth. Narrow, factual questions favour retrieval. Questions requiring synthesis across the whole document favour long context with caching.

Managing conversations against the limit

Long conversations approach the context limit from below, and the request that fails is the last one — typically at the worst possible moment, mid-session, for your most engaged user.

The failure is also expensive to recover from. Handling it after the fact means an error, a truncation, and a retry the user experiences as the assistant losing its memory.

Manage it proactively. Track the token count of the accumulated conversation on every turn. Set a threshold well below the model's limit — leaving room for a full-length response — and when it is crossed, summarise or truncate deliberately rather than waiting for an error. This is covered alongside the cost implications in multi-turn conversation costs.

Note that the calculator checks the peak single-request size in multi-turn mode — the final, largest turn — because that is the request that would actually fail.

Choosing how much context to send

A reasonable default policy:

1. Send the least context that reliably works. Start small, add only what measurably improves output on a real evaluation set. This is the opposite of the natural instinct.
2. Put critical material at the edges. Instructions and the most important documents at the beginning or end; supporting material in the middle.
3. Restate key instructions at the end for long prompts. The cost is a few dozen tokens; the reliability gain is real, and this is one of the highest-value cheap tricks in prompt engineering.
4. Filter aggressively. Drop retrieved chunks below a relevance threshold rather than padding to a fixed count.
5. Test at your actual context length. A prompt that works at 5,000 tokens may not work at 100,000. Evaluate at the length you will deploy.
6. Do not choose a model on window size alone. A 200,000-token model that reasons reliably beats a million-token model that does not, and you will rarely use the extra capacity well.

When large windows genuinely earn their cost

To be clear, they are sometimes exactly right. Analysing an entire codebase for cross-cutting concerns; reviewing a long contract for internal inconsistencies; processing a lengthy transcript where any part may be relevant; tasks where building a retrieval pipeline costs more engineering than the context tokens cost in inference.

The common thread is that relevance genuinely cannot be determined in advance. When you cannot know which part matters, sending all of it is correct. When you can, retrieval is usually cheaper and better.

Further reading

• Prompt caching — how to make a large repeated context affordable.
• Multi-turn conversation costs — how conversations approach the limit.
• How tokenization works — what fills the window faster than you expect.