Why output tokens cost 3-5x more than input tokens

This is not arbitrary price discrimination. It falls directly out of how transformer inference runs on a GPU — and once you see the mechanism, several counterintuitive cost optimisations become obvious.

Pricing fundamentals · 8 min read · Last reviewed 2026-07-28

Key takeaways

Inference has two phases: prefill (reading your prompt) and decode (generating the reply). They have completely different performance characteristics.
Prefill processes every prompt token in parallel and is compute-bound, so it is fast and cheap per token.
Decode generates one token at a time, re-reading the entire model's weights for each one, and is memory-bandwidth-bound. That is the expensive part.
Decode also cannot be batched as efficiently, and holds GPU memory for the whole generation.
Practical upshot: shortening outputs saves 3–5× more than shortening prompts, and "think step by step" has a direct, measurable price.

Look at any provider's rate card and the same pattern appears: output tokens cost three to five times input tokens. It holds across OpenAI, Anthropic, Google, xAI, Meta hosts and DeepSeek, and across price tiers from budget models to flagships.

When every competitor in a market independently arrives at the same ratio, it is usually not a pricing strategy. It is a cost structure. And in this case the cost structure comes from a specific fact about how transformer inference runs on a GPU.

Two phases, not one

When you send a request, the model does two quite different things.

Prefill: reading the prompt

First it processes your entire prompt to build up internal state. The crucial property here is that this happens in parallel. The transformer can attend to all prompt tokens simultaneously, in one or a few large matrix multiplications, because every input token is already known.

Large matrix multiplications are exactly what GPUs are built for. Prefill saturates the compute units, achieves high arithmetic intensity, and processes thousands of tokens in the time it takes to do a handful of sequential operations. It is compute-bound, and modern accelerators have compute in abundance.

The output of prefill is the KV cache: the key and value vectors for every token in your prompt, stored so they do not need recomputing.

Decode: generating the reply

Then it generates the response, and here the parallelism is gone. Producing token n+1 requires knowing token n, because the model is autoregressive — it feeds its own output back in. There is no way around this; it is what "generative" means.

So generating 500 output tokens requires 500 separate forward passes through the network, strictly in sequence. And each of those forward passes has an awkward property: it processes a single token but must read every weight in the model to do it.

That is the crux. For a model with tens or hundreds of billions of parameters, each single-token step moves an enormous quantity of data from GPU memory into the compute units, then does a comparatively tiny amount of arithmetic with it. The compute units sit largely idle waiting for memory. Decode is memory-bandwidth-bound, and memory bandwidth is the scarcest, most expensive resource on an accelerator.

The core asymmetry

Prefill does a large amount of arithmetic per byte of weights loaded. Decode does almost none. One is limited by how fast the chip can compute; the other by how fast it can move data. The second is far more expensive per token produced.

Why batching does not rescue decode

Providers serve many users concurrently, and batching is how they amortise cost. If the GPU is loading all the model weights anyway to generate one token for you, it can generate a token for many other requests in the same pass at almost no extra memory-bandwidth cost. This is why inference at scale is economically viable at all.

But batching decode is harder than batching prefill, for reasons that all cost money:

Sequences finish at different times. Requests in a batch produce different response lengths. Naive batching wastes capacity waiting for the longest one. Continuous batching mitigates this and adds scheduling complexity.

The KV cache consumes memory for the entire generation. Every active request holds its keys and values in GPU memory for as long as it is generating, and that memory grows with each token produced. KV cache capacity, not compute, is typically what limits how many concurrent requests a GPU can serve. Long outputs occupy that scarce memory for longer, directly reducing how many other users the same hardware can serve.

Latency requirements cap batch size. Larger batches improve throughput and worsen time-per-token. A provider promising responsive streaming cannot batch arbitrarily.

Prefill has none of these problems. Prompt lengths are known upfront, the work is one-shot, and it can be chunked and scheduled efficiently.

Roughly, in numbers

On typical serving hardware, prefill throughput is commonly measured in thousands of tokens per second per GPU, while decode throughput for a single sequence is tens of tokens per second. That is a difference of one to two orders of magnitude in raw speed.

Batching closes most of that gap in aggregate cost terms, which is why the price ratio you actually see is 3–5× rather than 50×. But it does not close it entirely, and the 3–5× you pay is a reasonable reflection of the underlying difference in cost to serve.

What this means for how you build

Output length is the dominant cost variable

If output costs four times input, then cutting 100 tokens from your response saves as much as cutting 400 from your prompt. Most people optimise the prompt, because that is the part they wrote and can see.

Ask for what you need and nothing more. Set max_tokens as a real constraint, not a safety net. Explicitly instruct brevity where brevity is acceptable — "answer in one sentence" is a price reduction. Request structured output rather than prose with structure embedded in it: a JSON object with three fields is far cheaper than three paragraphs explaining the same three fields.

Chain-of-thought has a price tag

"Let us think step by step" works because the model generates intermediate reasoning tokens, and those tokens are billed at the output rate. Reasoning quality is bought with output tokens, at the most expensive rate on the card.

That can absolutely be worth it. But it should be a deliberate trade, applied where accuracy justifies it, rather than a default appended to every prompt. Reasoning models take this to its conclusion — see why your bill is 2–5× the token count you estimated.

Long prompts are cheaper than they feel

The corollary is more useful than it sounds. Adding context is relatively inexpensive: few-shot examples, detailed instructions, retrieved documents, formatting specifications. All input, all at the cheap rate, and all can be cached to become cheaper still.

So the instinct to trim a good prompt for cost reasons is usually misplaced. A longer prompt that reliably produces a short, correct, correctly-formatted answer on the first attempt is cheaper than a terse prompt that needs a retry. Spend input tokens to save output tokens and retries.

Prefix caching attacks the cheap half — and still wins

Prompt caching discounts input, which is already the inexpensive side. It is nonetheless the biggest lever most applications have, because input volume in a real chat or agent application dwarfs output volume — the entire history resends on every turn. See the caching guide and multi-turn costs.

Streaming changes nothing about cost

Streaming improves perceived latency by delivering tokens as they are generated. The same number of tokens is generated and billed. It is a user experience decision, not a cost one.

A worth-knowing consequence: why input:output ratio picks your model

Because the two rates differ by 3–5×, and because providers do not all use the same multiple, the cheapest model for your workload depends on your ratio.

Consider a model priced $1.25 input / $10.00 output against one priced $1.25 input / $2.50 output. On a heavily input-weighted task — summarising a long document into a sentence — they cost almost the same. On an output-weighted task — generating long-form content from a short brief — the second is close to four times cheaper.

Same two models, same rate cards, opposite conclusions. This is why comparing headline input prices is not just imprecise but actively misleading, and why comparing on your real workload is the only method that works.

Try it

The calculator separates input and output cost and lets you vary expected response length. Set it short, then long, and watch the model ranking reorder — that reordering is the whole point of this article.