Home › Guides › Cost optimisation

Prompt caching: how it works at OpenAI, Anthropic and Google

Prompt caching is the single largest cost lever available to most production LLM applications, and the easiest to accidentally get no benefit from at all.

Cost optimisation · 12 min read · Last reviewed 2026-07-28

Prompt caching is the largest cost lever available to most production LLM applications. It is also the one most commonly implemented in a way that produces no benefit at all, because the requirement it imposes on prompt structure is strict and easy to violate by accident.

What is actually being cached

When a model processes your prompt, it computes internal state for every token — the key and value vectors that later tokens attend to, collectively the KV cache. This is the expensive part of the prefill phase (see why output costs more than input).

Crucially, in a causal transformer, the state for token i depends only on tokens 1 through i. Nothing later can affect it. So if two requests begin with the same first 2,000 tokens, the internal state for those 2,000 tokens is identical in both cases, regardless of how the requests differ afterwards.

That is the entire basis of prompt caching. The provider stores the computed state for a prefix, and on a subsequent request that starts with the same prefix, loads it instead of recomputing it. Loading from a cache is far cheaper than recomputing attention over thousands of tokens, and the saving is passed on.

How the providers differ

The mechanisms fall into two families, and the difference matters for how you implement.

Automatic caching (OpenAI, Google implicit)

Nothing to configure. The provider detects repeated prefixes and applies a discount — typically around 50% off the input rate for cached portions. There is usually a minimum prefix length before caching engages, on the order of 1,000 tokens, and matching happens in blocks rather than at arbitrary boundaries.

The response includes a field reporting how many input tokens were served from cache. Log this. It is the only way to know whether your caching is working, and the gap between assumed and actual hit rates is routinely large.

Advantages: zero implementation effort, no risk of misconfiguration. Disadvantages: no control over what gets cached or for how long, and a smaller maximum discount than explicit schemes.

Explicit caching (Anthropic, Google explicit context caching)

You mark where the cacheable prefix ends — Anthropic uses a cache_control breakpoint on a content block. The discount on reads is much steeper, on the order of 90% off the input rate.

But there is a catch that has no equivalent in the automatic schemes: writing to the cache costs more than a normal input token, on the order of 1.25× the base rate. So the economics are:

uncached:        1.00 × base rate, every request
          cached, 1st hit: 1.25 × base rate (the write)
          cached, later:   ~0.10 × base rate (the reads)

Which means caching a prefix that gets used once costs you 25% more than not caching it. Break-even arrives on the second read; after that the savings compound rapidly. Cache a prefix you expect to reuse; do not cache reflexively.

Google's explicit context caching adds a different dimension: you pay a storage charge per token per hour for the cached content, separate from the read discount. That makes it well suited to a large document queried many times over an extended period, and poorly suited to short-lived prefixes.

How applications defeat their own caching

In rough order of how often it happens.

Dynamic content at the top of the prompt

This is the big one. A system prompt that opens with the current date, a session identifier, a request ID, or the user's name produces a different prefix on every single request. Cache hit rate: zero. The prompt looks almost identical to a human and is completely different to a cache.

The fix is trivial once you see it: move all of it to the end. Static instructions, tool definitions and reference material first; per-request variables last.

Worth checking whether you actually need the dynamic value at all. "Today's date is..." injected into every system prompt is a common pattern that costs an entire caching strategy in exchange for information most prompts never use.

Non-deterministic serialisation

If you build the prompt from a dictionary or map with non-deterministic iteration order, the bytes differ between requests even though the content is identical. Same for JSON serialisers that do not sort keys, floating-point formatting that varies, and any construction path that produces logically equivalent but textually different output.

Byte-identical means byte-identical. Serialise deterministically, and ideally assert on it in tests.

Reordered tool definitions

Tool and function schemas are usually a large, perfectly static block — an ideal caching candidate, sitting right at the front of the prompt. If they are assembled from an unordered collection, or the set varies by request, that advantage is lost. Fix the order and, where possible, the set.

Sliding-window history truncation

A common mitigation for multi-turn cost growth is dropping the oldest exchange each turn. But that changes the prefix every turn, so it fights caching — and caching is usually the bigger win.

Truncate in infrequent large chunks instead of continuously. Drop ten exchanges every tenth turn rather than one every turn: you get nine cached turns and one cache miss, instead of ten misses.

Expiry in low-traffic applications

Caches live for minutes. A high-traffic application keeps its common prefixes permanently warm; an application handling a request every ten minutes may miss almost every time and see no benefit whatsoever from a correctly implemented caching strategy.

This is not fixable by better prompt structure. If your traffic is genuinely sparse, measure before assuming caching will help, and consider a longer-TTL option where the provider offers one.

Structuring a prompt for caching

Order the prompt from most to least stable:

1. Tool and function definitions — typically the largest fully static block.
2. System instructions — role, rules, output format.
3. Few-shot examples — static, and often substantial.
4. Reference documents — static if the same corpus is used across requests.
5. Conversation history — grows by appending, so everything up to the latest turn is cacheable.
6. Retrieved context for this specific query.
7. The current user message.
8. Dynamic values — timestamps, IDs, user metadata.

With explicit caching, place the breakpoint after the last block you expect to be reused. With automatic caching, this ordering is all you need.

Which workloads benefit most

Excellent: document Q&A, where one large document is queried repeatedly; agents with large static tool definitions; any application with a long system prompt and steady traffic; multi-turn chat with active users; few-shot classification at volume.

Poor: single-shot requests with short, unique prompts; low-traffic applications where caches expire between requests; workloads where each request's context is genuinely different from every other's; prompts below the minimum cacheable length.

Measure it

Every provider reports cached token counts in the response. Log them, compute your actual hit rate, and alert on regressions.

This matters more than it sounds, because caching fails silently. A refactor that moves a timestamp to the top of the system prompt will not break a single test. It will just quietly multiply your input costs, and without hit-rate monitoring you will discover it on an invoice weeks later.

The gap between an assumed hit rate and a measured one is routinely substantial. Treat any cost model built on an assumed rate as provisional until you have real numbers.

Further reading

• Multi-turn conversation costs — the problem caching most effectively solves.
• Why output costs more than input — what the KV cache is and why computing it is expensive.
• LLM API pricing explained — where caching fits among the other discounts.