Home › Guides › Cost modelling

The hidden cost of chatbots: why multi-turn conversations grow quadratically

A per-request cost estimate will understate a real chatbot by an order of magnitude. The reason is a single design fact about how LLM APIs work: they have no memory.

Cost modelling · 11 min read · Last reviewed 2026-07-28

Here is the single most common cost-modelling mistake in LLM applications. Someone works out that an average request is 500 input tokens and 300 output tokens, prices that correctly, multiplies by expected daily conversations, and budgets accordingly. Then the first invoice arrives at several times the estimate.

The error is not in the arithmetic. It is in the assumption that a conversation is a sequence of independent requests.

The API has no memory

LLM APIs are stateless. A chat completions endpoint receives a list of messages, produces one response, and forgets everything. There is no session, no conversation ID that persists state, nothing retained between calls.

So when a user sends their fifth message in a chat, your application does not send that message. It sends the system prompt, plus the first user message, plus the first assistant reply, plus the second user message, plus the second assistant reply, and so on, plus the new fifth message. All of it, as input tokens, at the input rate.

The conversational memory your users experience is an illusion your application maintains at its own expense, by re-transmitting the entire history every single turn.

The arithmetic

Let S be the system prompt in tokens, u the average user message, a the average assistant reply, and n the number of turns.

On turn i, the input is the system prompt, plus all i−1 previous exchanges, plus the current message:

input(i) = S + (i-1) × (u + a) + u

Summing over all n turns gives total input tokens for the conversation:

total input = n×S + n×u + (u + a) × n(n-1)/2

The n(n-1)/2 term is the important one. It is quadratic. Double the conversation length and the history component roughly quadruples.

Total output, by contrast, is simply n × a — linear. So as conversations lengthen, an application that started out output-dominated becomes overwhelmingly input-dominated.

A worked example

Take a system prompt of 500 tokens, average user messages of 50 tokens, and average assistant replies of 300 tokens.

By turn 20, the conversation has consumed 77,500 input tokens against a naive estimate of 11,000 — a factor of seven. By turn 50 it is nearly seventeen. Note also that the single request on turn 50 is 17,700 tokens, which matters for context window limits as well as cost.

If your product has a small number of users who hold very long conversations, they can plausibly account for the majority of your bill. Cost per user in chat applications is extremely unevenly distributed.

What to do about it

1. Prompt caching (largest effect, least disruption)

This is the right first move in almost every case. Every major provider will discount input tokens that repeat a prefix it has recently processed — commonly 50%, up to 90% for explicit cache reads.

A conversation is close to an ideal caching workload: turn n's input is turn n−1's input plus a bit on the end, so almost the entire prompt is a repeat of a prefix the provider just saw. Applied well, this substantially flattens the quadratic curve — the tokens still accumulate, but most of them bill at a fraction of full rate.

Two requirements. The prefix must be byte-identical, so anything dynamic must go at the end of the prompt, never the beginning. And caches expire, typically after a few minutes of inactivity, so a user who returns after a coffee break pays full price again. The caching guide covers provider specifics, minimum sizes, and the case where a cache write costs more than not caching.

2. Truncate the history (simple, effective, lossy)

Keep the system prompt and the most recent k exchanges; drop the rest. This converts quadratic growth into linear growth, because request size stops at a ceiling.

It is crude but frequently adequate. For most support, Q&A and task-oriented assistants, turn 30 rarely needs turn 2. Choose k by looking at actual transcripts and asking how far back references genuinely reach.

Watch two things: dropping history mid-conversation invalidates the cached prefix, so a naive sliding window can fight your caching — drop in larger infrequent chunks rather than one exchange per turn. And users notice when the assistant forgets something they said, so be deliberate about where the boundary falls.

3. Summarise older history (better quality, added complexity)

Rather than discarding old turns, periodically replace them with a compact summary: 40 turns become a 300-token synopsis plus the last 5 verbatim exchanges.

This preserves more continuity than truncation, and it is what most production assistants with genuinely long sessions do. The costs are that the summarisation call is itself an LLM request, summaries lose specifics (exact figures, names and decisions are the usual casualties), and each new summary changes the prefix and therefore invalidates the cache. Summarise infrequently and in large blocks.

Use a cheap model for the summarisation step. It is a genuinely easy task and does not need your flagship.

4. Structural changes

Retrieve instead of retaining. Store the transcript externally and inject only the passages relevant to the current message. Turns 50 costs the same as turn 5. Adds a retrieval system and the risk of missing relevant context.

Cap conversation length. A "start a new conversation" prompt after a certain number of turns is a legitimate product decision, not just a cost hack — very long chats often indicate a task that has drifted anyway.

Tier by turn depth. Early turns matter most for first impressions; deep turns are often follow-ups and clarifications. Routing later turns to a cheaper model is sometimes viable, though it needs testing because tone shifts are noticeable.

Agents have the same problem, worse

Everything above applies to agentic loops, with an additional multiplier.

An agent that answers a question by calling three tools makes four API requests: one to decide on the first tool, one after receiving its result, and so on. Each request carries the system prompt, the tool definitions, the user's request, and every prior tool call and result. Tool results are frequently large — API responses, file contents, search results — which makes each step grow faster than a conversational turn does.

So a single user request to an agent can generate the token volume of a long conversation. Multi-step agents are the most expensive common LLM architecture, and per-request estimates understate them dramatically. Model them as a conversation whose "turns" are tool calls, and be aggressive about trimming tool output before it enters the context.

Further reading

• Prompt caching — the main mitigation, in detail.
• Context windows — long conversations eventually hit the limit, and get worse before they do.
• LLM API pricing explained — the other line items on the bill.