Home › Guides › Infrastructure

Self-hosting LLMs: working out the real break-even point

Self-hosting looks dramatically cheaper per token right up until you account for utilisation. Here is the calculation that decides it, and the costs people leave out.

Infrastructure · 12 min read · Last reviewed 2026-07-28

Open-weights models are free to download, and GPU rental is cheap enough that the per-token arithmetic looks devastating for commercial APIs. Then people actually do it, and a significant fraction migrate back.

The gap between the promising calculation and the disappointing outcome is almost always utilisation. Here is how to do the calculation properly before committing to hardware.

The fundamental difference

Everything follows from one structural fact: APIs bill per token, self-hosting bills per hour.

With an API, cost is exactly proportional to use. Zero traffic costs zero. Traffic doubling doubles the bill and requires nothing from you.

With self-hosting, you rent or buy a GPU and pay for wall-clock time whether it processes anything or not. Cost per token is therefore your hourly rate divided by tokens produced in that hour — which means it is entirely determined by how busy you keep it.

A GPU running at 10% utilisation has a cost per token ten times higher than the same GPU at 100%. This is the whole game, and it is why self-hosting comparisons based on peak throughput are meaningless.

Working out the break-even

Step 1: what hardware do you actually need?

Usually determined by VRAM, not compute. The model weights must fit in GPU memory, along with the KV cache for every concurrent request.

A rough weights calculation: parameters × bytes per parameter. At 16-bit precision that is 2 bytes per parameter, so a 70-billion-parameter model needs about 140 GB for weights alone — more than a single 80 GB accelerator, so multi-GPU. At 8-bit, roughly 70 GB, which fits on one. At 4-bit, roughly 35 GB, which fits comfortably with room for a decent KV cache.

Then add KV cache, which scales with context length and concurrency and is frequently underestimated. At long context and meaningful concurrency it can rival the weights in size. Budget for it explicitly, because running out of KV cache capacity is what limits your throughput in practice.

Quantisation is the main cost lever here. Reducing weights to 8-bit typically costs very little quality and halves memory. 4-bit is usually acceptable and sometimes not — it depends on the model and the task, and it needs testing rather than assuming. Below 4-bit, degradation becomes hard to ignore. Since the memory requirement often determines whether you need one GPU or four, quantisation frequently moves cost by a factor of several.

Step 2: measure real throughput

Not the benchmark number from a blog post. Your throughput, on your hardware, with your prompt lengths, your context sizes and your concurrency pattern.

Throughput depends heavily on things specific to your workload: batch size, sequence length, how bursty your traffic is, and which serving framework you use. A well-configured server with continuous batching can outperform a naive one by an order of magnitude on the same hardware. Aggregate tokens per second under realistic concurrency is the number you need.

Step 3: divide by utilisation

Cost per million tokens is:

cost per 1M tokens = (hourly GPU rate / tokens per hour at full load)
                               × 1,000,000 / utilisation

Take an illustrative example. Suppose you rent hardware at $2.00/hour and achieve 2,000 tokens per second aggregate throughput at full load — 7.2 million tokens per hour. At 100% utilisation that is about $0.28 per million tokens, which is genuinely cheaper than most commercial APIs.

At 5% utilisation, self-hosting costs twenty times what it does at full load — and more than most commercial APIs charge for comparable models.

Now ask what your utilisation will realistically be. Traffic follows daily and weekly cycles. If you serve business users in one timezone, your GPU is idle overnight and at weekends, which is already most of the week. And you must size for peak load, not average, or the service degrades when it matters — so average utilisation is structurally well below 100%.

Achieving genuinely high utilisation requires either steady round-the-clock traffic, or a batch workload you can use to fill the troughs. If you have neither, be honest that your number will be low.

The costs that are usually left out

In rough order of how much they distort naive comparisons.

Engineering time. The largest by a wide margin. Deploying a serving stack, tuning it, monitoring it, handling OOM conditions, managing model updates, and being on call for it is real ongoing work. At loaded engineering costs, a few days a month of attention can exceed the entire GPU bill. This is where most self-hosting business cases quietly fail.

Redundancy. One GPU is a single point of failure. Production availability means at least two, ideally in different zones, which roughly doubles hardware cost and halves utilisation.

Peak headroom. Sizing for peak means paying for capacity that is idle most of the time. Autoscaling GPUs helps, and cold-start times for loading large models into VRAM are long enough to be a real design constraint.

Storage and transfer. Model weights are tens to hundreds of gigabytes. Storing them, and moving them at instance start, has non-trivial cost.

Evaluation. Open-weights models generally require more prompt engineering and evaluation work to reach the output quality you get from a frontier model by default. Budget for it.

When self-hosting genuinely wins

It does win, in identifiable circumstances.

High, steady volume. Sustained round-the-clock traffic at scale, where you can hold utilisation high. This is the main case, and the economics can be compelling.

Data cannot leave your infrastructure. A regulatory or contractual requirement, in which case cost comparison is beside the point. Note that most major providers offer zero-retention and regional processing options that satisfy many such requirements — check before assuming self-hosting is the only route.

Batch workloads. Overnight processing with no latency requirement is close to ideal, because you can run at 100% utilisation by construction. Do compare against providers' batch endpoints, which are typically around 50% off list and require no infrastructure at all.

Custom fine-tunes. If you have fine-tuned a model on proprietary data and no API hosts it, self-hosting may be the only option. Model hubs with dedicated endpoints are the intermediate choice.

Local and edge. Running on hardware you already own — developer machines, on-premise servers — has a marginal cost of nearly zero. Tools like Ollama make this genuinely easy, and for development, testing and privacy-sensitive personal use it is excellent. It does not extend to serving production traffic at scale.

When it does not

Low or spiky volume — utilisation will be terrible. Early-stage products where traffic is unpredictable and engineering time is the scarcest resource you have. Cases where you need frontier capability, since the best open-weights models still trail the best proprietary ones on the hardest tasks. And small teams without GPU operations experience, where the learning curve is the real cost.

The honest summary

Self-hosting is an infrastructure decision that happens to have cost implications, not a cost optimisation. It makes sense when volume is high and steady, when you have the operational capability, or when data residency requires it.

For most teams below sustained high volume, commercial APIs are cheaper once engineering time is counted honestly — and the cheapest hosted open-weights models are inexpensive enough that the arbitrage self-hosting used to offer has largely closed. Inference providers hosting Llama, Mistral, Qwen and DeepSeek at low per-token prices have done the utilisation optimisation for you, across many customers, which is exactly the thing you would struggle to do alone.

Before buying GPUs, price the same workload on a cheap hosted open-weights model. The answer is frequently that you were optimising the wrong variable.

Further reading

• Gateways vs direct API access — the cheaper option to rule out first.
• Why output tokens cost more — the GPU economics underneath all of this.
• Context windows — why the KV cache determines your serving capacity.