Home › Guides › Infrastructure

LLM gateways vs direct API access: when the markup is worth it

A gateway charges you roughly 5% to not have to manage six API keys, six rate limits and six billing relationships. Whether that is a bargain or a waste depends on things that are easy to work out in advance.

Infrastructure · 11 min read · Last reviewed 2026-07-28

Once you use more than one model provider, you have a small infrastructure problem: multiple API keys, multiple SDKs with different request formats, separate rate limits, separate billing relationships, and no single place to see what you are spending. Gateways exist to solve that. The question is whether the solution costs less than the problem.

Three different things called "gateway"

The terminology is loose, and the categories have quite different economics.

Aggregating gateways

OpenRouter, LLM Gateway.

These do not host models. They sit in front of everyone else's APIs and give you one endpoint, one key and one bill for hundreds of models across dozens of providers. You send an OpenAI-shaped request naming any model, and they route it.

Revenue comes from a markup on the underlying provider's price — commonly around 5%, though it varies by model and some are passed through at no markup. Both also offer bring-your-own-key arrangements where you supply your own provider credentials and the routing layer is free or near-free.

Inference providers

Groq, Together AI, Fireworks AI.

These are genuinely different and often confused with the above. They host open-weights models — Llama, Mistral, Qwen, DeepSeek and others — on their own hardware and set their own prices. There is no markup because there is no upstream provider; they are the provider.

They compete on price, throughput and latency. Groq in particular runs custom inference hardware and delivers throughput that is difficult to match elsewhere, which matters for real-time and voice applications where time-to-first-token dominates the user experience.

They cannot serve proprietary models. If you need GPT or Claude, an inference provider is not an option — you need a direct relationship or an aggregating gateway.

Model hubs

Hugging Face.

A different proposition again: access to an enormous catalogue including fine-tuned and specialised models that no commercial API carries, via serverless inference or dedicated endpoints. The right answer when you need a specific fine-tune rather than a general-purpose model, and less suited to high-volume general inference.

What the markup actually buys

Taking around 5% as the typical figure, here is what you get for it.

One integration

The strongest argument. Providers' APIs differ in message format, parameter names, streaming conventions, error semantics and tool-calling schemas. Supporting four providers directly means four integrations and four sets of edge cases to maintain as they all evolve.

A gateway collapses that to one, and adding a fifth model becomes a configuration change rather than an engineering task. If you are actively evaluating models — which most teams should be, given how fast the landscape moves — this alone can justify the cost.

Automatic failover

Providers have outages, and they have rate limits you will hit at exactly the wrong moment. A gateway can retry against a different provider automatically. Building equivalent multi-provider failover yourself is not difficult in principle and is fiddly in practice, particularly getting the behaviour right under partial degradation rather than clean failure.

For anything user-facing, this is worth real money. 5% of your inference spend is usually cheaper than the engineering time plus the incidents.

Consolidated billing and visibility

One invoice, one usage dashboard, one place to set spend limits. Genuinely useful operationally, and more useful than it sounds if you have to attribute cost across teams or customers.

Access without commercial setup

Trying a new provider through a gateway requires no new account, no payment method, no minimum commitment. For evaluation, this removes a real amount of friction.

What it costs beyond the markup

Latency

An extra network hop adds latency — usually tens of milliseconds, which is immaterial against multi-second generation times, but not always. If you are optimising time-to-first-token for a voice application, going direct to a fast-inference provider is measurably better.

A third party sees your prompts

This is the consideration that most often decides the question. Your prompts transit the gateway's infrastructure, which means an additional party in scope for data protection review, an additional vendor to assess, and an additional set of retention and processing terms to read.

Some gateways offer strong guarantees and compliance certifications; check them rather than assuming. For regulated workloads, healthcare, or anything under a data processing agreement that enumerates sub-processors, this may be a hard blocker regardless of the economics.

Another dependency

The gateway can itself go down, and then everything behind it is unavailable. The failover argument cuts both ways: you have removed single-provider risk and added single-gateway risk. Mitigating properly means being able to fall back to a direct integration, which partly undoes the "one integration" benefit.

Feature lag

Gateways normalise across providers, which means the newest provider-specific features — a novel tool-calling mode, structured output guarantees, a new caching control — may not be exposed immediately or at all. If you depend on the leading edge of one provider's feature set, go direct.

This applies to prompt caching specifically, which is worth checking carefully: caching semantics are provider-specific and not all gateways expose the controls needed to use them well. Given that caching is often the largest cost lever available (see the caching guide), losing it to save 5% is a bad trade.

Deciding

The scale arithmetic

The markup is a percentage, so its absolute cost scales with spend while the engineering effort it saves is roughly fixed.

At $500 a month, 5% is $25 — less than an hour of engineering time, so using a gateway is obviously correct. At $50,000 a month, 5% is $2,500 monthly, or $30,000 a year, which funds a meaningful amount of direct-integration work.

The crossover is usually somewhere in the low thousands per month. Above it, look at bring-your-own-key arrangements, which often preserve most of the operational benefit at a fraction of the cost — you keep the unified interface and routing while paying providers directly.

A pragmatic pattern

Many teams converge on the same shape, and it is worth stating explicitly:

• Direct integration with the one provider handling the bulk of production traffic, where you want full feature access, caching control and the best price.
• A gateway for everything else — secondary models, experiments, evaluation, and fallback when the primary is degraded.

You pay the markup only on the minority of traffic where flexibility matters more than unit cost, and you keep the direct relationship where volume makes it worth the engineering. It is more work than either extreme, and for most teams past early scale it is the right amount of work.

Further reading

• Self-hosting break-even — the option beyond all of these.
• Prompt caching — check your gateway exposes it before trading it away.
• LLM API pricing explained — what you are marking up in the first place.