A single API and single bill across models from most major providers, with automatic fallback when one is down.
Directory
LLM Gateways and API Routers Compared
A gateway sits between your application and the model providers. Instead of calling Anthropic, OpenAI, and Google directly with three SDKs, three keys, and three billing relationships, you call one endpoint and the gateway routes the request.
The reason to add one is rarely the unified interface — most SDKs are similar enough that swapping them is an afternoon. It is the operational layer: automatic failover when a provider degrades, per-key budgets so one runaway job cannot spend a month of credit, caching, and a single place to see what every feature is costing you.
If you call one provider, run modest volume, and have no need to switch models at runtime, a gateway is a dependency and a hop of latency you do not need yet. Add it when the failure it prevents becomes real.
4 tools · 2 open source · 2 self-hostable
At a glance
| Tool | Best for | Pricing | Runs |
|---|---|---|---|
| OpenRouter | Trying many models quickly, or shipping something that must keep working when a single provider degrades. | Usage-based | Hosted |
| LiteLLMopen source | Teams that want gateway behaviour — budgets, key management, failover — without adding a third-party service to the request path. | Open source | Hosted or self-hosted |
| Portkeyopen source | Teams that want routing and request-level tracing from one system instead of integrating a gateway and an observability tool separately. | Free tier | Hosted or self-hosted |
| Cloudflare AI Gateway | Adding caching and cost visibility to calls you already make, especially if you are on Cloudflare already. | Free tier | Hosted |
LiteLLM
open sourcePython SDK and self-hostable proxy that puts a consistent interface over 100+ model APIs, with budgets and rate limits per key.
Portkey
open sourceAI gateway with routing, caching, retries, and guardrails, plus request-level observability.
Drop-in proxy that adds caching, rate limiting, and analytics to provider calls from Cloudflare’s edge.
How to choose
Hosted or self-hosted
A hosted gateway is another service in your critical path — when it is down, you are down. A self-hosted proxy keeps the failure domain yours but is now yours to run. Some options offer both; pick before you build around one.
What it costs you in latency
Every gateway adds a network hop. That is usually single-digit milliseconds and irrelevant next to generation time, but it matters for high-volume classification where the model call itself is short.
Spend controls that actually bind
The most valuable gateway feature is a hard budget per key, per user, or per feature. Check whether limits are enforced or merely reported — an alert after the fact does not stop the bill.
Whether you keep provider features
Gateways normalise across providers, and normalisation loses things: prompt caching, extended thinking, structured outputs, and provider-specific tool types are the usual casualties. Confirm the features you rely on survive the abstraction.
Frequently asked
What is an LLM gateway?
A service that sits between your application and one or more model providers, exposing a single API. It typically adds routing, failover, caching, rate limiting, spend controls, and request logging.
Do I need an LLM gateway?
Not at first. If you use one provider at modest volume, calling it directly is simpler and faster. A gateway earns its place when you need failover across providers, hard per-key budgets, or the ability to change models without a deploy.
Does a gateway make requests slower?
It adds one network hop, usually a few milliseconds. Against a generation that takes seconds this is noise. It is worth measuring on short, high-volume calls where overhead is a larger share of total time.
Can I self-host a gateway?
Yes — several of the options here are open source and run as a proxy inside your own infrastructure, which keeps both your keys and your uptime under your control.
Tooling choices rarely dominate a bill; the model does. Compare model prices per million tokens or price your own workload before optimising anything here.