A gateway or router for model calls

One proxy, two jobs

An LLM gateway is a proxy between your application and the model providers it calls. It holds the API keys, applies rate limits and budgets, logs every call with its cost, retries and falls back when a provider fails, and gives your code one interface for many providers. A router is the part that decides which model or provider handles a given call, by price, by task, or by rules you write. Many products do both. You need one once you call more than one provider or need to know what each feature spends.

Before that point a gateway is an extra network hop on every call and one more service for someone on your team to keep running.

The gateway half: control in one place

Without a gateway, every service that calls a model carries its own key, its own retry loop and its own idea of logging. Rotating a key means a deploy in each service. Finding out which feature spent the money means reading invoices and guessing.

A gateway moves those concerns into one place. Keys live there and nowhere else. Each call is tagged with the team or feature that made it, so spend can be capped per tag and a runaway loop in one service cannot drain the shared budget. Logs carry the prompt size, the response size and the price, which turns the monthly bill into something you can explain.

It is also the natural place for guardrails that inspect prompts and replies, and for caching identical requests. Each of those deserves its own decision; the gateway only gives them somewhere to live.

The router half: choosing per call

Routing exists because the same work can often be done by different models at very different prices. A short classification can go to a small cheap model; a long reasoning task can go to a larger one. Rules can be static, by endpoint or tenant, or dynamic, by a classifier that looks at the request first.

Routing also works across sellers of one model. The LLM prices page shows which models are sold by more than one provider (models on record with several sellers: 93; providers with a token price on record: 16). A router can send a call to whichever seller is cheaper or currently healthy.

Be careful with that last idea. The same model name at different providers may differ in precision, context limit or version, and outputs can change between them. Test each seller you route to, as you would test a new model.

Fallback is a policy, and policies need tests

A feature many gateways advertise is fallback: when the primary provider errors or slows, send the call somewhere else. It sounds like free resilience. In practice it is a second production path that only runs during an outage, which is the worst time to discover it was never tested.

Write the fallback down as a rule. Which errors trigger it, how long to wait before giving up on the primary, which model or seller takes over, and whether the prompt needs changing for that model. Then trigger it on purpose, on a quiet afternoon, and read the replies.

Watch the bill as well. A fallback model may be priced higher than the primary, and an outage that lasts a day moves a day of traffic onto that price. The gateway log will show it afterwards; a budget cap on the fallback route stops it during.

When it earns its place, and when it does not

A gateway earns its place when you call several providers, when several teams share a budget, when you need an audit log of every prompt for compliance, or when a provider outage has already cost you a day. Any one of those is enough.

It does not earn its place for a single service calling a single provider with one key. There, the provider's own client library may already retry, and the provider's console may already show spend. Adding a gateway adds latency on every call, a new failure point in the request path, and in some cases a place where prompts are stored that your data policy has to cover.

The honest middle case is a team that expects to need one soon. There, a thin wrapper in your own code with a single function for model calls costs an afternoon and makes adopting a gateway later a one-line change. It also gives you somewhere to log cost from the first day.

What to check before adopting one

Choose between a hosted gateway and one you run. A hosted one is less work and sees every prompt you send. A self-run one keeps data in your network and puts the gateway on your on-call rota.

Start small whichever you pick. Route one feature through the gateway, compare its logged cost with the provider invoice for a week, and only then move the rest. A gateway that counts tokens differently from the provider will mislead every budget built on it.

Check which providers and API shapes it supports, how it counts tokens for cost (its count should match the provider invoice), what happens to requests when the gateway itself is down, and whether logs can be turned off for sensitive routes. The gateway and router category lists the products in the directory with facts read from vendor pages. It does not rank them, and nor does this post.