LLM Provider Routing: What a Gateway Won't Fix

In July 2026 the frontier moved fast enough that one benchmark outlet published head-to-head comparisons of three different flagship models on a single day. If adopting any of them costs you a sprint, the architecture is wrong: model choice belongs in configuration, behind a routing layer. But LLM provider routing normalises the wire format and very little else. Your prompt cache is scoped to the model you just routed away from, identical text tokenizes differently on each model, and a request that works on one model returns a 400 on its own sibling. Routing is necessary. It isn't sufficient, and the difference is where the money goes.

Three flagship models, one benchmark cycle

On 23 July 2026, LLM Stats published a head-to-head of Claude Opus 5 against GPT-5.6 Sol and, the same day, Opus 5 against Claude Fable 5. Three flagship models, current enough to be worth benchmarking against each other, covered in one afternoon.

That cadence is the operating condition now, not an unusual week. A July 2026 comparison of gateways put the stakes plainly: switching models without redeploying is "the difference between a 30-minute decision and a 30-day project."

There's a duller reason to care, too. Major providers each experience multiple incidents per month, from elevated latency through partial degradation to complete API unavailability. And new releases are only one of five triggers that should make you reconsider a model; the others are rising costs, latency or UX degradation, a widening set of task types, and governance changes.

Couple your application code to a model string and every one of those five becomes a migration. Put the model behind a routing layer and four of them become a config change. The fifth, quality, is the one this post spends most of its time on.

What an LLM gateway actually is

An LLM gateway is a single API endpoint between your application and every model provider, translating one request shape into each vendor's own and handling routing, failover, and usage accounting on the way through.

That's the whole idea. Here's the honest scope of what it fixes.

Model choice becomes a slug

Routing is expressed per request. The Vercel AI Gateway exposes order to try providers in sequence and fail over on error, only to restrict traffic to named providers, and models to supply a fallback model list when the primary is unavailable:

import { generateText } from "ai";
import { gateway } from "@ai-sdk/gateway";

const { text } = await generateText({
  // The model is configuration, not code: a config rollout, not a code change.
  model: gateway(process.env.SUMMARY_MODEL ?? "anthropic/claude-opus-5"),
  prompt,
  providerOptions: {
    gateway: {
      order: ["bedrock", "anthropic"],                       // provider failover for this model
      models: ["openai/gpt-5.6-terra", "google/gemini-3.6-flash"], // model-level fallback
      tags: ["feature:summary", "env:production"],           // cost attribution
      user: userId,
    },
  },
});

Note what SUMMARY_MODEL is doing. The slug is an input, so a model swap ships as a configuration change rather than a code change, and how fast that lands is a property of your deployment system rather than of the gateway. Resist the urge to hard-code it, and resist the urge to guess: Vercel's own AI Gateway guidance is that slugs, routing rules, and capabilities change often enough that you should read the current model list rather than assume last quarter's names still resolve.

Failover stops being bespoke code

If a provider degrades, the gateway fails over to the same model on another provider, so a bad afternoon at one vendor doesn't take your product down. Kong's AI Proxy Advanced plugin does the same job as declarative config: provider targets, priority-based load balancing, fallback chains, traffic redirected with no code changes. OpenRouter treats mixed-provider fallback chains as table stakes, and the comparison above is blunter: "a gateway without failover is only half a gateway." Given the incident rate, that half earns its keep on its own.

Cost attribution per request

Both tags and user in the snippet above are accounting, not routing. They let you answer "which feature spent that" without instrumenting anything yourself. Pricing inside a single vendor's lineup spans an order of magnitude, from Claude Haiku 4.5 at $1 in and $5 out per million tokens up to Claude Fable 5 at $10 and $50, which is what makes per-task model selection a genuine cost lever rather than a micro-optimisation. Pick the cheap model for the mechanical pass and the expensive one for the judgement call, and tag both so you can prove it later. That's the same reasoning behind our two-stage web scraping pipeline, where a cheap model does the structuring pass and nothing more expensive touches the page.

On the commercial side, the Vercel gateway passes through provider list pricing with zero markup and supports BYOK for direct provider billing at no additional fee. The routing layer doesn't have to be a tax.

The portability tax: what LLM provider routing doesn't cover

Here is the part the gateway comparisons leave out.

A DigitalOcean piece from 31 July 2026 states it better than we could: "OpenAI-compatible" only describes the wire format, while prompt wording, error handling, caching, and cost estimates are all calibrated to one provider's specific behaviour, "and that calibration is where the real migration cost lives."

Call that residue the portability tax. It's the work that remains after the request shapes match, and it doesn't appear on any feature comparison table. Four parts of it are worth knowing before you route anything.

The gateway solves the bottom layer. The four above it are still yours.

Your prompt cache belongs to the model you just left

Prompt caches are model-scoped. Anthropic's caching documentation is explicit that changing the model string invalidates the existing cache and the first request on the new model writes it fresh, and that this one has no workaround because caches are scoped to a model by design.

The arithmetic matters. Cache reads cost roughly 0.1x the base input price, while writes cost 1.25x at the five-minute TTL and 2x at the one-hour TTL. On the five-minute TTL you break even at two requests (1.25x + 0.1x = 1.35x, against 2x uncached); on the one-hour TTL it takes three (2x + 0.2x = 2.2x, against 3x uncached). Route away before you've earned that back and you pay the write premium a second time on the other side.

Then there's a quieter failure. The minimum cacheable prefix varies by model, and it doesn't move in one direction as models get newer:

Model	Minimum cacheable prefix
Claude Opus 5, Claude Fable 5	512 tokens
Claude Opus 4.8, Claude Sonnet 5, Claude Sonnet 4.6, Claude Sonnet 4.5	1,024 tokens
Claude Opus 4.7	2,048 tokens
Claude Opus 4.6, Claude Opus 4.5, Claude Haiku 4.5	4,096 tokens

Those are the current models; the caching documentation lists older and preview ones at the same four floors. Read the two ends of the table together. A 3,000-token system prompt caches on Opus 5 and on Sonnet 5, and silently doesn't cache on Opus 4.6 or Haiku 4.5. Silently is the operative word: a prefix under the minimum raises no error and simply reports cache_creation_input_tokens: 0. Route a cache-optimised workload onto a model with a higher floor and your bill changes shape with nothing in the logs to explain it.

The same text costs a different number of tokens

Tokenizers differ between models, including between generations of the same family. Anthropic's migration guide reports that the Opus 4.7 tokenizer produces roughly 1x to 1.35x as many tokens for identical text as its predecessor, and that Claude Sonnet 5 produces about 30% more tokens than Sonnet 4.6 at unchanged per-token pricing. Its advice is to re-measure with a token-counting endpoint per model and specifically not to apply a blanket multiplier.

Sit with what that does to a routing decision. A model advertised at a lower price per token can produce a more expensive request, because the price fell less than the token count rose. Any cost model built on price-per-token alone will point you at the wrong model with total confidence.

A valid request returns 400 on its own sibling

This is the fact that ends the "routing is just swapping the slug" argument, and it doesn't even require crossing vendors. On Claude Opus 4.7 and later, an extended-thinking request carrying budget_tokens is rejected outright, and temperature, top_p, and top_k are no longer accepted at all. Same provider, same endpoint, same wire format, newer model, hard 400.

We hit our own version of this. Our judge panel talks to an OpenAI-compatible gateway endpoint, and sending response_format returns a 400 there, so the prompt enforces JSON-only and the wrapper extracts the object from whatever comes back. Compatible in shape, not in feature set.

Capability skew runs deeper than parameters. Feature availability differs by model tier and by platform, so a fallback target can be the "same" model on paper and still be missing the tool your prompt depends on.

Structured outputs and streaming drift underneath you

Two failure modes worth reading before you fan out across many providers, both catalogued in a May 2026 write-up on routing across ten of them. Routing the same structured-output request to different providers does not keep the JSON schema consistent. And when a router switches providers mid-conversation, say because provider A timed out, the streaming response format changes underneath the consumer.

If you want proof that "unified" abstractions are really a stack of per-provider translators, read LiteLLM's internals. It carries a separate transformation config per provider, and its own contributor docs use prompt-caching translation as the worked debugging example, comparing how two backends each handle a cache_control field. That's a mature, well-built library. The per-provider quirks didn't vanish, they moved into files with names.

Quality regresses without telling you

The most expensive item on the list has no error code. Routing to cheaper models produces silent quality regression: answers degrade in ways that surface as customer tickets days later rather than on a dashboard.

Prompts are a large part of why, and the best evidence comes from a vendor documenting its own lineup. Anthropic's migration guide devotes a section to prompt-behaviour changes between models: instructions written to overcome an older model's reluctance overtrigger on a newer one, while verbosity, tone, tool-use rate, and how eagerly the model delegates to subagents all shift between versions. Its remedy is to re-run the workload with the older scaffolding stripped out rather than porting prompts across unchanged.

That is the optimistic case, one vendor describing its own models with the receipts published. Between vendors you get the same drift with no changelog. And nothing in the response will tell you: every request still returns 200.

Providers now ship their own routing

Something changed recently that none of the gateway round-ups mention, including the one published on 21 July.

Anthropic now offers a server-side fallbacks parameter in beta. A declined request is re-run on another model inside the same call, and a default mode picks the fallback by refusal category rather than making you maintain a list. Two things about it are interesting for anyone building a routing layer.

First, the accompanying fallback-credit mechanism bills the previously cached span at cache-read rates on the retry instead of charging a cold write on the new model. That is a direct answer to the cache-locality problem from two sections ago, and it's hard for a third-party router to match, because the cache belongs to the provider rather than to the layer in front of it.

Second, a refusal is an HTTP 200, carrying stop_reason: "refusal". Any routing or retry logic that branches on status codes will sail straight past it, and code that reads content[0] unconditionally will break on the empty content array.

This doesn't replace a gateway. It's single-vendor, it's beta, and it won't fail you over to a different company when that company is the one having the outage. But it does change the build-versus-buy question for the failover half, and it solves a cost problem that no gateway can.

What we run, and what broke

We should be concrete about where our own numbers come from.

The engine that produced this post runs every draft through a quality gate: three models from three different vendors score it on six axes, and the post ships only if the average clears 8.5 with no single axis below 7. All three run through one gateway, and the panel is one environment variable:

JUDGE_MODELS="openai/gpt-5.6-terra google/gemini-3.6-flash xai/grok-4.5"

Swapping a judge is editing that line. No code changes, no redeploy, and the alternates we've tried are recorded next to it. Cost comes from each response's own usage.cost field rather than a price table we'd have to maintain, which matters given how fast the prices in this post will date. A clean three-model pass costs us about $0.05, and external API cost for a finished post lands around $0.30 to $0.45.

Now the part that's more useful than the numbers.

On our first real run, a slow model intermittently hung the gateway call. Our wrapper set no curl timeout, so one stuck member blocked the entire pass past the shell's foreground timeout and left orphaned processes still appending to the cost ledger. The fix was to drop that model from the panel, which took one line, and the lesson took longer: routing added a failure mode our single-provider code never had. A fan-out is only as fast as its slowest member unless you set the timeout yourself, and a partially failed fan-out can keep spending after you think it stopped. There was a smaller sting alongside it. Our loader re-reads the env file and overrides any inline variable passed on the command line, so the "quick test" of a different panel silently tested the old one.

Two smaller notes, because they're the portability tax in miniature. One judge occasionally returns prose instead of JSON, so the panel degrades to whichever models parsed and still gates rather than failing the run. And the pattern extends past LLM calls: our search layer routes across several providers with an explicit fallback order, and while researching this post one provider's key was missing, so the run degraded to the remaining four and finished with cost tracking intact. Underneath that abstraction one search API returns results under .data[] rather than the nested key its sibling uses, and another rejects an unexpected field with extra_forbidden instead of ignoring it.

One caveat, stated plainly: this panel is a content quality gate, not a user-facing production request path. The failure modes generalise. The traffic volume doesn't, and we're not going to pretend otherwise. The same questions come up on the client builds and on our own products, JobMason and JobSieve, under tighter latency budgets than a blog post deserves.

What to build alongside routing

Routing gets you a slug and a failover chain. These three things get you the rest.

A capability contract, not a model list

Write down what a task actually needs, then check it, rather than pinning a model name and hoping. Capabilities are queryable at runtime: the Models API returns max_input_tokens, max_tokens, and a capabilities tree with an explicit supported flag per feature. A fallback candidate that fails the contract should be excluded by code, not discovered by an incident.

Keep the contract behind logical aliases resolved from configuration rather than literal slugs scattered through the codebase. That's what makes resilience testable: you can point SUMMARY_MODEL at the fallback in staging and watch what breaks on a Tuesday afternoon instead of during an outage. Check the rate limits you're routing into, too. Limits are bucketed per model, and a newer model can sit in a bucket of its own that inherits nothing from the one you're leaving, so a fallback chain can route a surge straight into zero headroom at the exact moment the primary is already struggling. Shifting traffic across doesn't free capacity on the old bucket either.

An eval gate in front of every swap

If quality regression is silent, the only defence is a test that isn't. The recommendation from the routing analysis above is a pre-merge eval gate of 50 to 500 cases, and it's the thing that makes cheap-model savings safe to take. Pair it with stable baselines and side-by-side runs under identical conditions, or you'll be comparing a model change against a prompt change against yesterday's traffic mix.

Our judge panel is one shape this can take, and its imperfections are instructive. Model-graded evals disagree with each other, which is exactly why we require a floor on every axis rather than a good average. A gate that can be passed by being excellent in five dimensions and broken in one isn't a gate.

Meter cost from the response

Read the usage the provider hands back rather than multiplying tokens by a number you typed in a config file last quarter. A local price table goes stale on the day the vendor changes pricing, and it was already wrong across models the moment tokenizers diverged. Tag each request by feature and environment so attribution survives the swap, then compare cost per successful outcome rather than cost per million tokens. Those two metrics disagree more often than you'd like.

Frequently asked questions

What is an LLM gateway? A single API endpoint between your application and multiple model providers. It translates one request shape into each vendor's own and adds routing, failover, and usage accounting, so switching models is a configuration change instead of an integration.

Does an LLM gateway let me switch models without code changes? For the request shape, yes. For anything calibrated to a specific model, no. A request carrying budget_tokens, temperature, or top_p that works on one Claude model returns a 400 on a newer one from the same vendor, and feature availability varies by model tier and hosting platform. Validate the target before you route to it.

Why did my costs go up after switching to a cheaper model? Tokenizers differ, so identical text produces more tokens on some models, around 30% more on Sonnet 5 than Sonnet 4.6 at unchanged per-token pricing. Caches are also model-scoped, so the swap re-pays a cold write at 1.25x to 2x base input price before any discount resumes.

Do I still need evals if I'm using a gateway? Yes, and more than before, because routing makes silent quality changes easy to ship. A pre-merge eval gate of 50 to 500 cases is what makes cheaper routing safe to take. Nothing in a gateway tells you the answers got worse.

Should I use a gateway or my provider's built-in fallback? They cover different failures. Provider-native fallback handles a decline within one vendor's lineup and can re-price the existing cache on the retry, which a third-party gateway cannot do because it doesn't own that cache. A gateway is what you want when the vendor itself is degraded, and when you need more than one company in the chain.

Does routing through a gateway add cost or latency? Not necessarily in cost: at least one major gateway passes through provider list pricing with zero markup and supports BYOK at no extra fee. Latency depends on your fan-out. If you call several models in parallel, the call is as slow as its slowest member, and if you haven't set an explicit timeout, one hung provider can block the whole thing. Ask us how we know.

The next one is already coming

The question worth answering isn't which model you're running today. It's how long a swap takes, and what it costs you in cache, tokens, and confidence when you make it. LLM provider routing is the cheap half of that answer, and it's worth building early. The capability contract, the eval gate, and honest per-request cost metering are the half that decides whether the swap is actually free.

It's a question the AI features we build at Vantaso run into most weeks, and we're glad to compare notes with anyone working through the same thing.