Most .NET applications that call an LLM call exactly one, chosen once, wired into DI and never revisited. That works right up until one of three things happens: the bill arrives, the provider has a bad afternoon, or someone asks why a one-line classification is going to the same expensive model as a full report.
All three have the same fix — a routing layer between your application and the models — and Microsoft.Extensions.AI makes it small enough that there is no real excuse not to have one.
What a bad afternoon actually looks like
Be specific about the failure you are insuring against, because “the provider went down” is almost never the shape of it. Providers rarely go dark. What happens instead is that a deployment starts returning 429 on a fraction of requests, or latency on the flagship model quietly triples while the small one is fine, or a single region degrades while the status page stays green for another forty minutes.
Your application’s experience of that is worse than the underlying event, because a retry policy tuned for a transient blip keeps a user’s request open for the full backoff schedule before it fails. One provider having a mediocre hour turns into your API timing out, your thread pool filling with waiting requests, and a support queue that says the product is broken. The model was available the entire time — just not that deployment, and your process had no way to say so.
Routing does not make the provider healthier. It gives you somewhere to put the decision to stop waiting, which is the part you currently do not have.
The abstraction that makes this cheap
IChatClient is the piece that matters. Every provider package — Azure OpenAI, OpenAI, Ollama, Anthropic via community adapters — exposes the same interface, which means a router is just another IChatClient that happens to delegate. Your calling code never learns that routing exists.
If Microsoft.Extensions.AI is new to you, the explainer covers the abstraction itself; this piece assumes it.
Step 1: name your tiers, not your models
The first mistake is routing to model names. Model names change every few months, and a router full of gpt-4o-mini string literals becomes a migration project. Route to tiers that describe the job, and bind tiers to models in configuration.
public enum ModelTier
{
Small, // classification, extraction, routing, short rewrites
Standard,// the default for user-facing answers
Large // long reasoning chains, code generation, hard synthesis
}{
"Models": {
"Small": { "Deployment": "gpt-4o-mini", "MaxInputTokens": 8000 },
"Standard": { "Deployment": "gpt-4o", "MaxInputTokens": 60000 },
"Large": { "Deployment": "o4", "MaxInputTokens": 180000 }
}
}Swapping a model is now a config change and a redeploy, not a code review.
Step 2: route on signals you already have
Resist the urge to classify with a model call. You almost always know enough already: which endpoint was hit, how long the input is, whether tools are in play, what the customer is paying you.
public sealed class TierSelector
{
public ModelTier Select(ChatRequest request) => request switch
{
// Structured extraction and classification never need a large model.
{ Kind: RequestKind.Classify or RequestKind.Extract } => ModelTier.Small,
// Anything with a tool loop benefits from stronger instruction-following.
{ RequiresTools: true } => ModelTier.Standard,
// Long documents need the context window before they need the reasoning.
{ EstimatedInputTokens: > 50_000 } => ModelTier.Large,
// Explicit user intent wins over every heuristic above.
{ UserRequestedDeepAnalysis: true } => ModelTier.Large,
_ => ModelTier.Standard,
};
}Two rules keep this honest. Escalation must be explicit and logged — if the router quietly upgrades a request to the expensive tier, your bill will surprise you and nobody will know why. And there must be an override, because the first time the router gets a request badly wrong you will want to pin it without shipping code.
Step 3: the router is just another client
public sealed class RoutingChatClient(
IReadOnlyDictionary<ModelTier, IChatClient> clients,
TierSelector selector,
ILogger<RoutingChatClient> log) : IChatClient
{
public async Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken ct = default)
{
var tier = selector.Select(ChatRequest.From(messages, options));
log.LogInformation("Routing to {Tier}", tier);
// Ordered fallback: the chosen tier first, then anything else that can
// plausibly serve the request. Never fall back to a smaller context window.
foreach (var candidate in FallbackChain(tier))
{
try
{
return await clients[candidate].GetResponseAsync(messages, options, ct);
}
catch (Exception ex) when (IsProviderFailure(ex))
{
log.LogWarning(ex, "{Tier} unavailable, trying next", candidate);
}
}
throw new InvalidOperationException("Every model tier failed.");
}
private static IEnumerable<ModelTier> FallbackChain(ModelTier start) => start switch
{
ModelTier.Large => [ModelTier.Large, ModelTier.Standard],
ModelTier.Standard => [ModelTier.Standard, ModelTier.Large],
_ => [ModelTier.Small, ModelTier.Standard],
};
// Cancellation is not a provider failure — do not burn a second model on it.
private static bool IsProviderFailure(Exception ex) =>
ex is not OperationCanceledException &&
ex is HttpRequestException or TimeoutException or TaskCanceledException;
}The IsProviderFailure filter earns its place. Without it, a user closing the browser tab triggers a retry against a second provider, and you pay for a completion nobody will read.
Notice too that Standard falls back up to Large rather than down to Small. Falling back to a smaller context window means the request cannot succeed — you would be spending money to produce a truncation error.
Getting the exception filter wrong is the usual bug
That filter as written is a starting point, and it has a hole worth knowing about: provider SDKs do not throw HttpRequestException. The current OpenAI and Azure OpenAI clients are built on System.ClientModel and surface failures as ClientResultException; older Azure SDK clients built on Azure.Core throw RequestFailedException instead. If your filter only names the System.Net.Http types, every real provider failure sails past it as an unhandled exception and the fallback chain never runs. The symptom is a router that looks perfect in unit tests — where you throw HttpRequestException from a fake — and does nothing whatsoever in production.
Catch the provider’s own exception type and switch on its status code, because the status code is what tells you whether falling back is even the right idea:
429— the clearest fallback signal there is, provided the quota is per-deployment. Retrying the same deployment does not create capacity; a different one might have some.500,502,503and timeouts — retry once at the resilience layer, then fall back.400— never fall back. A malformed request or acontext_length_exceededwill be equally malformed at the next model, so you pay twice and fail anyway. The only sensible response tocontext_length_exceededis a tier with a larger window, and that is a routing decision made before the call, not a fallback after it.- Content filter rejections — not an availability problem. Failing over to a second provider so it can also refuse is a good way to double your spend on requests you did not want served. Surface these to the caller.
The general rule: fall back on failures that are about the deployment, and fail fast on failures that are about the request.
One provider’s tiers are not independent
There is an uncomfortable truth underneath the fallback chain above. If Standard and Large are two deployments in the same provider account, they share a control plane, an auth path and frequently a regional dependency. The failure that takes out one has a decent chance of taking out the other, which means tier-to-tier fallback protects you against a single model being busy and not much else.
Real availability comes from crossing a boundary the outage does not: a second region, a second subscription, or a second vendor entirely. IChatClient makes that arrangement cheap to express — the router does not care whether two entries in the dictionary point at the same account — so the question is only whether you have provisioned the second path, not whether your code can use it. Provision it before you need it; quota approval is not a thing you want to be requesting during an incident.
Fallback has a latency cost, so add a breaker
Naive fallback makes your p99 worse, not better. A request that fails over waits out the full timeout on the primary and then pays the full latency of the secondary. During a genuine outage that is every request, so a thirty-second timeout plus a normal completion becomes the experience of every user until someone intervenes.
A circuit breaker on each client is what stops that. After a handful of consecutive failures the breaker opens, subsequent calls to that tier fail instantly rather than after the timeout, and the router moves straight to the next candidate — turning a thirty-second penalty into a microsecond one. It also closes itself when the provider recovers, which matters more than it sounds, because the alternative is a manual config change that someone has to remember to revert. The Polly guide covers the breaker configuration; the point here is that the breaker belongs on the client and the routing decision belongs above it.
Streaming breaks the illusion
Everything above assumes you can retry invisibly. Once you are streaming tokens to a browser, you cannot. The moment the first chunk is flushed, a failure at token 200 is visible — you can either abandon the response mid-sentence or start a second one that contradicts the first half already on screen.
There are two honest options. Buffer until the first chunk arrives and only then begin flushing, which preserves silent failover for the common case (connection failures and rate limits happen at the start, not in the middle) at the cost of a little time-to-first-token. Or accept that mid-stream failures are user-visible and design the UI for it, with a retry affordance that resends the whole turn. What does not work is pretending the problem is not there, which is the default behaviour of most streaming implementations. Streaming agent responses to a web UI has the transport side.
A fallback can silently change your output shape
The last thing routing costs you is uniformity. Prompts are tuned, deliberately or otherwise, against the model you developed on. Fail over to a different one and the request still succeeds — with different results in ways that pass every availability check you have. Structured output support varies between models and providers, tool-calling strictness varies, and a system prompt that reliably produces bare JSON on one model produces JSON inside a markdown fence on another.
The consequence is a failover that “works” and then breaks the parser two layers downstream, where nothing mentions models at all. Two habits contain it. Validate the response shape after every call rather than trusting the tier that produced it, so a format problem surfaces as a format error rather than a null reference in a mapper. And keep tier-specific prompt overrides available even if you start with one prompt for all tiers — you will eventually need a slightly different instruction for the small model, and discovering that is much easier than retrofitting the ability to have one. A scored eval set run against each tier turns this from a production surprise into a known quantity.
Step 4: keep retries out of the router
Transient failures belong in the resilience pipeline attached to each individual client, not in the routing loop. If both layers retry, one blip produces a multiplicative burst of traffic against a provider that is already unhealthy.
Attach Polly per client and let the router see only the final verdict — the Polly guide has the pipeline configuration.
services.AddKeyedChatClient(ModelTier.Small, sp => /* small deployment */)
.UseLogging()
.UseOpenTelemetry(); // per-client resilience configured here
services.AddSingleton<IChatClient, RoutingChatClient>();Step 5: measure the thing you built it for
A router you cannot measure is a router you will eventually mistrust and rip out. Three counters are enough:
- Requests per tier. If ninety-five per cent of traffic lands on
Standard, your heuristics are not doing anything and the complexity is unpaid-for. - Fallback rate per tier. A rising rate is provider degradation, usually before the status page admits it.
- Cost per tier. The number that justifies the whole exercise. Cost control in production covers per-request attribution.
Tag every span with the tier and whether a fallback fired. When someone reports a strange answer, the first question is always which model produced it, and without the tag you cannot answer. OpenTelemetry for agents covers getting the spans in the first place.
When not to build this
Routing is a seam, and seams are not free. Three situations do not want one yet.
A single low-volume workload. If the whole application is one internal endpoint serving a few thousand requests a month, the cost saving from tiering is smaller than the time you will spend maintaining the selector, and an outage is survivable. Wire the seam — one IChatClient behind DI — and skip the router.
No measurement. A router you cannot attribute cost to is a router built on a guess. If you cannot currently answer “what do we spend per request, per endpoint”, build that first; it frequently reveals that the expensive traffic is not where anyone assumed, and the routing rules you would have written would have optimised the wrong path. Cost control in production is the prerequisite.
Quality is non-negotiable per request. For workloads where a worse answer is more expensive than an error — anything feeding a downstream automated decision, anything regulated — silent degradation is the wrong failure mode. Route if you like, but do not fall back. Fail loudly and let the caller decide.
The related question of which model to put in each tier is its own exercise; choosing a model for a .NET agent works through it. Do not let that decision block building the seam, which is the part that lets you change your mind cheaply afterwards.
Note: provider SDKs and
Microsoft.Extensions.AIare both still evolving; the builder extension names in particular have shifted between previews. Verify against the Microsoft.Extensions.AI documentation before wiring this up. The shape — tiers in config, deterministic selection, ordered fallback, per-client resilience — has outlived several API renames.
Takeaway
Routing is not a scaling feature you add later. It is the seam that lets you change your mind about models without changing your application, and the first time a provider has an outage during business hours it pays for itself.
Start with two tiers and one deterministic rule. That is enough to prove the seam works, and it is a far better position than discovering you need it during an incident.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.