Resilient LLM Calls in .NET with Polly

LLM APIs fail in ways you must plan for: 429 rate limits under load, occasional timeouts, and transient 5xx errors. In production, an unhandled one of these becomes a failed user request. The .NET answer is Polly — the resilience library — and this article shows how to use it for AI calls specifically, where the failure modes have their own quirks.

Why a model call is not a normal HTTP call

The standard retry advice was written for cheap, fast, idempotent requests. A model call is none of those, and every one of those differences changes what a correct policy looks like.

It is not cheap. You are billed for the input tokens on every attempt, including the ones that fail after the model has already read your prompt. A three-retry policy on a request with a 30,000-token context can cost four times what a successful call costs, and it does so precisely when the system is under stress and everyone is making the same call. Retry counts are a spend decision, not just a reliability one.

It is not fast. A generation that takes six seconds is normal. Layer exponential backoff on top — two, then four, then eight seconds of waiting — and a three-attempt policy has a worst case somewhere north of thirty-five seconds before it gives up. Your user left at about eight. A retry policy that outlives the user’s patience is not resilience, it is a slow way to burn tokens on a response nobody will read.

And it is only conditionally idempotent. Asking a model a question twice is harmless. Re-running an agent turn that already called send_invoice before failing on the follow-up model call is not. Where you put the retry matters more than how you configure it.

The failures you’ll actually hit

• 429 Too Many Requests — you’ve exceeded the provider’s tokens-per-minute or requests-per-minute quota. The most common one at scale.
• Timeouts — a slow generation or network blip.
• 5xx — transient provider-side errors.
• 400 with a context-length error — your prompt is too long. Retrying is pure waste.
• Content filter rejections — deterministic. The same input will be rejected again.

Each wants a slightly different response, and getting the retry behaviour wrong can make things worse (retry storms).

The distinction that costs people real money is inside the 429. Some 429s mean “you are going too fast right now” and clear in seconds. Others mean “you have exhausted your quota for the period” or “your billing is not in order” and will not clear no matter how patiently you back off. Treating both as transient means your service spends the rest of the hour hammering an endpoint that is never going to say yes. If your provider distinguishes them in the error body, use that; if not, a circuit breaker is what saves you, which is the section after next.

Retry with exponential backoff and jitter

For 429s and transient errors, retry — but back off exponentially and add jitter, so a fleet of clients doesn’t retry in sync and hammer the API:

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromSeconds(2),
        MaxDelay = TimeSpan.FromSeconds(20),
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .HandleResult(r => IsTransientOrRateLimited(r))
    })
    .Build();

var response = await pipeline.ExecuteAsync(async ct =>
    await chatClient.GetResponseAsync(prompt, cancellationToken: ct));

MaxDelay is not decoration. Without it, exponential growth plus jitter can produce a wait longer than the request that is waiting for it, and you have built a very expensive Task.Delay.

Jitter deserves more credit than it usually gets. Without it, every instance that got rate-limited at the same moment retries at the same moment, which produces another synchronised 429, and the pattern self-reinforces until something breaks the cycle. Polly’s UseJitter spreads each computed delay by roughly a quarter in either direction — enough to decorrelate a fleet. Turning it on costs nothing and it is the difference between a bad thirty seconds and a bad ten minutes.

Respect Retry-After. Rate-limit responses often tell you how long to wait — honour that header instead of your own guess when it’s present. Blindly retrying a 429 immediately just earns another 429. Polly’s DelayGenerator exists for exactly this: return a delay to override the computed one, or null to fall back to the strategy’s own backoff.

DelayGenerator = args =>
{
    if (args.Outcome.Exception is ClientResultException ex &&
        ex.GetRawResponse() is { } raw &&
        raw.Headers.TryGetValue("retry-after", out var value) &&
        int.TryParse(value, out var seconds))
    {
        return new ValueTask<TimeSpan?>(TimeSpan.FromSeconds(seconds));
    }

    return new ValueTask<TimeSpan?>((TimeSpan?)null);   // use the built-in backoff
}

Two practical notes. The header is sometimes seconds and sometimes an HTTP date, so parse defensively rather than assuming. And clamp whatever you read — a provider telling you to wait sixty seconds is information, not an instruction you must obey inside a user-facing request. Log it, cap the wait at something your latency budget can survive, and fail the request rather than hold a connection open for a minute.

Add a circuit breaker

If the provider is having a genuine outage, retrying every request is pointless and wasteful. A circuit breaker trips after repeated failures and fails fast for a cooldown, giving the provider time to recover:

.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,
    MinimumThroughput = 10,
    BreakDuration = TimeSpan.FromSeconds(30)
})

Order matters: retry outer, circuit breaker inner — so the breaker’s fail-fast isn’t itself retried. In Polly v8 the strategy you add first is the outermost, so the order above reads correctly as written.

Two things about breakers around model APIs that generic guidance misses.

First, scope the breaker to the thing that can actually fail independently. One breaker per model deployment or per provider, not one global breaker for “AI”. If you route between a primary and a fallback model and they share a breaker, the primary’s outage takes down the fallback that exists specifically to survive it. Register named pipelines and resolve them per destination.

Second, decide what an open circuit looks like to the user before you turn it on. Polly throws BrokenCircuitException and it arrives instantly, which is the point — but an instant unhandled exception is a worse experience than a slow success. Catch it and do something deliberate: serve a cached answer, degrade to a cheaper model, queue the work and return a job id, or render an honest “this feature is temporarily unavailable”. The value of failing fast is entirely in what you do with the time you saved.

Timeouts, in three layers

Set a timeout so a hung call can’t block forever — but size it for LLMs, which are legitimately slow. A three-second timeout will fail healthy generations; think tens of seconds, longer for big responses.

There are really three timeouts and they do different jobs. The per-attempt timeout sits inside the retry so one stuck attempt is abandoned and retried rather than blocking the whole budget. The total timeout sits outside everything and bounds the entire operation including all backoff waits — this is the one that protects your latency SLA, and the one people forget. And there is the underlying HttpClient.Timeout, which must be longer than both or it will fire first and turn your carefully classified failures into an undifferentiated TaskCanceledException.

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(45))     // total budget, outermost
    .AddRetry(retryOptions)
    .AddCircuitBreaker(breakerOptions)
    .AddTimeout(TimeSpan.FromSeconds(20))     // per attempt, innermost
    .Build();

That ordering — total timeout, retry, breaker, attempt timeout — is the same shape the standard resilience handler in Microsoft.Extensions.Http.Resilience uses, and it is worth copying rather than inventing.

Streaming is the exception that breaks the model. The timeout you want applies to establishing the stream and to the gap between tokens, not to the total duration, because a legitimately long generation is supposed to take two minutes. Worse, retry semantics stop making sense the moment the first token reaches the browser: retrying restarts generation, and the user watches half a sentence get replaced by a different half sentence. Wrap the call that opens the stream; once tokens are flowing, a failure is a visible error, not something to paper over.

Where to put the retry in an agent loop

This is the mistake that turns a resilience improvement into a data incident.

An agent turn is: call the model, get a tool call, execute the tool, call the model again. If you wrap that whole loop in a retry and it fails on the second model call, the retry replays the loop from the start — and executes the tool again. If the tool sent an email, created an order or posted to a webhook, you have just done it twice, and the trigger was a transient 429.

Put the retry around the individual model call, inside the loop. Keep the loop itself un-retried, or retry it only if every tool it can reach is genuinely idempotent — which is a property you enforce with idempotency keys, not one you assume. The same reasoning applies to anything that wraps an agent turn, including model routing and fallback: the router decides which model to try, the pipeline decides how hard to try it, and neither should retry the other’s work.

Prevention beats retrying

Resilience handles failures, but the best fix for chronic 429s is to not hit the limit. A concurrency limiter in front of the client is more effective than any retry policy, because it converts “many requests fail expensively” into “requests queue briefly and succeed”:

.AddConcurrencyLimiter(permitLimit: 8, queueLimit: 32)

Pick the permit count from your quota and typical request size rather than from a round number, and cap max-replicas to match when deploying — twenty pods each politely limiting themselves to eight concurrent calls is still one hundred and sixty concurrent calls. Add response caching for repeated prompts and keep an eye on spend. Retries smooth over blips; they can’t fix a system that’s structurally over its quota.

Whatever you build, log it. OnRetry gives you the attempt number, the outcome and the delay — emit that as a metric and you will find out that a “reliable” integration has been quietly retrying eight per cent of calls for a month, which is a bill and a latency problem you cannot see any other way. Pair it with OpenTelemetry so retries show up as spans rather than log lines nobody reads.

When this is the wrong tool

In-process retry is for interactive requests where a human is waiting. If the work is a batch — embedding a corpus, classifying a backlog, running an eval suite — a queue with visibility timeouts and a dead-letter destination is strictly better. It survives your process restarting, it gives you a place to inspect what failed, and it lets you drain at the rate the provider will actually accept instead of discovering that rate through repeated rejection. Reaching for Polly there is building a worse queue inside your web server.

Retries absorb failures but do not reduce load. If you are seeing sustained 429s rather than occasional ones, rate limiting LLM calls by tokens is the other half of this — shape the traffic first, then retry the residue.

Note: Polly’s API (v8 resilience pipelines) and provider error shapes evolve; verify against the Polly docs. The strategy — backoff+jitter retry honouring Retry-After, circuit breaker inside retry, LLM-sized timeouts — is stable.

Takeaway

LLM APIs rate-limit and hiccup, so wrap your calls in Polly: retry transient errors and 429s with exponential backoff and jitter, honour Retry-After through a DelayGenerator, and cap the delay so backoff cannot outlive your latency budget. Put a circuit breaker inside the retry, scoped per model deployment, and decide in advance what an open circuit shows the user. Use three timeouts — total, per-attempt and a longer HttpClient.Timeout — and remember that streaming only tolerates retries before the first token. Keep the retry around the model call rather than the agent loop, or a transient error will re-execute a tool with side effects. And treat retries as a smoother, not a cure: a concurrency limiter, caching and honest capacity planning are what stop the 429s in the first place.