AAgentic.NET
.NETAI AgentsResilienceC#

Resilient LLM Calls in .NET with Polly

LLM APIs rate-limit, time out, and hiccup. Make your .NET AI calls resilient with Polly — retries with backoff, circuit breakers, and timeouts done right.

MB
Manikandhan Balakrishnan
·3 min read

LLM APIs fail in ways you must plan for: 429 rate limits under load, occasional timeouts, and transient 5xx errors. In production, an unhandled one of these becomes a failed user request. The .NET answer is Polly — the resilience library — and this article shows how to use it for AI calls specifically, where the failure modes have their own quirks.

The failures you’ll actually hit

  • 429 Too Many Requests — you’ve exceeded the provider’s tokens-per-minute or requests-per-minute quota. The most common one at scale.
  • Timeouts — a slow generation or network blip.
  • 5xx — transient provider-side errors.

Each wants a slightly different response, and getting the retry behaviour wrong can make things worse (retry storms).

Retry with exponential backoff and jitter

For 429s and transient errors, retry — but back off exponentially and add jitter, so a fleet of clients doesn’t retry in sync and hammer the API:

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromSeconds(2),
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .HandleResult(r => IsTransientOrRateLimited(r))
    })
    .Build();

var response = await pipeline.ExecuteAsync(async ct =>
    await chatClient.GetResponseAsync(prompt, cancellationToken: ct));

Respect Retry-After. Rate-limit responses often tell you how long to wait — honour that header instead of your own guess when it’s present. Blindly retrying a 429 immediately just earns another 429.

Add a circuit breaker

If the provider is having a genuine outage, retrying every request is pointless and wasteful. A circuit breaker trips after repeated failures and fails fast for a cooldown, giving the provider time to recover:

.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,
    MinimumThroughput = 10,
    BreakDuration = TimeSpan.FromSeconds(30)
})

Order matters: retry outer, circuit breaker inner — so the breaker’s fail-fast isn’t itself retried.

Timeouts

Set a timeout so a hung call can’t block forever — but size it for LLMs, which are legitimately slow. A 3-second timeout will fail healthy generations; think tens of seconds, longer for big responses. For streaming, the timeout applies to establishing the stream, not the whole generation.

The bigger picture: don’t just retry — prevent

Resilience handles failures, but the best fix for chronic 429s is to not hit the limit: cap concurrency, cap max-replicas to your quota when deploying, and control spend/volume (cost control). Retries smooth over blips; they can’t fix a system that’s structurally over its quota.

Note: Polly’s API (v8 resilience pipelines) and provider error shapes evolve; verify against the Polly docs. The strategy — backoff+jitter retry honouring Retry-After, circuit breaker inside retry, LLM-sized timeouts — is stable.

Takeaway

LLM APIs rate-limit and hiccup, so wrap your calls in Polly: retry transient errors and 429s with exponential backoff and jitter (honouring Retry-After), add a circuit breaker inside the retry to fail fast during real outages, and set timeouts sized for slow generations. But treat retries as a smoother, not a cure — the durable fix for constant 429s is staying within your quota by design.

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.

Related articles