Rate Limiting LLM Calls in .NET: Tokens, Not Requests

Here is the shape of the problem. You add a rate limiter to your agent, configure it for the 300 requests per minute your maths says is safe, deploy it, and still get 429s. The limiter is working correctly. It is just counting the wrong thing.

Model providers meter you in tokens per minute, not requests per minute. A limiter that permits 300 requests will permit 300 one-line questions or 300 forty-page document summarisations, and those differ by two orders of magnitude in the only unit the provider cares about.

Reserve tokens, then reconcile

System.Threading.RateLimiting supports this directly, because AcquireAsync takes a permit count. Use tokens as the permit unit:

// A TPM budget expressed as a token bucket. tokensPerMinute is your deployment
// quota with headroom taken off the top, not the number on the portal page.
private readonly TokenBucketRateLimiter _tpm = new(new TokenBucketRateLimiterOptions
{
    TokenLimit          = tokensPerMinute,          // burst ceiling
    TokensPerPeriod     = tokensPerMinute,          // refill rate
    ReplenishmentPeriod = TimeSpan.FromMinutes(1),
    QueueLimit          = 64,
    QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
    AutoReplenishment   = true,
});

The estimate before the call is the part people get wrong. You know your input exactly — you can count the tokens — and you do not know your output at all. So reserve against the only ceiling you control:

public async Task<ChatResponse> SendAsync(
    IEnumerable<ChatMessage> messages, ChatOptions options, CancellationToken ct)
{
    var maxOutput = options.MaxOutputTokens ?? 1024;
    var estimate  = CountInputTokens(messages) + maxOutput;

    using var lease = await _tpm.AcquireAsync(estimate, ct);
    if (!lease.IsAcquired)
        throw new RateLimitedException("token budget exhausted");

    var response = await _inner.GetResponseAsync(messages, options, ct);

    // Reconcile: hand back what we reserved and did not use.
    var actual = (int)(response.Usage?.TotalTokenCount ?? estimate);
    if (actual < estimate) Refund(estimate - actual);

    return response;
}

Two things follow from this that are worth stating plainly. Always set MaxOutputTokens. Without it there is no upper bound to reserve against and your estimate is a guess. And the reconciliation matters more than it looks — reserving 1024 output tokens for a call that produces 80 wastes 92% of that reservation, and at scale that is the difference between using your quota and leaving most of it idle.

Concurrency is a second, separate limit

Token throughput and simultaneous connections are different constraints, and a design that only handles the first will still fall over. Twenty concurrent calls that each stream for forty seconds can sit comfortably inside a TPM budget while exhausting connections, provider-side concurrency and your own thread pool.

Chain them:

var limiter = new ChainedRateLimiter(
    new ConcurrencyLimiter(new ConcurrencyLimiterOptions
    {
        PermitLimit = 8,
        QueueLimit  = 32,
        QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
    }),
    _tpm);

Concurrency first. If you take the token reservation and then block waiting for a concurrency slot, you are holding budget you are not spending, and every other caller sees a quota that looks full while the deployment sits idle.

Where it goes in the pipeline

Outermost. Before retries, before function invocation, before anything.

builder.Services.AddChatClient(sp => baseClient)
    .Use(inner => new RateLimitedChatClient(inner, limiter))   // outermost
    .UseFunctionInvocation()
    .UseLogging();

The reason is that an agent turn is not one call. A tool-using turn is a call, a tool execution, another call, possibly several more. Each of those consumes quota. A limiter placed inside function invocation sees one of them and lets the rest through unmetered, which is a limiter that reports success while doing approximately nothing.

If you have not read your own traces on this, do — the number of model calls behind a single user request is routinely two or three times what people assume.

Fairness, and the noisy tenant

QueueProcessingOrder.OldestFirst gives you FIFO, which is fair in the sense that nobody starves. It is not fair in the sense that matters commercially, because one tenant submitting a hundred bulk jobs will fill the queue and every interactive user waits behind them.

If you are multi-tenant, partition:

var perTenant = PartitionedRateLimiter.Create<AgentRequest, string>(req =>
    RateLimitPartition.GetTokenBucketLimiter(req.TenantId, _ =>
        new TokenBucketRateLimiterOptions
        {
            TokenLimit          = 20_000,
            TokensPerPeriod     = 20_000,
            ReplenishmentPeriod = TimeSpan.FromMinutes(1),
            QueueLimit          = 8,
            AutoReplenishment   = true,
        }));

Now a tenant burning their allocation slows only themselves. The global limiter still sits behind this as the protection for the deployment quota; the partitioned one is about not letting one customer become everyone else’s latency.

The distributed problem, stated honestly

System.Threading.RateLimiting is in-process. Your quota is not.

Three replicas each holding a limiter sized for the full quota will collectively use three times it, and you will get 429s from a system that believes it is behaving. There are two honest answers:

Divide the budget. Give each replica quota / replicaCount. It is trivial, it works, and it wastes capacity whenever load is uneven — which is most of the time.

Share the counter. Redis with a sliding window, or your gateway doing the metering. Correct, and it adds a network hop and a dependency to every model call.

For most teams the first is the right starting point and the second is what you move to when the waste becomes measurable. What you should not do is run per-process limiters sized for the whole quota and wonder why the 429s persist.

Backpressure: fail fast or queue

When the budget is gone you have two options and they are a product decision, not a technical one.

Queue, and the user waits — fine for a background summarisation job, awful for a chat box where a spinner that never resolves reads as broken. Reject with a clear error and a Retry-After, and the user knows where they stand, which is usually kinder.

Whichever you pick, set QueueLimit to something finite. An unbounded queue under sustained overload does not degrade, it accumulates, and then it falls over all at once with a memory profile nobody enjoys debugging.

What to instrument

Three numbers tell you whether any of this is working:

• Reservation accuracy — estimated versus actual tokens. Persistently over-estimating means you are throttling yourself for no reason.
• Queue wait time at the 95th percentile. This is the latency your limiter is adding, and it belongs in your budget explicitly.
• 429s that got through anyway. Should trend to zero. If it does not, either the limiter is per-process against a shared quota, or something is bypassing the chain.

That third one is the useful alarm. A rate limiter that is silently ineffective looks exactly like a rate limiter that is working, right up until the provider disagrees.

Takeaway

Count tokens, not requests, because tokens are the unit your bill and your quota are denominated in. Reserve against MaxOutputTokens and reconcile against real usage, or you will leave most of your quota unused. Chain a concurrency limiter in front of the token bucket, and put the whole thing outermost in the client pipeline so that multi-call agent turns are actually metered.

Then be honest about the distributed case. An in-process limiter across three replicas is a third of a solution, and the fix is either arithmetic or Redis — not a better local limiter.

Once the shaping is right, Polly handles the residue, and model routing and fallback gives you somewhere to send the overflow instead of just making people wait.

Note: Quota units, burst behaviour and the exact retry headers differ between OpenAI, Azure OpenAI and other providers, and Azure quotas are per deployment rather than per subscription. Verify the current limits and headers for your specific deployment against the provider’s documentation before sizing any of this.

---

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.