“The agent is slow” is one of the least useful bug reports in software, and one of the most common. It is also usually true.
The instinct is to reach for a smaller model. Sometimes that is right. More often, the model was never the largest term in the sum, and you can find that out in an afternoon.
First, get a real breakdown
You cannot fix a number you have not decomposed. Before changing anything, instrument the phases so you have a per-request distribution rather than an anecdote.
using var activity = ActivitySource.StartActivity("agent.turn");
var sw = Stopwatch.StartNew();
var context = await retriever.SearchAsync(query, ct);
activity?.SetTag("phase.retrieval_ms", sw.ElapsedMilliseconds);
sw.Restart();
var completion = await chatClient.CompleteAsync(messages, options, ct);
activity?.SetTag("phase.inference_ms", sw.ElapsedMilliseconds);
activity?.SetTag("tokens.input", completion.Usage?.InputTokenCount);
activity?.SetTag("tokens.output", completion.Usage?.OutputTokenCount);
sw.Restart();
var toolResults = await ExecuteToolsAsync(completion, ct);
activity?.SetTag("phase.tools_ms", sw.ElapsedMilliseconds);
activity?.SetTag("tools.count", toolResults.Count);Two tags matter more than the timings themselves: tools.count and tokens.input. Nearly every genuinely slow agent is slow because one of those two is larger than anyone realised. The OpenTelemetry guide covers exporting this properly.
Look at p95, not the mean. Agent latency distributions have long tails, and the mean hides exactly the requests that make users complain.
Three ways this measurement goes wrong
The first is averaging across request shapes. A support agent handling “what are your opening hours” and “reconcile these three invoices against the ledger” in the same histogram produces a mean that describes neither. Split the metric by tool-call count and the distribution usually resolves into two clean populations, one of which is fine and one of which is your actual problem.
The second is measuring only server-side. Your span starts when the request reaches your handler and ends when you write the last byte, which can look excellent while the user waits three seconds for TLS setup, a cold CDN edge, and a client that re-renders the whole transcript on every token. If the gap between your p95 and the browser’s is large, the fix is not in the agent at all.
The third is measuring in development. Local runs have a short conversation history, a warm process, a nearby endpoint and one user. Every one of those flips in production. A prefill cost you never saw locally shows up as the dominant term the first time someone has a forty-turn thread open.
The five places the seconds actually go
Round trips through the model. A turn that calls four tools sequentially is four full inference passes, each re-reading the entire conversation. This is almost always the biggest term, and it is structural — no amount of model tuning touches it.
Prefill. Time to first token scales with how much input you send. A 40,000-token context has a meaningfully slower first token than a 4,000-token one, before a single word is generated.
Output length. Generation is sequential, token by token. A 1,200-token answer takes roughly four times as long to finish as a 300-token one. Verbose system prompts produce verbose answers, and you pay for the verbosity in seconds.
Retrieval. Usually 50–300ms for the search, which is fine. The real cost is the tokens the results add to the prompt, which lands back in prefill.
Cold starts and connection setup. Serverless hosts, fresh HttpClient handlers, and cross-region calls each add a fixed tax that is invisible in local testing and obvious in production.
The latency that hides from your breakdown
There is a category your phase timers will happily absorb without telling you: retries. If your SDK or your resilience pipeline transparently retries a rate-limited or transient-failure response, the second attempt’s wait is counted inside phase.inference_ms and looks like a slow model. It is not a slow model. It is one fast call, a backoff, and another fast call.
This is worth separating because the fix is completely different. A genuinely slow model wants a smaller model or a shorter prompt; a retry storm wants concurrency control and a look at your quota. Tag the retry count on the span and the two stop being confusable:
activity?.SetTag("llm.retry_count", attempts - 1);
activity?.SetTag("llm.rate_limited", sawTooManyRequests);The trap here is that the obvious latency fix makes it worse. Fanning out ten parallel calls to shave wall-clock time is exactly the pattern that trips a per-minute token limit, and the resulting backoff costs more than the serialisation you removed. If your p99 is several times your p95 and the gap appears under load rather than under long prompts, suspect throttling before you suspect the model.
The same applies to a shared endpoint. Two teams pointing at one deployment share one quota, and the team that ships a batch job on Tuesday morning becomes the reason your interactive agent got slow on Tuesday morning. Separate deployments for interactive and background traffic cost nothing and remove a whole class of unexplained incident.
Fix them in this order
1. Stream, before anything else
Streaming does not reduce total time. It changes what the user experiences from “nine seconds of nothing” to “text moving after four hundred milliseconds”, and that is the difference between a product that feels broken and one that feels fast.
Nothing else on this list moves perceived latency as far for as little work. Streaming agent responses to a web UI has the ASP.NET Core wiring.
The failure mode to watch for is streaming that silently is not streaming. The symptom is unmistakable once you know it: the endpoint works perfectly against curl on your machine, and in the deployed app the entire answer appears at once after eight seconds. Something between your handler and the browser is buffering the response — response compression middleware that waits to fill a block, a reverse proxy with buffering left on, a CDN or gateway that treats the response as a document rather than a stream, or your own code accumulating chunks into a StringBuilder before writing. Each of these turns a stream back into a single delivery and undoes the only change on this list that users actually feel.
Diagnose it by moving the observation point. Watch the response arrive at your own server first, then at the edge, then in the browser’s network panel; whichever hop first shows a single large chunk instead of a trickle is the one buffering. The fixes are unglamorous — disable buffering for that route, exclude the streaming endpoint from compression, flush explicitly after each chunk — but you should confirm the behaviour in the deployed environment rather than assuming it survived the trip.
While the model is working, say what it is doing. A line that reads “Searching orders…” then “Reading the March invoice…” costs nothing and changes a blank wait into visible progress. Tool-heavy turns are the slowest ones you have, and they are also the ones with the most to narrate.
2. Collapse the tool loop
This is the biggest real reduction available. Three sequential tool calls cost three inference passes; one batch call costs one.
// Sequential in the transcript: three turns, three full context re-reads.
[Description("Get one customer's order history.")]
public Task<Orders> GetOrdersAsync(string customerId);
// One turn. The model asks once, you fan out internally.
[Description("Get order history for several customers at once.")]
public async Task<IReadOnlyDictionary<string, Orders>> GetOrdersAsync(
IReadOnlyList<string> customerIds, CancellationToken ct)
{
var tasks = customerIds.Select(async id => (id, await FetchAsync(id, ct)));
return (await Task.WhenAll(tasks)).ToDictionary(x => x.id, x => x.Item2);
}Two things happen at once: you remove model turns, and the work inside the tool now runs concurrently instead of one customer at a time. Also check whether your framework version executes independently-requested tools in parallel — it is not always the default, and enabling it is a one-line change with no downside.
3. Cut the input, not just the model
Prefill scales with input, so every token you do not send is latency you do not pay.
- Retrieve fewer, better chunks. Twenty chunks where five would do is a latency decision wearing a relevance costume. Chunking strategies covers getting more from fewer.
- Compact the history. Long-running threads accumulate turns nobody needs; summarise the old ones. Agent memory and state has the patterns.
- Shorten the system prompt. Prompts grow by accretion — every incident adds a rule and none are ever removed. Re-read yours; a third of it is usually dead.
4. Ask for less output
Say so explicitly, and enforce it with MaxOutputTokens. If the answer feeds a UI rather than a human reader, use structured output — a JSON object with four fields is far fewer tokens than four paragraphs describing the same thing, and it parses. See structured output for .NET agents.
5. Then, and only then, change the model
Now that the structure is right, a smaller or faster model is a clean trade you can evaluate on quality alone. Doing it first means you never find out whether you needed it — you just accept worse answers and keep the round trips.
Route per request rather than switching globally: classification and extraction almost never need your largest model. Model routing and fallback covers the wiring, and choosing a model covers the trade.
6. Remove the fixed taxes
Reuse HttpClient through IHttpClientFactory. Put your app in the same region as your model endpoint — a cross-continent hop is 100–200ms on every call, including every tool round trip. On serverless, keep an instance warm or accept the cold start; the Azure Functions guide covers the trade-off honestly.
A budget worth holding yourself to
For a tool-using agent answering a typical question:
| Phase | Target (p95) |
|---|---|
| Retrieval | under 300ms |
| Time to first token | under 800ms |
| Tool execution, per call | under 500ms |
| Model turns per request | 1, occasionally 2 |
| Complete response | under 6s |
The row that matters most is model turns per request. If that number is four, no amount of tuning on the others will save you — and if it is one, most requests come in comfortably under budget without any further work.
When this is the wrong thing to work on
Not every agent is a chat box. If yours enriches records overnight, classifies a queue of documents, or runs on a schedule, per-request latency is close to irrelevant and optimising it is wasted effort. What matters there is throughput and cost per item, and the two goals pull in opposite directions: batching aggressively and running high concurrency is bad for a single request’s p95 and excellent for a nightly job’s completion time. Decide which one you are building before you tune anything.
There is also a floor you cannot argue with. A turn that genuinely requires reading a document, calling an external API that takes two seconds, and then writing three paragraphs is not going to come in under four seconds, and pretending otherwise leads to cutting the parts of the answer that made it useful. When you hit that floor, the remaining work is expectation-setting rather than engineering — show progress, let the user start reading the first paragraph while the third is still generating, and stop.
The other case for leaving it alone is when latency is not what people are actually complaining about. “Slow” is frequently the word users reach for when they mean “I asked twice and got two different answers” or “it took three exchanges to get what I wanted”. A turn that takes six seconds and answers the question outright beats three two-second turns that do not. Check whether your slow requests and your unsuccessful requests are the same requests before you spend a sprint on milliseconds.
Note: provider latency characteristics change with capacity, region and model version; treat published figures as a starting point and measure your own p95 against your own traffic. The ordering of the fixes has held up across providers because it follows the structure of the problem rather than any one vendor’s performance profile.
Takeaway
Perceived latency is a structural property, not a model property. Stream first so the wait becomes legible, then delete round trips, then cut tokens, and only then consider a different model.
If you measure one thing tomorrow, measure tool calls per request. It is the number most likely to be larger than you think, and the one with the shortest path to a fix.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.