Agents bill by the token, and tokens scale with success. The more users your agent helps, the bigger the invoice — and because a single agent turn can involve several model round-trips (reasoning, tool calls, follow-ups), costs climb faster than request counts suggest. The good news: most agent spend is waste you can remove without hurting quality. This article is a practical playbook for controlling the cost of a production .NET agent.
Cost tracking pairs naturally with observability — you can’t optimize what you don’t measure.
First: measure per-request cost
You can’t control what you can’t see. Before optimizing, make token usage visible — the AI telemetry exposes prompt and completion token counts per call. Emit them as metrics tagged by endpoint and model, and you’ll immediately spot which flows are expensive. Often one or two paths drive most of the bill, and that’s where to focus.
1. Right-size the model (the biggest lever)
The single largest cost mistake is running a frontier model for everything. As covered in choosing a model, route simple turns — classification, extraction, short answers — to a small, cheap model and reserve the expensive one for genuinely hard reasoning. Because it’s all IChatClient, a per-request router is cheap to build and often cuts spend by more than half with no quality loss users would notice. Do this before micro-optimizing anything else.
2. Cache what repeats
Agents answer the same things over and over. Two kinds of caching help:
- Response caching — for deterministic or near-deterministic queries (FAQ-style questions, repeated lookups), cache the answer keyed on a normalized input. A cache hit costs nothing and returns instantly.
- Prompt caching — many providers now cache a stable prompt prefix (your long system prompt, tool definitions, retrieved context) so repeated calls are billed at a steep discount for the cached portion. Structure your prompts so the stable part comes first and the variable part last, and you get this largely for free.
3. Trim the context you send
You pay for every token in the prompt, on every call. Bloated context is silent, recurring waste:
- Cap conversation history. Don’t resend the entire thread forever — keep a window of recent turns and summarize older ones. A 40-turn conversation resending all 40 turns each time is paying to re-read history the model barely needs.
- Retrieve less, better. In RAG, sending the top-10 chunks when the top-3 would do triples that part of the bill. Tune K down and use re-ranking so the few chunks you send are the right ones.
- Prune tool definitions. Every tool’s schema is tokens in every request. An agent with 30 tools pays for 30 descriptions each call — expose only the tools a given flow actually needs.
4. Stop runaway loops
An agent stuck calling tools in circles can burn a fortune on a single request. Defend against it:
- Set a max-turns / max-tool-calls limit per run so a confused agent fails fast instead of spinning.
- Alert on high token-per-request — a sudden jump is often a loop or a prompt regression, and catching it early saves real money.
- Add timeouts so a slow or looping run can’t run unbounded.
5. Put a budget in place
Finally, make cost a first-class constraint, not a monthly surprise:
- Per-user or per-tenant rate limits cap how much any single caller can spend — and double as abuse protection.
- Provider-side budget alerts on your Azure OpenAI / model spend give you a backstop.
- A monthly cost dashboard next to your quality metrics keeps the trade-off honest: you want the cheapest configuration that still passes your evals.
Note: provider pricing, prompt-caching behavior, and quotas change often. Verify specifics against your model provider’s current docs; the levers here — right-size the model, cache, trim context, cap loops, budget — are durable and apply across providers.
Takeaway
Agent cost isn’t fixed — it’s mostly removable waste. Measure per-request token spend first, then pull the levers in order of impact: right-size the model (the big one), cache repeated work and stable prompt prefixes, trim history and retrieval and tool schemas, cap runaway loops, and enforce per-user budgets. Done together, these routinely cut a bill by half or more while keeping quality intact — turning “the agent is getting expensive” into a number you control.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.