“Which model should I use?” is the question people agonize over first and should worry about least — because the Microsoft Agent Framework builds on Microsoft.Extensions.AI’s IChatClient abstraction, swapping models is a config change, not a rewrite. That frees you to choose pragmatically and change your mind. This article is a framework for making that choice well, and for not over-thinking it.
If you’re building the agent itself, start with the tool-using agent tutorial; this is about what powers it.
The four axes that actually matter
Every model decision trades off four things. Rank them for your use case before comparing models:
- Capability — can it reliably do the reasoning and tool-calling your task needs?
- Latency — how fast does it respond? Critical for interactive agents, less so for batch.
- Cost — price per token, multiplied by your real traffic.
- Deployment — hosted API, your cloud, or self-hosted? This is often decided by data-residency and compliance, not performance.
Most teams over-index on #1 and ignore #2 and #3 until the bill arrives. Don’t.
Capability is not one number
Treating capability as a single score is the mistake underneath most bad model choices. For an agent, it splits into at least four separate abilities, and models are uneven across them. There is instruction following — does it respect a system prompt that says “never mention pricing” on turn thirty as well as on turn one? There is tool selection — given nine tools with similar-sounding descriptions, does it pick the right one, or does it call the search tool for everything? There is argument construction — does it emit arguments matching your schema, including the awkward cases like an optional enum or a nested object? And there is long-context recall, which is not the same as a large context window: a model advertising a very large window may still lose a detail buried in the middle of a long transcript.
A model can be excellent at prose and mediocre at emitting valid tool arguments. That combination is common, and it is invisible on a general leaderboard while being fatal to an agent, because a malformed argument is not a slightly worse answer — it is an exception, a retry, and a doubled bill. If your agent leans on structured output or a lot of tool calls, test those specifically. They are the capabilities that decide whether the thing works.
Small errors also compound in a way single-turn testing hides. An agent that chains five tool calls needs every step right. At 95% per-step reliability, end-to-end success is roughly 0.95 to the fifth — about 77%. At 99% per step it is 95%. The gap between “usually fine” and “reliable” per step is the gap between a broken product and a working one, and it widens with every step you add to the chain.
Match the model to the job, not the hype
A common and expensive mistake is running the largest frontier model for everything. Right-size instead:
- Simple, high-volume tasks (classification, routing, extraction, short answers) — a small/efficient model is faster and substantially cheaper, with no quality loss you’d notice. Most agent turns are simpler than they feel.
- Complex reasoning, multi-step tool use, tricky planning — a larger frontier model earns its cost. This is where capability failures actually hurt.
- Mixed workloads — use both. Route easy turns to a small model and escalate hard ones to a big model. Because it’s all one
IChatClientinterface, a router that picks the model per request is a modest amount of code, not an architecture project.
The instinct “bigger is safer” is how agent costs balloon. Start smaller than you think and move up only where evals show you need to.
The cascade deserves an honest caveat, though. Escalation is only cheaper if the small model knows when it failed. Work through the arithmetic with your own numbers: if the large model costs twenty times the small one and the small one handles four requests in five, then routing costs roughly one unit of small-model spend plus a fifth of twenty units of large-model spend — around five, against twenty for using the large model throughout. Good. But the escalated fifth now pays for two calls and two round trips of latency, and if you cannot detect the failure cheaply, you are paying for both models on requests you never salvaged. Cascades work when failure is detectable — a schema validation error, a low-confidence classification, an empty tool result — and fall apart when failure looks like a confident wrong answer. Model routing and fallback covers the mechanics.
Latency is three numbers, not one
For an interactive agent, “how fast is the model” decomposes into time to first token, generation speed after that, and the number of round trips a turn requires. They pull in different directions. A model that starts fast and generates slowly feels better than the reverse, because streaming hides generation speed but cannot hide the initial wait. Reasoning-style models invert this deliberately: they spend tokens thinking before producing anything visible, which is fine for a batch job and painful in a chat box.
The multiplier people forget is the agent loop. One user question that triggers three tool calls is four model calls, not one, each re-sending a conversation that grew since the last. Per-call latency multiplies by loop depth, and so does per-call cost. Before blaming the model for a slow agent, count the round trips — that is usually where the seconds went, and it is the subject of the latency budget article.
Cost per task, not cost per token
Published prices are per token, but the only number that matters is what one completed user task costs. Get there by multiplying: tokens per call, calls per task, and the share of tasks that need a retry. Three details usually dominate. Output tokens typically cost several times what input tokens cost, so a verbose model is expensive twice over — once at generation, and again when its verbose reply becomes input on the next turn. Prompt caching changes the arithmetic sharply for agents, because an agent re-sends a large stable system prompt and tool schema on every single call, which is exactly the shape caching rewards. And offline work — evals, backfills, bulk classification — often qualifies for cheaper batch handling, so a workload you were sizing for interactive pricing may not need it.
Capacity is a fourth cost axis that only appears in production. Hosted models come with request and token rate limits per deployment and region, and the model you want may not have the throughput you need where you need it. A slightly weaker model you can actually get capacity for beats a better one you spend your peak hour being throttled by. Check the quota before you commit the architecture.
Where the model runs
Deployment is frequently the deciding constraint:
- Hosted APIs — least ops, fastest to ship, usually the default. Running through your own cloud tenant resolves many enterprise concerns; see Azure OpenAI vs OpenAI for how that comparison actually plays out.
- Your cloud (Azure AI, Vertex AI, Bedrock) — keeps traffic and data inside your cloud boundary; good for compliance.
- Self-hosted open-weight models — maximum control and data isolation, and no per-token fee, but you own the GPUs, scaling, and reliability. Choose this for hard data-residency requirements or genuinely huge volume where the infrastructure pays for itself — not by default.
Note that model availability varies by region, so “we must stay in this geography” can quietly remove your first choice from the list. For most .NET teams, a hosted model in your own cloud tenant is the pragmatic starting point. Running a small model locally with Ollama is worth doing regardless, as a development-time convenience that keeps your inner loop free and offline.
What swapping a model does not make free
The IChatClient abstraction makes the code portable. It does not make your prompts portable, and that is the part that costs a week. A system prompt tuned against one model — its tolerance for terse instructions, how it handles a long tool list, whether it needs to be told explicitly not to invent parameters — will behave differently on another. Tool-calling conventions differ underneath the abstraction too, in how aggressively a model calls tools, whether it can issue several in parallel, and how strictly it honours a JSON schema.
Plan for two other realities. Models get retired on published timelines, so anything you build has an expiry date you should record somewhere other than someone’s memory. And aliases can move under you: pointing at a floating name means your behaviour can change without a deployment, which is convenient until it silently breaks a prompt. Pin explicit versions in production, keep the alias for staging, and treat a model upgrade as a change that goes through your eval gate like any other.
Let evals decide, not vibes
Don’t pick a model from a leaderboard or a demo. Pick it with your eval set:
// Same agent code, different model — because it's all IChatClient.
foreach (var model in new[] { "gpt-4o-mini", "gpt-4o", "your-small-model" })
{
var agent = MakeAgent(model);
var score = await RunEvals(agent, evalCases);
Console.WriteLine($"{model}: quality {score.Quality:P0}, avg latency {score.LatencyMs}ms");
}Run your tasks through each candidate and compare quality, latency, and cost on real cases. Three things make this loop worth the effort. Freeze the eval set so results stay comparable across months. Record p95 latency rather than the average, because the average hides the turn that made a user give up. And record cost per completed task, not per call, so a model that needs two attempts is scored honestly. Then keep the harness — it is the regression gate you run when a provider ships a new version, and the evidence you point at when someone asks why you are not using whatever launched this week.
Note: available models and prices change constantly. Don’t hard-code a model choice as permanent; the point of the
IChatClientabstraction is that re-evaluating as new models ship is cheap. Verify current options against your provider’s docs.
Takeaway
Model choice matters less than people fear, because the Agent Framework makes it swappable — but swappable applies to your code, not to your prompts or your tool-calling reliability, so budget time to re-tune when you move. Rank capability, latency, cost, and deployment for your case, and split each one open rather than treating it as a single number: capability means instruction following, tool selection, argument construction and long-context recall; latency means time to first token times the number of round trips; cost means tokens per task after caching, not the published rate per million. Right-size to the task, cascade only where failure is detectable, pin explicit model versions, and let a frozen eval set make the final call. Start smaller and cheaper than your instinct says, and let evidence tell you when to scale up.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.