Choosing a Model for Your .NET Agent

“Which model should I use?” is the question people agonize over first and should worry about least — because the Microsoft Agent Framework builds on Microsoft.Extensions.AI’s IChatClient abstraction, swapping models is a config change, not a rewrite. That frees you to choose pragmatically and change your mind. This article is a framework for making that choice well, and for not over-thinking it.

If you’re building the agent itself, start with the tool-using agent tutorial; this is about what powers it.

The four axes that actually matter

Every model decision trades off four things. Rank them for your use case before comparing models:

1. Capability — can it reliably do the reasoning and tool-calling your task needs?
2. Latency — how fast does it respond? Critical for interactive agents, less so for batch.
3. Cost — price per token, multiplied by your real traffic.
4. Deployment — hosted API, your cloud, or self-hosted? This is often decided by data-residency and compliance, not performance.

Most teams over-index on #1 and ignore #2 and #3 until the bill arrives. Don’t.

Match the model to the job, not the hype

A common and expensive mistake is running the largest frontier model for everything. Right-size instead:

• Simple, high-volume tasks (classification, routing, extraction, short answers) — a small/efficient model is faster and often 10× cheaper, with no quality loss you’d notice. Most agent turns are simpler than they feel.
• Complex reasoning, multi-step tool use, tricky planning — a larger frontier model earns its cost. This is where capability failures actually hurt.
• Mixed workloads — use both. Route easy turns to a small model and escalate hard ones to a big model. Because it’s all one IChatClient interface, a router that picks the model per request is a modest amount of code, not an architecture project.

The instinct “bigger is safer” is how agent costs balloon. Start smaller than you think and move up only where evals show you need to.

Where the model runs

Deployment is frequently the deciding constraint:

• Hosted APIs (Azure OpenAI, and other providers) — least ops, fastest to ship, usually the default. Azure OpenAI keeps data in your tenant, which resolves many enterprise concerns.
• Your cloud (Azure AI, Vertex AI, Bedrock) — keeps traffic and data inside your cloud boundary; good for compliance.
• Self-hosted open-weight models — maximum control and data isolation, and no per-token fee, but you own the GPUs, scaling, and reliability. Choose this for hard data-residency requirements or genuinely huge volume where the infrastructure pays for itself — not by default.

For most .NET teams, a hosted model in your own cloud tenant is the pragmatic starting point.

Let evals decide, not vibes

Don’t pick a model from a leaderboard or a demo. Pick it with your eval set:

// Same agent code, different model — because it's all IChatClient.
foreach (var model in new[] { "gpt-4o-mini", "gpt-4o", "your-small-model" })
{
    var agent = MakeAgent(model);
    var score = await RunEvals(agent, evalCases);
    Console.WriteLine($"{model}: quality {score.Quality:P0}, avg latency {score.LatencyMs}ms");
}

Run your tasks through each candidate and compare quality, latency, and cost on real cases. The cheapest model that passes your quality bar is the right one — and you’ll often be surprised how far down the size ladder that is.

Note: available models and prices change constantly. Don’t hard-code a model choice as permanent; the point of the IChatClient abstraction is that re-evaluating as new models ship is cheap. Verify current options against your provider’s docs.

Takeaway

Model choice matters less than people fear, because the Agent Framework makes it swappable. Rank capability, latency, cost, and deployment for your case; right-size the model to the task instead of defaulting to the biggest; put the model where your compliance needs require; and let your eval set — not a leaderboard — make the final call. Start smaller and cheaper than your instinct says, and let evidence tell you when to scale up.

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.