“How do you test something that gives a different answer every time?” is the first objection every engineer raises about shipping agents. It’s a fair worry, but it hides a false assumption: that an agent is one untestable blob. It isn’t. An agent is a deterministic shell (your tools, your routing, your guardrails) wrapped around a non-deterministic core (the model). You test each with the right tool. This article lays out a practical testing and evaluation strategy for .NET agents.
If you’re building agents with tools, this pairs with our tool-using agent tutorial.
The test everyone writes first, and why it rots
The instinct is to write one integration test: call the agent with a real question, assert the answer contains “shipped”. It passes locally. It passes in CI. Then one morning it fails on a pull request that only touched a stylesheet, because that time the model said “on its way”. Someone wraps it in a retry attribute. It fails again a fortnight later. Someone marks it [Fact(Skip = "flaky")], and now you own a test that costs tokens on every push and tells you nothing.
The mistake isn’t the assertion. It’s asking one test to verify two unrelated things — whether your code wired the agent up correctly, and whether the model’s answer was any good. Those have different failure modes, different run frequencies and very different price tags. Separate them and both become tractable.
Layer 1: Unit-test your tools (the easy win)
Your tools are ordinary C# methods. They have no LLM in them. Test them like any other code — this is where most of your real logic lives, and it’s fully deterministic:
[Fact]
public void GetOrderStatus_UnknownId_ReturnsNotFound()
{
var result = OrderTools.GetOrderStatus("does-not-exist");
Assert.Contains("No order found", result);
}If your tools hit a database or API, mock those dependencies as usual. The point: the moment you push logic out of the prompt and into well-tested tools, most of your agent’s behavior becomes conventional, verifiable code. That’s a design goal, not just a testing one.
Test the error paths harder than the happy path
The happy path of a tool is rarely where agents come apart. What matters is what the model sees when a tool fails, because that string goes straight back into the conversation as context. A tool that lets a SqlException escape hands the model a stack trace, and a model holding a stack trace tends to apologise and then answer from its own training data anyway — which is precisely the hallucination the tool existed to prevent.
Make tools total over their inputs. Catch what you can, and return a short, unambiguous message the model can act on.
[Theory]
[InlineData("")]
[InlineData(" ")]
[InlineData("'; DROP TABLE Orders; --")]
public void GetOrderStatus_MalformedId_ReturnsActionableMessage(string id)
{
var result = OrderTools.GetOrderStatus(id);
Assert.StartsWith("Invalid order id", result);
Assert.DoesNotContain("Exception", result);
}The second assertion is the one worth stealing. It’s a cheap guard against the day someone refactors the tool and lets an internal error message — commonly carrying a connection string, a table name or a file path — leak into the model’s context, and from there into an answer a customer reads.
Layer 2: Test agent wiring by mocking the model
To test the agent’s plumbing — does it call the right tool, does it handle a tool error, does it respect a guardrail — you don’t want a real, paid, flaky model call. Swap in a fake IChatClient that returns a scripted response:
// A stub chat client that always "decides" to call GetOrderStatus
var fakeClient = new StubChatClient(response: ToolCall("GetOrderStatus", new { orderId = "ORD-1" }));
var agent = fakeClient.CreateAIAgent(
name: "Test",
instructions: "…",
tools: [AIFunctionFactory.Create(OrderTools.GetOrderStatus)]);
var reply = await agent.RunAsync("Where is ORD-1?");
Assert.Contains("shipped", reply.ToString());Because the framework builds on Microsoft.Extensions.AI’s IChatClient abstraction, substituting a deterministic fake is straightforward. These tests are fast, free, and run in CI — they verify your code around the model without depending on the model’s mood.
Script a sequence, not a single response
That snippet hides something that bites in practice: an agent with tools is a loop. The model returns a tool call, the framework runs the tool, feeds the result back, and asks again. A stub that returns the same tool-call response every time never terminates — the framework runs the tool, sends the result, gets the identical tool call back, and your CI job hangs until the build times out twenty minutes later.
Script a queue instead, and fail loudly when it runs dry:
sealed class ScriptedChatClient(Queue<ChatResponse> script) : IChatClient
{
public List<ChatMessage> Seen { get; } = [];
public Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages,
ChatOptions? options = null,
CancellationToken cancellationToken = default)
{
Seen.AddRange(messages);
if (script.Count == 0)
throw new InvalidOperationException(
"Agent asked for more turns than the script provides — probable tool loop.");
return Task.FromResult(script.Dequeue());
}
// GetStreamingResponseAsync, GetService and Dispose omitted for brevity.
}Two things fall out of this. The exception turns a twenty-minute hang into a one-line diagnosis. And Seen gives you the thing genuinely worth asserting on — not the prose, but the arguments:
Assert.Equal("ORD-1", capturedToolCall.Arguments["orderId"]);Tool-argument correctness is where a surprising share of real agent bugs live. The model passes the whole user sentence where the tool expects an id, or an ISO date where the tool parses dd/MM/yyyy, and the tool dutifully reports “not found”. The final answer reads perfectly well and is wrong.
Test the failure handling too
Script a turn where the tool returns an error, then assert what the agent does with it. The behaviour you want is that it tells the user the lookup failed. The behaviour you frequently get is a confident invented answer, because nothing in the instructions told it otherwise. That gap is a wiring and prompt problem, it reproduces perfectly against a fake client, and checking it costs nothing on every commit.
Layer 3: Evaluate the model’s actual output
The two layers above never call a real model, so they can’t tell you whether the agent’s answers are any good. That needs evaluation — a different discipline from unit testing. Instead of asserting exact strings (which non-determinism makes impossible), you score outputs against criteria over a fixed set of test cases. Pinning temperature to zero narrows the variation but doesn’t promise byte-identical output, so exact-string assertions stay off the table even for a configuration you have deliberately locked down.
Build an eval set: a collection of representative inputs with expected properties.
record EvalCase(string Input, string[] MustContain, string[] MustNotContain);
var cases = new[]
{
new EvalCase("Where is ORD-4821?", ["ORD-4821"], ["I don't know"]),
new EvalCase("What's your refund policy?", ["30 days"], []),
};Then run each case through the real agent and check the properties. For fuzzy judgments (“is this answer helpful and on-topic?”), a common technique is LLM-as-judge: a second model call scores the first’s output against a rubric.
Build the set from production, not from imagination
An eval set written from imagination tests the questions you already thought about, which are the questions your agent already handles. The useful set comes out of traces — real user inputs, weighted heavily towards the ones that produced a thumbs-down, a support ticket, or an unusually long tool loop. Thirty cases drawn from live traffic beat five hundred invented ones, and they’re cheap enough to run often. If you’re not capturing traces yet, OpenTelemetry for agents is the prerequisite.
The set should then change slowly. Every time you add a case because of a bug, you’ve converted an incident into a permanent check. That conversion is the entire return on the exercise.
What the .NET evaluation libraries actually give you
Microsoft.Extensions.AI.Evaluation is the core abstraction, and the evaluators live in satellite packages. Microsoft.Extensions.AI.Evaluation.Quality carries the model-graded ones — relevance, truth, completeness, fluency, coherence, retrieval, equivalence and groundedness — which are the rubric-driven judges you’d otherwise hand-roll and argue about. Microsoft.Extensions.AI.Evaluation.Safety builds on the Azure AI Foundry evaluation service and covers content-safety categories including hate and unfairness, violence, self-harm, sexual content, protected material, code vulnerability and indirect attack. When you want deterministic text-overlap metrics rather than a judge, Microsoft.Extensions.AI.Evaluation.NLP implements BLEU, GLEU and F1. Microsoft.Extensions.AI.Evaluation.Reporting stores results and generates reports, and — the part that shows up on your bill — caches model responses, so re-running a set doesn’t re-pay for the cases nothing changed for.
For a RAG agent, groundedness and retrieval are the two that earn their place immediately. Groundedness tells you whether the answer stayed inside the retrieved passages; retrieval tells you whether the right passages were there to begin with. Scored separately, they tell you which half of the pipeline to fix — and that distinction saves more time than any other measurement here.
The judge has its own failure modes
An LLM judge is a model, so it inherits everything you dislike about models. Three show up reliably.
It rewards length. Offered two answers, judges lean towards the longer, more hedged one. If your rubric doesn’t explicitly penalise padding, your scores will quietly push the agent towards waffle over several release cycles.
It agrees with itself. Judging output from a model using that same model shares the blind spots — the misreading that produced the answer also approves it. Judge with a different model, ideally a stronger one, and pin its version. Changing the judge silently re-bases every historical score, which makes your trend line worthless unless you re-run the baseline too.
Half the time you don’t need it. Valid JSON, a required citation marker, a forbidden phrase, a numeric field inside a range — those are a regex or a JsonDocument.Parse inside a try block. Every check you can make deterministic is free, instant, and never drifts. Keep the judge for the genuinely fuzzy questions and nothing else.
The cost arithmetic, before you schedule it
An eval run costs (cases × model calls per case) plus (cases × judge calls), multiplied by however many configurations you’re comparing. An agent that works through three or four tool calls per question is three or four model calls per case, not one — people consistently under-budget this by an order of magnitude. Two hundred cases across two candidate models is a four-figure call count, and it’s slow, because these runs hit rate limits and end up largely sequential.
That’s why layer 3 belongs on a schedule while layers 1 and 2 belong on every commit. Nightly against main, plus an on-demand run before a release or a model swap, gives you the regression signal without attaching a token bill to every push.
Gate on the right statistic
Don’t gate a release on a mean score. A mean cheerfully absorbs one case going from correct to catastrophically wrong, which is exactly the failure the set exists to catch. Gate on two numbers instead: the floor — no individual case scores below your minimum — and the delta — no case dropped by more than a small margin against the last accepted baseline. Store that baseline alongside the release, so you’re comparing against something you actually shipped rather than against yesterday’s experiment.
When the gate fails, check the eval before you check the agent. Eval sets rot. A policy changes, the expected string becomes genuinely wrong, and the test is now asserting last quarter’s truth with total confidence.
A worked example: swapping the model underneath
Say you want to move the default tier from a flagship model to a cheaper one to cut spend — model routing covers the mechanics. Layers 1 and 2 both pass, because none of your code changed. That’s not a gap in those tests; it’s the boundary of what they can see. This class of regression is invisible to them by construction.
What the eval run surfaces is rarely “the answers got worse” in any sweeping sense. It’s narrower and stranger than that. Format compliance usually cracks first: the smaller model occasionally wraps its JSON in a markdown fence, and your parser rejects it. Tool accuracy tends to go next, particularly when several tools have similar descriptions and the smaller model picks the neighbouring one. Groundedness often holds up perfectly well — which is exactly why a team eyeballing a dozen answers concludes the swap is fine and ships it.
With scores you get a decision instead of a feeling: this metric moved, that one didn’t, here is the size of the move. Is an occasional parse failure worth the saving, or do you keep the cheaper model and pair it with structured output plus a retry on parse failure? Both answers are defensible. Neither is available to you without the numbers.
What to actually measure
Don’t try to measure “correctness” in the abstract. Measure the things that break trust in production:
- Groundedness — does the answer stick to the tool/retrieved data, or does it invent facts?
- Tool accuracy — did it call the right tool with the right arguments?
- Refusal behavior — does it decline out-of-scope or unsafe requests?
- Format compliance — if you need JSON, is it valid JSON every time?
Track these scores over time. The real value of an eval set isn’t a one-time grade — it’s a regression net. When you change a prompt, swap a model, or upgrade the framework, re-run the evals and catch quality drops before your users do.
Where each layer runs
- Layers 1 & 2 (unit tests, mocked model) run on every commit in CI — fast and deterministic.
- Layer 3 (real-model evals) is slower and costs tokens, so run it on a schedule or before releases, not on every push. Treat a drop in eval scores like a failing test: block the release.
When not to build any of this
A spike you’ll throw away next week doesn’t need an eval set. Nor does an internal tool with five users who will tell you directly when it’s wrong, and who can tolerate it being wrong for an afternoon. Building a scored regression net before you have traffic to draw cases from is a reliable way to spend a week measuring nothing.
The threshold is roughly this: the moment more than one person can edit the prompt, or the moment a bad answer reaches somebody outside the team, you need the net. Prompts have no compiler and no type system. A two-word edit to a system prompt can change behaviour across every path at once, and nothing in your toolchain will warn you — which is a strange position for a .NET team to be in, and the reason this discipline is worth the effort even though it feels unfamiliar.
Layers 1 and 2, by contrast, have essentially no threshold. They’re ordinary xUnit tests against ordinary code, they cost nothing to run, and skipping them is just skipping tests.
Note: exact type names in the evaluation libraries evolve. Verify against the current .NET AI documentation; the strategy — deterministic tests for your code, eval sets for the model’s output, tracked as a regression net — is stable regardless of API churn.
Takeaway
Non-determinism doesn’t make agents untestable; it means you test them in layers. Push logic into tools and unit-test them. Mock the model to test your wiring in CI. And evaluate the real model’s output against an eval set that doubles as a regression net. Do all three and “how do you test this?” stops being a blocker and becomes a checklist.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.