An agent gave a user a wrong answer. Why? With plain logs, you’re guessing: did it pick the wrong tool, get bad data back, or just reason poorly? An agent is a chain of decisions, and to debug it you need to see the chain. That’s what observability gives you — and for .NET agents, OpenTelemetry is the way to get it. This article instruments an agent so every run becomes a trace you can inspect.
It complements the testing guide: tests catch problems before release, observability explains them after.
Why traces, not just logs
Logs are flat lines of text. A trace is a tree of timed spans that mirrors what actually happened: the request span contains a model-call span, which contains tool-call spans, each with attributes (which tool, what arguments, how long, success or failure). For an agent, that structure is exactly the shape of the problem you’re debugging.
The reason this matters more for agents than for ordinary services is that an agent run is variable-shaped. A conventional endpoint does roughly the same work every time, so a log line per stage is enough to reconstruct it. An agent might make one model call or nine, might call three tools in sequence because the first two returned nothing useful, and might loop. The interesting fact is almost never “what happened at step four” — it’s “how many steps were there, and why”. Only a tree answers that.
The good news: the .NET AI stack emits this telemetry using standard OpenTelemetry conventions, so you mostly enable it rather than hand-instrument it.
The conventions do the hard part
Both Microsoft.Extensions.AI and the Microsoft Agent Framework emit spans that follow the OpenTelemetry GenAI semantic conventions. That’s the single most useful thing to know, because it means your dashboards are not tied to a library. The same span names and attribute keys come out of a Python agent, a Go agent, or a third-party instrumentation library, so a query you write today survives a framework change.
Three span names carry almost all the signal:
invoke_agent <agent name>— the top-level span for one agent invocation. Everything else nests under it.chat <model name>— one call to the underlying model. An agent turn with tool use produces several of these.execute_tool <function name>— one function/tool invocation.
The attributes are equally standardised. gen_ai.operation.name tells you which of the above you’re looking at, gen_ai.provider.name and gen_ai.request.model identify what you called, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens give you cost, and gen_ai.tool.name, gen_ai.tool.call.id and gen_ai.response.finish_reasons give you the behavioural detail. There’s also gen_ai.conversation.id, which is the attribute you’ll wish you had set the first time someone reports “the agent went weird halfway through my conversation”.
Two metrics come along for free and are worth putting on a dashboard before anything else: gen_ai.client.operation.duration (a histogram, in seconds) and gen_ai.client.token.usage (a histogram, in tokens). Between them you get p95 latency and token spend without writing a line of custom instrumentation.
Wiring up OpenTelemetry
There are two halves to this. First, turn on instrumentation in the AI pipeline. Second, configure a tracer provider that actually listens to it. Getting one without the other is the most common reason people conclude “it doesn’t work”.
Instrumentation goes on the chat client, the agent, or both:
using Microsoft.Extensions.AI;
using Microsoft.Agents.AI;
const string SourceName = "MyApplication";
IChatClient chatClient = baseClient
.AsBuilder()
.UseOpenTelemetry(sourceName: SourceName)
.Build();
AIAgent agent = chatClient
.AsAIAgent(instructions: "You are a terse support agent.", tools: [/* … */])
.WithOpenTelemetry(sourceName: SourceName);Then subscribe to that source and export it:
using OpenTelemetry.Trace;
builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddSource(SourceName) // must match the name above
.AddAspNetCoreInstrumentation() // the incoming request
.AddHttpClientInstrumentation() // outbound model/API calls
.AddOtlpExporter()) // send to your backend
.WithMetrics(metrics => metrics
.AddMeter(SourceName)
.AddOtlpExporter());Point the OTLP exporter at whatever you use — Azure Monitor, Jaeger, Grafana Tempo, Honeycomb, or the Aspire dashboard on localhost during development. The value is the same everywhere: one trace per agent run, drillable down to each step.
The first failure mode is here. If you omit the sourceName argument when instrumenting, the libraries fall back to their default source names — Experimental.Microsoft.Extensions.AI for the chat client and Experimental.Microsoft.Agents.AI for the agent layer — and if your AddSource call doesn’t list those exact strings, you get a perfectly healthy application emitting no AI spans at all. There is no warning. The trace just contains your HTTP spans and a suspicious gap. Either pass an explicit source name in both places, or add all the default names:
.AddSource("Experimental.Microsoft.Extensions.AI")
.AddSource("Experimental.Microsoft.Agents.AI")
.AddSource("Microsoft.Extensions.AI")The second failure mode is the mirror image: instrument the chat client and the agent, and you get the same conversation recorded twice, once on the invoke_agent span and once on the chat span. That’s harmless when only metadata is captured and genuinely expensive when content capture is on, because you’re paying to store every prompt twice. Pick the layer you care about. Agent-level spans are better for product debugging; chat-level spans are better when you’re tuning a specific prompt.
The third is subtler and only bites outside ASP.NET Core: in a console app, a worker, or a short-lived function, the process can exit before the exporter flushes. You lose exactly the traces from the run you were investigating. Dispose the tracer provider, or call its flush, before the process ends.
Capturing the AI-specific detail
Generic HTTP traces tell you a model call happened; they don’t tell you what it did. Prompts, completions, tool arguments and tool results are gated behind an explicit opt-in, because they routinely contain user data:
// Development only. This puts prompts, responses, tool arguments and tool
// results into your traces.
.UseOpenTelemetry(sourceName: SourceName, configure: c => c.EnableSensitiveData = true)The equivalent standard environment variable is OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, which some layers honour directly.
Enable content capture in development, where it turns a confusing trace into an obvious one in about ten seconds. In production, think about it properly. Your traces are now a copy of everything users typed, sitting in a system that usually has broader read access and shorter-lived retention policies than your actual database. If you’re subject to deletion requests, that copy is in scope. If a user pastes a password into a chat window — and they will — it’s now in your observability backend.
The middle path most teams land on is to capture content on a sample of traffic, route those traces to a separate, access-controlled backend with a short retention window, and leave the always-on production traces at metadata only. Metadata alone is enough to detect almost everything; you only need content to explain it, and by then you can usually reproduce.
What to actually watch
Instrumentation is only useful if you look at the right signals. For agents, four things earn their place on a dashboard.
Tool-call outcomes. Group execute_tool spans by gen_ai.tool.name and split by status. A spike in one tool’s error rate is the single most common root cause behind a wave of bad answers, and it’s invisible in aggregate latency because a failing tool is usually a fast tool. Watch for the quieter version too: a tool that succeeds but returns an empty result, which the model then papers over with a confident guess. That one needs an explicit check inside the tool, because OpenTelemetry will happily record it as a success.
Token usage per run. The best cost signal you have, and an early warning when prompts bloat. Track the p95, not the mean — the mean hides the runs where an agent looped ten times, and those runs are both your cost problem and your quality problem. (See cost control.)
Latency, broken down. Is the slow part the model, a tool, or your own code? The trace tree answers this directly, and the answer is often surprising: teams that assume the model is the bottleneck frequently find a synchronous database call inside a tool costing more than the generation. Our latency budget guide goes deeper on what to do once you know.
Turn counts. Count chat spans per invoke_agent span. One or two is normal. Eight means the agent is thrashing — usually because a tool description is ambiguous or a tool keeps returning something the model can’t use. This is the metric that catches a bad deployment before your users do, because loop behaviour degrades long before answer quality does.
Two things that will bite you at scale
Sampling defeats the purpose if you do it naively. Head-based sampling at, say, 10% decides whether to keep a trace before anything interesting has happened, so nine times out of ten you throw away the run that failed. For agents, either sample at 100% and control cost through retention, or use tail-based sampling so the decision happens after you know the run errored, looped, or ran long. If your backend supports neither, the pragmatic compromise is 100% sampling of runs that produced an error or exceeded a token threshold, and a low fixed rate for the rest.
Watch metric cardinality. It is very tempting to add gen_ai.conversation.id or a user ID as a metric dimension so you can slice spend per customer. Do not. Every distinct value creates a new time series, and conversation IDs are unbounded, so you will either bankrupt your metrics bill or get silently rate-limited by your vendor. Identifiers belong on spans, which are sampled and stored differently. Aggregate per-customer spend from the trace store or from your own billing records instead.
Closing the loop
Once the telemetry flows, make it reachable. Build a dashboard with tool-error rate, p95 latency, token spend and turn count, and alert on the two that predict incidents: tool-error rate climbing, and tokens-per-request jumping. The second is worth an alert on its own — a step change in token usage is almost always either a prompt regression someone shipped or an agent stuck in a loop, and both are cheaper to catch in the first ten minutes.
The last piece is correlation. Return the trace ID to the caller (a response header is fine) and log it alongside any support ticket. When a user says “the agent gave me nonsense on Tuesday”, you want to paste an ID into your backend rather than reconstruct the session from timestamps. This costs one line of code and saves an afternoon roughly once a quarter.
Note: exact activity-source names and telemetry switches evolve as the .NET AI libraries mature, and several of the Agent Framework packages are still shipping as prerelease. Verify against the current .NET AI docs and the Agent Framework observability guide. The GenAI semantic conventions themselves — the span names and
gen_ai.*attributes — are the stable, vendor-neutral part, and they are what your dashboards should be built on.
Takeaway
You can’t operate what you can’t see, and an agent’s failures live in the chain of decisions between request and response. OpenTelemetry turns that chain into a trace: instrument the chat client or the agent, make sure AddSource matches the source name you passed, export to your backend, and build your dashboards on the gen_ai.* conventions rather than on library-specific names.
Then watch four things — tool errors, tokens per run, latency by span, and turn count — and keep content capture out of your always-on production traces unless you have decided, deliberately, that you can hold that data. Do all of it before you have traffic, not after your first incident, because the first time an agent misbehaves in production a good trace is the difference between a five-minute fix and an afternoon of guessing.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.