Agentic RAG in .NET: When One Retrieval Is Not Enough

Most retrieval-augmented generation in production looks like this: take the user’s question, embed it, run one vector search, paste the top five passages into the prompt, and send it. That pipeline is a genuinely good default. It is cheap, it is one round trip, and it answers the majority of questions people actually ask a documentation bot.

It also has a failure mode that is easy to miss during a demo and impossible to miss once real users arrive — and the fix has a name now.

What one-shot retrieval quietly assumes

The single-search pipeline carries three assumptions, and none of them are stated anywhere in the code.

That the question is one question. “How do I configure retries, and does that interact with our timeout policy?” is two lookups wearing one sentence. Embed it whole and you get a vector that sits somewhere between the two topics, which is frequently near neither.

That the answer lives in one place. Comparison questions — “what changed between v3 and v4” — need passages from two documents that a single similarity search has no reason to return together. The top five will usually come from whichever document happens to be more verbose.

That you can tell in advance how much context is enough. topK = 5 is a guess made at design time and applied identically to “what is the default port” and “walk me through the migration”.

None of that is an argument against one-shot retrieval. It is an argument for knowing which of your questions violate the assumptions, because that fraction is exactly the fraction that agentic retrieval is worth paying for.

The patterns, cheapest first

“Agentic RAG” covers at least three distinct designs with very different costs. Treat them as a ladder and stop climbing as soon as your evaluation stops improving.

1. Query decomposition

One model call turns the question into two to four focused subqueries, you run those in parallel, merge and deduplicate the results, then answer from the union.

This is the pattern with the best cost-to-benefit ratio by a wide margin. It adds one small model call and the subqueries run concurrently, so the added latency is roughly the slowest search rather than the sum. It directly fixes the compound-question and comparison cases, which between them account for most of what one-shot retrieval gets wrong.

// A small, cheap model is the right choice here - decomposition is a
// structural task, not a reasoning one.
private static readonly ChatOptions DecomposeOptions = new()
{
    ResponseFormat = ChatResponseFormat.ForJsonSchema<SubQueries>(),
    Temperature = 0,
};

public sealed record SubQueries(string[] Queries);

public async Task<string[]> DecomposeAsync(string question, CancellationToken ct)
{
    var response = await _cheapClient.GetResponseAsync(
        [
            new ChatMessage(ChatRole.System,
                """
                Split the user's question into the smallest set of independent
                search queries that together cover it. Return one query if the
                question is already atomic. Never return more than four.
                """),
            new ChatMessage(ChatRole.User, question),
        ],
        DecomposeOptions, ct);

    var parsed = JsonSerializer.Deserialize<SubQueries>(response.Text)!;

    // Trust the cap in code, not in the prompt. A model that ignores
    // "never more than four" should cost you nothing.
    return parsed.Queries.Take(4).ToArray();
}

Then fan out. The merge step matters more than it looks — the same passage will frequently rank highly for two subqueries, and paying for it twice in the context window is pure waste:

var subQueries = await DecomposeAsync(question, ct);

var results = await Task.WhenAll(
    subQueries.Select(q => _store.SearchAsync(q, topK: 4, ct)));

var passages = results
    .SelectMany(r => r)
    .DistinctBy(p => p.ChunkId)          // dedupe before ranking, not after
    .OrderByDescending(p => p.Score)
    .Take(8)
    .ToList();

Note that the final Take(8) is larger than the topK: 5 a one-shot pipeline would use, but smaller than the twelve to sixteen passages the fan-out produced. Decomposition is not a licence to stuff the window — it is a way to make the passages you do include more likely to be relevant.

2. Iterative retrieval

Retrieve, let the model judge whether it can answer, and if not let it issue a follow-up search informed by what it just read. Repeat under a hard cap.

This handles the genuinely multi-hop question — the one where you cannot know the second search until you have read the result of the first. “Which of our services still uses the deprecated auth library, and what is the migration path for that one?” needs the first answer before the second query exists.

It is also substantially more expensive than decomposition, and the expense is the bad kind: the rounds are sequential, so latency adds up rather than overlapping. Two rounds is usually a real improvement, three is occasionally, and beyond that you are almost always watching a model fail to find something that is not in the corpus.

// The cap is not a suggestion. Without it, a question whose answer simply
// is not in the index will happily consume your entire token budget
// searching for it.
const int MaxRounds = 3;

var gathered = new List<Passage>();
for (var round = 0; round < MaxRounds; round++)
{
    var hits = await _store.SearchAsync(currentQuery, topK: 5, ct);
    gathered.AddRange(hits.Where(h => gathered.All(g => g.ChunkId != h.ChunkId)));

    var verdict = await AssessAsync(question, gathered, ct);
    if (verdict.CanAnswer) break;

    if (verdict.NextQuery is null) break;   // the model gave up; so should you
    currentQuery = verdict.NextQuery;
}

Log the round count as a metric. A rising average is the clearest early signal you will get that your corpus has drifted away from what users are asking.

3. Retrieval as a tool

Rather than orchestrating the loop yourself, expose search as a function the model can call, and let the agent loop decide when and how often.

This is the least code and the least control, which is exactly the trade. It shines when retrieval is one capability among several — the agent can search the knowledge base, or query the database, or call an API, and the interesting decisions are about which. It is a poor fit when retrieval is the only thing the agent does, because you have handed the model a decision it did not need to make and pay for it on every request.

[Description("Search the internal knowledge base for passages relevant to a question.")]
public async Task<string> SearchKnowledgeBase(
    [Description("A focused search query. Prefer several specific searches over one broad one.")]
    string query,
    CancellationToken ct = default)
{
    var hits = await _store.SearchAsync(query, topK: 5, ct);
    return JsonSerializer.Serialize(hits.Select(h => new { h.Source, h.Text }));
}

The instruction in the parameter description is doing real work there. Models left to their own devices issue one broad query that looks a lot like the user’s original question, which puts you back where you started with extra steps.

Wiring it into Microsoft Agent Framework

Agent Framework has a first-class seam for this. Rather than manually stuffing passages into the prompt, you attach a context provider to the agent through the AIContextProviders option, and the framework injects retrieved context on each turn. TextSearchProvider is the supplied implementation that covers the common case.

var agent = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Instructions = "Answer strictly from the retrieved context. If it is not there, say so.",
    AIContextProviders = [new TextSearchProvider(vectorStore)],
});

The reason to prefer this over hand-rolled prompt assembly is that it keeps retrieval out of your prompt-building code, which means you can change retrieval strategy without touching the prompt and vice versa. Those two things change for different reasons and on different schedules, and coupling them is a decision you will regret about six weeks in.

That instruction line is not decoration either. Retrieval quality problems and hallucination problems look identical from a bug report, and an agent that is explicitly told to say when the context is insufficient tells you which one you have. There is more on that separation in testing and evaluating .NET agents.

Letting the search service do the planning

Azure AI Search added agentic retrieval as a service-side feature: you send the conversation, and it handles decomposition into subqueries, runs them in parallel, and returns a merged result already shaped for a chat model. Parts of it reached general availability in the 2026-04-01 REST API.

The case for it is straightforward. The planner is the part of agentic RAG that looks trivial in a prototype and turns into a maintenance surface in production — subquery quality, dedupe rules, ranking across heterogeneous results, and a steady trickle of “why did it search for that”. If you are on Azure AI Search already, letting the service own that is a real saving.

The case against is equally straightforward: it is a service-side decision you can no longer see into or unit-test locally, and it binds a core behaviour of your application to one vendor’s implementation. If your retrieval logic is a differentiator, keep it in your code where you can evaluate it.

Either way the decision is reversible, which is the useful part. Both sides of it produce the same thing — a set of passages — so keep that boundary clean and the swap stays cheap. Comparing vector databases for .NET covers the store-level side of that choice.

What it actually costs

Be honest about this before you build it, because the numbers are not small.

Decomposition costs one extra model call — use a cheap model, it is a structural task — plus two to four searches instead of one. Vector searches are usually a few milliseconds against a warm index, so the real cost is the model call. Call it 10-20% more per request, mostly latency.

Iteration costs one model call per round to assess, plus the searches, plus a context window that grows with every round. Three rounds against a corpus that does not contain the answer is roughly four times the cost of a one-shot answer that would have said “I do not know” immediately.

Tool-based retrieval costs whatever the model decides, which is the honest answer and the reason to bound it. Cap tool-call iterations in the agent configuration, not in the prompt.

Across all three, the number to watch is not cost per request but cost per answered request. A pipeline that costs 30% more and answers 20% more questions correctly is a good trade; one that costs 30% more and mostly retrieves harder before giving the same answer is not. Cost control for production agents covers the instrumentation that makes that measurable rather than a matter of opinion.

Do the cheap thing first

There is a strong temptation to reach for agentic retrieval when the real problem is upstream, and it is worth resisting because the upstream fixes are cheaper and compound.

Chunking. If your chunks split mid-procedure or bury headings, more queries just return more fragments of the same broken shape. Chunking strategies for .NET RAG is the higher-leverage fix, and it costs nothing at request time.

Hybrid search. Combining vector similarity with keyword matching fixes a large share of the cases people reach for agentic RAG to solve — particularly exact identifiers, error codes and product names, where embeddings are genuinely weak. One search, better results.

Metadata filtering. A great many “the model retrieved the wrong thing” reports are really “it retrieved the right thing from the wrong version of the docs”. A WHERE version = @v clause fixes that permanently, for free.

Work through those three before adding a planner. If retrieval is still failing afterwards, you now have a clean baseline to measure the planner against — and without that baseline you will not be able to tell whether it helped.

When not to build this at all

Your questions are homogeneous. An FAQ bot over 200 support articles gets asked one atomic question at a time. Decomposition on an atomic question returns one subquery and you have paid a model call to learn what you already knew.

Latency is the product. In an interactive assistant where people are watching a cursor, the second round trip is more expensive than a slightly worse answer. Ship one-shot, measure, revisit.

You cannot evaluate yet. Every pattern here is a trade of cost for quality, and if you cannot measure quality you cannot tell whether you made the trade or just spent the money. Build the evaluation set first — a hundred real questions with known-good answers is enough to make every decision on this page empirical instead of aesthetic.

Note: Agent Framework’s context-provider surface — AIContextProviders, TextSearchProvider and the options used to attach them — has moved between previews, and Azure AI Search’s agentic retrieval is versioned by REST API date. Verify the exact names and API version against the Agent Framework RAG documentation and the agentic retrieval overview before wiring anything up. The patterns — decompose, iterate under a cap, or expose retrieval as a tool — have outlived several API renames.

Takeaway

Agentic RAG is not an upgrade you apply to a RAG pipeline. It is a decision to move the “how much retrieval does this need” question from design time to runtime, and it is worth making only when your questions genuinely vary enough to justify the bill.

Start with query decomposition. It is one extra call, it runs in parallel, and it fixes the compound and comparison questions that break one-shot retrieval — which is most of what people mean when they say their RAG bot is unreliable. Add iteration only when you have a measured multi-hop failure it would solve, and cap it in code the first time you write it, not after the incident.

---

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.