Giving Your .NET Agent a Knowledge Base: RAG in C#

An agent knows what its model was trained on — and nothing about your product docs, policies, or codebase. Ask it about your internal refund policy and it will confidently invent one. Retrieval-Augmented Generation (RAG) is how you fix that: before the model answers, you fetch the relevant snippets from your own data and hand them to it as context. This article builds a practical RAG pipeline for a .NET agent.

If you’ve followed the tool-using agent tutorial, think of retrieval as a special kind of tool — one that supplies knowledge instead of taking an action.

The RAG pipeline in three parts

RAG has two phases. Indexing (done ahead of time): chop your documents into chunks, turn each into an embedding vector, and store them. Retrieval (at query time): embed the user’s question, find the nearest chunks, and inject them into the prompt. Let’s build both.

Step 1: Chunk and embed your documents

You can’t embed a whole 40-page PDF as one vector — retrieval would be too coarse. Split documents into overlapping chunks (a few hundred tokens each), then generate an embedding for every chunk. In .NET, the Microsoft.Extensions.AI abstractions give you an IEmbeddingGenerator you can point at Azure OpenAI or another provider:

using Microsoft.Extensions.AI;

IEmbeddingGenerator<string, Embedding<float>> embedder = /* Azure OpenAI embeddings client */;

foreach (var chunk in ChunkDocuments(docs))   // your splitter
{
    var embedding = await embedder.GenerateAsync(chunk.Text);
    await store.UpsertAsync(chunk.Id, embedding.Vector, chunk.Text, chunk.Metadata);
}

A little overlap between chunks (say 10–15%) keeps a sentence that straddles a boundary from being lost. Store the original text and useful metadata (source, section, URL) alongside the vector — you’ll want them at answer time for grounding and citations.

Two decisions here are expensive to change later

The first is the splitter. Splitting on a fixed character count is the fastest thing to write and the one you’ll regret, because it cuts through the middle of tables, separates a heading from the paragraph it introduces, and orphans list items from their stem. A chunk that reads “…must be submitted within 14 days” with no indication of what “must” refers to will still embed, still match, and still get handed to the model as fact. Split on structure first — headings, sections, paragraph boundaries — and fall back to character counts only inside a section that’s genuinely too long. Our guide to chunking strategies goes through the options properly.

The second is the embedding model, and this one is a trap that catches teams once each. Vectors from different embedding models are not comparable. Swap the model and every vector already in your store is meaningless against the new queries — the numbers still have the right shape, the search still returns results, and those results are noise. There’s no exception thrown and no warning logged; retrieval quality just collapses. Changing the embedding model means a full re-index, so budget for that before you pick one, and record the model id in your chunk metadata so a mismatch is at least detectable. Embeddings explained covers what the vectors actually represent.

Size the storage before you’re surprised by it. A common embedding dimension is 1,536 floats, which at four bytes each is roughly 6 KB per chunk before index overhead and before the original text you’re also keeping. A hundred thousand chunks is therefore well under a gigabyte of vectors — genuinely small — but ten million chunks is a different infrastructure conversation, and the index memory, not the disk, is what runs out first.

Step 2: Store the vectors

You need a store that does similarity search — “find the vectors nearest this one.” Options for .NET teams:

• A dedicated vector database (Qdrant, Milvus, Pinecone) — purpose-built, scales well.
• A database you already run — pgvector on PostgreSQL, or vector search in Azure AI Search, MongoDB, or Redis. Often the pragmatic choice: no new infrastructure.

Microsoft.Extensions.VectorData provides a common abstraction over many of these, so your retrieval code doesn’t hard-couple to one vendor — you can start on Postgres and move later without rewriting. For most .NET teams starting out, pgvector on a database you already operate is the right first choice; a dedicated store earns its operational cost at scale, and the vector database comparison works through where that line sits.

Filtering is not optional, and it is a security control

Pure similarity search over one big collection works fine in a demo and is dangerous in a multi-tenant product. If chunks from every customer share a collection and your query is nothing but “nearest five vectors”, then customer A’s question can and eventually will retrieve customer B’s document — and because the model was told to answer from the context, it will summarise it politely and cite it. Nothing in the pipeline treats this as an error. The similarity score was perfectly good.

The fix is to make the tenant a filter applied inside the search, not a check applied to the results afterwards. Post-filtering wastes your top-K on rows you then throw away, so a heavily filtered query silently returns two usable chunks instead of five. Store tenant id, document class, effective date and any permission scope as filterable metadata from day one — retrofitting it means re-indexing, and you’ll be doing it under incident pressure.

The same mechanism solves quieter problems. Filtering on document status keeps superseded policies out of answers. Filtering on language stops a query matching the German edition of the same page. Filtering on date range is how you answer “what was the policy in March” without the model blending two versions together.

Step 3: Retrieve and ground the answer

At query time, embed the question, pull the top matches, and build a prompt that instructs the model to answer only from those snippets:

var queryVector = (await embedder.GenerateAsync(question)).Vector;
var hits = await store.SearchAsync(queryVector, topK: 5);

var context = string.Join("\n\n", hits.Select(h => $"[{h.Source}] {h.Text}"));

var prompt = $"""
    Answer the question using ONLY the context below. If the answer isn't in
    the context, say you don't know. Cite the [source] you used.

    Context:
    {context}

    Question: {question}
    """;

var answer = await agent.RunAsync(prompt);

That instruction — “answer only from the context, and say you don’t know otherwise” — is the heart of RAG. It’s what converts a confident hallucinator into a system that stays grounded in your data and admits its limits.

Handle the empty result explicitly

Vector search always returns something. Ask a knowledge base of shipping policies about the weather and it will hand back the five least-unrelated chunks it owns, with scores that are poor in absolute terms but are still the top five. Drop those into the prompt under a heading that says “Context:” and you’ve told the model, structurally, that these passages are relevant. Some of the time it will construct an answer out of them.

Apply a similarity floor and branch on it. If nothing clears the threshold, don’t build a context block at all — return “I don’t have anything on that” without a model call, which is both more honest and free. The threshold value itself is corpus-specific and there’s no universal number worth quoting; find yours by scoring a few dozen known-good and known-irrelevant questions and looking at where the two distributions separate.

The question you embed is rarely the question the user asked

In a single-shot API this doesn’t arise. In a conversation it arises constantly. The user asks “what’s the refund window?”, gets an answer, and follows up with “and for enterprise?”. Embedding “and for enterprise?” on its own retrieves nothing useful — the words that carry the meaning are two turns back.

The standard fix is a rewrite step: a cheap model call that folds the recent conversation into one self-contained query (“refund window for enterprise customers”) before you embed it. It costs a small-model round trip per turn, which is real latency you’re adding in front of the main call, so route it to your smallest tier and skip it entirely on the first turn of a conversation where there’s nothing to fold in.

Retrieval as an agent tool

For a conversational agent, don’t bolt retrieval onto every message — expose it as a tool the agent calls when it needs facts. Then the agent decides: chit-chat needs no lookup, but “what’s our refund window?” triggers a SearchKnowledgeBase("refund policy") call. This is cleaner and cheaper than retrieving on every turn, and it composes naturally with the agent’s other tools.

Where RAG quality actually comes from

Most “RAG isn’t working” problems aren’t the model — they’re retrieval. Watch these:

• Chunking — too big and matches are diluted; too small and context is lost. Tune it against real questions.
• Retrieval quality — if the right chunk isn’t in the top-K, the model can’t use it. Consider hybrid search (keyword + vector) and re-ranking for hard cases.
• Grounding discipline — without a firm “only use the context” instruction, the model blends its own training data back in and you lose the guarantee.
• Freshness — re-index when documents change, or your agent cites last quarter’s policy.

Measure retrieval separately from the answer

This is the single highest-leverage habit in RAG work, and most teams skip it. When an answer is wrong you have two suspects — retrieval fetched the wrong passages, or the model misused the right ones — and one output in which to tell them apart. You can’t, so you tune the prompt, because the prompt is the thing that’s easy to edit. Weeks disappear this way while the actual defect sits in the splitter.

Build a small labelled set instead: thirty or so real questions, each tagged with the chunk (or document) that genuinely answers it. Then measure whether that chunk appears in your top-K at all, without involving the model. If the right chunk is missing, no amount of prompt work will save the answer, and you now know to look at chunking, embeddings or filtering. If it’s present and the answer is still wrong, the problem is downstream and prompt work is exactly the right response. Testing and evaluating agents covers wiring this into a scored regression set — the .NET evaluation libraries ship retrieval and groundedness evaluators that map onto precisely this split.

When top-K similarity isn’t enough

Pure vector search has a known weakness: it’s excellent at meaning and poor at exact tokens. Ask for error code SB-4021, a part number, a surname, or the specific phrase from a contract clause, and semantic similarity will cheerfully return passages about error codes rather than the one containing yours. Keyword search has the mirror-image weakness — it nails the exact token and misses the paraphrase entirely.

Hybrid search runs both and merges the rankings, and for corpora full of identifiers it’s less an optimisation than a requirement. Most managed stores support it directly; on pgvector you’re combining a vector operator with full-text search in one query.

Re-ranking is the other lever. Retrieve more candidates than you need — twenty rather than five — then score those twenty against the query with a model that reads the query and passage together rather than comparing pre-computed vectors, and keep the best five. It’s more accurate than embedding similarity because nothing was compressed into a vector before the comparison. It also adds a call to the critical path, so it belongs on the queries that need it rather than on every turn.

Both of these are ways of buying precision, and neither is worth adding until you’ve measured that precision is what you’re short of. Reaching for a re-ranker to fix a chunking problem is a common and expensive detour.

What it costs, in latency and money

Indexing is a one-off embedding call per chunk, and embedding models are among the cheapest things you can call — the bill for a corpus of tens of thousands of chunks is usually unremarkable. The exception is the re-index, which is why the embedding-model decision above matters so much: it converts a cheap one-off into a recurring one.

Query time is where the cost shows up in a way users feel. A grounded answer is at minimum an embedding call plus a vector search plus the main model call, and adding conversational rewriting and re-ranking makes it four or five sequential network hops before a single token reaches the browser. The main model call still dominates, but you’ve moved the floor. If perceived latency matters, stream the response so the retrieval delay is hidden behind the first tokens rather than added to a blank screen.

The token cost is the other half. Five chunks of a few hundred tokens each is a meaningful addition to every request’s input, and it’s paid on every turn where retrieval fires — which is the practical argument for exposing retrieval as a tool rather than running it unconditionally. Prompt caching helps with the stable system prompt but not with the retrieved passages, since those change per query. Cost control for production agents has the measurement side.

When not to build RAG

Retrieval is the right answer often enough that it gets applied reflexively, and three cases genuinely don’t want it.

Your whole corpus fits in the context window. If the knowledge is an employee handbook, a product spec and a pricing page, put all of it in the system prompt and use prompt caching. You get perfect recall with no index, no embedding model, no re-index job and no retrieval failures — a class of bug you’ve simply chosen not to have. Reach for retrieval when the corpus stops fitting, not before.

The answer lives in structured data. “How many orders shipped late last month” is not a retrieval problem, and no chunking strategy will make it one. It’s a query, and the agent should be generating one — text-to-SQL is the pattern. Teams sometimes embed database rows as sentences and are then puzzled that aggregate questions fail. Similarity search cannot count.

There’s one document. Chatting with a single PDF doesn’t need a vector store at all; for anything short you can pass the whole thing, and chat with a PDF covers the middle ground. Standing up infrastructure for one file is work you can decline.

The related decision — whether the problem calls for retrieval or for training the model — is in RAG vs fine-tuning. The short version: missing knowledge is a retrieval problem, wrong behaviour is not.

Note: exact interfaces (IEmbeddingGenerator, Microsoft.Extensions.VectorData) are still maturing. Verify against the current .NET AI docs; the pipeline — chunk, embed, store, retrieve top-K, ground the prompt — is stable across providers and versions.

Takeaway

RAG is what turns a general-purpose agent into one that knows your world. The recipe is consistent: split documents into chunks, embed them into a vector store, and at query time retrieve the closest snippets and instruct the model to answer only from them. Expose retrieval as a tool so the agent fetches knowledge only when it needs it — and remember that when RAG underperforms, the fix is almost always in retrieval, not the model.

Next: RAG vs fine-tuning if you are weighing this against training a model, or agentic RAG in .NET if one search per question is not answering them.

---

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.