Giving Your .NET Agent a Knowledge Base: RAG in C#

An agent knows what its model was trained on — and nothing about your product docs, policies, or codebase. Ask it about your internal refund policy and it will confidently invent one. Retrieval-Augmented Generation (RAG) is how you fix that: before the model answers, you fetch the relevant snippets from your own data and hand them to it as context. This article builds a practical RAG pipeline for a .NET agent.

If you’ve followed the tool-using agent tutorial, think of retrieval as a special kind of tool — one that supplies knowledge instead of taking an action.

The RAG pipeline in three parts

RAG has two phases. Indexing (done ahead of time): chop your documents into chunks, turn each into an embedding vector, and store them. Retrieval (at query time): embed the user’s question, find the nearest chunks, and inject them into the prompt. Let’s build both.

Step 1: Chunk and embed your documents

You can’t embed a whole 40-page PDF as one vector — retrieval would be too coarse. Split documents into overlapping chunks (a few hundred tokens each), then generate an embedding for every chunk. In .NET, the Microsoft.Extensions.AI abstractions give you an IEmbeddingGenerator you can point at Azure OpenAI or another provider:

using Microsoft.Extensions.AI;

IEmbeddingGenerator<string, Embedding<float>> embedder = /* Azure OpenAI embeddings client */;

foreach (var chunk in ChunkDocuments(docs))   // your splitter
{
    var embedding = await embedder.GenerateAsync(chunk.Text);
    await store.UpsertAsync(chunk.Id, embedding.Vector, chunk.Text, chunk.Metadata);
}

A little overlap between chunks (say 10–15%) keeps a sentence that straddles a boundary from being lost. Store the original text and useful metadata (source, section, URL) alongside the vector — you’ll want them at answer time for grounding and citations.

Step 2: Store the vectors

You need a store that does similarity search — “find the vectors nearest this one.” Options for .NET teams:

• A dedicated vector database (Qdrant, Milvis, Pinecone) — purpose-built, scales well.
• A database you already run — pgvector on PostgreSQL, or vector search in Azure AI Search, MongoDB, or Redis. Often the pragmatic choice: no new infrastructure.

Microsoft.Extensions.VectorData provides a common abstraction over many of these, so your retrieval code doesn’t hard-couple to one vendor — you can start on Postgres and move later without rewriting.

Step 3: Retrieve and ground the answer

At query time, embed the question, pull the top matches, and build a prompt that instructs the model to answer only from those snippets:

var queryVector = (await embedder.GenerateAsync(question)).Vector;
var hits = await store.SearchAsync(queryVector, topK: 5);

var context = string.Join("\n\n", hits.Select(h => $"[{h.Source}] {h.Text}"));

var prompt = $"""
    Answer the question using ONLY the context below. If the answer isn't in
    the context, say you don't know. Cite the [source] you used.

    Context:
    {context}

    Question: {question}
    """;

var answer = await agent.RunAsync(prompt);

That instruction — “answer only from the context, and say you don’t know otherwise” — is the heart of RAG. It’s what converts a confident hallucinator into a system that stays grounded in your data and admits its limits.

Retrieval as an agent tool

For a conversational agent, don’t bolt retrieval onto every message — expose it as a tool the agent calls when it needs facts. Then the agent decides: chit-chat needs no lookup, but “what’s our refund window?” triggers a SearchKnowledgeBase("refund policy") call. This is cleaner and cheaper than retrieving on every turn, and it composes naturally with the agent’s other tools.

Where RAG quality actually comes from

Most “RAG isn’t working” problems aren’t the model — they’re retrieval. Watch these:

• Chunking — too big and matches are diluted; too small and context is lost. Tune it against real questions.
• Retrieval quality — if the right chunk isn’t in the top-K, the model can’t use it. Consider hybrid search (keyword + vector) and re-ranking for hard cases.
• Grounding discipline — without a firm “only use the context” instruction, the model blends its own training data back in and you lose the guarantee.
• Freshness — re-index when documents change, or your agent cites last quarter’s policy.

Note: exact interfaces (IEmbeddingGenerator, Microsoft.Extensions.VectorData) are still maturing. Verify against the current .NET AI docs; the pipeline — chunk, embed, store, retrieve top-K, ground the prompt — is stable across providers and versions.

Takeaway

RAG is what turns a general-purpose agent into one that knows your world. The recipe is consistent: split documents into chunks, embed them into a vector store, and at query time retrieve the closest snippets and instruct the model to answer only from them. Expose retrieval as a tool so the agent fetches knowledge only when it needs it — and remember that when RAG underperforms, the fix is almost always in retrieval, not the model.

---

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.