AAgentic.NET
.NETRAGEmbeddingsC#

Chunking Strategies for RAG (with .NET Examples)

Bad chunking is the #1 reason RAG underperforms. A practical guide to chunking strategies — size, overlap, and structure-aware splitting — for .NET RAG apps.

MB
Manikandhan Balakrishnan
·3 min read

When RAG gives vague or wrong answers, people blame the model. Usually the real culprit is chunking — how you split documents before embedding them. Chunk badly and retrieval returns diluted or fragmented context, and no model can answer well from bad context. This article covers chunking strategies that actually work.

Why chunking matters so much

You embed chunks, not whole documents, and retrieval pulls back the chunks nearest the question. So a chunk is the unit of retrieval. If a chunk is:

  • Too big, its embedding averages several topics together — the meaning blurs, and a match brings in irrelevant text.
  • Too small, it loses the surrounding context needed to make sense — a sentence retrieved without its paragraph.

The goal is chunks that are self-contained and single-topic — big enough to stand alone, small enough to be focused.

Strategy 1: Fixed-size with overlap (the baseline)

Split into fixed-size chunks (by tokens or characters) with a bit of overlap so a thought that straddles a boundary isn’t lost:

IEnumerable<string> ChunkFixed(string text, int size = 500, int overlap = 75)
{
    for (int start = 0; start < text.Length; start += size - overlap)
        yield return text.Substring(start, Math.Min(size, text.Length - start));
}

Simple, works surprisingly well, and a fine default. Start here: ~300–500 tokens with 10–15% overlap.

Strategy 2: Structure-aware splitting (usually better)

Documents have natural boundaries — paragraphs, headings, sections. Splitting on those keeps each chunk coherent instead of cutting mid-sentence:

  • Split on paragraphs or headings first.
  • If a section is too big, sub-split it (falling back to fixed-size within the section).
  • If a paragraph is tiny, merge it with its neighbour.

This “respect the structure, then size-bound” approach beats blind fixed-size splitting for most real documents (Markdown, docs, articles).

Strategy 3: Match the chunk to the content

Different content wants different chunking:

  • Prose/docs → paragraph/section-based.
  • Code → by function or class, not arbitrary lines.
  • Tables/structured data → keep rows with their headers.
  • FAQs → one Q&A pair per chunk (they’re already perfectly chunked).

Tuning against real questions

There’s no universal best chunk size — tune it against your data and your questions. Build a small set of real questions with known answers, then try a couple of chunking configs and measure whether the right chunk shows up in the retrieved results. This is retrieval evaluation, and it’s the fastest way to fix “RAG isn’t accurate.” (It pairs with the testing/eval mindset.)

Store metadata with each chunk

Keep the source, section, and position alongside each chunk (and its embedding in pgvector). You’ll use it to cite sources in answers and to filter retrieval — “search only this document/section.”

Note: exact tokenizer and splitter utilities vary; verify against your libraries. The principles — self-contained single-topic chunks, overlap, respect structure, tune against real questions — are stable and matter more than any specific tool.

Takeaway

Chunking quality determines RAG quality. Start with fixed-size chunks (~300–500 tokens, 10–15% overlap), move to structure-aware splitting on paragraphs and headings for coherence, and match the strategy to the content type. Then tune against real questions and store metadata for citations and filtering. Get chunking right and most “RAG isn’t working” problems disappear — because the model was never the problem.

Next: choosing a vector database for .NET for where the chunks then live.

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.

Related articles