Chunking Strategies for RAG (with .NET Examples)

When RAG gives vague or wrong answers, people blame the model. Usually the real culprit is chunking — how you split documents before embedding them. Chunk badly and retrieval returns diluted or fragmented context, and no model can answer well from bad context. This article covers chunking strategies that actually work.

Why chunking matters so much

You embed chunks, not whole documents, and retrieval pulls back the chunks nearest the question. So a chunk is the unit of retrieval. If a chunk is:

• Too big, its embedding averages several topics together — the meaning blurs, and a match brings in irrelevant text.
• Too small, it loses the surrounding context needed to make sense — a sentence retrieved without its paragraph.

The goal is chunks that are self-contained and single-topic — big enough to stand alone, small enough to be focused.

The reason “too big” hurts is worth understanding properly, because it isn’t obvious. An embedding is a single fixed-length vector regardless of how much text you feed it. Embed one paragraph about refund windows and you get a vector that points firmly at “refunds”. Embed a 3,000-word page covering refunds, shipping, warranty and contact details and you get a vector that points at the average of all four — which is to say, at nothing in particular. It will rank mediocrely for every one of those queries and win for none of them.

What bad chunking looks like from the outside

You rarely get an error. You get one of these three symptoms, and recognising them tells you which end of the problem you’re on.

The answer says the information isn’t available when you can see it sitting in the source document. That’s a retrieval miss — the right chunk existed but ranked below the cut-off, usually because it was buried inside a large multi-topic chunk.

The answer is confidently wrong in a specific way: it mixes two products, two versions, or two customers. That’s boundary bleed — a chunk spans a section break, so the retrieved text contains a heading for one thing and the details of another.

The answer is vague and hedging, restating the question without committing. That’s fragment retrieval — the chunks came back too small to contain a complete thought, and the model is doing its best with three disconnected sentences.

All three get blamed on the model. None of them are the model.

Strategy 1: Fixed-size with overlap (the baseline)

Split into fixed-size chunks (by tokens or characters) with a bit of overlap so a thought that straddles a boundary isn’t lost:

IEnumerable<string> ChunkFixed(string text, int size = 500, int overlap = 75)
{
    for (int start = 0; start < text.Length; start += size - overlap)
        yield return text.Substring(start, Math.Min(size, text.Length - start));
}

Simple, works surprisingly well, and a fine default. Start here: roughly 300–500 tokens with 10–15% overlap.

Note the unit mismatch hiding in that sentence, because it catches almost everyone. The code above counts characters; the recommendation is in tokens. For ordinary English prose a token averages somewhere around four characters, so 500 characters is nearer 125 tokens — a quarter of what you intended. The ratio is much worse for code, for JSON, and for languages that don’t use a Latin alphabet, where a single character can consume a token or more. If you size by characters and validate on English marketing copy, your chunks will be four times too small the day someone indexes a Japanese manual.

Count tokens properly:

// dotnet add package Microsoft.ML.Tokenizers
// dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase
using Microsoft.ML.Tokenizers;

// Create once and reuse — construction is expensive.
private static readonly Tokenizer Tok = TiktokenTokenizer.CreateForModel("gpt-4o");

static int CountTokens(string text) => Tok.CountTokens(text);

Use the tokenizer that matches the model doing the embedding, and check your embedding model’s maximum input length. Exceeding it is a nasty failure because it is often quiet: some pipelines truncate the input and embed only the first portion, so the chunk is indexed under the meaning of its opening lines and everything after the cut is invisible to search forever. Nothing errors. Retrieval simply never finds it.

The real cost of overlap

Overlap is presented as free insurance. It isn’t, and knowing the price stops you setting it to 50 percent “just to be safe”.

Overlap multiplies your corpus. Fifteen percent overlap means fifteen percent more chunks to embed, store, and search — a one-off embedding cost and a permanent storage and index cost. That’s usually acceptable. The part that isn’t acceptable is what heavy overlap does to your results: near-duplicate chunks compete for the same slots. Ask for the top five chunks and you can get five overlapping windows of the same paragraph, having spent your entire context budget on one idea while the second relevant section never appears.

Ten to fifteen percent is the sweet spot for prose. Go lower for structure-aware splitting, where boundaries are already meaningful — if you’re splitting on paragraphs, a thought rarely straddles the break, and overlap buys you very little.

Strategy 2: Structure-aware splitting (usually better)

Documents have natural boundaries — paragraphs, headings, sections. Splitting on those keeps each chunk coherent instead of cutting mid-sentence:

• Split on paragraphs or headings first.
• If a section is too big, sub-split it (falling back to fixed-size within the section).
• If a paragraph is tiny, merge it with its neighbour.

This “respect the structure, then size-bound” approach beats blind fixed-size splitting for most real documents (Markdown, docs, articles).

The general form is a recursive splitter: try the most meaningful separator first, and only fall back to a cruder one when a piece is still too large.

static readonly string[] Separators = ["\n## ", "\n\n", "\n", ". ", " "];

IEnumerable<string> Split(string text, int maxTokens, int depth = 0)
{
    if (CountTokens(text) <= maxTokens || depth >= Separators.Length)
    {
        yield return text;
        yield break;
    }

    var parts = text.Split(Separators[depth], StringSplitOptions.RemoveEmptyEntries);

    foreach (var buffer in Coalesce(parts, Separators[depth], maxTokens))
        foreach (var chunk in Split(buffer, maxTokens, depth + 1))
            yield return chunk;
}

Coalesce is the part that matters and the part most implementations skip: after splitting, greedily re-join adjacent pieces while they still fit under the limit. Without it, splitting on "\n\n" gives you one chunk per paragraph, and a document of one-line paragraphs produces hundreds of useless fragments. Splitting is the easy half; re-assembling to a target size is what makes the output usable.

Give every chunk its context back

This is the highest-return improvement on this page and the one most pipelines are missing.

A chunk taken from the middle of a document is full of references to things it no longer contains. “This limit applies only to enterprise plans” is meaningless when the chunk doesn’t say which limit. A paragraph beginning “It must be renewed annually” embeds as a vector about renewal in the abstract, matching nothing anyone would actually ask.

The fix is to prepend the document’s identity and heading path to each chunk before you embed it:

var contextualised = $"""
    Document: {doc.Title}
    Section: {string.Join(" > ", headingPath)}

    {chunkText}
    """;

A chunk that reads “Contoso Billing Guide > Refunds > Enterprise — This limit applies only to enterprise plans…” is a genuinely self-contained unit. It embeds near questions about enterprise refund limits, which is exactly what you wanted. The cost is a handful of tokens per chunk and no additional complexity.

A stronger version of the same idea uses a cheap model to write a one-sentence summary of how each chunk fits into its parent document and prepends that instead. It works well, and it costs one model call per chunk at index time. That’s a real bill on a large corpus, so treat it as an upgrade you apply after the free version — heading paths — has been measured.

Strategy 3: Match the chunk to the content

Different content wants different chunking:

• Prose/docs → paragraph/section-based.
• Code → by function or class, not arbitrary lines.
• Tables/structured data → keep rows with their headers.
• FAQs → one Q&A pair per chunk (they’re already perfectly chunked).

Tables deserve a specific warning because they break silently. Split a forty-row table by size and every chunk after the first loses the header row, leaving you with columns of numbers whose meaning is gone. The retrieved context then contains figures with no labels, and the model will happily attribute them to the wrong column. Either keep small tables whole, or repeat the header row at the top of each chunk. A short serialised form — one line per row, with column names inlined — often retrieves better than the original layout, because the embedding then contains the words a user would search for rather than pipe characters and whitespace.

Code has the mirror problem. A function split across two chunks yields two halves that are individually meaningless, and the imports at the top of the file — which say what the code actually is — end up in a chunk of their own that matches nothing. Split on declaration boundaries, and prepend the file path and enclosing type to each chunk for the same reason you prepend headings to prose.

Tuning against real questions

There’s no universal best chunk size — tune it against your data and your questions. Build a small set of real questions with known answers, then try a couple of chunking configs and measure whether the right chunk shows up in the retrieved results. This is retrieval evaluation, and it’s the fastest way to fix “RAG isn’t accurate.” (It pairs with the testing/eval mindset.)

Be concrete about the measurement, because “seems better” is not a result. Take thirty real questions, note for each one which chunk should be retrieved, then measure how often that chunk appears in the top five results. That single number — recall at five — is what a chunking change moves. Measure it before you change anything so you have a baseline, then change one variable at a time. Thirty questions is enough to see a real difference and small enough that you’ll actually build the set, which matters more than statistical purity.

Re-run it whenever the corpus changes materially. A chunking strategy tuned on tidy Markdown documentation behaves quite differently once someone adds a folder of PDFs exported from PowerPoint.

When chunking is not the problem

Diminishing returns arrive faster than people expect, and it’s worth knowing when to stop tuning splitters.

If the right chunk is in your top twenty but not your top five, chunking is fine — ranking is the problem, and a reranking pass over the candidates will do more than any splitter change. If retrieval fails only on questions phrased differently from the source text, that’s a vocabulary mismatch, better solved with hybrid search combining keyword and vector matching, or by rewriting the query before searching. And if the retrieved chunks are correct but the answer still isn’t, chunking has done its job and the problem is in the prompt or the model.

There are also documents that shouldn’t be chunked at all. If a document is short enough to fit comfortably in the context window and your corpus is small, retrieving whole documents is simpler and strictly better. Chunking exists to work around a size constraint; where that constraint doesn’t bind, you’re adding failure modes for nothing.

Store metadata with each chunk

Keep the source, section, and position alongside each chunk (and its embedding in pgvector). You’ll use it to cite sources in answers and to filter retrieval — “search only this document/section.”

Store more than you think you need. The source URL and heading path let you show citations, which is the single biggest driver of user trust in a RAG system. The ordinal position lets you fetch a chunk’s neighbours at query time — retrieve on the precise small chunk, then expand to the surrounding context before sending it to the model, which gets you precision in ranking and completeness in the answer. A content hash lets you re-index only what changed instead of rebuilding the corpus nightly. A last-modified date lets you filter out documents that are years stale, which is otherwise a permanent source of confidently outdated answers.

Note: exact tokenizer and splitter utilities vary; verify against your libraries. The principles — self-contained single-topic chunks, overlap, respect structure, tune against real questions — are stable and matter more than any specific tool.

Takeaway

Chunking quality determines RAG quality. Start with roughly 300–500 tokens and 10–15% overlap, counting tokens with a real tokenizer rather than guessing from character length. Move to structure-aware recursive splitting that respects headings and paragraphs and then coalesces the pieces back up to your target size. Prepend the document title and heading path to every chunk before embedding — it’s the cheapest large improvement available. Match the strategy to the content type, and treat tables and code as special cases rather than hoping. Then measure recall at five against thirty real questions, change one thing at a time, and stop tuning the splitter once the misses are ranking problems rather than retrieval ones. Get chunking right and most “RAG isn’t working” problems disappear — because the model was never the problem.

Next: choosing a vector database for .NET for where the chunks then live, and agentic RAG in .NET for when a single query over good chunks still is not enough.