Building Semantic Search in C# with Embeddings

Traditional search matches words. Ask a keyword search for “how to cancel my plan” and it’ll miss a document titled “ending your subscription” — same meaning, different words. Semantic search fixes that by matching on meaning using embeddings. This guide builds a minimal semantic search in C#.

How it works

Two phases:

1. Index — turn every document into an embedding vector and store it.
2. Search — turn the query into a vector, then find the stored vectors closest to it.

“Closest” means most similar in meaning. That’s the entire idea.

Step 1: Index your documents

Generate an embedding per document (or per chunk) and keep it with the original text:

using Microsoft.Extensions.AI;

record Doc(string Text, ReadOnlyMemory<float> Vector);

IEmbeddingGenerator<string, Embedding<float>> embedder = /* your embeddings client */;

async Task<List<Doc>> IndexAsync(IEnumerable<string> texts)
{
    var docs = new List<Doc>();
    foreach (var text in texts)
    {
        var e = await embedder.GenerateAsync(text);
        docs.Add(new Doc(text, e.Vector));
    }
    return docs;
}

Step 2: Search by similarity

Embed the query, then rank documents by cosine similarity and return the top matches:

async Task<List<string>> SearchAsync(string query, List<Doc> docs, int topK = 3)
{
    var q = (await embedder.GenerateAsync(query)).Vector;
    return docs
        .Select(d => (d.Text, Score: Cosine(q.Span, d.Vector.Span)))
        .OrderByDescending(x => x.Score)
        .Take(topK)
        .Select(x => x.Text)
        .ToList();
}

Now searching “how do I cancel” returns “ending your subscription” — because they mean the same thing, even with zero shared keywords.

The similarity function itself

Cosine is the piece everything above depends on, and it is short enough that there is no reason to pull in a library for it. It measures the angle between two vectors, ignoring their length, which is exactly what you want when comparing meaning:

static float Cosine(ReadOnlySpan<float> a, ReadOnlySpan<float> b)
{
    if (a.Length != b.Length)
        throw new ArgumentException("Vectors must have the same dimensions.");

    float dot = 0f, magA = 0f, magB = 0f;
    for (var i = 0; i < a.Length; i++)
    {
        dot  += a[i] * b[i];
        magA += a[i] * a[i];
        magB += b[i] * b[i];
    }

    var denominator = MathF.Sqrt(magA) * MathF.Sqrt(magB);
    return denominator == 0f ? 0f : dot / denominator;
}

The result runs from -1 to 1, and in practice everything you see from a modern embedding model lands between roughly 0.6 and 1.0. That matters when you go looking for a relevance threshold: 0.5 is not the midpoint you would expect it to be. Sample real queries against your own corpus and pick the cut-off from what you observe, because a threshold borrowed from a blog post will either let everything through or nothing.

One shortcut worth knowing: if your provider returns unit-length vectors — several do — then magA and magB are both 1, and cosine similarity reduces to the dot product. Skipping the two square roots is a measurable saving once you are scanning tens of thousands of vectors per query.

Before you hand-optimise that loop, note that .NET already ships a vectorised version. System.Numerics.Tensors exposes TensorPrimitives.CosineSimilarity(ReadOnlySpan<float>, ReadOnlySpan<float>), along with TensorPrimitives.Dot for the normalised case, and both use SIMD instructions for the architecture you are running on:

using System.Numerics.Tensors;

float score = TensorPrimitives.CosineSimilarity(query.Span, doc.Vector.Span);

Write the loop yourself when you want to understand what is happening, then use the built-in one in production. It throws ArgumentException on empty input, and — worth knowing before you debug it at midnight — it returns NaN if any element of either vector is infinity or NaN, rather than throwing. A single corrupt vector in your index therefore produces a score that loses every comparison silently instead of surfacing as an error. Validate vectors at index time, not at query time.

Batch your embedding calls

The indexing loop above makes one HTTP request per document, which is fine for a demo and painful for a real corpus. Embedding APIs accept arrays, and one request carrying a hundred texts is dramatically faster than a hundred requests:

async Task<List<Doc>> IndexAsync(IReadOnlyList<string> texts, int batchSize = 100)
{
    var docs = new List<Doc>(texts.Count);

    foreach (var batch in texts.Chunk(batchSize))
    {
        var embeddings = await embedder.GenerateAsync(batch);
        docs.AddRange(batch.Zip(embeddings, (t, e) => new Doc(t, e.Vector)));
    }

    return docs;
}

Keep the batch order intact — Zip relies on the provider returning embeddings in the order you sent the inputs, which every mainstream provider does, but it is worth an assertion if you are indexing something you cannot easily re-check.

Batching introduces the failure that ends most first indexing runs: rate limits. Fire a hundred batches concurrently at an embeddings endpoint and you will collect 429 Too Many Requests responses, and because the default behaviour of many HTTP stacks is to surface that as a failed task, you end up with a partially indexed corpus and no clear record of which documents made it. Bound your concurrency to something modest, retry 429s with exponential backoff honouring the Retry-After header when the provider sends one, and — most importantly — write each batch’s results to durable storage as it completes rather than accumulating a list in memory and saving at the end. Re-indexing a large corpus from scratch because the process died at 90% is an avoidable afternoon.

Batches also have a token limit, not just a count limit. A batch of a hundred short FAQ entries is fine; a batch of a hundred long document chunks may exceed the per-request token ceiling and fail as a unit. Size batches by estimated tokens rather than by item count if your documents vary in length.

Step 3: Scale it with a vector database

The in-memory approach above is perfect for hundreds of documents. For thousands or millions, computing similarity against every vector on each query is too slow. That’s what vector databases are for — they index vectors for fast approximate nearest-neighbour search. See vector search with pgvector for a Postgres-based option that most .NET teams can adopt without new infrastructure.

It helps to know roughly where the wall is, and the arithmetic is simple enough to do on the back of an envelope. A brute-force scan does one multiply-add per dimension per document, so 10,000 documents at 1,536 dimensions is about 15 million floating-point operations per query — fast enough that you will not notice it. A million documents at the same width is 1.5 billion operations per query, which you very much will.

Memory is usually the constraint that bites first. A 1,536-dimension vector of float is 6 KB, so 100,000 documents is roughly 600 MB of vectors alone, before the original text. On .NET those arrays land on the large object heap and stay there, so an in-memory index that “works fine” in testing can turn into GC pressure and a memory ceiling in a container with a 1 GB limit. Halving the dimension count halves both the memory and the scan time, which is the practical reason to care about it.

The other reason to move to a real index has nothing to do with speed: a restart. An in-memory index has to be rebuilt from scratch on every deploy, and rebuilding means re-embedding — paying the API bill again and waiting for it before the service can serve traffic. Persisting vectors is what turns embedding cost into a one-off rather than a per-deploy tax.

The filtering problem nobody warns you about

Real search is rarely pure similarity. Users want results from their own tenant, from the last year, in documents they have permission to read. The obvious implementation is to take the top 10 by similarity and then filter — and it is wrong.

Post-filtering means the constraint is applied after the ranking, so a user whose documents are 1% of the corpus will routinely get zero results back from a top-10 scan that contained nothing of theirs. The search does not error; it just returns nothing, and the bug report says “search is broken for one customer”.

The fix is to filter first and rank second: restrict the candidate set by metadata, then compute similarity over what remains. With an in-memory list this is a Where before the Select. With a vector database it is whatever pre-filtering the engine supports, and the quality of that support is the single most important thing to check when choosing one — an engine that can only post-filter will force you to over-fetch and hope. Security filters especially must be pre-filters, because “the permission check is applied to whatever survived ranking” is not a permission check.

Making results better

Semantic search quality comes down to a few knobs:

• Chunking — split long documents so a match points to the relevant passage, not a whole file.
• Hybrid search — combine keyword and vector search; keywords catch exact terms (product codes, names) that embeddings can blur.
• Re-ranking — for hard queries, re-score the top candidates with a more precise model.

Most “semantic search isn’t accurate” problems are chunking or retrieval issues, not the embedding model — the same lesson as RAG.

Hybrid search deserves particular attention, because it fixes the one failure mode that surprises people. Embeddings encode meaning, and identifiers have none: SKU-4417 and SKU-4471 are near-identical strings that a model maps to near-identical vectors, so a semantic search for one will happily return the other. Anything your users type verbatim — part numbers, error codes, surnames, version strings — needs a keyword path alongside the vector one. Run both and merge, rather than choosing.

When keyword search is still the right answer

Semantic search is not a strict upgrade. If your users search by identifier — order numbers, SKUs, error codes, file names — an inverted index does that job perfectly and an embedding model does it badly. If your corpus is a few hundred documents with consistent vocabulary, keyword search will match what people type because everyone uses the same words. And if you have to justify a result, keyword search can point at the matching term while a vector search can only say the angle was small, which matters more than it sounds in regulated or audited systems.

The honest failure mode of semantic search is confident irrelevance: there is always a nearest vector, so an unanswerable query returns the least-unrelated document with a respectable-looking score rather than nothing. Keyword search returns zero results and tells the user the truth. If you replace one with the other, add a relevance floor and a “no good match” path, or you have traded missed results for wrong ones.

Two operational details that bite later

Store the model name next to the vector. Vectors from different embedding models are not comparable, and if you upgrade models without re-indexing you get a search that returns confident nonsense with no error anywhere. A column recording which model produced each vector makes the mismatch obvious instead of mysterious.

Decide your dimension count deliberately. Several providers let you request a shorter vector, and going from 3072 dimensions to 1024 typically costs a small amount of accuracy while cutting both storage and scan time by roughly two-thirds. On a large corpus that is the difference between a search that fits in memory and one that does not — but measure the accuracy cost on your own data rather than trusting a general figure.

Note: IEmbeddingGenerator APIs are still settling; verify against the current .NET AI docs. The two-phase design — index vectors, search by nearest vector — is fundamental.

Takeaway

Semantic search matches meaning, not words: embed your documents, embed the query, and return the nearest vectors. It’s a couple dozen lines of C# for a small corpus, and a vector database when you scale. Add chunking, hybrid search, and re-ranking to sharpen results — and you have the retrieval engine that powers everything from site search to RAG-grounded agents.