Caching LLM Responses in .NET

LLM calls are the slowest and most expensive part of an AI app — often a second or more of latency and real money per request. Yet a surprising fraction of those calls ask the same thing the app already answered. Caching is the easy win most teams skip. This article covers caching LLM responses in .NET.

Two kinds of caching

• Response caching — you store the whole answer and skip the model entirely on a repeat. Cheapest and fastest (a cache hit costs nothing), but only safe for deterministic or near-deterministic queries.
• Prompt caching — a provider feature that discounts the repeated, stable prefix of your prompt (long system prompt, tool definitions, retrieved context). You still call the model, but pay less.

Use both where they fit.

Response caching in C#

Wrap your model call with a cache keyed on the normalized input. .NET’s IMemoryCache (or a distributed cache like Redis) does the job:

async Task<string> AskCachedAsync(string question)
{
    var key = "llm:" + Normalize(question);   // lowercase/trim for better hit rate
    if (_cache.TryGetValue(key, out string? cached))
        return cached!;

    var answer = (await _chat.GetResponseAsync(question)).ToString();
    _cache.Set(key, answer, TimeSpan.FromHours(24));
    return answer;
}

For FAQ-style questions, classification, or any query with a stable answer, this turns a slow paid call into an instant free one.

Why the in-memory version disappoints in production

That snippet is the right shape and the wrong storage, and the gap between them is where most caching efforts quietly die.

IMemoryCache lives inside one process. Run three replicas behind a load balancer and you have three caches, each seeing roughly a third of the traffic, so your hit rate is a third of what a shared cache would give you — and every deploy, scale event or pod restart throws all three away. It also competes for the same heap as everything else in your app; under memory pressure the cache is evicted first, precisely when you were relying on it to reduce load.

Then there is the hit rate itself. Exact-match caching on free-text questions performs far worse than people expect, because humans do not phrase things identically. “How do I cancel my subscription”, “how to cancel subscription” and “cancel my sub” are three cache misses. Lowercasing and trimming, as above, recovers a little. Collapsing whitespace and stripping trailing punctuation recovers a little more. None of it changes the fundamental fact that exact-match caching pays off on repeated inputs, and free text repeats far less than machine-generated input does.

That points at where response caching actually earns its keep: suggested questions, buttons, menu items, classification of a bounded label set, enrichment of records that repeat, and anything an automated caller sends. A support widget with six canned starter prompts will cache beautifully. An open-ended chat box will not. Measure your real hit rate before building anything elaborate — a day of logging normalized inputs and counting duplicates tells you whether this is a 40% saving or a 2% one.

Use a distributed cache (Redis, or anything behind IDistributedCache) so the cache is shared across replicas and survives restarts. You trade an in-process dictionary lookup for a network round trip of a millisecond or two, which against an LLM call of a second or more is not a trade you need to think about.

Do it as middleware

The clean way is a decorator around IChatClient, so caching is transparent to your app code and composes with logging and telemetry. Microsoft.Extensions.AI ships this: DistributedCachingChatClient sits in the pipeline, and when the same history and options come through again it returns the stored response instead of forwarding the call.

IChatClient client = baseClient
    .AsBuilder()
    .UseLogging(loggerFactory)       // outer: sees hits and misses alike
    .UseDistributedCache(cache)      // an IDistributedCache
    .Build();

Pipeline order matters more than it looks. The builder nests each stage inside the previous one, so anything registered after the cache is skipped entirely on a hit. Put logging or telemetry after it and a cache hit produces no trace at all, so your dashboards report latency and token counts only for misses — flattering, and wrong. Put function invocation after it and a hit replays a stored answer without ever executing the tool, which is fine if the tool reads stable reference data and badly wrong if it writes to a database. Decide deliberately what each layer is allowed to short-circuit; the ordering above keeps observability outside the cache so you can actually measure your hit rate.

If you need control over the key or over which requests are cacheable at all, the abstract CachingChatClient base class exposes the hooks: GetCacheKey to compute the key, and EnableCaching to decide per-request whether to bother. CoalesceStreamingUpdates controls whether a streamed response is stored as one combined response or as the original sequence of updates — the combined form is smaller and cheaper, at the cost of replaying the answer in one lump rather than token by token, which users notice.

One thing to understand before you switch it on: this caches on the whole message list plus options. In a multi-turn conversation, every turn carries all prior turns, so no two requests after the first are ever identical and the hit rate is essentially zero. That is correct behaviour — you cannot reuse turn five of one conversation in another — but it means the built-in cache pays off on stateless single-shot calls, not on chat. If your workload is chat, cache the expensive stateless pieces inside it: retrieval, classification, summarization of a fixed document.

When NOT to cache

Caching assumes a repeated input should yield the same output. Skip it — or key more carefully — when:

• The answer depends on fresh data (current stock, live RAG results) — cache the retrieval, not the final answer, or use a short TTL.
• Responses should feel varied (creative writing, brainstorming).
• The input includes user-specific context — include that context in the cache key, or you’ll serve one user another’s answer.

That last one is a real bug and a privacy risk: your cache key must include everything that changes the answer, including the user or tenant when relevant.

Everything that belongs in the key

It is worth being exhaustive here, because a cache-key omission does not fail loudly — it returns a plausible answer that happens to be somebody else’s, or last month’s, or the one from before you rewrote the system prompt.

The obvious inputs are the user’s text and the tenant or user identity for anything personalised. The ones people forget:

• The model id. Switch from a small model to a large one and, without the model in the key, you keep serving the small model’s answers from cache while paying nothing and wondering why quality didn’t improve.
• Everything in ChatOptions. Temperature, max tokens, response format, the tool list. A structured-output request and a free-text request with the same prompt are different calls.
• A version stamp for your system prompt. Change the prompt and every cached answer is stale, but the key is unchanged so you keep serving the old behaviour until the TTL expires. Bump a constant alongside the prompt and put it in the key; the old entries then age out harmlessly.
• A version stamp for your retrieved corpus, if you cache anything downstream of RAG. Re-index the knowledge base and the cache is now serving answers grounded in documents you have replaced.

The general rule: if changing it would change the answer, it goes in the key. Hash the composite rather than concatenating it, so keys stay a bounded length and you are not storing prompt text in Redis key names.

The stampede you’ll hit on launch day

A cold cache plus a burst of identical requests — everyone opening the same page, a batch job starting, a link going round the office — means every one of those requests misses, every one calls the model, and the cache is written the same value N times. You have paid N times for one answer and possibly tripped your rate limit doing it.

The fix is single-flight: let the first caller through and make the rest wait on its result. A SemaphoreSlim per key, or a ConcurrentDictionary<string, Task<string>> holding the in-flight task, both work. Keep it modest — it only matters where a genuinely hot key exists, and a lock keyed on unbounded user input is its own memory leak, so evict entries when their task completes.

Semantic caching: the tempting next step

Once exact-match caching disappoints, the obvious idea is to cache on meaning — embed the question, look for a stored question within some cosine similarity threshold, and serve its answer. It works, and it raises hit rates substantially on FAQ-shaped traffic. It also introduces a failure mode plain caching does not have: a near-miss serving a confidently wrong answer.

“Can I cancel my subscription?” and “Can I cancel my subscription for free?” are close in vector space and have different answers. So are “does this work on Windows?” and “does this work on Windows Server?”, or any pair distinguished by a negation, a version number, or a plan tier. Embeddings capture what text is about, not the details that flip the answer — so the threshold you pick is a direct trade between hit rate and wrong answers, and there is no setting that eliminates both.

If you go this way: set the threshold high, keep it configurable, log every semantic hit with both questions and the score so you can audit what it is actually matching, and restrict it to a domain where a near-miss is survivable. It does not belong in front of anything billing, legal or medical. And note the cost is no longer zero — you pay an embedding call and a vector search on every request, hit or miss, so the saving has to clear that floor.

Prompt caching: structure for the discount

To benefit from provider prompt caching, put the stable content first (system prompt, tool schemas, fixed context) and the variable part (the user’s question) last. Providers cache the unchanging prefix, so repeated calls are billed at a steep discount for that portion — free performance if you structure prompts right. See cost control.

The word doing the work is prefix. Matching is on an exact, byte-identical run from the very start of the prompt, so a single varying character near the front discards the discount for everything after it. The usual culprits are all things that felt harmless when someone added them:

• A timestamp or a “today’s date is…” line at the top of the system prompt. Now the prefix changes every day at best, every request at worst.
• The user’s name, tenant, or a request id injected into the system message for logging.
• Tool definitions serialized from a dictionary whose enumeration order isn’t stable, so the JSON differs run to run.
• Retrieved context placed before the system prompt rather than after it — retrieval varies per query, so nothing before it is ever shared.

Put all of that after the stable block, and put the longest genuinely fixed content first. The other half of the trick is that prefixes are only shared when calls actually repeat within the provider’s cache window, which is typically short — so this rewards steady traffic against one prompt version and does nothing for a low-volume endpoint called twice an hour.

Prompt caching also composes badly with aggressive prompt trimming. If you drop the oldest turns to control context length, you are changing the prefix each time you trim. Trimming from the middle — keeping the system prompt and early context intact and summarizing the middle of the conversation — preserves the cacheable prefix and costs you nothing in quality that trimming the front wasn’t already costing.

Note: provider prompt-caching behaviour, cache windows, discount rates and minimum prefix lengths vary by provider and change over time; verify against your provider’s docs and the .NET caching APIs. The strategy — response cache for repeats, prompt-prefix structure for the provider discount, careful keys — is stable.

Takeaway

Caching is the most overlooked LLM optimization. Add response caching (via IMemoryCache/Redis, ideally as an IChatClient decorator) to turn repeated questions into instant, free answers — but key on everything that affects the output, including the user. Structure prompts with the stable part first to earn provider prompt-caching discounts. Together they cut both latency and spend with no quality cost, which is about as close to free lunch as production AI gets.