AAgentic.NET
.NETAI AgentsCostPerformance

Caching LLM Responses in .NET

LLM calls are slow and cost money. Add caching to your .NET AI app — response caching and prompt caching — to cut latency and spend without losing quality.

MB
Manikandhan Balakrishnan
·3 min read

LLM calls are the slowest and most expensive part of an AI app — often a second or more of latency and real money per request. Yet a surprising fraction of those calls ask the same thing the app already answered. Caching is the easy win most teams skip. This article covers caching LLM responses in .NET.

Two kinds of caching

  • Response caching — you store the whole answer and skip the model entirely on a repeat. Cheapest and fastest (a cache hit costs nothing), but only safe for deterministic or near-deterministic queries.
  • Prompt caching — a provider feature that discounts the repeated, stable prefix of your prompt (long system prompt, tool definitions, retrieved context). You still call the model, but pay less.

Use both where they fit.

Response caching in C#

Wrap your model call with a cache keyed on the normalized input. .NET’s IMemoryCache (or a distributed cache like Redis) does the job:

async Task<string> AskCachedAsync(string question)
{
    var key = "llm:" + Normalize(question);   // lowercase/trim for better hit rate
    if (_cache.TryGetValue(key, out string? cached))
        return cached!;

    var answer = (await _chat.GetResponseAsync(question)).ToString();
    _cache.Set(key, answer, TimeSpan.FromHours(24));
    return answer;
}

For FAQ-style questions, classification, or any query with a stable answer, this turns a slow paid call into an instant free one.

Do it as middleware

The clean way is a decorator around IChatClient, so caching is transparent to your app code and composes with logging and telemetry:

// A caching IChatClient wraps the real one and short-circuits on a hit.
public class CachingChatClient(IChatClient inner, IMemoryCache cache) : IChatClient
{
    // Check cache -> on miss call inner -> store -> return.
}

Now every caller benefits without changing.

When NOT to cache

Caching assumes a repeated input should yield the same output. Skip it — or key more carefully — when:

  • The answer depends on fresh data (current stock, live RAG results) — cache the retrieval, not the final answer, or use a short TTL.
  • Responses should feel varied (creative writing, brainstorming).
  • The input includes user-specific context — include that context in the cache key, or you’ll serve one user another’s answer.

That last one is a real bug and a privacy risk: your cache key must include everything that changes the answer, including the user or tenant when relevant.

Prompt caching: structure for the discount

To benefit from provider prompt caching, put the stable content first (system prompt, tool schemas, fixed context) and the variable part (the user’s question) last. Providers cache the unchanging prefix, so repeated calls are billed at a steep discount for that portion — free performance if you structure prompts right. See cost control.

Note: provider prompt-caching behaviour and cache APIs vary; verify against your provider’s docs and the .NET caching APIs. The strategy — response cache for repeats, prompt-prefix structure for the provider discount, careful keys — is stable.

Takeaway

Caching is the most overlooked LLM optimization. Add response caching (via IMemoryCache/Redis, ideally as an IChatClient decorator) to turn repeated questions into instant, free answers — but key on everything that affects the output, including the user. Structure prompts with the stable part first to earn provider prompt-caching discounts. Together they cut both latency and spend with no quality cost, which is about as close to free lunch as production AI gets.

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.

Related articles