Running a Local LLM in .NET with Ollama

Sometimes you don’t want to send data to a cloud API — for privacy, for offline work, to avoid per-token costs while prototyping, or just to experiment freely. Ollama lets you run open-weight models locally with one command, and because it exposes an OpenAI-compatible API, calling it from C# is nearly identical to calling OpenAI. This guide connects .NET to a local model and covers when local actually makes sense.

Step 1: Install Ollama and pull a model

Download Ollama for your OS from ollama.com, then pull and run a model:

ollama pull llama3.2      # or another open-weight model
ollama run llama3.2       # test it in the terminal

Ollama now serves a local API at http://localhost:11434. Your machine is the whole backend — no account, no key, no network.

One thing to understand before you go further: the tag you pull is not just a model, it’s a specific quantisation of that model. The default tag for most models is a 4-bit build chosen to fit comfortably in consumer memory, and you can request others explicitly by tag. That choice is the main dial between quality, speed and memory footprint, and it’s the reason a local model can feel noticeably worse than the same model name you’ve read about elsewhere. When a local answer disappoints, check whether you’re comparing a heavily quantised 3-billion-parameter build against a frontier hosted model — because you usually are.

Step 2: Call it from C# (the OpenAI-compatible way)

Because Ollama mimics the OpenAI API, you can use the official OpenAI SDK and just point it at localhost with any placeholder key. If you’ve read how to call OpenAI from C#, this will look familiar:

using OpenAI.Chat;
using System.ClientModel;

var client = new ChatClient(
    model: "llama3.2",
    credential: new ApiKeyCredential("ollama"),  // any non-empty value
    options: new() { Endpoint = new Uri("http://localhost:11434/v1") });

var completion = await client.CompleteChatAsync("Explain dependency injection in one paragraph.");
Console.WriteLine(completion.Content[0].Text);

The only differences from a cloud call: the endpoint points at localhost:11434/v1, the model name is your local model, and the key is a placeholder Ollama ignores.

Step 3: Or use Microsoft.Extensions.AI

If you want to swap between local and cloud without changing your app code, go through the IChatClient abstraction. There’s a dedicated Ollama client, and the OpenAI-compatible client also works — either way your downstream code (and any agent built on IChatClient) is identical whether the model runs locally or in the cloud. That portability is the real payoff: prototype against a free local model, then flip a config value to a hosted model for production.

The four things that surprise people first

The code above works on the first try, which is exactly why the next few problems catch teams off guard. All four are configuration rather than code, and all four look like bugs in your app.

The first request after a pause is slow. Ollama keeps a model resident for five minutes after its last use and then unloads it. The next call has to pull several gigabytes of weights back into memory before a single token is generated, so a request that normally takes two seconds takes considerably longer — and it happens exactly when you come back from lunch and conclude the model got worse. You can change this per request with the keep_alive parameter, or globally with the OLLAMA_KEEP_ALIVE environment variable; setting it to a long duration keeps the model pinned at the cost of holding that memory permanently. For a dev machine, pinning is usually right. For a shared box running several models, it isn’t.

The context window is smaller than the model’s advertised maximum. Ollama’s default context length has historically been 4,096 tokens regardless of what the model itself supports, and newer versions may size it from available memory instead. Either way, don’t assume — set OLLAMA_CONTEXT_LENGTH explicitly, or pass the per-request option, so you know what you’re getting. This matters because exceeding the window doesn’t throw. Older turns simply fall out of the transcript, and the symptom is an assistant that has quietly forgotten the beginning of the conversation or the document you pasted in. If you’re doing RAG against a local model, this is the failure that will bite you: retrieved chunks silently truncated, and an answer confidently drawn from whatever survived.

Requests queue rather than run in parallel. OLLAMA_NUM_PARALLEL governs how many requests a single loaded model handles concurrently, and the default is small — one, in the current documentation. Fire ten requests at your local endpoint from a load test and they serialise; the ninth caller waits for the first eight. OLLAMA_MAX_LOADED_MODELS is the related dial, defaulting to three models resident at once (or three per GPU). Both are bounded by memory in the end, and raising them past what your hardware holds just trades queuing for thrashing. This is the single biggest reason a local setup that felt fast in a console app feels broken behind a web front end.

Localhost isn’t localhost from inside a container. Ollama binds 127.0.0.1:11434 by default, which means it accepts connections from the host machine only. Run your .NET app in Docker while Ollama runs on the host and you’ll get a connection refused — on Windows, “No connection could be made because the target machine actively refused it”. The fix is either to point the container at host.docker.internal instead of localhost, or to set OLLAMA_HOST=0.0.0.0:11434 so the server listens on all interfaces. Do the second one advisedly: an Ollama endpoint has no authentication whatsoever, so binding it to 0.0.0.0 on a machine reachable from your network hands anyone who finds it free use of your GPU.

The other error worth recognising is a 404 from what looks like a perfectly good call. That’s almost always a model-name mismatch — the string in your ChatClient has to match a tag you’ve actually pulled, llama3.2 and llama3.2:3b being different tags as far as the server is concerned. ollama list settles the argument in one command.

Tool calling and structured output work, conditionally

If you’re building an agent rather than a chat box, the question that matters is whether the local model can reliably emit tool calls and valid JSON. Ollama supports both, but support is per-model: the tag has to be a build trained for tool use, and even then the smaller ones are markedly less reliable at it than hosted frontier models. The failure isn’t an exception. It’s a model that describes the tool it would like to call in prose instead of emitting a call, or that emits one with a hallucinated parameter name, and your agent loop simply produces a chatty non-answer.

Test this early, because it determines whether local is viable for your use case at all. Write the smallest possible agent — one tool, one unambiguous prompt — and see whether the model reaches for it ten times out of ten. If it manages six, no amount of prompt tuning is going to make it a production dependency, and the honest answer is a larger local model or a hosted one. Reliability at tool selection scales with model size much more steeply than conversational quality does, which is why a small model can hold a pleasant conversation and still be useless as an agent.

The same caution applies to the compatibility layer generally. It covers the chat-completions surface that most .NET code uses, but it isn’t a byte-for-byte reimplementation of the OpenAI API, and an unusual parameter you depend on may not behave identically. Check the compatibility documentation before you build on anything beyond messages, streaming and tool calls.

When local actually makes sense

Local isn’t automatically better — it’s a trade-off. Choose it when:

• Privacy/compliance requires that data never leaves your infrastructure.
• Offline or air-gapped environments rule out a cloud API.
• Cost during development matters — iterate freely with zero per-token charges.
• Full control over the model and versioning is a hard requirement.

The cost argument deserves more scrutiny than it usually gets, because “free” is only true at the margin. Inference on your own laptop has no per-token charge, and for a developer iterating on prompts all day that genuinely saves money — prompt engineering is a loop you run hundreds of times, and running it against a hosted model is a bill you pay for work that never reaches a user. Self-hosting to serve production traffic is a different proposition entirely: now you’re paying for GPU instances that are idle most of the day, plus the engineering time to run them, and you’re comparing that against a hosted price that only accrues when someone actually asks something. The break-even sits at high, steady utilisation. Below it, hosted wins on cost as well as on capability, and the local case has to be made on privacy or offline requirements instead.

Stick with a hosted model when you need top-tier capability (the best frontier models aren’t open-weight), no infrastructure to manage, or elastic scale — a laptop GPU won’t serve production traffic. Many teams do both: local for dev, hosted for production, switching via the IChatClient config. See choosing a model for the full trade-off.

A note on hardware

Local model quality is bounded by your hardware. Small models (a few billion parameters) run comfortably on a modern laptop; larger, more capable models want a serious GPU and lots of VRAM. Start with a small model to validate your code, and size up only if quality demands it — the same “start small” advice that applies to cloud models applies doubly here.

The specific cliff to watch for is a model that doesn’t fit in VRAM. When it fits, generation runs on the GPU and feels quick. When it doesn’t, part of the model spills to system memory and CPU, and throughput drops by an order of magnitude — same model name, same code, output that now arrives at reading speed or slower. Nothing errors; it just crawls. If a model that behaved yesterday is suddenly glacial, check whether something else is holding VRAM, or whether a larger context window has pushed the key-value cache over the edge, because the context length you set is itself a memory cost that grows with the conversation.

There’s also a difference between “runs” and “runs while you work”. A model that saturates your GPU makes the rest of the machine unpleasant, and on a laptop it will heat up, throttle, and get slower over a long session — so a benchmark taken in the first minute won’t represent the tenth. If local inference is going to be part of your daily loop, a dedicated machine that you talk to over the network beats sharing the one you’re typing on, and OLLAMA_HOST is how you make that machine reachable.

Note: SDK options for pointing a client at a custom endpoint evolve; verify against the OpenAI .NET SDK and Ollama docs for your versions. The approach — Ollama’s OpenAI-compatible endpoint on localhost, called via a standard client — is stable.

Takeaway

Running an LLM locally in .NET is genuinely easy: install Ollama, pull a model, and point the standard OpenAI client (or Microsoft.Extensions.AI) at localhost:11434. You get privacy, offline capability, and zero token cost — bounded by your hardware. Build on IChatClient and you can develop against a free local model and deploy against a hosted one without touching your app code.

Next: running the whole local stack with Aspire to start Ollama and a vector store with one keypress.