AAgentic.NET
.NETOllamaLocal LLMC#

Running a Local LLM in .NET with Ollama

Run an LLM locally and call it from C# with Ollama — no API key, no cloud, no per-token cost. How to connect .NET to a local model and when local makes sense.

MB
Manikandhan Balakrishnan
·3 min read

Sometimes you don’t want to send data to a cloud API — for privacy, for offline work, to avoid per-token costs while prototyping, or just to experiment freely. Ollama lets you run open-weight models locally with one command, and because it exposes an OpenAI-compatible API, calling it from C# is nearly identical to calling OpenAI. This guide connects .NET to a local model and covers when local actually makes sense.

Step 1: Install Ollama and pull a model

Download Ollama for your OS from ollama.com, then pull and run a model:

ollama pull llama3.2      # or another open-weight model
ollama run llama3.2       # test it in the terminal

Ollama now serves a local API at http://localhost:11434. Your machine is the whole backend — no account, no key, no network.

Step 2: Call it from C# (the OpenAI-compatible way)

Because Ollama mimics the OpenAI API, you can use the official OpenAI SDK and just point it at localhost with any placeholder key. If you’ve read how to call OpenAI from C#, this will look familiar:

using OpenAI.Chat;
using System.ClientModel;

var client = new ChatClient(
    model: "llama3.2",
    credential: new ApiKeyCredential("ollama"),  // any non-empty value
    options: new() { Endpoint = new Uri("http://localhost:11434/v1") });

var completion = await client.CompleteChatAsync("Explain dependency injection in one paragraph.");
Console.WriteLine(completion.Content[0].Text);

The only differences from a cloud call: the endpoint points at localhost:11434/v1, the model name is your local model, and the key is a placeholder Ollama ignores.

Step 3: Or use Microsoft.Extensions.AI

If you want to swap between local and cloud without changing your app code, go through the IChatClient abstraction. There’s a dedicated Ollama client, and the OpenAI-compatible client also works — either way your downstream code (and any agent built on IChatClient) is identical whether the model runs locally or in the cloud. That portability is the real payoff: prototype against a free local model, then flip a config value to a hosted model for production.

When local actually makes sense

Local isn’t automatically better — it’s a trade-off. Choose it when:

  • Privacy/compliance requires that data never leaves your infrastructure.
  • Offline or air-gapped environments rule out a cloud API.
  • Cost during development matters — iterate freely with zero per-token charges.
  • Full control over the model and versioning is a hard requirement.

Stick with a hosted model when you need top-tier capability (the best frontier models aren’t open-weight), no infrastructure to manage, or elastic scale — a laptop GPU won’t serve production traffic. Many teams do both: local for dev, hosted for production, switching via the IChatClient config. See choosing a model for the full trade-off.

A note on hardware

Local model quality is bounded by your hardware. Small models (a few billion parameters) run comfortably on a modern laptop; larger, more capable models want a serious GPU and lots of VRAM. Start with a small model to validate your code, and size up only if quality demands it — the same “start small” advice that applies to cloud models applies doubly here.

Note: SDK options for pointing a client at a custom endpoint evolve; verify against the OpenAI .NET SDK and Ollama docs for your versions. The approach — Ollama’s OpenAI-compatible endpoint on localhost, called via a standard client — is stable.

Takeaway

Running an LLM locally in .NET is genuinely easy: install Ollama, pull a model, and point the standard OpenAI client (or Microsoft.Extensions.AI) at localhost:11434. You get privacy, offline capability, and zero token cost — bounded by your hardware. Build on IChatClient and you can develop against a free local model and deploy against a hosted one without touching your app code.

Next: running the whole local stack with Aspire to start Ollama and a vector store with one keypress.

Have a correction or a topic you want covered? Email mani.bc72@gmail.com.

Related articles