Run Your Whole AI Agent Stack Locally with Aspire (Ollama + Qdrant + Traces)

Getting a .NET AI agent running on a new machine usually looks like this: install Ollama, pull a model, run a Qdrant container, copy an appsettings.Development.json from a colleague, discover the ports collide, and then find out someone’s API key expired. Multiply by every new joiner.

Aspire collapses that into F5. It is not an AI framework — it is orchestration and observability for the supporting cast around your agent, and that turns out to be exactly the painful part.

What you get

An AppHost project describes your local topology in C#, and Aspire starts everything, wires the connection strings, and gives you a dashboard with logs, traces and metrics across all of it.

For an agent, that topology is usually:

• an LLM (Ollama locally, a hosted model in production)
• a vector store for RAG
• your API and maybe a worker
• optionally a cache and a relational database

Two properties make this worth setting up rather than clever-but-optional. First, the AppHost is C#, so it refactors, autocompletes and fails at compile time — none of which a docker-compose.yml does. Second, the dashboard shows model calls on the same trace timeline as your HTTP and database spans, which is how you discover that the “slow LLM” was actually three sequential retrieval round-trips.

The AppHost

Add the integrations you need. Aspire ships a CLI for this:

aspire add qdrant

Then describe the stack:

var builder = DistributedApplication.CreateBuilder(args);

// Local inference. Aspire runs the Ollama container and pulls the model on startup.
var ollama = builder.AddOllama("ollama")
    .WithDataVolume();                      // keep pulled models across restarts

var chatModel = ollama.AddModel("llama3");

// Vector store for retrieval.
var qdrant = builder.AddQdrant("qdrant")
    .WithLifetime(ContainerLifetime.Persistent);   // Qdrant is slow to start; keep it warm

builder.AddProject<Projects.AgentApi>("agent-api")
    .WithReference(chatModel)
    .WithReference(qdrant)
    .WaitFor(qdrant);

builder.Build().Run();

That is the entire local environment. No compose file, no port allocation, no README section titled “Local setup (please read carefully)”.

A few details that save time:

• WithDataVolume() on Ollama is not optional in practice. Without it, every restart re-downloads the model. That is several gigabytes and a lot of goodwill.
• WithLifetime(ContainerLifetime.Persistent) on Qdrant keeps the container alive between debug sessions. Qdrant’s startup is slow enough that you notice it every single time otherwise.
• WaitFor(qdrant) stops your API racing the database on startup — the classic “works on the second run” bug.
• GPU: .WithGPUSupport() for Nvidia, or .WithGPUSupport(OllamaGpuVendor.AMD). On a laptop without one, expect small models to be slow but usable for wiring work.

The consuming project

The model resource name is derived from the model you added: AddModel("llama3") under a server named "ollama" produces the resource ollama-llama3. That derived name is the connection name you use.

Worth knowing before it confuses you: the naming rules strip the tag suffix, take the last path segment of a slash-separated name, and replace dots with hyphens — so phi3.5 becomes ollama-phi3-5. If a connection name is not resolving, that transformation is usually why.

var builder = WebApplication.CreateBuilder(args);
builder.AddServiceDefaults();

// Registers IChatClient over the Ollama model resource.
builder.AddOllamaApiClient("ollama-llama3")
       .AddChatClient();

// Embeddings for retrieval — same resource, different abstraction.
builder.AddOllamaApiClient("ollama-llama3")
       .AddEmbeddingGenerator();

Need more than one model — a small fast one for classification, a larger one for answers? Register them keyed:

builder.AddKeyedOllamaApiClient(connectionName: "chat");
builder.AddKeyedOllamaApiClient(connectionName: "embeddings");

and resolve with [FromKeyedServices("chat")].

The important thing about all of this: what lands in DI is IChatClient and IEmbeddingGenerator<string, Embedding<float>> from Microsoft.Extensions.AI — the same interfaces Azure OpenAI, OpenAI and Anthropic connectors register. Nothing downstream knows or cares that a container is doing the inference. See Microsoft.Extensions.AI explained for why that abstraction is the load-bearing piece.

Your agent code does not change

Because the abstraction is the same, the agent is written once:

public sealed class SupportAgent(IChatClient chat)
{
    private readonly AIAgent _agent = chat.AsAIAgent(
        instructions: "You answer support questions from the provided context only.",
        tools: [AIFunctionFactory.Create(LookupOrder)]);

    public async Task<string> AskAsync(string question, AgentSession session)
    {
        AgentResponse response = await _agent.RunAsync(question, session);
        return response.Text;
    }
}

Local Ollama in development, Azure OpenAI in production, and this class is untouched. That is the payoff.

One caveat, stated plainly: a small local model is a way to exercise your plumbing — tool schemas, retrieval, streaming, error paths. It is not a proxy for production answer quality. Run your evaluation set against the model you will actually deploy before you believe any quality number.

And check tool support before you commit to a local model. Not every small model handles function calling reliably, and an agent whose tools silently never fire is a confusing afternoon.

Where the local model and the hosted one diverge

The “same interface, different backend” story is true at the type level and leaky in four specific places. Knowing them up front saves the afternoon where everything works locally and nothing works in staging.

Context window. Small local models frequently have a much smaller usable context than the hosted model you will deploy against. A RAG prompt carrying eight retrieved chunks that fits comfortably in production can be silently truncated locally — and truncation tends to take your system prompt with it. The symptom is an agent that ignores its instructions only on your machine, which people diagnose as almost anything before they check the token count.

Tool calling. Worth repeating because it is the most commonly wasted day: many small models either do not support tool calling or support it unreliably, and the failure is not an exception you can catch. The model answers from its own knowledge, fluently and wrongly, and never emits a tool call at all. If you are testing whether your tools work, assert that the tool ran. Do not read the answer and conclude the plumbing is fine.

Structured output. Coaxing a small model into reliably schema-conformant JSON is markedly harder than getting a frontier model to do it. If your agent depends on structured output, expect more parse failures locally and make sure your retry-and-repair path is genuinely exercised rather than assumed. This one is arguably a feature: local development forces you to handle the failure you would otherwise discover in production.

Latency shape, inverted. A hosted model gives you meaningful network latency and fast inference. A local model gives you no network and, on a laptop CPU, a great deal of inference. Timings measured locally do not transfer in either direction. Use the local stack to find structural latency problems — the loop making five embedding calls, the retrieval that runs twice — and measure absolute numbers against the deployment you will actually ship.

The embedding trap

This one earns its own heading because it produces the most confusing failure of the lot. Embeddings from different models are not interchangeable. If you index your knowledge base locally with an Ollama embedding model and then query in production with something like text-embedding-3-small, the vectors are not comparable — different dimensionality, and even where the dimensionality happens to match, a different semantic space entirely.

With Qdrant you usually get a clean error, because a collection is created with a fixed vector size and an insert of the wrong length is rejected outright. That is the good outcome. Where the sizes coincidentally match, there is no error at all: retrieval runs, returns results, and the results are confident nonsense. Nothing in the stack can tell you this has happened.

Treat the embedding model as part of the index’s identity. Record which model produced a collection — in the collection name is crude and effective — and re-index whenever it changes. The same rule applies to a production model upgrade, which is the version of this problem that arrives with a change nobody thought was risky.

The part that pays for itself: traces

AddServiceDefaults() wires OpenTelemetry across the whole app. Add instrumentation to the chat client and every model call becomes a span:

builder.Services.AddChatClient(sp =>
    sp.GetRequiredService<IChatClient>()
      .AsBuilder()
      .UseOpenTelemetry(configure: o => o.EnableSensitiveData = true)
      .UseLogging()
      .Build());

EnableSensitiveData records prompts and completions in the trace — development only, obviously; it puts user input into your telemetry. But during development it answers the question you actually have, which is not “was the model slow” but “what exactly did the model receive?” Nine times out of ten the answer explains the bad output: the retrieved chunk was wrong, or the tool result got truncated, or the system prompt lost a line.

In the dashboard those spans sit on the same timeline as your HTTP requests and Qdrant queries. Nesting makes the shape of a request obvious — and a RAG request that looked like one slow LLM call frequently turns out to be five embedding calls in a loop. For the production version of this, see observability for AI agents with OpenTelemetry.

The workflow this enables is worth spelling out, because it inverts how most people debug an agent. When an answer is wrong, do not start with the prompt. Open the trace for that request and read it top to bottom: how long retrieval took, how many chunks came back, what text actually reached the model, which tools fired and with what arguments. Nine times in ten the defect is visible before you get as far as the model call — the query embedded was the raw user question rather than the rewritten one, retrieval hit an empty collection because the indexing job never ran, or a tool returned an error string that the agent then summarised as though it were data. Prompt engineering is where people start and rarely where the bug is.

The AppHost as a test fixture

The AppHost pays a second dividend that is easy to miss. Because your topology is a program rather than a document, a test can start it. The Aspire.Hosting.Testing package provides DistributedApplicationTestingBuilder, which launches the AppHost in the background, randomises proxied resource ports so several instances can run side by side, and tears everything down on dispose.

What that buys is an integration test running against the real Qdrant and a real model, rather than a mock IChatClient returning a canned string — which proves your mapping code compiles and nothing else. Testing an agent is awkward for reasons of its own, and non-determinism means you assert on shape rather than exact text: that the expected tool fired with the expected arguments, that retrieval returned something from the right collection, that a malformed model response takes the repair path instead of throwing. Those assertions are stable across model versions, and they catch the regressions that actually occur.

The caveat is obvious once you run it: these tests pull containers and perform inference, so they are slow and resource-hungry next to unit tests. Keep them as a small, deliberate suite over the critical paths rather than the default way you test everything, and be realistic about whether your CI agents have the memory to run a model at all.

Where Aspire stops

Be clear about the boundary. Aspire is local orchestration and dev-time observability. It is not a production runtime, not an agent framework, and not a replacement for your deployment story — you still ship to Container Apps, GKE or wherever you were already going.

Which is fine, because the problem it solves is real and specific: the distance between “clone the repo” and “the agent answers a question.” On an AI project that distance is unusually long — more moving parts than a typical CRUD service, and several of them are containers with awkward startup behaviour. Closing it to one keypress is worth an afternoon.

When it is not worth the afternoon

If your agent is one ASP.NET Core project calling a hosted model over HTTPS, with no vector store and no worker, Aspire is solving a problem you do not have. One project, one key in user secrets, F5 — bolting an AppHost onto that buys a dashboard and an extra project to maintain. The value curve starts somewhere around the second container.

Be honest about machine resources too. Running a model locally costs no money and a great deal of RAM, and a model that does not fit does not fail cleanly: it swaps, inference degrades from slow to unusable, and it takes the rest of your machine with it. On a laptop with integrated graphics, expect small models to be fine for wiring work and painful for anything resembling iteration. If most of the team is on hardware like that, the pragmatic setup is a shared hosted model for daily work and the local stack for offline days and for the moments you need to see exactly what went over the wire.

And Aspire is a .NET development-time tool. If half your services are Node or Python owned by other teams, the AppHost describes your slice while something else still describes theirs — you have removed compose from your repository, not from the organisation. Still worth having, just a smaller win than the demo implies.

Takeaway

Aspire does not make your agent smarter. It makes the ten things around your agent start together, wire themselves up, and report what they are doing on one timeline. Describe Ollama and Qdrant in the AppHost, consume them as IChatClient and IEmbeddingGenerator so the same code runs against a hosted model in production, and turn on OpenTelemetry so you can see the prompt that actually reached the model. Remember the two boundaries: a small local model validates plumbing, not answer quality, and Aspire is a development-time tool — production deployment is still your own.