The .NET job market changed quickly. Roles that said “microservices, Azure, SQL” in 2024 now say “experience integrating LLMs” in 2026, and a lot of very good engineers are walking into interviews unsure what the bar even is.
The bar is lower than the hype suggests and different from what people prepare for. Nobody is asking you to explain transformer attention. They are asking whether you can build an LLM feature that does not fall over — which is an engineering question, and you probably already have most of the instincts.
These are the questions being asked, grouped by round, with the answer that lands and a note on what is really being tested.
Fundamentals
1. What is a token, and why should you care?
A token is the unit a model reads and bills in — roughly ¾ of an English word, though code and non-English text tokenise less efficiently. You care for three reasons: cost is per token, the context window is measured in tokens, and latency scales with output tokens. Testing: whether your cost and latency intuitions have any grounding. See counting tokens in C#.
2. What is an embedding?
A fixed-length vector of floats representing the meaning of a piece of text, such that semantically similar texts land close together. It is what makes “how do I get my money back” match a document titled “Refund policy”. Testing: can you explain semantic search without hand-waving.
3. What happens when you exceed the context window?
The request fails, or the provider silently truncates — both bad. You handle it by budgeting: cap retrieved context, summarise or window older conversation turns, and count tokens before sending rather than catching the error afterwards. Testing: whether you have built something long-running or only a single-turn demo.
4. What does temperature do, and what do you set it to?
It controls sampling randomness. Near 0 for anything where correctness matters — extraction, classification, SQL generation, tool arguments. Higher only for genuinely creative output. Follow-up you should expect: “does temperature 0 make it deterministic?” No. It makes it much more consistent; batching, floating-point non-determinism and model updates mean identical output is not guaranteed.
5. Why can’t you unit-test an LLM feature the normal way?
Because the output is not deterministic, so exact-match assertions are useless. You test the deterministic parts normally — parsing, validation, tool execution, prompt assembly — and test model behaviour with an evaluation set: fixed inputs scored on properties like “contains the right figure”, “cites a source”, “refuses when it should”. Testing: this is the single strongest signal between someone who has shipped and someone who has demoed.
The .NET AI stack
6. What is Microsoft.Extensions.AI and why does it exist?
A set of provider-agnostic abstractions — chiefly IChatClient and IEmbeddingGenerator<string, Embedding<float>> — that do for AI providers what ILogger did for logging. Write against the interface, swap OpenAI for Azure OpenAI for a local Ollama model by changing a registration. Testing: do you know the .NET-native answer or only the raw OpenAI SDK. Full explanation here.
7. Why is that abstraction worth having in practice?
Three concrete payoffs: you can run local models in development and hosted ones in production without a code change; you can A/B a different model as a configuration experiment; and middleware — caching, telemetry, retries — composes onto the client for every provider at once:
IChatClient client = baseClient
.AsBuilder()
.UseDistributedCache(cache)
.UseOpenTelemetry()
.Build();Testing: whether you understand the decorator pattern is the actual point.
8. Semantic Kernel or the Microsoft Agent Framework?
Agent Framework for new work — it is the GA’d successor to both Semantic Kernel and AutoGen, and where Microsoft’s investment goes. Semantic Kernel is not deprecated and existing production apps do not need a panic migration. Testing: currency. Getting this wrong suggests knowledge that stopped updating in 2025. Details: SK vs Agent Framework and the migration walkthrough.
9. How do you create an agent in the Agent Framework?
Off the chat client, with tools passed in:
AIAgent agent = chatClient.AsAIAgent(
instructions: "You are a terse support agent.",
tools: [AIFunctionFactory.Create(GetOrderStatus)]);
AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync(question, session);Testing: have you actually written this, or only read about it. The give-away is knowing that RunAsync returns one AgentResponse with a .Messages list, not an async stream.
10. What is MCP?
The Model Context Protocol — an open standard for exposing tools and data to any AI client over a defined protocol, so a capability written once is callable by your agent, someone’s Python agent, or a Copilot session. Testing: awareness that the ecosystem is standardising. Building an MCP server in C#.
11. When would you use MCP rather than a plain C# function tool?
A local function when the capability belongs to one app; an MCP server when it needs to be shared across apps, teams or languages, or when a third party already publishes one. Do not stand up a protocol server for a method only your own agent calls. Testing: judgement, not knowledge. The full trade-off.
RAG
12. Explain RAG end to end.
Offline: chunk documents, embed the chunks, store vectors. At query time: embed the question, retrieve the nearest chunks, put them in the prompt as context, generate an answer grounded in them. Testing: baseline competence. You should be able to say this in thirty seconds.
13. Where does RAG usually go wrong?
Retrieval, not generation. The model is usually fine at answering from context it was given; the failure is being given the wrong context. Which is why you measure retrieval quality — is the right chunk in the top-k? — separately from answer quality. Testing: this is the senior/junior line in RAG questions.
14. How do you chunk documents?
By structure before size. Split on semantic boundaries — headings, sections, paragraphs — rather than every N characters, keep some overlap so a sentence spanning a boundary is not orphaned, and attach metadata (source, section, date) so you can filter and cite. Then tune against your eval set, because the right size is corpus-dependent. Testing: whether you know “512 tokens with 50 overlap” is a starting point and not an answer. Chunking strategies.
15. Which vector database?
The one your team already runs. Microsoft.Extensions.VectorData abstracts the store, so the decision is reversible — pgvector if you have Postgres, SQL Server 2025’s native VECTOR type if your data is there, Azure AI Search if you need hybrid keyword-plus-vector retrieval, Qdrant when you have genuinely outgrown those. Testing: pragmatism. Naming an exotic store unprompted is a mild red flag. Full comparison.
16. Why is pure vector search sometimes worse than keyword search?
Because embeddings capture meaning and are weak on literal strings. A user searching an error code or a part number wants exact matching, and semantic similarity returns things that are about the same topic instead. Hybrid retrieval — vector plus BM25, fused — fixes it. Testing: depth. Most candidates do not have this one.
17. RAG or fine-tuning?
RAG changes what the model knows; fine-tuning changes how it behaves. Missing knowledge is a RAG problem. Wrong tone or format is a fine-tuning problem — though usually a better prompt or structured output fixes it far more cheaply. Testing: whether you can resist the framing that these compete. The full decision framework.
18. How do you stop it answering from outside the provided context?
Instruct it explicitly to answer only from context and to say so when the answer is not there; keep retrieved passages clearly delimited; and evaluate the refusal case deliberately, because “did it correctly decline?” is a test most eval sets forget to include. It reduces the failure rate; it does not eliminate it. Testing: calibration — do you oversell the fix.
Tools and agents
19. How does tool calling actually work?
You send tool definitions — name, description, JSON schema — with the request. The model responds with a structured call rather than text. Your code executes it; the model never runs anything. You send the result back and it continues. Testing: the “your code executes it” part. Candidates who think the model runs the function have not built this.
20. What makes a tool description good?
It is a prompt, not a comment. The description and parameter descriptions are exactly what the model uses to decide whether and how to call your tool, so vague descriptions produce wrong calls. Write them like API docs for a capable but literal-minded reader. Testing: practical experience. This is the number-one cause of “my agent won’t use my tool”.
21. What do you do when the model calls a tool with bad arguments?
Validate, and return a descriptive error to the model rather than throwing. “OrderId must match the format ORD-nnnn” lets it correct itself on the next turn; an exception just fails the request. Cap the correction loop so it cannot cycle forever. Testing: whether you have debugged a real agent.
22. How do you stop an agent looping?
Bound it: a maximum number of turns, a token budget per conversation, and a timeout. Then alert when any of them trips, because a loop is a bug in your tools or prompt and the cap is a containment measure, not a fix. Testing: production instinct.
23. What is prompt injection and how do you defend against it?
An attacker puts instructions into content your agent processes — a document, a web page, a support ticket — and the model follows them, because it cannot reliably distinguish your instructions from data. You cannot fix this at the prompt layer. The defence is architectural: least privilege on every tool, human confirmation for consequential actions, treating all retrieved content as untrusted, and never granting an agent a capability whose worst-case misuse you have not accepted. Testing: security maturity, and increasingly a screening question in regulated industries. Securing AI agents.
24. When do you use multiple agents instead of one?
When responsibilities genuinely differ and each needs its own instructions and tools — and not before. Multi-agent adds latency, cost and debugging difficulty; one well-equipped agent handles more than people expect. Testing: resistance to architecture astronautics. Graph workflows when you do need it.
Production
25. How do you control cost?
Measure first — per-feature token spend, not one monthly bill. Then: route easy requests to a cheaper model, cache aggressively, trim retrieved context, cap output tokens, and set per-tenant budgets so one runaway workload cannot consume the quarter. Testing: whether you think about unit economics. Cost control in production.
26. How do you handle provider failures?
Like any unreliable dependency, with one twist: retry with jitter and honour Retry-After on 429s, use a circuit breaker so a provider outage does not exhaust your threads, and have a fallback — a cheaper model, a cached answer, or a graceful degradation. The twist is that LLM calls are slow, so a naive retry policy can triple a tail latency that was already seconds. Testing: do you apply normal distributed-systems discipline. Resilient LLM calls with Polly.
27. What do you monitor?
Token usage and cost per request, latency including time-to-first-token, error and rate-limit rates, tool call frequency and failure rate, and quality proxies — refusal rate, retry rate, user thumbs-down. Trace prompts and completions with OpenTelemetry so a bad answer can be reconstructed. Testing: seriousness. “We log errors” is not an answer. Observability with OpenTelemetry.
28. How do you get reliable JSON out of a model?
Use the provider’s structured-output or JSON-schema mode rather than asking politely in the prompt, define the schema from your C# type, and validate on receipt anyway. Never JsonSerializer.Deserialize model output without a try/catch and a repair path. Testing: have you been burned yet. Structured output.
29. Users say the agent “got worse”. How do you investigate?
First, establish whether it did — run the eval set and compare to the baseline. If quality genuinely dropped, the usual suspects are a model version change on the provider’s side, a corpus change that degraded retrieval, or a prompt edit nobody evaluated. If the numbers held, you are looking at a distribution shift in what users are asking. Testing: this is the best senior question on this list. It separates people who can debug a probabilistic system from people who can only build one.
30. Design a support agent over 50,000 internal documents.
Sketch: ingest and chunk by structure, embed, store with metadata for filtering; hybrid retrieval; grounded generation with citations; tools for the things retrieval cannot answer (order lookup, ticket creation); per-user permission filtering enforced at the data layer, not the prompt; caching on common questions; evaluation set with retrieval and answer metrics tracked separately; cost and latency dashboards; explicit escalation to a human. Testing: breadth and, critically, whether you raise permissions and evaluation without being prompted. Candidates who mention both unprompted are the ones who get offers.
How to prepare if you have not shipped AI yet
Build one small thing end to end and understand its failure modes. A RAG bot over documentation you know well, with an eval set of thirty questions and a measurement of what it costs per query, gives you a genuine answer to almost every question on this list — because the follow-ups are all “what went wrong, and how did you know?”
Five tutorials you followed cannot answer that. One project you debugged can.
Takeaway
These interviews are not testing machine-learning knowledge; they are testing whether you can engineer around a component that is non-deterministic, occasionally confidently wrong, and billed by the token. The candidates who do well are the ones who bring up evaluation, cost, permissions and failure handling without being asked — because that is exactly the gap between a demo and a system. If you have shipped anything in .NET that had to survive production, you already have the instincts. Point them at this new dependency.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.