Not every AI feature needs a full service running around the clock. For an endpoint that gets sporadic traffic — a “summarize this” button, an occasional classification call — Azure Functions is ideal: it scales to zero when idle (you pay nothing) and spins up on demand. This guide deploys a serverless OpenAI-powered API in .NET, and then covers the four platform limits that decide whether it survives.
Why serverless for AI
AI endpoints are often bursty and I/O-bound (most of the time is spent waiting on the model). That’s a great fit for Functions:
- Scale to zero — no traffic, no cost. Perfect for internal tools or low-volume features.
- Automatic scale-out — a burst of requests spins up more instances.
- No servers to manage — you ship a function, not infrastructure.
For steady, high-volume traffic, an always-on container (Azure Container Apps or GKE) is often better — but for spiky workloads, serverless wins.
One correction to the usual pitch, though, because it changes the maths. Serverless billing is memory multiplied by wall-clock duration, and a model call is almost entirely wall clock spent waiting on someone else’s GPU. You are billed for the waiting. That is fine — it is still cheaper than idle capacity at low volume — but it means the “I/O-bound work is cheap on serverless” instinct from a database-backed API does not transfer cleanly. A function that waits six seconds for a generation costs six seconds of allocated memory, every time, and halving your model latency halves your hosting bill as a side effect.
Pick the plan first
Start on the Flex Consumption plan. The original Consumption plan is now legacy, Microsoft points new serverless apps at Flex, and for an AI endpoint the differences are exactly the ones that matter: a 30-minute default execution timeout instead of five, up to 4 GB per instance instead of 1.5 GB, virtual network integration, and always-ready instances that let you keep a warm instance without moving to a Premium plan. Choosing the legacy plan for a new AI endpoint is picking the tighter limits on purpose.
Premium earns its keep when the app is running nearly continuously, when you need more CPU or memory per instance than a serverless tier offers, or when you want a custom Linux image. At that point you are close enough to always-on that a container is worth comparing honestly, because the pricing gap narrows and containers give you full control of startup.
The function
An HTTP-triggered function that calls the model is refreshingly small:
public class SummarizeFunction
{
private readonly IChatClient _chat;
public SummarizeFunction(IChatClient chat) => _chat = chat;
[Function("Summarize")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req)
{
var input = await req.ReadFromJsonAsync<SummarizeRequest>();
var reply = await _chat.GetResponseAsync(
$"Summarize this in two sentences:\n\n{input!.Text}");
var res = req.CreateResponse();
await res.WriteAsJsonAsync(new { summary = reply.ToString() });
return res;
}
}
record SummarizeRequest(string Text);Register the IChatClient in Program.cs and inject it — the same Microsoft.Extensions.AI pattern as anywhere else.
Register it as a singleton. This is the mistake that turns a working function into a production incident. Constructing a model client, or an HttpClient, per invocation leaks sockets — each one holds a connection in TIME_WAIT after disposal, and under any real burst the instance runs out. Outbound connections are capped per instance, and on the legacy Consumption plan the cap is low enough that a moderately busy per-request client hits it in minutes. The symptom is nasty to diagnose: intermittent SocketException or connection-timeout errors that appear only under load and vanish when you look at a single request. Resolve the client once from the container and reuse it; the SDK clients are designed for concurrent use.
Do the same for credentials. DefaultAzureCredential walks a chain of possible sources, and some of those probes involve network calls that fail before it reaches the one that works — which is fine once and wasteful on every cold start. In a deployed Function App you know the answer is managed identity, so construct that credential explicitly and cache the client. It is a small saving per instance, but instances are created constantly on a scale-to-zero plan, so it lands on exactly the requests you care about.
Keep the key out of the function
Don’t put your API key in app settings as plain text. Use a managed identity: give the Function App an identity and grant it access to Azure OpenAI, then authenticate with a token credential. No key exists to leak — the same keyless pattern from securing agents. Nothing rotates, nothing appears in a portal blade someone screenshots, and revoking access is a role assignment rather than a redeploy.
The 230-second wall
This is the limit that catches people, and it is not the one they are watching.
Regardless of what you set functionTimeout to in host.json, an HTTP-triggered function has about 230 seconds to produce a response. The cap comes from the platform load balancer’s idle timeout, not from the Functions runtime, so raising the function timeout to 30 minutes changes nothing about it. Your function may well keep running past that point; the caller has already received an error.
For a summarise endpoint you will never come close. For an agent that makes four model calls with tool executions in between, you can — and the failure is horrible, because it is timing-dependent, it only happens on the hard requests, and the logs show the function completing successfully after the client gave up. Any request whose worst case is measured in minutes should not be a synchronous HTTP function.
The pattern to use instead is the async one: the HTTP trigger validates the request, writes a message to a queue, and returns 202 Accepted with a job id immediately. A queue-triggered function does the slow work and writes the result somewhere the client can poll or a webhook can deliver. It is more moving parts, and it is the difference between a feature that degrades gracefully under a slow model and one that fails at exactly the moment it is doing the most valuable work.
The async split buys you something else for free: retries. HTTP triggers get one attempt — if the instance dies mid-generation, the work is gone and the user’s tokens are spent. Queue triggers retry automatically and park repeatedly failing messages in a poison queue where you can inspect them. Set the retry count low, because every retry is another paid model call, and pair it with a proper resilience pipeline so you are not retrying at two layers at once.
Cold starts, honestly
The received wisdom is that cold starts do not matter for AI because the model call dominates, and it is half right. A model call takes seconds; a warm start takes milliseconds. But a genuine cold start on a .NET function means allocating an instance, starting the host, and starting the worker process — and the platform gives the language worker up to 60 seconds to come up before it gives up, which tells you something about the range involved. Added to a multi-second generation, it is the difference between a snappy feature and one users describe as “sometimes it just hangs”.
Three things actually help. Use always-ready instances on Flex Consumption or pre-warmed instances on Premium for any user-facing path — this is the real fix and everything else is a rounding error next to it. Deploy from a package rather than unpacking files at start. And keep the startup path thin: a Program.cs that reads configuration from three services and builds an object graph before the first request is paying that cost on every new instance. Background and event-driven functions can happily cold start; it is only the button a human is watching that needs the warm instance.
Cap the scale-out, or your quota will
Automatic scale-out is the feature and, with a model behind it, the trap.
A burst of traffic can take a Flex Consumption app to a very large number of instances. Every one of those instances calls the same model deployment, against the same tokens-per-minute quota. The platform is scaling to meet demand your model provider has no intention of meeting, so past a certain point every additional instance produces 429s rather than answers — and you have spent the scale-out on generating errors faster.
Set a maximum instance count that reflects your model quota rather than your traffic, and tune per-instance concurrency in the same breath: instances multiplied by concurrency is your real peak call rate, and it should sit under your provisioned throughput with room to spare. Then handle the 429s that remain, ideally by queueing rather than retrying in place. This is the single most common way a serverless AI endpoint fails its first real traffic spike, and it looks like a model problem when it is a scaling configuration problem.
Streaming is where this stops being the right host
A model streaming tokens to a browser is the natural shape of a chat UI, and it is the workload Functions fits worst. The 230-second ceiling still applies to a streamed HTTP response, so a long generation can be cut off mid-sentence. Billing is per second of allocated memory, so holding a connection open for the duration of a generation is the expensive case rather than the cheap one. And the isolated worker model needs the ASP.NET Core integration to work with HttpRequest and IResult directly before streaming is even ergonomic.
None of that makes it impossible, and for short streamed responses it is fine. But if streaming is the primary interaction, a container is the better home — you get an unbounded response duration, connection handling you control, and a cost model that does not charge you by the token’s arrival time. The details are in streaming AI responses to a web UI.
When to choose it
Reach for Functions when your AI endpoint is occasional, bursty, or event-driven — a queue message, a blob upload, a scheduled classification run, a button nobody presses at 3am. The economics are unbeatable at low volume, and the event-driven triggers mean a lot of AI plumbing you would otherwise write as a hosted service becomes a trigger attribute.
Choose an always-on container when traffic is steady and high (scale to zero stops being a saving when you never scale to zero), when responses stream, when a request can run long, or when you need startup control that a managed host will not give you. Many systems use both, and that is the right answer more often than picking one: Functions for the background enrichment, the ingestion pipeline and the internal tools; a container for the chat endpoint on the front page.
Note: Azure Functions APIs and plan limits change; verify against the Azure Functions docs. The design — HTTP-triggered function, injected
IChatClient, managed-identity auth, mind cold starts and timeouts — is stable.
Takeaway
Azure Functions gives an AI endpoint serverless economics: scale to zero, pay per use, no servers. Start on Flex Consumption rather than the legacy Consumption plan, inject a singleton IChatClient so you do not exhaust the per-instance connection cap, and authenticate with a managed identity so there is no key to leak. Then design around the 230-second HTTP response ceiling — it is a platform limit that the function timeout setting does not override, so anything that might run long belongs behind a queue with a job id rather than in a synchronous request. Cap the maximum instance count against your model quota instead of your traffic, or scale-out will simply produce 429s faster. Use always-ready instances for anything a human is waiting on. For bursty, event-driven AI features this is the cheapest, simplest home — and it pairs naturally with always-on containers for the streaming, high-volume traffic that needs them.