Azure Container Apps is the fastest path to hosting a .NET agent, but plenty of teams live on Google Cloud, already run GKE, and want their agent alongside the rest of their microservices — same cluster, same mesh, same observability. This guide deploys a .NET agent to Google Kubernetes Engine the production way, with a focus on the two things people get wrong: identity and scaling.
It assumes you have an agent exposed over HTTP, like the one from our tool-using agent tutorial. For the Azure equivalent, see deploying to Azure Container Apps.
Step 1: Containerize and push to Artifact Registry
Use the same multi-stage Dockerfile you’d use anywhere, then push to Google Artifact Registry (GKE’s native registry):
gcloud artifacts repositories create agents \
--repository-format=docker --location=us-central1
docker tag agent-demo \
us-central1-docker.pkg.dev/PROJECT_ID/agents/agent-demo:v1
docker push us-central1-docker.pkg.dev/PROJECT_ID/agents/agent-demo:v1Step 2: Identity — the part that matters most
The instinct is to create a service-account key JSON, drop it in a Kubernetes Secret, and mount it. Don’t. Exported keys are the most common cloud-credential leak, and they’re unnecessary on GKE.
Use Workload Identity instead: it lets a Kubernetes service account impersonate a Google service account, so your pod authenticates as itself with short-lived, automatically-rotated credentials — no key files anywhere. If your agent calls Vertex AI (or any Google API), this is how it should get its credentials.
# Bind the Kubernetes SA to a Google SA
gcloud iam service-accounts add-iam-policy-binding \
agent-gsa@PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:PROJECT_ID.svc.id.goog[default/agent-ksa]"Then annotate the Kubernetes service account so the binding takes effect:
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-ksa
annotations:
iam.gke.io/gcp-service-account: agent-gsa@PROJECT_ID.iam.gserviceaccount.comGrant agent-gsa only the roles the agent needs (for Vertex AI, roles/aiplatform.user). In .NET, the Google client libraries pick up these credentials automatically — your code contains no keys.
Workload Identity Federation for GKE can also skip the Google service account entirely and grant IAM roles straight to the Kubernetes service account’s principal, which looks like principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME. One fewer identity to create, name and audit. The impersonation route above is still the one to use when several workloads share a set of permissions, or when an existing GSA already holds the grants you need — but for a new agent with its own narrow permissions, binding directly is cleaner.
When it doesn’t work, it’s one of three things
Workload Identity is the step people get stuck on, and the failures look alike from inside the pod: the Google client library reports that it couldn’t find default credentials, or a call comes back 403 for a role you’re certain you granted. Three causes cover nearly all of it.
The node pool wasn’t enabled. Workload Identity Federation has to be on at the cluster level and the GKE metadata server has to be enabled on the node pool, via --workload-metadata=GKE_METADATA. Enabling it on the cluster and forgetting the existing node pools is the single most common mistake, and it fails silently in the worst way — pods on updated pools work, pods on the old pool don’t, so the behaviour depends on which node the scheduler happened to pick. Autopilot clusters have it on permanently, which is one good reason to prefer Autopilot for this.
The pod asked too early. The GKE metadata server takes a few seconds after a pod starts before it will answer, so a container that authenticates immediately at startup can fail while the identical call succeeds seconds later. If your agent builds its model client in Program.cs before the first request, this is a startup crash loop that clears itself the moment you add a retry. Google’s documented remedy is an init container that waits for the metadata server to become ready; a retry with backoff around your first credential use achieves the same thing with less YAML.
The binding names don’t match. The member string embeds the namespace and the Kubernetes service account name — PROJECT_ID.svc.id.goog[default/agent-ksa] — so deploying into a namespace other than default without updating the binding produces a pod that authenticates as nobody. The annotation on the KSA and the IAM binding have to agree exactly, and neither will warn you when they don’t.
One more for the mental file: the metadata server runs as a DaemonSet whose memory use scales with the number of Kubernetes service accounts in the cluster. On a large, busy cluster it can be evicted, at which point authentication fails cluster-wide for reasons that have nothing to do with your deployment.
Step 3: Deployment and Service
A standard Deployment referencing the service account, with health probes and sane resource requests:
apiVersion: apps/v1
kind: Deployment
metadata:
name: agent-demo
spec:
replicas: 2
selector: { matchLabels: { app: agent-demo } }
template:
metadata: { labels: { app: agent-demo } }
spec:
serviceAccountName: agent-ksa
containers:
- name: agent
image: us-central1-docker.pkg.dev/PROJECT_ID/agents/agent-demo:v1
ports: [{ containerPort: 8080 }]
readinessProbe: { httpGet: { path: /health, port: 8080 } }
resources:
requests: { cpu: "250m", memory: "256Mi" }
limits: { cpu: "1", memory: "512Mi" }
---
apiVersion: v1
kind: Service
metadata: { name: agent-demo }
spec:
selector: { app: agent-demo }
ports: [{ port: 80, targetPort: 8080 }]The /health readiness probe matters: it keeps traffic from hitting a pod before the agent has warmed up its model client.
Two numbers in that manifest that bite .NET specifically
The memory limit is not a suggestion. .NET reads the container’s cgroup memory limit and sizes its heaps against it, but a limit that’s too low doesn’t produce a manageable OutOfMemoryException — the kernel kills the process. What you see is a pod restarting with exit code 137 and absolutely nothing in your application logs explaining why, because the process never got a chance to write anything. If an agent pod restarts under load with no stack trace, check kubectl describe pod for OOMKilled before you look anywhere else. Agent workloads make this likelier than typical web services do, because a long conversation transcript and a few large tool results are real allocations that grow with usage rather than with request rate.
The CPU limit deserves as much thought. A limit of 1 doesn’t slow the pod down gracefully when it needs more; it throttles it, and throttling shows up as latency spikes that correlate with nothing in your traces. Agent pods are mostly waiting on the model, so their steady-state CPU is genuinely low — but startup, JSON serialisation of large payloads and garbage collection are bursty. Setting requests low and limits generously above them usually gives better tail latency than a tight limit that looks tidy in the manifest.
Step 4: Expose it
Point an ingress at the Service. On GKE you can use the built-in GCE ingress, or — if you’re already running a service mesh like Istio — route the agent through your existing gateway so it inherits your mTLS, routing, and policy. Putting the agent behind the same mesh as your other services is the whole reason to choose GKE over a standalone container platform; don’t bolt on a separate ingress if you already have one.
The 30-second timeout that catches every agent
If you use the GCE ingress, the backend service timeout defaults to 30 seconds. Any request that takes longer is cut by the load balancer, and the client gets a 504 while your pod carries on working on a response nobody will receive.
Thirty seconds is comfortable for a single short model call and nowhere near enough for an agent that chains three tool calls, retries a rate-limited request, or summarises a long document. The symptom is maddening precisely because it’s partial: quick questions work, hard questions fail, and the failure is attributed to “the model being slow” rather than to infrastructure.
Raise it with a BackendConfig, and note that it attaches to the backing Service, not to the Ingress — so the Service from Step 3 gains an annotation:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: agent-backendconfig
spec:
timeoutSec: 180
---
apiVersion: v1
kind: Service
metadata:
name: agent-demo
annotations:
cloud.google.com/backend-config: '{"default": "agent-backendconfig"}'
spec:
selector: { app: agent-demo }
ports: [{ port: 80, targetPort: 8080 }]Streaming the response is the better answer where you can manage it, because a streamed reply starts producing bytes almost immediately and the user stops waiting on a total that no longer matters. Raising the timeout is what you do for the requests you can’t stream.
Expect 502s in two other situations, both normal rather than broken. A freshly created ingress returns them for several minutes while the load balancer provisions — wait before you start debugging. And during scale-down or a rolling update you’ll see them when pods are terminated before the load balancer has stopped sending them traffic, because deregistering a backend takes longer than Kubernetes takes to kill a pod. The fix is a preStop hook that sleeps long enough for deregistration to propagate, plus a terminationGracePeriodSeconds comfortably longer than your slowest in-flight agent request. Killing a pod mid-tool-call is worse here than in a typical service: the model call has already been paid for, and the user gets nothing for it.
Step 5: Autoscale on the right signal
Add a Horizontal Pod Autoscaler, but scale on the metric that reflects agent load:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: agent-demo }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: agent-demo }
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }CPU works for synchronous endpoints, but agent calls are I/O-bound (waiting on the model), so CPU may stay low while requests queue. For those workloads, scale on a custom metric like in-flight requests or queue depth via the Custom Metrics API. And cap maxReplicas to respect your model provider’s tokens-per-minute quota — scaling pods past the quota just relocates the bottleneck, exactly as it would on any platform.
Be concrete about how badly CPU-based scaling fails on this shape of workload, because the manifest above is the one most people ship. A replica handling twenty concurrent agent requests, every one of them blocked on a Vertex AI call, may sit at 15% CPU. The HPA sees a comfortable, well-provisioned pod and does nothing while p99 latency climbs into the tens of seconds. The autoscaler isn’t broken — it’s measuring the resource that isn’t scarce. The scarce resource is concurrent capacity against a remote service, and the only signals that track it are in-flight request count, queue depth, or your own latency.
There’s a second layer of delay to plan around. When the HPA does add pods and the cluster has no room, the cluster autoscaler has to provision a node first — that’s minutes, not seconds, and then your image has to be pulled onto it. Scaling out is not a response to a traffic spike; it’s a response to sustained load. Absorbing the spike itself is what headroom is for, which is why minReplicas: 2 in that manifest is a floor to raise rather than a number to be proud of.
While you’re there, add a PodDisruptionBudget. Node upgrades and autoscaler consolidation will evict your pods, and without a budget nothing stops both replicas from going at once. minAvailable: 1 costs a line of YAML and prevents an outage during routine cluster maintenance that you didn’t schedule and won’t be watching.
When GKE is the wrong answer
Everything above is a fair amount of machinery, and it’s worth naming when it isn’t warranted.
If you’re not already running Kubernetes, don’t start because of an agent. Cloud Run gives you the same container, the same Workload Identity model, scale-to-zero, and an ingress that already works, with none of the manifests, the node pools, or the upgrade cycle. The break-even isn’t about traffic volume — it’s about whether a team already owns a cluster and the operational habits that go with one. On Azure the equivalent trade is Container Apps versus AKS, and it resolves the same way.
GKE earns its complexity when the agent has to live with your other services: when it calls internal APIs that are only reachable inside the cluster, when your mesh already provides mTLS and traffic policy that the agent should inherit, when compliance requires network policies you control, or when your deployment pipeline, secrets and observability are all built around Kubernetes and a second platform would be the odd one out. Those are good reasons. “Kubernetes is where serious workloads go” is not.
Note:
gcloudflags and GKE APIs shift over time; verify against the current GKE and Workload Identity docs. The architecture — Workload Identity over keys, health probes, mesh-native ingress, quota-aware autoscaling — is the durable part.
Takeaway
GKE is the right home for a .NET agent when it belongs with the rest of your Google Cloud microservices. Get two things right and the rest is standard Kubernetes: authenticate with Workload Identity so no keys ever exist, and autoscale on a signal that reflects real agent load within your model quota. Do that, and your agent is just another well-behaved workload in your cluster — observable, secured by your mesh, and scaling with demand.
Have a correction or a topic you want covered? Email mani.bc72@gmail.com.