All posts
AI Engineering

EKS and AI Agents: Running Agents on Kubernetes, and Letting Agents Operate Your Clusters

By

November 2, 2026

7 min read

Two different problems. Running agents on EKS is about long-lived workers, GPUs, sandboxes for code they write, and tight egress. Letting an agent operate EKS, through the EKS MCP server or kubectl, is about read-only by default, separate roles for writes, and approvals. The setups that are safe enough to use.

EKS
AI Agents
MCP
Kubernetes
Security

Part one: running agents on EKS

Requests

API
SQS queue

Agent tasks are jobs, not requests: queue them so a long run never holds a connection.

Orchestration

Agent workers (Deployment)
Step Functions or durable functions for long flows

Execution

Sandbox node pool
Short-lived pods per task
No service account token mounted

Code an agent writes runs on nodes that hold nothing else.

Models

Amazon Bedrock
GPU node pool for self-hosted models

Karpenter or Auto Mode brings GPU nodes up only when work arrives.

Guardrails

Egress network policies
Pod Identity with narrow roles
Per-tenant token budgets

Agents as workloads: queue in, isolated workers, narrow egress

  • Treat code an agent writes as untrusted. Run it in short-lived pods on a separate node pool, with no service account token, no cloud credentials, and network policies that allow only the egress the task needs.

  • Containers are not a security boundary, as AWS's own Pod Identity docs say; nodes are. Separate node pools for untrusted execution are the minimum.

  • GPU capacity is the expensive part. Scale GPU node pools to zero between jobs, and use split cost allocation data to see GPU spend per namespace.

  • Cap spend per tenant and per task in code: tokens, tool calls, wall-clock time.

Want a second opinion on this for your system?

Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.

Part two: letting agents operate EKS

AWS publishes an open-source EKS MCP server that gives coding assistants tools to inspect and manage clusters: list and describe resources, read logs and events, generate and apply manifests, and run troubleshooting. Two flags define how much it can do, and both default to off: --allow-write for anything that changes the cluster, and --allow-sensitive-data-access for logs, events and Secrets.

Setup

What the agent can do

When

Read-only, no sensitive data

Describe resources and cluster state

Default for everyone; answering questions about a cluster

Read-only with logs and events

Troubleshoot: read pod logs, events, CloudWatch

On-call investigation, with the logs' contents in mind

Write, against a non-production cluster

Apply manifests, scale, restart

Development and staging

Write against production

Changes, each approved by a person

Only with an approval step and a role scoped to specific namespaces

  1. Give the agent its own IAM role and its own access entry, with an access policy scoped to the namespaces it may touch. Never your admin credentials.

  2. Separate roles for read and write, as the EKS MCP server's documentation recommends.

  3. Route production writes through an approval: a pull request to the GitOps repository rather than kubectl apply.

  4. Keep CloudTrail and the control plane audit log on; every agent action is attributable to its role.

hoolco Monthly Review

Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.

As of November 2, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.