Agent Substrate for On-Demand Kubernetes Agents
AI agents are becoming real infrastructure workloads. They need identity, storage, networking, isolation, and observability, but many spend most of their time waiting for a user, model call, tool result, or scheduled trigger. Agent Substrate tackles that mismatch by running agent-like workloads on Kubernetes without binding every logical agent to a dedicated always-on pod.
What Is Agent Substrate?
Agent Substrate is an open source system built on top of Kubernetes for running independent, stateful container actors at higher density. Kubernetes still provisions capacity and manages the base infrastructure. Substrate adds agent-specific scheduling and lifecycle control outside the Kubernetes scheduler hot path.
The core idea is simple: map many actors onto fewer ready workers. Actors can be created, suspended, resumed, assigned to workers in real time, and reached through routing. The project uses gVisor for sandboxed execution and supports preserving filesystem and memory state through snapshots.
The maintainers are clear that the project is very early and not production ready. That matters. Treat it as a design signal and lab target, not a replacement for mature workload platforms today.
Why SRE Teams Should Care
Agent isolation is necessary, but it is not the whole problem. A platform that runs hundreds of idle agents as hundreds of pods still pays for idle memory, node pressure, scheduling churn, and operational noise.
Substrate points at a different runtime shape:
- Worker pool multiplexing: Many logical actors can share fewer physical pods when they are not active at the same time.
- Suspend and resume: Actors can snapshot state, release the worker slot, then resume later without starting from scratch.
- Kubernetes-native capacity: Clusters still provide nodes, pods, networking, and storage primitives.
- Sandboxed execution: gVisor gives a stronger boundary for tool-calling agents than a plain process on a shared host.
- Framework flexibility: The README calls out ADK, LangChain, Claude Code, Codex, and MCP servers because the unit of execution is an OCI container.
Installation
For development, the upstream README documents a kind-based setup:
hack/create-kind-cluster.sh
hack/install-ate-kind.sh --deploy-ate-system
hack/install-ate-kind.sh --deploy-demo-counter
go install ./cmd/kubectl-ate
kubectl ate create atespace demo
kubectl ate create actor my-counter-1 -a demo --template ate-demo-counter/counter
kubectl port-forward -n ate-system svc/atenet-router 8000:80
The kagent docs also show a Helm path for running Substrate v0.0.6 and creating a SandboxAgent on a worker pool. That is useful if your team wants to compare plain agent pods with substrate-backed actors.
Operational Evaluation Checklist
Start with a non-production cluster and measure the boring details:
kubectl get pods -n ate-system
kubectl get workerpool -A
kubectl get sandboxagent -n kagent
Track wakeup latency, snapshot size, object storage cost, worker saturation, and failure behavior during suspend or resume. Also inspect how identities, egress policy, logs, metrics, and traces attach to a logical actor after it moves between workers.
The roadmap is worth reading before any serious trial. High-priority items include actor versioning, autoscaling worker pools, network policy, credential injection, audit logging, Prometheus metrics, OTLP correlation, and runtime modularity. Those are the surfaces SRE teams need before a runtime can carry production responsibilities.
Conclusion
Agent Substrate is early, but the model is important. Running agents safely is only one part of the platform problem. Running them efficiently, with durable state and low idle cost, is what makes large agent fleets operationally plausible.
At Akmatori, we build AI agents for SRE teams that help investigate alerts, inspect infrastructure, and automate operational workflows. If you want a managed edge and cloud foundation for resilient systems, explore Gcore for infrastructure that pairs well with modern automation stacks.