HolmesGPT: A CNCF SRE Agent for Incident Triage
Incident response starts with scattered evidence. The alert is in PagerDuty or Alertmanager, logs are in Loki or Datadog, traces live somewhere else, and the useful runbook might be hidden in Confluence. HolmesGPT is built for that gap: an AI agent that can query production context and help find root causes.
What Is HolmesGPT?
HolmesGPT is an open-source SRE agent and a CNCF Sandbox project. It was originally created by Robusta.dev, with major contributions from Microsoft. The project focuses on production incident investigation, not generic chat. It connects to live data sources and turns noisy context into a diagnosis responders can review.
The important detail is scope. HolmesGPT works with Kubernetes, VMs, cloud services, databases, and SaaS platforms. It does not require every workload to run inside Kubernetes, although Operator Mode runs in a cluster for background health checks.
Key Features
- Deep data integrations: Toolsets cover Kubernetes, Prometheus, Grafana, Datadog, Loki, PagerDuty, Jira, GitHub, cloud platforms, SQL databases, and more.
- Large-output handling: Filtering, JSON traversal, transformers, and output budgeting help keep huge telemetry responses out of model context.
- Background checks: Operator Mode can run scheduled health checks and deployment verification, then message teams in Slack.
- Provider flexibility: Teams can connect OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, and other model providers.
- Production posture: The project emphasizes read-only access by default and respects existing RBAC permissions.
Installation
The official docs list several installation paths. For a quick CLI setup:
pipx install holmesgpt
holmes ask "Why is checkout latency high?"
Check the HolmesGPT installation guide before using it in production, since provider keys and data-source permissions need to match your environment.
Usage For SRE Teams
The strongest use case is incident triage with controlled access. Connect HolmesGPT to read-only sources first: Kubernetes events, Prometheus metrics, log search, traces, and runbooks. Then test it against old incidents. Ask it to explain a known outage, compare its answer with the postmortem, and note what context was missing.
Operator Mode is useful around deployments. A health check can run after a rollout, inspect live service signals, and report regressions before users notice. With the GitHub integration, the workflow can open a pull request for a suggested fix, although production teams should keep human review in front of any change.
Operational Tips
Start narrow. Give the agent access to one service, one alert class, and one set of runbooks. Log every investigation, track false positives, and require links back to evidence. If it suggests remediation, separate diagnosis from execution so responders can approve the action.
Treat model choice as an operational setting. Cheap models may work for log summaries, while cross-service root-cause analysis may need stronger reasoning.
Conclusion
HolmesGPT is worth watching because it moves AI-assisted SRE from chat prompts toward connected incident workflows. Its CNCF Sandbox status, broad toolset support, and Operator Mode make it relevant for teams building AI into on-call operations.
If your team wants AI-assisted incident workflows with strong operational context, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.