SRE Task LLM Leaderboard
Which AI actually finds the root cause? We take real incidents that SRE teams deal with in production, recreate each fault in a live environment, and let every model investigate it through Akmatori — the same way an on-call engineer would. No mocked data, no multiple-choice: the model digs through the system and names the cause. An independent judge grades every investigation against written ground truth.
First edition · August 2026
| # | Model | Score ↓ | Root causes found | Cost / root cause | Median time |
|---|---|---|---|---|---|
| 1 | Qwen3.8 Max Alibaba | 93.6 | 100% | $0.17 | 1m 58s |
| 2 | Claude Opus 5 Anthropic | 93.0 | 100% | $0.57 | 1m 33s |
| 3 | Claude Fable 5 Anthropic | 92.4 | 100% | $0.82 | 1m 35s |
| 4 | GPT-5.6 Sol OpenAI | 91.5 | 100% | $0.63 | 2m 6s |
| 5 | DeepSeek V4 Pro DeepSeek | 88.9 | 90% | $0.03 | 1m 11s |
| 6 | Kimi K2.6 Moonshot AI | 88.5 | 90% | $0.07 | 51s |
| 7 | Claude Haiku 4.5 Anthropic | 87.1 | 90% | $0.15 | 1m 1s |
| 8 | GPT-5.6 Terra OpenAI | 86.7 | 90% | $0.18 | 1m 4s |
| 9 | GLM-5.2 Z.ai | 85.6 | 90% | $0.11 | 1m 48s |
| 10 | Kimi K3 Moonshot AI | 82.0 | 90% | $0.38 | 2m 35s |
| 11 | DeepSeek V4 Flash 0731 DeepSeek | 81.1 | 80% | $0.04 | 32s |
| 12 | GPT-5.6 Luna OpenAI | 80.2 | 80% | $0.02 | 1m 17s |
| 13 | Claude Sonnet 5 Anthropic | 76.0 | 80% | $0.30 | 1m 30s |
How we measure
- Real incidents, not quizzes. Every task is drawn from production experience: crash loops, broken configuration, failing health checks, exhausted resources, dead services — and multi-hop traps where the obvious first answer is wrong and only a deeper investigation finds the true cause.
- Identical harness for every model. Each model gets a fresh Akmatori installation with a clean database and no accumulated memory, the same investigation tools, and the same alerts — inside a real environment that we recreate from scratch for every test. Only the model changes.
- Judged against ground truth. A pinned judge model (Claude Opus 5, constant across all runs) compares each final investigation report with the scenario's written root cause. Naming the broken component is not enough — the model must find why.
- One overall score. The score blends root-cause accuracy (90%) with cost efficiency (5%) and investigation speed (5%), normalized across the field — accuracy dominates, because a cheap wrong answer is still wrong. Click any column to re-sort.
- Cost and speed are first-class. Cost is metered API usage where the provider reports it and public market rates otherwise, divided by successful root-cause identifications. Time is the median wall-clock investigation, alert to answer.
Single run per task in this edition; scores on repeated runs may vary. Provider prices change frequently — cost figures reflect rates at the time of the run. Want your model or endpoint benchmarked? Get in touch.
Run these investigations yourself
Every score above comes from the same open-source Akmatori that you can self-host. Point it at your own model and your own incidents — it is Apache 2.0, and your data stays in your infrastructure.
View Akmatori on GitHub