Logo of Akmatori
Skip to main content

SRE Bench

SRE Bench is a benchmark that measures what SRE teams actually need from an AI: can it take a live incident — a real alert, in a real broken environment — investigate it end to end, and name the root cause? We recreate incidents drawn from production experience, let every model work them through Akmatori the same way an on-call engineer would, and grade each investigation against written ground truth. No mocked data, no multiple-choice — and naming the broken component is not enough: the model must find the cause behind it.

Second edition · updated November 2026

SRE Bench score

90% accuracy, 5% cost, 5% speed · Higher is better

1GPT-6.2 Sol97.0
2GPT-6 Sol96.8
3GPT-6.1 Sol95.7
4Claude Fable 5.295.4
5GPT-6.2 Astra95.1
6Gemini 4 Neon94.7
7Grok 4.794.3
8GPT-6 Luna93.9
9GPT-6 Astra93.8
10GPT-5.6 Sol92.9
11Gemini 4 Hydrogen92.3
12Claude Opus 592.0
13Gemini 3.7 Flash91.7
14Gemini 4 Carbon91.1
15Muse Spark 1.290.8
16DeepSeek V4.1 Flash90.4
17Claude Opus 5.590.3
18DeepSeek V4.1 Flash 101790.2
19GPT-5.6 Terra89.6
20MiniMax M3.1 Flash89.6
21Claude Haiku 5.589.0
22Claude Fable 5.188.7
23Muse Wonder88.5
24Qwen3.8 Max88.1
25DeepSeek V4 Pro 081384.5
26Kimi K3.1 Pro84.1
27Mistral Large 482.5
28Grok 4.681.5
29GLM-5.381.5
30Claude Fable 580.7
31DeepSeek V4 Pro80.6
32Kimi K2.680.2
33GPT-5.6 Luna79.2
34Claude Haiku 4.577.1
35GLM-5.276.4
36Kimi K373.7
37Claude Sonnet 568.5

Score vs. cost per root cause

Cost on a log scale · Top left is best

Most attractive quadrantPareto frontierOn the frontierOther models
65707580859095100$0.01$0.02$0.05$0.1$0.2$0.5$1Cost per root cause (USD, log scale)ScoreGPT-6 LunaGPT-6.2 Sol

Quadrant lines are the field medians. Click a bar or a dot to highlight one model in both charts.

How we measure

  • Real incidents, not quizzes. Every task is drawn from production experience: crash loops, broken configuration, failing health checks, exhausted resources, dead services — and, expanded in this edition, multi-hop traps where the obvious first answer is wrong and only a deeper investigation finds the true cause. The trap tasks are what separate the field: most models find the broken component; far fewer find the cause behind it.
  • Identical harness for every model. Each model gets a fresh Akmatori installation with a clean database and no accumulated memory, the same investigation tools, and the same alerts — inside a real environment that we recreate from scratch for every test. Only the model changes.
  • Judged against ground truth. A pinned judge model (Claude Opus 5, constant across all runs) compares each final investigation report with the scenario's written root cause. Naming the broken component is not enough — the model must find why.
  • One overall score. The score blends root-cause accuracy (90%) with cost efficiency (5%) and investigation speed (5%), normalized across the field — accuracy dominates, because a cheap wrong answer is still wrong. Use the chart tabs to rank by each metric on its own.
  • Cost and speed are first-class. Cost is metered API usage where the provider reports it and public market rates otherwise, divided by successful root-cause identifications. Time is the median wall-clock investigation, alert to answer.

Single run per task in this edition, with one exception; scores on repeated runs may vary. In November 2026 we added two tasks to make the benchmark harder: a configuration-drift incident, and the same incident with a memory of a past incident whose fix no longer works. The memory task is run three times and counts as its pass rate. The two new tasks ran on a newer Akmatori build than the first twelve; cost and time figures come from the first twelve tasks. DeepSeek V4 Flash 0731 was removed because its endpoint is no longer available. Provider prices change frequently — cost figures reflect rates at the time of the run. In November 2026 we corrected the costs of nine models (GPT-6 Luna, Sol, 6.1 Sol and Astra, Claude Opus 5.5 and Haiku 5.5, Gemini 3.7 Flash, Grok 4.6, DeepSeek V4 Pro 0813): their earlier figures left out prompt-cache tokens. The corrected figures are estimates, because those runs did not record cache reads and writes separately. Want your model or endpoint benchmarked? Get in touch.

Run these investigations yourself

Every score above comes from the same open-source Akmatori that you can self-host. Point it at your own model and your own incidents — it is Apache 2.0, and your data stays in your infrastructure.

View Akmatori on GitHubDiscuss results in #srebench