SRE Bench

How we measure

• Real incidents, not quizzes. Every task is drawn from production experience: crash loops, broken configuration, failing health checks, exhausted resources, dead services — and, expanded in this edition, multi-hop traps where the obvious first answer is wrong and only a deeper investigation finds the true cause. The trap tasks are what separate the field: most models find the broken component; far fewer find the cause behind it.
• Identical harness for every model. Each model gets a fresh Akmatori installation with a clean database and no accumulated memory, the same investigation tools, and the same alerts — inside a real environment that we recreate from scratch for every test. Only the model changes.
• Judged against ground truth. A pinned judge model (Claude Opus 5, constant across all runs) compares each final investigation report with the scenario's written root cause. Naming the broken component is not enough — the model must find why.
• One overall score. The score blends root-cause accuracy (90%) with cost efficiency (5%) and investigation speed (5%), normalized across the field — accuracy dominates, because a cheap wrong answer is still wrong. Click any column to re-sort.
• Cost and speed are first-class. Cost is metered API usage where the provider reports it and public market rates otherwise, divided by successful root-cause identifications. Time is the median wall-clock investigation, alert to answer.

Single run per task in this edition; scores on repeated runs may vary. Provider prices change frequently — cost figures reflect rates at the time of the run. Want your model or endpoint benchmarked? Get in touch.