The Model Got Dumber Today: Canary Tests for Your LLM Provider

On November 5, a developer in a large chat of engineers who use coding agents every day wrote that both of their models were "dumb as hell" and did not understand simple questions. Three days earlier, the same work went well. Another person replied that for them the model worked exactly as before. The next morning, the same developer wrote that the agent now proposed things they had not thought of, and did by itself what they had to push it to do the day before.

Was the model worse on November 5? Nobody in the chat could say. One person asked for "a Downdetector for model intelligence". Another asked what you would even do with the answer.

For a coding assistant, a bad day is an annoyance. For an AI SRE agent that investigates production incidents, the LLM is a production dependency. If it degrades silently, the agent writes confident but wrong root causes at 3 AM. You monitor every other dependency with a health check and an SLO. This post shows how to do the same for the model, with a canary test that costs cents per run.

Silent Degradation Is Real

"The model got dumber" is often just a hard task, a long context or a bad prompt. But real degradations happen, and the provider does not always see them first.

In September 2025, Anthropic published a postmortem of three infrastructure bugs that reduced Claude's response quality for weeks:

• A routing error sent some Sonnet 4 requests to servers configured for a different context window. At the worst hour, 16% of Sonnet 4 requests were affected. The routing was sticky, so a user who hit the bug once was likely to hit it again.
• A TPU misconfiguration sometimes gave a high probability to tokens that should rarely appear, for example Thai or Chinese characters in English answers and syntax errors in code.
• A compiler bug in approximate top-k token selection affected Claude Haiku 3.5 and possibly a subset of Sonnet 4 and Opus 3 requests.

Two lines from that postmortem matter for anyone who runs an agent. First: "The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes." Second, the fix: they will run quality evaluations "continuously on true production systems". Anthropic also states that it never reduces model quality because of demand, time of day or server load. The cause was bugs, not a policy, and a bug is exactly what a canary is for.

The Same Model Name Is Not the Same Model

The second source of drift is the serving stack. An open-weight model runs at many providers, each with its own inference engine and settings. Moonshot AI built the K2 Vendor Verifier because users reported tool-call problems with Kimi K2 at third-party providers. In its test of kimi-k2-thinking from 2025-11-15, Moonshot's own API returned a schema-valid tool call 100% of the time. Third-party providers ranged down to 83.05%. For an agent, an invalid tool call is a failed step.

Routers add one more variable. By default, OpenRouter chooses a provider for each request by effective price, throughput, latency and reliability, and it falls back to other providers when one is unavailable. You can filter providers by quantizations, pin them with order, and turn off fallbacks with allow_fallbacks: false. If you do not, two requests to the same model name can go to different stacks. The same page warns that quantized models "may exhibit degraded performance for certain prompts".

What a Canary Must Measure

The chat raised a good objection: a full benchmark costs a whole usage limit to find out whether the model is worse. A canary is not a benchmark. It is a smoke test with these properties:

• Small and fixed. 20 to 50 tasks that never change, so today's result compares with last week's.
• Known answers, checked by code. No judge model. The canary must not depend on the thing it tests.
• Step-level checks. The postmortem shows why: an agent loop recovers from single mistakes, so task success can hide a degradation. Check single steps: is the tool call valid, are the arguments right, is the extracted value exact.
• Your own task shapes. Use the prompts, tools and context sizes of your real workload. A model can pass a public benchmark and still fail your tool schema.
• The full signal, not only accuracy. Record pass count, errors (timeouts, overload responses), median latency and tokens. A model that suddenly needs twice the tokens for the same answer has changed, even if it still passes.

A Minimal Canary

This script sends each task to any OpenAI-compatible endpoint, checks the answer or the tool call, and prints one JSON line per run.

#!/usr/bin/env python3
"""LLM canary: run fixed known-answer tasks and print one JSON line per run."""
import json, os, statistics, sys, time, urllib.request

BASE = os.environ["LLM_BASE_URL"]   # any OpenAI-compatible endpoint
KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]
EXTRA = json.loads(os.environ.get("LLM_EXTRA", "{}"))  # e.g. provider pinning

def ask(task):
    body = {"model": MODEL, "messages": [{"role": "user", "content": task["prompt"]}], **EXTRA}
    if "tools" in task:
        body["tools"] = task["tools"]
    req = urllib.request.Request(
        f"{BASE}/chat/completions", json.dumps(body).encode(),
        {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    start = time.monotonic()
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r), time.monotonic() - start

def passed(task, resp):
    msg = resp["choices"][0]["message"]
    if "expect_tool" in task:
        call = (msg.get("tool_calls") or [{}])[0].get("function", {})
        if call.get("name") != task["expect_tool"]:
            return False
        args = json.loads(call.get("arguments") or "{}")
        return all(args.get(k) == v for k, v in task["expect_args"].items())
    return task["expect"].lower() in (msg.get("content") or "").lower()

tasks = json.load(open(sys.argv[1]))
ok, errors, tokens, latency = 0, 0, 0, []
for task in tasks:
    try:
        resp, seconds = ask(task)
    except Exception:
        errors += 1          # HTTP error, timeout, overload
        continue
    latency.append(seconds)
    tokens += resp.get("usage", {}).get("total_tokens", 0)
    try:
        ok += passed(task, resp)
    except (KeyError, IndexError, ValueError):
        pass                 # malformed answer counts as a failure
print(json.dumps({
    "ts": int(time.time()), "model": MODEL, "passed": ok, "total": len(tasks),
    "errors": errors, "tokens": tokens,
    "p50_s": round(statistics.median(latency), 1) if latency else None,
}))

Two example tasks, one answer check and one tool-call check:

[
  {
    "prompt": "Log line: 'upstream connect error, reset reason: overflow, cluster=payments-api'. Which service is failing? Answer with the service name only.",
    "expect": "payments-api"
  },
  {
    "prompt": "The /var disk on host db-3 is 97% full. Find the largest directories.",
    "tools": [{"type": "function", "function": {
      "name": "run_command",
      "description": "Run a read-only shell command on a host",
      "parameters": {"type": "object",
        "properties": {"host": {"type": "string"}, "command": {"type": "string"}},
        "required": ["host", "command"]}}}],
    "expect_tool": "run_command",
    "expect_args": {"host": "db-3"}
  }
]

Run it from cron and append the output to a file, or push the numbers to your metrics system:

LLM_BASE_URL=https://openrouter.ai/api/v1 LLM_MODEL=<model> LLM_API_KEY=... \
LLM_EXTRA='{"provider": {"order": ["<provider>"], "allow_fallbacks": false}}' \
  python3 canary.py tasks.json >> canary.jsonl

Pin the provider in the canary, as above, so a change in the result points at one stack. Then run a second canary without pinning, the way production calls the model, to see what your users get.

Alert on a Trend, Not on One Run

Models are not deterministic, so one failed task means little. With 20 tasks and a model that passes 90% of them on a normal day, one or two failures are normal noise. A useful rule:

• Collect a baseline over a week of runs.
• Alert when two runs in a row are well below the baseline, for example five or more failures out of 20 when the normal is one or two.
• Alert separately on errors and on latency or tokens that double. These often show a capacity problem or a stack change before accuracy drops.

Cost stays small. 20 short tasks of about 2,000 tokens each are 40,000 tokens per run. At $1 per million tokens that is $0.04, so four runs a day cost less than $5 a month.

What to Do When the Canary Fires

The chat asked the right question: "If the model got dumber, what is the next step?" Prepare the answer before the alert:

1. Check that it is not you. Did your prompt, tools or harness version change? The canary runs a fixed input, so if it fails with no change on your side, the cause is upstream.
2. Switch the stack, not the workflow. Keep a second provider for the same model, or a second model that you already measured on your tasks. Our model routing cheat sheet explains how to choose one.
3. Keep the evidence. Save the failing requests and responses. A provider can act on a reproducible case. Anthropic's postmortem says that privacy controls made user reports hard to reproduce.
4. Re-run before you switch back. Wait for the canary to return to the baseline.

Canaries and Real Incidents

A canary catches broad degradation. It does not prove that the model handles your incidents well. For that, you need real investigations with known root causes. This is what SRE Bench does: a fresh environment, a real fault, and a judge that compares the report with the written root cause. Our table is one run per task, and the methodology notes that repeated runs may vary. That is the same lesson at a larger scale.

Akmatori records the model, input and output tokens, cache tokens, cost and duration of every agent run. Plot tokens per investigation over time: if the same kind of alert suddenly needs twice the tokens, the agent is working harder for the same answer. The token cost post shows how to read those numbers.

Conclusion

Do not argue about whether the model got dumber. Measure it. A canary of 20 fixed, code-checked tasks, run on a schedule and pinned to one provider, costs a few dollars a month. It turns a feeling into a trend line, and it gives you a plan when the trend drops.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.