Skip to main content
November 21, 2026

Skills Regress on Model Upgrades: A Minimal Eval Loop for SRE Agents

Skills Regress on Model Upgrades: A Minimal Eval Loop

You tune a skill until the agent handles your disk-pressure alerts well. Three months later you switch to a new model release that scores higher everywhere. The skill has not changed, but the agent now restarts the database before it reads the archive config. Nobody notices until an incident review.

A skill is written for one model, whether you meant it to be or not. This post shows why, where models hide the damage, and a small eval loop that turns each upgrade into a measured change.

Why a Model Upgrade Changes Your Skill

The vendors say it plainly. Anthropic's skill authoring best practices state: "Skills act as additions to models, so effectiveness depends on the underlying model. Test your Skill with all the models you plan to use it with."

OpenAI's migration guides say the same about prompts. The GPT-4.1 prompting guide says the model follows instructions "more closely and more literally than its predecessors" and that "existing prompts optimized for other models may not immediately work with this model." In the GPT-5 prompting guide, Cursor reports that a prompt section that helped earlier models had to be softened for GPT-5, because it became counterproductive. The current latest-model guide for GPT-6.2 tells you to "revisit instructions like 'be concise'" when you migrate.

A skill is a prompt with a file name. Every one of those warnings applies to it.

The Quiet Failure: Contradictions Get Resolved, Not Reported

Long skills collect contradictions. Someone adds "never restart a pod without approval" in March. Someone else adds "if a pod is OOMKilled three times, restart the deployment" in June. Both lines are reasonable. Together they conflict.

The model does not stop and ask which one you meant. It picks a resolution, and different models pick differently:

  • The GPT-4.1 guide says that with conflicting instructions, the model "tends to follow the one closer to the end of the prompt."
  • The GPT-5 guide says contradictory instructions are more damaging to GPT-5, because it "expends reasoning tokens searching for a way to reconcile the contradictions" instead of picking one at random. Its example is a scheduling prompt where a consent rule collides with an auto-assign rule.

So the same contradictory skill can behave one way on the old model and another on the new one, and neither run tells you there was a conflict. Neither vendor documents that the model will flag it. Make your harness flag it:

  1. Add one line to every skill: "If two instructions in this skill conflict for this incident, name both, say which one you followed, and why."
  2. Keep one seeded case per important skill with a deliberate conflict, and check that the report names it. If a new model stops reporting the seed, its silence means nothing.
  3. Treat every reported conflict as a skill edit, not a model problem.

The Loop

The loop has four steps, and none of them needs a platform.

1. Every new model triggers a comparison run. Before you switch production, run your current skills on your current cases with the old and the new model, same harness, same tools, several times per case. Anthropic's guide to agent evals notes that "because model outputs vary between runs, we run multiple trials," and defines pass^k, the chance that all k trials pass. For skills that drive production actions, consistency is the metric: use pass^k, not best of k.

2. Notable cases go into an evals repo. A case is notable when the two models disagree, when a human corrected the agent during a real incident, or when the agent reported a conflict. Write the case down while the transcript is fresh.

3. The repo grows from real tasks. Do not invent cases. The guide says "20-50 simple tasks drawn from real failures is a great start" and points at your bug tracker and support queue. For an SRE agent, those are your incident tickets and postmortems.

4. Once the set is large, generate a candidate revision and test it. Feed the current SKILL.md and the failing transcripts to a model and ask for a minimal diff. Anthropic's best-practices page calls this "Claude A" refining the skill while a fresh "Claude B" runs it on real tasks. Run the candidate against the whole set, on every model you use. Wait until the set covers the skill's main alert types: a revision tuned on the five newest failures can break ten older cases, and only a large set shows it.

A Minimum Harness

Keep it in git next to the skills:

skill-evals/
  skills/                      # the SKILL.md files production runs
  cases/
    disk-pressure-db3.yaml
    oom-loop-checkout.yaml
  runs/
    2026-11-21_old-model.jsonl
    2026-11-21_new-model.jsonl
  compare.py

One case per file, built from a real incident:

id: disk-pressure-db3
source: incident from 2026-11-03     # real task, not invented
kind: regression                    # or capability
skill: investigate-disk-pressure
input: |
  Alert DiskSpaceLow on db-3: /var is 97% full.
expect:
  root_cause: WAL archive not pruned after the backup job change
  must_not: ["rm -rf", "systemctl restart postgresql"]
note: new model restarted postgres before reading the archive config

Grade the outcome, not the path: the evals guide calls step-order checks "overly brittle" because agents find valid routes you did not plan. Check three things: the root cause matches the reference answer, no forbidden command ran (check that in code, not with a judge), and any conflict was reported. If a case fails on every model and every trial, suspect the case first: the guide says a 0% pass rate "is most often a signal of a broken task."

Your runner writes one JSON line per case and trial. The comparison is short:

#!/usr/bin/env python3
"""Compare two eval runs: python3 compare.py runs/old.jsonl runs/new.jsonl"""
import json, sys
from collections import defaultdict

def load(path):
    trials = defaultdict(list)
    for line in open(path):
        r = json.loads(line)  # {"case": "...", "trial": 1, "pass": true}
        trials[r["case"]].append(r["pass"])
    return {case: all(passes) for case, passes in trials.items()}  # pass^k per case

old, new = load(sys.argv[1]), load(sys.argv[2])
for case in sorted(old.keys() | new.keys()):
    a, b = old.get(case, False), new.get(case, False)
    if a != b:
        print(("REGRESSED " if a else "IMPROVED  ") + case)
print(f"pass^k: old {sum(old.values())}/{len(old)}, new {sum(new.values())}/{len(new)}")

The total hides what matters: a new model can pass more cases and still fail the one that keeps it from deleting data. Read every REGRESSED line.

Reading the Comparison

Result What it usually means Action
Passes on old, fails on new The skill leaned on the old model's habits Read the transcript, fix the skill, rerun both models
Fails on both Broken case or a real capability gap Check the reference answer; move it to capability cases
Fails on old, passes on new New model is better, or the skill over-explains Keep the case; try trimming the skill
Conflict reported The skill contradicts itself Fix the text and add the conflict as a case
Seeded conflict not reported The new model resolves conflicts silently Do not trust its silence; lint skills by hand

A Review Cadence

Trigger What to run Gate
New model, new provider, or new serving stack All cases, old vs new, 3 trials each No regression case flips to failing
Skill edit Cases for that skill, production model Same as above
Agent was wrong in a real incident Add a case within a week Case reproduces the failure
Set has grown enough to cover the skill Generate and test a candidate revision No regressions, capability cases improve

The serving stack row matters for multi-provider setups: the same model name at another provider can behave differently, as our post on canary tests for LLM providers shows. A canary catches a broad drop. This loop catches the narrow one: your skill, on your incidents.

Applying It to Akmatori Skills

Akmatori skills are SKILL.md files in the open Agent Skills format. You can put a directory with a SKILL.md in the skills directory and press Sync, so the skills your evals test can be the same files production runs. Three details from the docs matter for this loop:

  • Akmatori rewrites the frontmatter to name, description and metadata, so a version key is dropped. Version skills in git, and record the commit in each eval run.
  • Akmatori appends generated sections to each skill: the memory manifest and a pointer to the connected-tools skill. Run comparisons through the real harness, not a bare API call, or you test text the agent never sees.
  • Models are configured in Settings → AI models (providers include OpenAI, Anthropic, Google, OpenRouter and custom OpenAI-compatible endpoints). Run the comparison on a staging install with the new model active, and switch production only when the gate passes.

The docs advise testing a skill by asking the agent about a known problem and reading the run log. Each eval case is that check, written down. SRE Bench applies the idea at benchmark scale.

Conclusion

Skills are not model-neutral, and models resolve contradictions quietly and differently. Keep a repo of real incident cases, compare old and new models on every upgrade, make the agent report conflicts, and change skills only through a candidate that passes the whole set. Then a model upgrade is a diff you can read.

If your team wants AI-assisted incident workflows with skills you can test across model upgrades, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.

Automate incident response and prevent on-call burnout with AI-driven agents!