Fable 5.2 Memorized the Test: How to Add a Contamination Check to Your Agent Evals

Anthropic released Claude Fable 5.2 on November 9, 2026. A week later, Endor Labs ran it through their security coding benchmark and published a result that should change how you read any model upgrade: Fable 5.2 is cheaper and faster, not more secure, and memorizes much more.
On raw scores, Fable 5.2 looks like a clear win over Fable 5.1. Once Endor removed the solutions the model recalled from training data instead of working out, Fable 5.2 fell behind its predecessor on both functional and security pass rates. The run still cost less than a third as much.
This post covers what Endor measured, how they flagged memorized solutions, and how to build the same check into your own evals.
What Endor Measured
The Agent Security League has 200 tasks from 108 open-source Python projects across 77 CWE classes. It extends the SusVibes benchmark. Each one is built from a real vulnerability fix: the agent gets a copy of the repository with a feature removed, a feature request and the functional tests. The security tests are held back for grading.
Two scores come out of each run:
- FuncPass: the code passes the functional test suite.
- SecPass: the code also passes the hidden security tests. It counts only on top of a functional pass.
Endor ran Fable 5.2 through Claude Code at the CLI default effort, with one attempt per task. Here are the numbers with confirmed recall counted as passes ("kept") and with it removed:
| Claude Code with | Recalled solutions | FuncPass kept | FuncPass removed | SecPass kept | SecPass removed | Run cost |
|---|---|---|---|---|---|---|
| Fable 5.2 | 65 | 94.4% | 63.7% | 60.3% | 35.8% | $187 |
| Fable 5.1 | 18 | 93.9% | 86.6% | 41.3% | 36.9% | $672 |
| Fable 5.0 | 42 | 77.1% | 59.2% | 31.8% | 18.4% | $417 |
The "kept" columns say Fable 5.2 beats 5.1 on security by 19 points. The "removed" columns say it is about one point behind on security and 23 points behind on working code. Same runs, same tasks. The only difference is whether recall counts as a pass.
Fable 5.2 also worked very differently. It averaged 10.9 tool calls per task against 42.4 for Fable 5.1. Its mean time per task was 3.6 minutes against 9.5. In 159 tasks it never called Claude Code's file-editing tools and wrote every change through the shell, and 88.5% of its tool calls were Bash.
How Endor Decided a Solution Was Memorized
Endor's rule is that the fix must come from the agent's own reasoning. A fix recovered from git history, the web, a copy in the workspace or training recall counts as cheating. Their whitepaper and the Fable 5.2 post describe a pipeline in two stages:
- Automatic signals flag suspicious runs. They check the agent's trajectory for git or web retrieval, measure how similar the patch is to the known upstream fix, and look for passes on strict tests that assert exact messages. For memorization they also check what the agent wrote first, and whether it wrote a string before it ever saw that string. Their earlier post on training recall lists the giveaways: long copied comments, unusual phrasing, typos copied from the original, exact configuration names, and CVE identifiers that were never in the prompt.
- An LLM judges each flagged run. It gets the trajectory, the problem statement and both patches. Only runs it confirms are removed from the score.
The pipeline involves judgment. Endor counted one OpenStack Neutron fix as solved, even though its ownership check was character-identical to upstream's, after a manual review of the trajectory. They also report 20 "recall-then-diverge" cases, where the model's first edit was close to the upstream fix and later edits moved away from it.
Why Shell-Only Edits Matter
Endor notes that because Fable 5.2 wrote most edits through the shell, with inline Python or heredocs, the pipeline "has to reconstruct most edits from shell commands." A dedicated edit tool leaves a clean record of each change. With cat > file <<EOF, the change is buried in a command string.
This is a monitoring problem as well as an eval problem. If your agent guardrails, approval prompts or audit logs only watch the file-edit tool, a model that edits through Bash skips them. When you upgrade models, check which tools the new model actually calls, not just whether the task passed.
Cheaper Per Task Is Not More Capable
Endor's table puts the cost per secure pass at about $2.9 for Fable 5.2, against about $10.2 for Fable 5.1. That is a real saving. Since the two models are nearly tied on SecPass after removal, for this benchmark you get the same security outcome for a lot less money.
It does not make Fable 5.2 the better coder. For working code, the gap is 63.7% against 86.6%, and API cost does not show what each failure costs you in people's time. Here is a worked example. The rates are Endor's; the review cost is our assumption:
| Per 100 tasks on unseen code | Fable 5.2 | Fable 5.1 |
|---|---|---|
| Functional passes (removed rate) | 64 | 87 |
| Failures an engineer must redo | 36 | 13 |
| API cost (Endor mean, $0.94 vs $3.36 per task) | $94 | $336 |
| Engineer time at 20 min per failure, $100/hour | $1,200 | $433 |
| Total | $1,294 | $769 |
Change the assumptions and the answer can flip. Where most failures are cheap to catch, the cheaper model wins. Where a failure means a broken deploy or a missed root cause, it does not. Do the sum yourself instead of reading "28% of the cost" as "better."
Two caveats. First, the removed score gives Fable 5.2 no credit for the 65 recalled tasks, though it might have solved some of them on its own. So its real capability on those tasks sits somewhere between the two columns, and Endor's data cannot say where. Second, Endor says the cost figures are CLI reports at list price, and they tested one harness at one effort setting.
Add a Contamination Check to Your Own Evals
Your internal agent evals likely have the same weakness. If a task comes from a public CVE fix, a well-known GitHub issue or a published postmortem, the model may have seen the answer. These checks are ordered from cheapest to most expensive:
| Check | What it catches | Effort | False-positive risk |
|---|---|---|---|
| Use private or post-cutoff tasks (your own incidents and postmortems) | Recall of public fixes | Medium, one-time | None |
| Remove git history and block network egress in the sandbox | Retrieving the upstream fix | Low | None |
| Canary strings: list unique comments, identifiers and CVE IDs from the reference fix, and grep the agent's output for them | Verbatim recall | Low | Low |
| "Written before seen": flag any canary string the agent writes before a tool output showed it | Recall that a diff alone hides | Medium, needs full trajectories | Low |
| Patch similarity against the reference fix, combined with few read calls before the first edit | Fixes copied without exploration | Medium | Medium: small fixes look alike |
| Mutated variant: rename the project, functions and config keys, then rerun | Recall tied to surface names | High | Low |
| LLM or human review of flagged runs only | Confirms the above | Per flag | Depends on reviewer |
The first two rows prevent leaks. The middle three detect them. Then report two numbers for every model, the raw pass rate and the pass rate with recall removed, and make decisions on the second. If the gap between them grows sharply with an upgrade, as it did here from 18 cases to 65, your eval measures memory as much as skill.
Make sure the trajectory log records shell writes, not just edit-tool calls. Without them, the "written before seen" check misses exactly the models that use the shell most.
What This Means for Incident Agents
An incident agent has to work out why this cluster is failing today, not recall an upstream fix. The safest eval tasks are your own past incidents, rebuilt in an environment the model has never seen.
That is the design behind SRE Bench. Every model gets a fresh Akmatori installation with no stored memory, inside an environment rebuilt from scratch for each test. A pinned judge model checks the final report against the scenario's written root cause. Cost is divided by successful root-cause identifications, not by tasks attempted, so a cheap model that misses the cause does not look cheap.
Our post on skill regressions after model upgrades describes a small eval loop for each upgrade, and reading the scenario code covers what to audit before you trust an eval. Add the recall check from this post to both.
Conclusion
Fable 5.2 is much cheaper and faster than Fable 5.1, and on Endor's benchmark it nearly matches 5.1 on security once memorized solutions are removed. It is not the better coder: without recall, it fails far more functional tasks. Endor caught that because they recorded trajectories, compared patches with the upstream fix and reviewed flagged runs. Build the same check into your own evals, report scores with and without recall, and work out cost per solved task, including the human time that failures cost.
If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.
