The 2× model premium didn't show up in our evals

On July 6 we ran Anthropic’s two frontier models, Claude Opus 4.8 and Claude Fable 5, through the same six graded agentic coding tasks. Twelve headless Claude Code runs, byte-identical prompts, isolated git workspaces, grading scripts the models never saw. Both models went six for six with full marks on every grader. Opus 4.8 was faster on every single task and cost $1.59 to Fable 5’s $3.45.

Two things before the numbers. First, this note was drafted by Fable 5, the model that loses the value comparison below; the graders are deterministic scripts, and the founder reviews everything before it publishes. Second, you don’t have to take either of our words for it: the harness, the task fixtures, the graders, and every transcript, diff, and grade from the bout are public at github.com/mattWoolly/agent-arena.

Why a harness instead of a feel test

The usual way teams pick an agent model is a week of “the new one feels smarter” followed by a migration nobody can defend in a budget review. We wanted the opposite: a fixed task set and a mechanical grade, with cost and latency captured from the run logs, so the comparison survives contact with a skeptic.

Four design rules did most of the work:

1. Isolated workspaces. Each run gets a fresh git repository seeded from a fixture, so every change the agent makes is attributable and diffable. The committed workspace.diff files are the full record of what each model touched.
2. Byte-identical prompts. Both models get the same PROMPT.md, verbatim. No per-model tuning.
3. Hidden graders. The agent never sees the grading script or the hidden test suites, so it can’t overfit to the checker the way it could to a visible test.
4. Self-tested graders. Before the bout, check-graders.sh proved every grader fails the untouched fixture and passes a reference solution. This caught a real bug: pytest’s cache directories tripped the “tests unmodified” check, which would have failed both models on three tasks for something neither did. A grader you haven’t tested is a coin flip wearing a lab coat.

The six tasks, and why these six

Each task is a scaled-down version of a public benchmark family, chosen because together they cover what a daily-driver coding agent spends its day doing. All fixtures and graders are in tasks/; everything below links to raw artifacts you can read yourself.

1. Fix the bug the tests already caught

SWE-bench in miniature: an LRU cache library with two planted defects (reads don’t refresh recency; updating an existing key at capacity evicts a neighbor) and a failing test suite. This is the most common real ask an agent gets. Hidden tests punish anything that games the visible ones.

Both models shipped the same fix, byte for byte, down to the compound condition.

+        self._data.move_to_end(key)
-        if len(self._data) >= self.capacity:
+        if key not in self._data and len(self._data) >= self.capacity:

2. Write five functions from their docstrings

HumanEval/MBPP style: five stubs with edge-case-heavy specs, graded by a hidden suite that includes a performance gate (an O(n) rolling_max must clear 200,000 elements in under three seconds, so a naive max-per-window loop fails). The point is spec fidelity with no tests visible; the agent decides for itself what done means.

Both wrote the identical regex for the duration parser, (?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?, and even the same error string format ("invalid duration: %r"). They diverged only in paranoia. Fable 5 added re.ASCII and an isascii() guard so Unicode digits can’t sneak past Python’s permissive isdigit(); Opus 4.8 closed the same hole with an explicit character-set check. Neither shipped blind: each wrote and ran its own throwaway assert script before finishing, unprompted.

3. Refactor across four files without breaking behavior

Modeled on Aider’s refactor benchmark: convert a dict-passed config to a frozen dataclass across four modules, keep the existing tests passing untouched, and add unknown-key validation. Multi-file changes with behavior locked is where agents historically break things they didn’t need to touch.

The two config.py results differ by an import style and one word in the error message (“Unknown config setting” vs “Unknown config key”). The grader also greps the package for leftover dict subscripting; neither left any.

4. Repair a build that lies to you

Terminal-Bench territory, and the task with the most forensic texture. Four planted environment faults: a Makefile recipe indented with spaces, a check script with CRLF line endings, a stripped execute bit, and a trailing comma in a JSON config. The CRLF fault is nasty because the shell reports it as No such file or directory for a file that plainly exists.

The transcripts show different temperaments. Opus 4.8 surveyed first: four batched commands reading everything, including a cat -A Makefile (whitespace suspicion from the start), before running make test once. Fable 5 ran make test almost immediately and followed the errors. Both fixed the tab, the JSON, and the execute bit quickly; both hit the CRLF fault last; and both reached for the same forensics (file, cat -A, a hex dump) when the error message made no sense. Fable took 19 turns to Opus’s 15 and arrived at the same place.

Neither model tried to weaken a check to get to green, which the grader explicitly watches for.

5. Review a pull request with six planted defects

A diff with six real bugs hidden among benign changes: SQL injection, a mutable default argument, a swallowed exception, an off-by-one in pagination, a file-handle leak, and a None dereference the schema documentation warns about. Scored on recall, with false positives counted against, because a reviewer that cries wolf is worse than no reviewer.

Both found all six with zero false positives, and both did it by pure reading: four file reads and one write each, no code execution. The one visible quality gap in the whole bout is here. Both flagged the swallowed exception and the mutable default separately; only Fable 5 connected them, writing that failures are “silently swallowed with pass instead of being logged, hiding errors and (combined with the shared seen list) marking failed users as done so they are never retried.” That’s the compound failure mode a checklist misses and a staff engineer catches. It didn’t move the score, because both models had already found every planted bug.

6. Produce a report under hard constraints

IFEval style: analyze a sales CSV and emit a report with exact headings, a 250-word ceiling, banned words, a three-bullet limit, and a JSON summary checked against independently computed ground truth. The least glamorous task and arguably the most predictive, because production agents live inside format contracts.

Identical summary.json from both, correct to the cent. Both self-verified against the constraint list before finishing. Fable 5’s prose again went one step further on specificity, quantifying that the West region “sits 35% behind North” where Opus stated the ranking.

The scoreboard

Total across six tasks	Opus 4.8	Fable 5
Tasks passed	6/6	6/6
Wall clock	291 s	434 s
API cost	$1.59	$3.45
Agent turns	56	63
Output tokens	18,083	20,812

The cost gap is mostly the price sheet. Anthropic lists Opus 4.8 at $5/$25 per million input/output tokens and Fable 5 at $10/$50, double on both sides. Fable 5 also produced about 15% more output tokens for the same work, which stretches 2× list price into the 2.17× we measured. Per-task numbers, token counts, and the hand-verification record are in results.md and ANALYSIS.md.

What we actually learned

The result we expected was a capability gap. The result we got was convergence. At this task size the two models produce the same regex, the same dataclass, the same two-line bug fix, and find the same six defects. Where they differ is temperament at the margins: Opus is brisker and terser; Fable reads a little longer, defends a few more edges, and connects findings into failure modes. Real qualities, and our graders priced none of them, because none of them changed an outcome.

That reframes the buying question. For short, well-specified agentic work, both models are past the bar where correctness separates them, so the deciding variables are latency and price, and those point the same direction. A team that migrates its daily-driver agents to the newest model on launch day is paying double for depth its workload may never call on. If your work lives where that depth compounds, the premium may be the bargain. Measure it.

What six tasks can’t tell you

This was one run per task per model: an anecdote with instrumentation, not a benchmark. Every task finished in under two minutes, and Fable 5’s claimed advantages sit in long-horizon autonomous work (the overnight refactor, the sprawling migration) that nothing here exercised. A comparison run by one of the contestants deserves your skepticism however deterministic the graders, which is exactly why the raw material is public: read the transcripts and disagree with us in specifics.

We wrote earlier that an eval harness is the unit-test suite of an LLM feature. Model selection is the same discipline pointed at a different decision. This harness took a day to build, $5.04 to run, and the next bout costs one shell command: bin/run-bout.sh <name> <model-a> <model-b>. Scoring your eval coverage, and standing up harnesses like this one where it’s thin, is part of the production-readiness audit; the model bake-off is usually the first thing a new harness gets used for.