We reran our own benchmark. The headline didn't survive.
Five serialized reruns per task turned 'Opus faster on every task' into two wins, two ties, two losses. The cost gap held. Single runs measure weather.
Last updated
On July 6 we published a model comparison with a tidy headline: Claude Opus 4.8 beat Claude Fable 5 on speed in all six of our graded agent tasks, at less than half the cost. One run per task per model. We called it “an anecdote with instrumentation” in the limitations section, then put the anecdote’s cleanest number in the title anyway.
This week we ran the replication. The speed half of the headline is gone.
What we changed before rerunning
The original bout ran each task once per model, with both models executing concurrently on the same API account. For the rerun we made three changes and wrote them down first: five repeats per task per model, one run at a time so nothing competes for rate-limit headroom, and a hermetic harness that pins the effort setting and shuts out this machine’s user-level Claude configuration. The hypotheses went into a design document committed before the first run, so we couldn’t quietly move the goalposts after seeing the data. 120 runs, about two hours, $32.47 in API spend.
The speed claim, before and after
| Task | July 6, one run each | July 7, mean of five ±sd |
|---|---|---|
| 01 bugfix | Opus 28s, Fable 45s | Opus 39 ±3, Fable 47 ±3 |
| 02 synthesis | Opus 67s, Fable 87s | Opus 118 ±42, Fable 87 ±14 |
| 03 refactor | Opus 39s, Fable 76s | Opus 44 ±6, Fable 54 ±3 |
| 04 terminal | Opus 76s, Fable 97s | Opus 92 ±12, Fable 93 ±11 |
| 05 review | Opus 38s, Fable 57s | Opus 97 ±44, Fable 94 ±16 |
| 06 instructions | Opus 44s, Fable 71s | Opus 91 ±25, Fable 54 ±4 |
“Faster on every task” became two Opus wins (19% each), two ties, and two Fable wins (26% and 41%). Sum the per-task means and the model that lost the original comparison now finishes the whole course first: 430 seconds for Fable 5 against 482 for Opus 4.8. Every grade stayed perfect on both sides, all sixty runs.
We are not replacing one headline with its mirror image. The honest reading is that per-task latency differences between these two models mostly sit inside the noise, and a single run will hand you whichever ordering the API felt like that afternoon.
How big is that noise?
Between identical runs, same model, same task, same prompt, the standard deviation of wall-clock time ran from 5% to 46% of the mean across our 24 cells, median 13%. The worst cell is worth staring at: Opus 4.8 on the code-review task. Four of its five runs landed between 73 and 84 seconds. The fifth took 177. Nothing about the task changed.
Our July 6 measurement of that same cell was 38 seconds, faster than all five reruns. We caught an unusually good afternoon and published it as a property of the model.
What replicated
The cost gap. The original bout measured Fable at 2.17× Opus’s cost; five repeats put it at 1.82×, both sitting near the 2× list-price ratio. Cost is metered on tokens, and token counts are something the model’s own behavior determines, so they barely move between runs. Wall-clock time belongs to the API’s load balancer. If you must compare models on one run, compare the token-metered numbers and treat the stopwatch as decoration.
What this means for your bake-off
If your team picked a model after a comparison with one run per task, and the winning margins were under about 30%, you measured weather. Rerun it with repeats before the migration budget gets spent. The protocol that caught our error is small: pin the configuration, serialize the runs, repeat each cell five times, and write the hypotheses down before you start. Our harness does all four and is public, with every transcript and grade from both bouts alongside it.
One disclosure the design demanded: the Claude Code CLI auto-updated itself partway through our rerun (2.1.202 to 2.1.203, affecting the final 17 of 120 runs, recorded per-run in the artifacts). The affected runs all sit in the final task group, where all four models ran on the updated version, so the update doesn’t rescue the original headline. Pin your tooling anyway; we now do.
We’ve added a correction to the original note. The rest of this series walks down the Claude price ladder until a model breaks, and then tests whether a four-line prompt can buy the expensive model’s judgment. A production-readiness audit starts with exactly this question: which of the numbers your team is betting on would survive five reruns?
Questions this raises
Straight answers.
- How many runs do you need before comparing two LLMs?
- Enough to know your noise floor. Across our 24 task-model cells, the standard deviation of wall-clock time between identical runs ran from 5% to 46% of the mean (median 13%). Any latency difference smaller than your measured variation is unresolved by a single run. We now treat five repeats as the floor for a latency claim, and we run them one at a time so concurrent runs never share rate-limit headroom.
- Did the cost comparison fail to replicate too?
- No. The measured cost ratio between Claude Fable 5 and Claude Opus 4.8 was 2.17× in the original single-run bout and 1.82× across five repeats, against a 2× list-price gap. Cost is metered on tokens, which the model's own behavior determines; wall-clock time depends on API load and queueing, which it doesn't. Token-metered numbers are stable; stopwatch numbers are not.
- What changed between the original run and the replication?
- Three things, all disclosed in the pre-registered design: five repeats per cell instead of one, strictly serialized execution instead of running both models concurrently, and a hermetic harness that pins effort and excludes machine-level configuration. Opus 4.8 came in markedly slower than its July 6 numbers on three tasks and we cannot fully attribute that to any single cause, which is itself an argument against trusting one run.
Production-Readiness Audit
The writeup has a service behind it.
If this is your situation, the production-readiness audit is where it gets fixed — by the person who wrote this.