Walking the Claude price ladder until a model failed
Four Claude models, six graded agent tasks, five runs each. The $0.35 rung guessed instead of computing, and the mid-tier stopped being cheap at list price.
Last updated
The cheapest and the most expensive model in Anthropic’s current lineup differ by 10× on the price sheet. We ran four of them down the same six graded agent tasks, five runs each under matched, recorded conditions, to find the rung where the ladder actually breaks. It cost $32 to find out. The break was not where we predicted, and the mid-tier price advantage evaporated on contact with the meter.
The tasks are scaled-down versions of public benchmark families (bug fixing, function synthesis, refactoring, environment repair, code review, constrained reporting), and every transcript, grade, and diff is in the public bout directory. Hypotheses were committed before the first run.
The scoreboard
| Model | Runs passed | Cost per full pass | Wall clock per pass |
|---|---|---|---|
| Haiku 4.5 | 28/30 | $0.35 | 196s |
| Sonnet 5 | 30/30 | $1.30 (≈$1.94 at list) | 328s |
| Opus 4.8 | 30/30 | $1.72 | 482s |
| Fable 5 | 30/30 | $3.13 | 430s |
Three of four models cleared everything, which mostly tells you our tasks stop discriminating above the bottom rung; we said as much in the pre-registered design. The information is in the two rows where something moved.
Sonnet 5 is cheap until the discount ends
Sonnet 5 went thirty for thirty and beat Opus 4.8 on measured cost, $1.30 against $1.72 per full pass. Case closed, migrate the daily drivers?
Check the meter, then the calendar. Sonnet 5 is billed at an introductory $2/$10 per million tokens through August 31, 2026; list price is $3/$15. Scale our measured spend to list and Sonnet costs about $1.94 per pass, above Opus. Same tasks, same grades, higher bill. The reason is appetite: Sonnet took more turns and read far more cached tokens per task than Opus on this workload, and a tier’s per-token discount can’t outrun a model that uses more tokens.
If you migrate to Sonnet 5 this summer, put a reminder in your calendar for September 1 and re-run the cost eval when the intro pricing lapses.
Haiku 4.5 guessed
Haiku’s two failures both came on the constrained-reporting task: compute revenue figures from a 30-order CSV, publish them in a report and a JSON summary. The grader checks against independently computed truth. Haiku’s run 1 answered $6,618.09 total revenue. Run 2 answered $5,288.55. The truth, both times, was $7,154.44.
Different wrong answers to the same arithmetic is a signature, and the tool logs confirm it. The two failing runs used only file reads and writes: the model looked at the CSV and wrote down totals it made up in its head, in about 14 seconds. The three passing runs did what the other three models did in every one of their runs on this task: wrote a script, ran it, reported what it printed. You can read the wrong answers yourself (run 1, run 2).
That’s a failure of diligence, not capability. And a rung that breaks two times in five is worse than one that breaks every time, because it will pass your one-shot eval and then miss in production on a schedule you can’t predict. A broken rung gets rejected; a flaky one gets deployed.
The repair is cheap, though. Haiku’s failure mode is specific and mechanical: it skips the computation step. A harness rule that requires code execution for any arithmetic, plus a verification pass that recomputes one figure independently, would have caught both failures before they shipped. At $0.35 per pass against Opus’s $1.72, there’s a lot of margin available to spend on guardrails.
How to run this on your own ladder
Pick the cheapest model you’d consider, run your actual workload against it five times, and read the failures before the pass rate. A 60% pass rate with a mechanical, guardable failure mode may be a better deal than a 100% pass rate at five times the price. A 60% pass rate from scattered, unpredictable errors is a rejection. The distinction only shows up with repeats, and it’s invisible in a leaderboard number.
Scoring whether your eval can even see that distinction is part of the production-readiness audit; the ladder walk is usually the first thing we run after standing the harness up.
Questions this raises
Straight answers.
- Is Claude Sonnet 5 cheaper than Claude Opus 4.8 for agent workloads?
- On our six-task agent workload, yes at today's introductory pricing ($1.30 vs $1.72 per full pass) and no at list price. Sonnet 5's intro rate of $2/$10 per million tokens runs through August 31, 2026; at its $3/$15 list rate our measured usage works out to about $1.94, above Opus 4.8. Sonnet took more turns and read far more cached tokens per task than Opus, and the meter bills the tokens a workload actually uses.
- Why did Claude Haiku 4.5 fail, and how often?
- Two runs out of five on one task, and the failures were skipped work rather than wrong ability. Asked to compute revenue figures from a CSV, the failing runs read the file and wrote answers without ever running code: different wrong totals each time ($6,618.09, then $5,288.55, against a true $7,154.44). The three passing runs used Bash to compute. Haiku can do the task; it doesn't reliably decide to.
- Should I use the cheapest model that passes my eval?
- Only if it passes repeatedly. A model that passes three runs out of five will pass a one-shot eval most of the time and still fail two production runs in five. If the cheap rung shows a skipped-computation failure mode, you can often keep it by adding a guardrail that forces the behavior — require code execution for arithmetic, and verify outputs against an independently computed answer.
Production-Readiness Audit
The writeup has a service behind it.
If this is your situation, the production-readiness audit is where it gets fixed — by the person who wrote this.