Walking the Claude price ladder until a model failed
Four Claude models, six graded agent tasks, five runs each. The $0.35 rung guessed instead of computing, and a 2.5× cheaper tier only billed 24% less.
Last updated
The cheapest and the most expensive model in Anthropic’s current lineup differ by 10× on the price sheet. We ran four of them down the same six graded agent tasks, five runs each under recorded conditions, serialized with rotating model order, to find the rung where the ladder actually breaks. It cost $32 to find out. The break was not where we predicted, and most of the mid-tier’s price advantage did not survive contact with the meter.
The tasks are scaled-down versions of public benchmark families (bug fixing, function synthesis, refactoring, environment repair, code review, constrained reporting), and every transcript, grade, and diff is in the public bout directory. Hypotheses were committed before the first run.
The scoreboard
| Model | Runs passed | Cost per full pass | Wall clock per pass |
|---|---|---|---|
| Haiku 4.5 | 28/30 | $0.35 | 196s |
| Sonnet 5 | 30/30 | $1.30 | 328s |
| Opus 4.8 | 30/30 | $1.72 | 482s |
| Fable 5 | 30/30 | $3.13 | 430s |
Three of four models cleared everything, which mostly tells you our tasks stop discriminating above the bottom rung; we said as much in the pre-registered design. The information is in the two rows where something moved.
Sonnet 5 is 2.5× cheaper per token and 24% cheaper per pass
Sonnet 5 went thirty for thirty and beat Opus 4.8 on measured cost, $1.30 against $1.72 per full pass. Case closed, migrate the daily drivers?
Check the size of the win first. Sonnet 5 lists at $2/$10 per million input and output tokens against Opus 4.8’s $5/$25, and the same 2.5× ratio holds on cache writes and cache reads (Anthropic pricing). Run Opus’s exact token usage through Sonnet’s rate card and a pass costs $0.69. We measured $1.30. The gap is appetite: Sonnet took more turns and read far more cached tokens per task than Opus on this workload, and a tier’s per-token discount can’t outrun a model that uses more tokens. A price sheet offering 60% off delivered 24%.
That 24% is durable now, which it wasn’t when we ran the bout. The $2/$10 we metered at was introductory pricing scheduled to rise to $3/$15 on September 1, 2026; on August 10 Anthropic cancelled the increase and made $2/$10 the standard price. The costs in the table above are list-price costs, and there is no cliff to plan around. Keep budgeting off your own meter: the price sheet said 2.5×, the meter said 1.3×.
Haiku 4.5 guessed
Haiku’s two failures both came on the constrained-reporting task: compute revenue figures from a 30-order CSV, publish them in a report and a JSON summary. The grader checks against independently computed truth. Haiku’s run 1 answered $6,618.09 total revenue. Run 2 answered $5,288.55. The truth, both times, was $7,154.44.
Different wrong answers to the same arithmetic is a signature, and the tool logs confirm it. The two failing runs used only file reads and writes: the model looked at the CSV and wrote down totals it made up in its head, in about 14 seconds. The three passing runs did what the other three models did in every one of their runs on this task: wrote a script, ran it, reported what it printed. You can read the wrong answers yourself (run 1, run 2).
One asymmetry is declared in the design doc and belongs here: Haiku 4.5
doesn’t accept the API’s effort parameter, so it ran at its native default
while the other three were pinned to xhigh. That is a plausible
contributor, since effort is the knob that governs how long a model thinks
before it answers. It doesn’t explain the split, though. The same model at
the same setting wrote and ran the script three times out of five.
That’s a failure of diligence, not capability. And a rung that breaks two times in five is worse than one that breaks every time, because it will pass your one-shot eval and then miss in production on a schedule you can’t predict. A broken rung gets rejected; a flaky one gets deployed.
The repair is cheap, though. Haiku’s failure mode is specific and mechanical: it skips the computation step. A harness rule that requires code execution for any arithmetic, plus a verification pass that recomputes one figure independently, would have caught both failures before they shipped. At $0.35 per pass against Opus’s $1.72, there’s a lot of margin available to spend on guardrails.
How to run this on your own ladder
Pick the cheapest model you’d consider, run your actual workload against it five times, and read the failures before the pass rate. A 60% pass rate with a mechanical, guardable failure mode may be a better deal than a 100% pass rate at five times the price. A 60% pass rate from scattered, unpredictable errors is a rejection. The distinction only shows up with repeats, and it’s invisible in a leaderboard number.
Scoring whether your eval can even see that distinction is part of the production-readiness audit; the ladder walk is usually the first thing we run after standing the harness up.
Questions this raises
Straight answers.
- Is Claude Sonnet 5 cheaper than Claude Opus 4.8 for agent workloads?
- Yes, on our six-task agent workload: $1.30 against $1.72 per full pass, both metered at the rates in force today. The gap is narrower than the price sheet implies. Sonnet 5 lists at $2/$10 per million input/output tokens and Opus 4.8 at $5/$25, so Sonnet is 2.5× cheaper on every rate category, but it took more turns and read far more cached tokens per task, and the saving came out at 24%. The meter bills the tokens a workload actually uses.
- Why did Claude Haiku 4.5 fail, and how often?
- Two runs out of five on one task, and the failures were skipped work rather than wrong ability. Asked to compute revenue figures from a CSV, the failing runs read the file and wrote answers without ever running code: different wrong totals each time ($6,618.09, then $5,288.55, against a true $7,154.44). The three passing runs used Bash to compute. Haiku can do the task; it doesn't reliably decide to.
- Should I use the cheapest model that passes my eval?
- Only if it passes repeatedly. A model that passes three runs out of five will pass a one-shot eval most of the time and still fail two production runs in five. If the cheap rung shows a skipped-computation failure mode, you can often keep it by adding a guardrail that forces the behavior — require code execution for arithmetic, and verify outputs against an independently computed answer.
Production-Readiness Audit
The writeup has a service behind it.
If this is your situation, the production-readiness audit is where it gets fixed — by the person who wrote this.