Pre-register your evals. The misses are the yield.

The thing that stopped us from shipping a second bad headline this week was not a better model or a smarter reviewer. It was a one-page text file, committed to git before the experiment it describes had produced a single number.

That file said, in effect: here are 120 runs we’re about to do, here are five things we believe will happen, and here is what we’ll disclose either way. One of its commitments was to put an error bar under a claim we’d already published (“Opus 4.8 was faster on every single task”) by rerunning it with five repeats instead of one. The claim didn’t survive the error bar. The follow-up experiment got the same treatment, five more predictions, before its first run.

The scorecard

Ten written predictions across the two experiments. Six held up. Two earned partial credit: latency noise was real but wider than our predicted band (5–46% per cell against a predicted 10–30%), and the cheap model did fail a task, just not either of the two tasks we named. One was an interpretation guardrail that can’t fail by design. And one was refuted outright: we predicted a four-rule prompt addition would move cost less than 15%, and it moved cost 28%.

The refuted one was the best of the ten. Chasing why cost jumped led to the observation that carried part three: thorough behavior costs output tokens no matter which model performs it, so part of a flagship’s price premium is the token bill of its own habits. We would never have looked without a wrong number staring back from a committed prediction.

A prediction that can’t miss isn’t a prediction. If your team’s eval reports have never once said “we expected X and got Y,” the expectations are being written after the results.

Deviations get a section, not a cover-up

A hundred and three runs into our 120-run bout, the CLI we use to drive the agents auto-updated itself. The last seventeen runs executed on a different tool version than the first hundred and three. That’s exactly the kind of detail that quietly disappears from writeups, because including it feels like weakening your own work.

Pre-registration inverts the incentive. The design doc promises a deviations list, so the update becomes a disclosed, per-run-recorded fact with an assessment of impact (in our case: the affected runs cluster in one task group, both failing runs span both versions, conclusion unchanged). The alternative, a reader discovering it in the raw logs, converts a footnote into a credibility problem. And you should assume readers of an eval will go looking; we publish every transcript precisely because the sort of client we want reads them.

The whole protocol, in six lines

1. Pin everything: model IDs, tool versions, settings, effort. Record them per run, mechanically.
2. Serialize runs that will back a latency claim; concurrent runs share rate-limit headroom.
3. Five repeats per cell minimum. Report spread, never bare means.
4. Commit the design doc, with numbered predictions, before run one.
5. Publish raw artifacts: transcripts, grades, diffs, judge samples.
6. Say who wrote the analysis. Ours are drafted by one of the models under test, which is why the graders are deterministic scripts and the raw data ships alongside every claim.

None of this needs infrastructure. The design docs are markdown; the harness is under 600 lines of shell and Python, public. What it needs is the willingness to be seen missing.

The same protocol pointed at your workload is most of a model bake-off, and the first deliverable of a production-readiness audit is usually exactly this: your eval, rebuilt so its conclusions survive a skeptic with access to the logs.