We built a flip-flop detector for Opus 5. It made the same mistake the internet did.

The complaint was specific enough to test, which is rarer than it should be. Claude Opus 5 flip-flops in agentic loops, the reports said, reversing the same decision round after round: one Hacker News comment described thirteen rounds of it and added “never saw anything remotely this bad on Opus 4.8.” So we built a detector. A coding task with genuine tension, a money formatter where the sign has to be decided after rounding so a careless fix to one case re-breaks another, and a metric that flags any run where the failing-test count goes back up after coming down. A fix that breaks something already green is the whack-a-mole the complaint describes, and it leaves a trace in the transcript.

The detector fired three times, all on Opus 5 at high effort, all on the task where it had to repair a broken build environment. Then we read the three transcripts.

Every one was the model deliberately breaking its own config to prove the check would catch it. “make test with a deliberately broken config (should FAIL),” then a restore. A bad batch_size written in on purpose, “must FAIL,” then reverted. In the third it changed a file’s exec bit back to confirm that was load-bearing, saw the permission error it expected, and set it right. Our detector had watched a model verify its own work with adversarial care and scored it as thrashing. That is a hypothesis about the complaints, not a proof of them: we cannot see the Hacker News commenter’s session, and thirteen rounds on a large real repository could be something our small tasks never provoke. What we can prove is that the exact shape people describe, a check that passes and then fails again, is also what rigorous self-verification produces, and that a metric built to catch the first will catch the second.

What the number actually is

So we read the rest. Not just the three that fired: across 145 runs where the model iterated at all, Opus 5 and Opus 4.8, pooled over every task and window we have, there were exactly three moments where a passing check later failed. All three are the ones above, all three intentional. Genuine flip-flops, a fix that broke something already passing: zero. Code reverts, the same lines toggled back and forth: zero. On this failure mode, in our runs, it did not reproduce.

Read that zero with the metric’s blind spot in view, because the whole point of this piece is that the metric has one. The detector only sees an oscillation that crosses the checker’s pass/fail line. A decision reversed between two states that fail the same test, or churn inside code the hidden checker never exercises, would leave the failing count flat and never fire. The revert check catches some of that, the same lines going back and forth, and it also found zero. But a reversal the tests cannot feel is a reversal we cannot count, and no amount of reading fixes that.

The controlled part is smaller than 145 and worth separating out. Pre-registered before it ran: one task built for the failure mode, a second harder one added when the first turned out to be solvable in a single edit, both graded by a hidden checker the model never sees. Four runs each of Opus 5 at low effort, Opus 5 at high effort, and Opus 4.8 at high effort, same window so nothing is compared across weeks. That grid is where the Opus 4.8 column sits next to Opus 5; the other 121 runs are older bouts pooled in for the flip-flop count, not the head-to-head. The detector, the design, and every transcript are public.

What does reproduce

Volume. On the environment-repair task, Opus 5 at high effort ran the checks 5.25 times per attempt. Opus 4.8 ran them 2.25 times. Same task, same window, more than double the checking.

A model that runs the tests three extra times before it stops, and that pauses in the middle to break its own fix and confirm the failure, looks on a live stream like a model that cannot make up its mind. It is not indecision. It is the same behavior we measured last week from the other side: Opus 5 verifies more than its predecessor, writes its own checks, and here it checks that the checks fail when they should. The diff never oscillates. The activity around the diff does.

The dial, and where it stops explaining things

The setting moves it, at least for Opus 5. At low effort Opus 5 ran the environment-repair checks exactly twice in every one of its four runs and made no extra edits. At xhigh, the tier this series pins, one step above the API’s own default, it ran them three to eight times. Dialed down, Opus 5 does roughly a third of the verification, on the same model, with every run still landing a passing solution. All thirty-two runs of the grid converged. They took different numbers of steps to be sure of it.

The cross-model version of the claim is where I have to stop. It would be tidy to say low-effort Opus 5 just does what Opus 4.8 does, except Opus 4.8’s own four low-effort runs came in at two, five, five, and six checks, noisier than tidy wants, and higher on average than 4.8 at high effort, which is the wrong direction for a clean story. At four runs a cell, those means wobble too much to carry an equivalence. What all four cells agree on is the number that was ever the question: zero reversed fixes.

About the grain of salt

None of this makes the original commenter wrong. A comment is one workflow, one codebase, one effort setting, run once, with no counterfactual sitting next to it. It is real evidence about that person’s afternoon. It is weak evidence about the model, because everything that would let you separate the model from the setup has been averaged away by the time it reaches a text box. Our 145 runs are not the last word either; they are controlled tasks, small ones, and a real repository is messier than a broken Makefile. What they have that the comment does not is a same-window Opus 4.8 sitting in the next column and a metric you can point at.

The tell is that our own detector, built by people who do this deliberately, made the identical error: it saw careful work and called it churn. A metric that can be fooled into a false alarm can also miss the real thing, so that humility runs in both directions, including toward our own zero. But it is the point about the timeline that it sharpens: if a purpose-built metric gets fooled by a model verifying itself, a person watching tokens scroll past is not better placed. That is the whole case for reading the timeline reports as weather. They are what the sky looked like from one window, on one afternoon.

We almost shipped the false positive. The number came back “three regressions, Opus 5 high effort only,” which is a clean, publishable, wrong headline, and the only thing between it and the draft was reading the three transcripts it came from. Do that before you believe your own tools, and before you believe a screenshot.