We planted four faults. The harness added a fifth.

Our terminal-repair task breaks make test four ways on purpose: a Makefile recipe indented with spaces instead of a tab, a check script stripped of its execute bit, CRLF line endings on that script’s shebang, and a trailing comma in a config JSON. The faults reveal themselves strictly one at a time; you cannot see error N+1 until you have fixed error N. We have now run 19 public attempts at it: five models (Claude Fable 5, GPT-5.6 Sol, Kimi K3, GLM-5.2, one Claude Opus 4.8 anchor run), three harnesses (Claude Code, Codex CLI, Kimi Code), three runs per cell. Every one of the 19 scored 4/4.

The scores carry no information. Everything this task knows about these models lives in the transcripts, and the sharpest thing it knows is about our own harness.

The onion, as one run saw it

Here is the whole planted sequence from one Sol run in Claude Code (bouts/2026-07-17-fable-sol-kimi/04-terminal/gpt-5.6-sol/run-1), trimmed to the checker calls and picked for legibility, not sampled:

$ make test
Makefile:4: *** missing separator.  Stop.
$ make test                                # after re-tabbing the recipe
make: ./scripts/run_checks.sh: Permission denied
$ chmod +x scripts/run_checks.sh && make test
make: ./scripts/run_checks.sh: No such file or directory
$ make test                                # after stripping \r
Traceback (most recent call last):
  File ".../tools/validate_config.py", line 18, in <module>
    main(sys.argv[1])
$ make test
3 passed

The third error is the one that teaches. The file exists; the kernel is looking for an interpreter literally named /bin/bash\r and blaming the script. Every run that hit that message diagnosed it correctly, and all 19 final reports named the real cause. With known ground truth and a deterministic grader, 4/4 is near a ceiling, so we don’t lean on it. The honesty datum that matters comes later.

The analysis was pre-registered before we read a single transcript (DESIGN, per the series method), and the scorecard came back 3 of 7, with the design itself flagging which hypotheses were partially derivable from published data. The misses are the article.

Peel or batch

We predicted at most 3 of 19 runs would fix a fault before its error message had appeared on screen. Ten did. That was the biggest miss, and it has structure.

A model can work this task two ways. It can peel: run the checker, read the error, fix exactly that, repeat. Fifteen of 19 runs paid the full serial price of five or more checker executions. Or it can batch: read the broken files directly, spot several faults at once, fix them before their errors ever fire. Spaces versus tab and a stray \r are invisible in a terminal render, but a model reading tool output sees raw bytes; cat is cat -A if you aren’t a human.

Where a run landed on that split tracked the harness more than the model, with one caveat we will get to. Sol batched in 3 of 3 runs under its own Codex CLI and 1 of 3 under Claude Code. Kimi K3 batched in 3 of 3 under Kimi Code and 1 of 3 under Claude Code. One Codex run, after the permission error, dumped the check script and the config in a single command; the script’s \r bytes and the config’s trailing comma were both sitting right there in the output, and it fixed both before either error could fire. (In this piece “home” means the vendor’s own CLI, Codex for Sol and Kimi Code for Kimi; our standing disclosure that Claude Code is home field for Anthropic models is the same word pointing the other way.)

Two bounds before that becomes a slogan. Only Sol and Kimi ran in two harnesses, and the Kimi home cell is contaminated by what the next section describes, so the clean evidence for “the harness changes the strategy, not just the bill” is Sol’s 3/3 versus 1/3, n=3 per side, on one task. It extends the pairing-sign finding from cost into method, and it is hypothesis-grade, not law. And batching is not free virtue: this task’s four faults are all real, so it contains no decoy that would punish a model for fixing something it never confirmed. GLM-5.2 read the damaged files in all three of its runs, cat -A included, and still paid all four errors serially every time. Surveying is not batching. Batching bought speed, never correctness: 4/4 was already everyone’s score.

The one Opus 4.8 run is the corpus extreme: two checker executions total, one red and one green, 12 tool calls, 77 seconds. Full inspection, one batch fix, one confirmation. It is a single run and we are not calling it a trait.

The fifth fault

Kimi K3 under its own CLI took a median 340 seconds on this task against 119 in Claude Code, and our published bout table has carried that row (386 ±111s) without an explanation. The walkthrough found it. The Kimi Code driver runs with a redirected HOME for auth isolation, and on this machine pytest lives in the real user’s ~/.local site-packages. Redirect HOME and python3 -m pytest stops resolving:

./scripts/run_checks.sh
config ok: data/config.json
all checks passed
python3 -m pytest -q tests/
/usr/bin/python3: No module named pytest

(bouts/2026-07-18-kimi-homegame/04-terminal/kimi-k3-kimicode/run-2)

A fifth fault, in exactly one harness, planted by nobody. The task’s chain puts pytest last, so it can only appear after all four planted faults are fixed; it masked nothing. The three runs solved it three different ways: one resolved the real user’s site-packages through the passwd entry, one prefixed the pytest invocation with PYTHONPATH, one symlinked five package directories into the workspace. That third workaround is why one run’s diff touches eight paths instead of three, and the ±111s spread is three remediation strategies, not one flaky model.

The driver’s timestamped wire log prices it. The four planted faults were done by t+68, t+82, and t+122 seconds in the three runs, right in line with the same model’s 102 to 136 second full walls in Claude Code (walls that are themselves mostly model latency between tool calls, about 97 of a 119 second median). The segment from the first pytest error to green ran 154 to 335 seconds: 51% to 65% of each run’s wall clock. On this task, most of the home-harness slowdown we had filed under style was our own sandbox’s fault; the published aggregate, Kimi Code slower but cheaper across eight tasks, still stands, with this task’s share of it now reattributed. The grader reruns make test in a normal environment, so every score stayed 4/4, and the same defect explains three “peek-check” warnings we published against those transcripts: the redirected HOME lives inside the repo tree, so the model’s site-packages archaeology printed repo paths and tripped the grader-asset scan.

One check we ran after the pre-registered analysis, labeled as such: the same error string appears in all three 01-bugfix and all three 03-refactor runs from that bout. Nine of its 24 runs hit the fifth fault. On those two tasks the agent invokes pytest itself, so it routed around the problem in seconds (one run built a fresh venv and moved on). 04-terminal is where it got expensive, because the contract says make test itself must pass, and you cannot route around a Makefile you are being graded on.

This lands in the week OpenAI retracted its recommendation of SWE-bench Pro after finding roughly 30% of tasks broken. Ours is a 19-run battery, not a leaderboard, but the mechanism is identical and it happened to us with four hand-planted faults and a deterministic grader: the environment drifted, the scores stayed perfect, and only the transcripts knew. The one reporting test in this corpus with any teeth is that all three Kimi Code runs wrote the unplanted fault into their SOLUTION.md unprompted, correctly attributed to the environment. The models disclosed our bug before we found it.

Every transcript, wire log, timestamp, and adjudication is in the public repo. The cells are n=3, the figures are medians, the rubric judge never ran (this task grades deterministically), and Fable 5 helped build the harness and this analysis while being a subject in it. The anchor run predates this week’s Opus 5 release, which we have not bouted yet.

If you run agents in a sandbox, treat your isolation choices as unplanted faults waiting for a benchmark: diff what the sandbox hides against the target environment (HOME, PATH, site-packages) before you trust any agent-debugging result from inside it, which is the first thing a production-readiness audit checks about a harness. And score the transcript, not the exit code. Ours put the bug in writing three times before we read it.