The correct ranking was on line 201. Our agent used line 1.

One sentence from the report our deep research agent finished on 23 August:

Claude-4.6-Opus achieved the highest overall score of 58.9% (86.1% travel, 80.3% shopping), while GPT-5.2-high scored 44.6% overall but relied on an average of ~224 tool calls per task.

The 44.6% is exact. So is the 224. Both are straight out of DeepPlanning, the Qwen team’s benchmark for long-horizon agent planning, where GPT-5.2-high is the top scorer at 44.6% average case accuracy and roughly 224 tool calls per task.

The other three numbers do not exist. The paper has no Claude Opus 4.6 entry at all; its Claude-4.5-Opus scores 26.3% without thinking and 33.9% with. So the sentence takes the benchmark’s winner, demotes it to second, and awards first place to a model that was never evaluated, at a score chosen high enough to win.

Our agent didn’t invent that number. It copied it, accurately, out of the search API.

What we ran

NVIDIA’s AI-Q deep researcher on the NeMo Agent Toolkit, two open-weight models (the lead and planner on openai/gpt-oss-120b, the researcher subagents on nvidia/nemotron-3-super-120b-a12b), Tavily for search. The question was where intelligence lives in 2026 agentic development: the model, the harness, or deterministic code.

Job 4b930b01, submitted 20:39, success at 21:05. Twenty-five minutes, 119 model calls, 2.64M prompt and 69K completion tokens, 39 web searches, 4,321 sources captured by the citation registry, eight references in the finished report.

The retrieval patch did what it was supposed to

The same question on the unpatched tool took 7 hours 33 minutes across 76 calls. Tavily’s content field is a relevance snippet of one or two thousand characters, which cannot hold a results table or a limitations section, so the agent searched, got fragments, and searched again. We patched the tool to ask for raw_content (the whole parsed page) and to use it in place of the snippet. One config flag and a generous length cap.

Wall clock went from 7h33m to 25 minutes. Better: every figure that was actually present in a retrieved page now came through exactly. 44.6 and 224 above, and Mem0’s 92.5 on LoCoMo, 94.4 on LongMemEval, and ~6.9k tokens per query, all three of which are on the page the agent read. In the earlier run, every effect size in the report had been made up.

Line 1 beat line 201

Tavily’s response opens with an answer field: an LLM-written synthesis that sits ahead of the documents, which our config had requested at advanced quality. Here is what it said.

Anthropic’s Claude-4.6-Opus (max) leads the leaderboard with a 58.9% overall score (86.1% travel-case success, 80.3% shopping-case success) while OpenAI’s GPT-5.2-high follows with 44.6% overall (88.5% travel, 83.3% shopping) but relies on an average of roughly 224 tool invocations per task

Two real numbers, four invented ones, and an inverted ranking, inside one grammatical sentence. Our report reproduces it almost word for word.

That whole search result came back at about 163,000 characters, too large for the context window, so the harness spilled it to a file and handed the agent a path. The agent called read_file with offset 0, limit 200. The <Answer> block is line 1.

Then it read the next 200 lines. From line 201 the file holds the transcript of a YouTube explainer about the same paper, and the machine transcript, ASR mangling of “Claude” included, gets it right:

The top model is listed as GPT 5.2 high. […] It got a case accuracy of about 44.6% on average, less than 50%. And that’s the best one. Right behind it was clawed 4.5 opus with thinking enabled.

Correct ranking, correct models, correct score, read into the agent’s context. Nineteen seconds later its reasoning trace begins: “We have retrieved a large result; the answer preview at the start gave concise summary,” and then quotes the fabricated leaderboard back in full. That version went into /shared/research_1.md, from there into the consolidated findings, and from there into section 2.1 of the report with two citation markers attached.

Ground truth was in the window. The pre-digested version was on top, shorter, and phrased like a conclusion. It won.

One more thing about that search: it captured 61 sources, of which 34 were Pexels photographer profile pages.

The plan promised a survey, so the writer produced one

Different failure, different mechanism. Section 6.1 of the report opens:

A 2026 practitioner poll (n = 213) revealed a split: 44% of respondents cite model quality […] while 48% emphasise framework orchestration. The remaining 8% highlighted deterministic code.

No such poll appears anywhere in the run. Not in a search result, not in a tool response, nowhere in 984KB of server log. This does, though, written to the plan file twenty minutes before any of the writing started:

{ "id": "6.1", "title": "Survey of Industry Opinions (2026)" }

The planner committed to a section. Retrieval came back with an opinion piece carrying no numbers. The writer, holding a plan that promised a survey, supplied one: three percentages that sum to 100 and a sample size odd enough to look like fieldwork.

A plan is a promise the writer keeps. When the evidence can’t keep it, the writer keeps it anyway.

All eight URLs resolved, which made it harder to catch

Every reference in the report returns HTTP 200. No invented URLs, which is exactly where the earlier run fell down. Read them, though, and four of the eight labels don’t describe what they point at.

Reference [6], “ENNS retrieval latency breakdown,” is arXiv 2604.13942, which is Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection, a robotics paper. Reference [7], “Toolformer math-agent profiling,” is arXiv 2505.02709, Evaluating Goal Drift in Language Model Agents. Reference [5], “Practitioner Survey on Model vs Framework Dominance (2026),” is AI Digest’s “A new Moore’s Law for AI agents,” an extrapolation of METR’s time-horizon curve with no survey in it. Reference [3], “Azure OpenAI Guardrails & Reflection post-mortem,” is a Microsoft Research announcement of CORPGEN, an agent framework. Reference [1], credited with the entire DeepPlanning section, contains none of that data. And the body cites a reference [10]; the list stops at eight.

Alongside them sit three enterprise case studies (Rakuten, Netflix and Wisedocs, with 27% cost reductions, 34% latency improvements and 97% drops in first-pass errors) cited to the Mem0 page, which mentions none of those companies and none of those figures. They came from the same place the 58.9% did: another answer field, in another search.

Dangling citations announce themselves. Resolving citations with wrong labels don’t. The retrieval fix improved the surface and left the attribution layer untouched.

The controls we’re adding

Drop the vendor’s answer field. It is a second, unaccountable model in the pipeline, and its output arrives first and short, phrased with more confidence than any source it summarises. We asked for it at advanced quality, which is to say we asked for it to sound authoritative. Documents only from here.

Order the context so evidence outranks summary. Where a synthesis has to be present, it goes last, and it gets labelled as generated rather than retrieved.

Require a quote and a locator for every number, then check it mechanically against the retrieved document text with the synthesis excluded. Under that rule, 44.6 and 92.5 pass and 58.9 fails, which is the whole job.

Give the writer a way to return nothing. Every section in that plan got filled because filling it was the only available move. A “no supporting evidence found” outcome has to survive from the researcher through to the report.

Verify each reference title against its own URL before publishing. Fetch it and compare. It takes seconds and it catches robotics papers filed as retrieval benchmarks.

None of that makes the output citable on its own. The report works as a lead generator: the sources it names are real, and reading them is how we found the errors. As a citation of record it is unusable, and no configuration change we’ve made has moved it closer. The break is in the attribution layer, and it will still be there when the models get better at retrieval.

Which is the same shape as every agent system we audit. Agents propose; verification and policy decide; a human approves the apply. A production-readiness audit asks the question this run answers badly: for each number your agent produces, what mechanically proves it came from the source it cites?