Home › Blog › 84% Precision Is Not 84% of Your Bugs
AI Agents

84% Precision Is Not 84% of Your Bugs

Two agent benchmarks landed on 2 November, one for QA bots and one for frontier models. The useful part of both is not the leaderboard. It is the fine print.

One signal a day. No noise. A 3-minute read when something genuinely shifts.
By Tyron Dizon · November 3, 2026 · 5 min read
Two agent benchmarks landed on 2 November, one for QA bots and one for frontier models. The useful part of both is not the leaderboard. It is the fine print.
Source: Momentic QA Bench, 2 Nov 2026. Corpus labelled by Momentic.

Two leaderboards arrived on the same day. Momentic, a Y Combinator W24 company, launched Mo, an autonomous QA agent that plans and explores web, iOS and Android flows, and scored it on its own QA Bench, a corpus of 1,396 bugs drawn from Gitea, Metabase, Twenty and Saleor. SemiAnalysis launched GDPx, an agentic benchmark spanning 11 domains from chip design to legal work. Both produced a headline number. Both numbers mean less than the headlines suggest, and the gap between the two is where the useful lesson lives.

The 84% that was never 84%

Early coverage said Mo "finds 84% of real bugs." That is not what Momentic's own figures say. In a 60-minute run, Mo matched 269 bugs. Of the bugs it reported, 84% turned out to be real. That is precision. Of the 1,396 bugs in the corpus, it found 19.3%. That is recall. The headline took the first number and described it as if it were the second.

Think of a smoke detector. Precision is how often the alarm means there is actually a fire. Recall is how many fires manage to set it off. A detector that only rings for the one kitchen fire you already knew about has perfect precision and terrible recall. Nobody would call that "catches 84% of fires."

If you are buying QA automation, you need both numbers, for different reasons. Precision tells you how much engineering time gets burned chasing false alarms. Recall tells you how much is still hiding in the product after the agent says it is done. One number dressed up as the other is the most expensive kind of marketing, because it looks like data.

The rest of the fine print

Momentic's comparison is still interesting once you read it properly. Codex with GPT-6.1 Sol and Playwright MCP matched 86 bugs at $0.80 each. Claude Code with Opus 5.5 matched 59 at $4.09 each. Devin matched 27. Mo came in at $2.67 per confirmed bug. Between the 30-minute mark and the 60-minute mark, Mo's match count rose 52% while Codex's rose 16%, which suggests the exploratory approach keeps paying off as the clock runs.

And then the caveat that should sit at the top of every chart: the bug corpus is Momentic's own labelling, not an independent benchmark. A vendor's agent was scored on a vendor's test. That does not make the result wrong. It makes it a claim rather than a verdict, and claims get verified before they get budget.

GDPx: a 12% that is honest about itself

SemiAnalysis took the opposite approach and published the uncomfortable parts. On GDPx, Claude Opus 5.5 leads at 12.0%, Gemini 4 Carbon scores 10.9% and GPT-6 Astra 9.6%. No model wins every domain. Gemini leads biology and software engineering. GPT-6 Astra leads materials science. Opus leads quant trading, hardware and chip design. Finance and legal scored zero for every model under binary grading, and the finance task is not a toy: it drops the model into a deal room of about 2,000 files from a real private equity transaction.

Cost changes the picture again. Opus 5.5 ran at $51.41 per task against roughly $17 for Gemini 4 Carbon. The authors also say safety classifiers lowered Opus and GPT-6 Astra scores in the cyber, biology and alignment domains, and a second leaderboard without those domains puts GPT-6 Astra slightly ahead. So the "winner" depends on which rows you count and whether you divide by dollars. For good measure, the same team says its QA process flags nearly 50% of Terminal Bench 4.0 tasks as broken, which is a quiet reminder that the tests themselves can be the bug.

A leaderboard is a photograph of one test on one day, taken by someone who chose the camera angle.

Why this matters for anyone buying an agent

These numbers will show up in sales decks within the month. The 84% will travel further than the 19.3% because it is a better story. The 12% will be rounded into "leads the industry" because that is a better story too. Neither is a lie, but both are the front of the cereal box, and what you actually need is the nutrition label on the back.

The back of the box, for any agent benchmark, is four lines. Which number is this: precision, recall, or a pass rate under binary grading? Who built and labelled the test, and do they sell the thing being tested? What did each success cost, in dollars and in minutes? And what happens to the ranking when the grading rule changes, as it did on GDPx the moment the safety-affected domains were removed?

Momentic deserves credit for publishing recall at all. Plenty of vendors would have stopped at precision. SemiAnalysis deserves credit for publishing the zeroes and the cost column. The useful habit for the rest of us is to treat every headline number as an invitation to find the second number it is hiding.

What a citation harness taught me about leaderboards

I built a measurement harness that runs a fixed set of buyer questions against ChatGPT, Perplexity and Gemini, detects whether a target brand or domain gets cited, and tracks that citation rate over time. The point was to turn answer-engine optimization from opinion into a baseline and a trend.

Three things from that work apply directly to this week's benchmarks.

Bugs matched in 60 minutes, out of a 1,396-bug corpusMomentic QA Bench, 2 Nov 2026. Cost shown is per confirmed bug.Mo (Momentic)269 · $2.67Codex + GPT-6.1 Sol86 · $0.80Claude Code + Opus 5.559 · $4.09Devin27Mo: 84% precision, 19.3% recall. The bug corpus is labelled by Momentic, not an independent body.Source: Momentic QA Bench
Source: Momentic QA Bench, 2 Nov 2026. Corpus labelled by Momentic.

One signal a day. No noise.

A 3-minute read when something genuinely shifts in AI, automation, or defense tech. Free, most weekdays.

Free, most weekdays. No spam, unsubscribe anytime.

Sources

  1. Momentic - QA Bench - https://momentic.ai/blog/qa-bench
  2. AI Weekly - Momentic launches Mo, YC W24 QA agent - https://aiweekly.co/alerts/momentic-launches-yc-w24-mo-qa-agent-finds-84-of-real-bugs-at-267-each-across
  3. SemiAnalysis - GDPx: measuring frontier model capabilities - https://newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities
  4. AI Weekly - SemiAnalysis launches GDPx, best frontier model hits 12% - https://aiweekly.co/alerts/semianalysis-launches-gdpx-best-frontier-model-hits-12
  5. OfficeChai - Claude Opus 5.5 tops GDPx, cost per task - https://officechai.com/ai/claude-opus-5-5-tops-semi-analysis-gdpx-benchmark-gemini-4-carbon-placed-second/

Quick answers

What is the difference between precision and recall for a QA agent?

Precision is the share of reported bugs that are real. Recall is the share of all real bugs the agent found. On Momentic's QA Bench, Mo scored 84% precision and 19.3% recall, which means most of its reports were genuine but most of the corpus went unfound.

Did Mo really find 84% of real bugs?

No. The 84% figure is precision. Mo matched 269 of the 1,396 bugs in Momentic's corpus in 60 minutes, which is 19.3% recall. Early headlines mixed the two up.

Which model leads the SemiAnalysis GDPx benchmark?

Claude Opus 5.5 leads at 12.0%, ahead of Gemini 4 Carbon at 10.9% and GPT-6 Astra at 9.6%. No model wins every domain, finance and legal scored zero for all models, and a second leaderboard that excludes the cyber, biology and alignment domains puts GPT-6 Astra slightly ahead.

What should a buyer ask when shown an AI agent benchmark?

Which metric the number is (precision, recall or pass rate), who built and labelled the test and whether they sell the product, what each success cost in dollars and minutes, and how the ranking changes when the grading rule changes.

Tyron Dizon is a Chief Product Officer, AI product builder, and Techstars-backed SaaS founder based in Baguio City, Philippines. He is Chief Product Officer of StoryPages and StoryFunnels and interim CTO of StoryPages, co-founded SanityDesk (Techstars Los Angeles 2021), and builds AI products, automation systems, and rapid prototypes hands-on. About · Work · Resume · LinkedIn