Your Benchmark Is Grading
the Wrong Thing

Many code-review AI systems are chasing the same leaderboards, and these leaderboards generally share a quiet assumption: the ground truth is a golden set, the issues human annotators wrote down when they reviewed the PR. Score above the other tools at matching those notes and you "win."

We compete on those benchmarks too, and we do well. But building for them taught us something uncomfortable: golden sets measure the human flagging function, agreement with the findings reviewers recorded, not an exhaustive inventory of the codebase's actual defects.

What the golden set leaves out

We audited a 50-PR corpus finding by finding: reading the code, tracing the call paths, and requiring a verbatim citation for every claim. Among the findings scored as false positives against that golden set, we found real, verifiable defects: silently swallowed errors, race conditions, contract drift, inverted logic. The annotators simply hadn't written them down. A human reviewer has a noise budget; they flag what clears their personal bar on that day, and everything else, real or not, goes unrecorded.

When a valid finding is absent from the golden set and the benchmark's judge counts unmatched findings as false positives, a verified true finding can lower your score.

Two different products hiding in one category

That realization split our product line in two:

RMCode Max simulates the human flagging function, tuned to surface what a busy senior reviewer would actually comment on. It's the right product for humans in the loop, and it's what our public benchmark claim (F1 74.7 on the canonical 50-PR set) measures.

RMCode Deep Audit optimizes for something else: broader, evidence-gated coverage. Before a finding reaches the Bugs section it must survive an adversarial audit that independently re-checks the claim against your code and attaches the exact supporting quote. The output is two sections:

Together, the two sections retained every finding our adjudication process verified as true on that same audited corpus: 100% retention. Nothing true gets thrown away.

Same engine ancestry, different objective functions. On the golden-set benchmark, Deep Audit scores lower than Max, because it reports verified findings that are absent from the golden set. That benchmark remains a fair measure of Max's objective, agreement with what a senior reviewer would flag. It just doesn't fully measure Deep Audit's broader evidence-coverage objective, and we'd rather explain the yardstick than cripple the product to fit it.

Why this matters now: agents don't have a noise budget

The human flagging function exists because human attention is scarce. Coding agents have their own compute and context limits, but they consume structured evidence at a different scale: an agent doesn't get tired, doesn't suffer alert fatigue, and can cheaply verify a finding before acting on it. For agent-driven workflows, what matters is exactly what golden-set optimization throws away: broader verified coverage with per-finding evidence the agent can check.

That's Deep Audit's design brief. Every Bugs-section finding ships with a citation, the exact quote from the diff or source used to support the claim. Feed it to your agent, and it has a concrete evidence trail to check before acting; the Other-issues section can be triaged separately.

Reviews for humans need an editor. Reviews for agents need an auditor. We now ship both, and we label which number measures which.

Available now in beta

Deep Audit is rolling out in beta on the Max and Enterprise plans at 40 credits per review. Reviews run asynchronously; an audit takes minutes, not seconds, because every finding is verified before it posts. The beta label stays while we collect production accept/dismiss telemetry on how the verification gate performs at scale; we plan to publish those results with their methodology and scope.