2026-11-21 · Evaluation notes

Our AI reviewer caught 10 out of 10 known vulnerabilities. Here is why I told people to ignore that number.

The method, the four reasons the score does not count for much, and what happened when we pointed the same reviewer at ordinary pull requests.

Every AI code review tool has the same demo. Someone opens a pull request with a planted bug, the bot comments on line 42, everyone nods. It is theatre. The only question that matters is whether the thing works on your code, on a Tuesday, when nothing was planted.

So when we built our own reviewer, we did the obvious thing: took ten commits that fixed real CVEs in widely used projects, reverted each one so the working tree contained the vulnerable code again, and asked the tool to review the diff with no hints. Then we published the result — including a section explaining why the result is close to meaningless.

We got 10/10. That is the least interesting sentence in this post.

10/10
vulnerabilities localised
1
severity under-called (rated low, shipped as a security release)
too easy
four reasons below

The method, because a number without a method is marketing

For each case:

  1. Take a commit that fixed a real vulnerability in a widely used project (Django, axios, lodash).
  2. git revert --no-commit it, so the diff looks like somebody just introduced the problem.
  3. Strip documentation and test changes from the patch. This matters more than it sounds: the release notes name the CVE, and a reviewer that reads them is reading the answer key, not the code.
  4. Run the reviewer on HEAD~1 with the security agent only. No repository context, no PR description, no hints.
  5. A case counts as caught when a finding names the vulnerable file and describes the right class of bug.

Ten cases, one pass each, security agent alone, a small and cheap model, run on 2026-11-20.

Two of the ten had to be thrown out and re-run. In the first attempt the patch still contained the release notes, and the reviewer quoted the CVE title back at us, verbatim. That is not review, that is OCR with confidence. It was the first hint that our benchmark was measuring something other than what we thought.

Why a perfect score is a warning sign

A reverted fix is suspicious by construction

Reverting a security fix removes a length check, a validation call, a sanitizer invocation. The diff says “I deleted the guard clause” in plain sight. Any reviewer that looks at the hunk will notice, because the shape of a security fix is itself a tell. Real bugs are not introduced by deleting guards. They are introduced by accident, at 11pm, inside a 400-line feature that was mostly about something else. That is a fundamentally harder detection problem.

It measures recall and never precision

Nothing in these ten runs says how often the reviewer comments on clean code. Most pull requests are fine. A tool that finds every planted bug and also comments on a third of all normal PRs is not half-good, it is unusable, because every false positive costs a human the minutes it takes to read it, decide it is wrong, and move on.

Ten cases is an anecdote with a table

Add two cases and the headline moves twenty points. There is no confidence interval in a sample that size, and no honest way to present one.

Severity labels are unreliable

One case — password hash disclosure to view-only admin users — was rated low by the model. Django shipped it as a security release. The finding text was correct; the colour was not. If your workflow is “only look at critical and high”, a mislabelled severity is functionally a miss even though the tool technically saw the bug.

All four of these are written on the public benchmark page, above the fold, next to the 10/10: “A perfect score here is a sign that the benchmark is weak, not that the tool is perfect.”

What happened on real pull requests

Then we ran the same reviewer over ordinary PRs, the kind where nothing is wrong. It produced a confident, plausible-sounding comment on roughly a third of them.

Some were real. Most were pattern matching on style, or a fabricated failure mode that the surrounding code already handled three lines down. Each one cost a human a few minutes of “wait, is this actually broken?” — which is exactly the review time that is supposed to be the thing we are saving.

That reframes the design problem. The question is not how do we find more bugs. It is when should the reviewer stay quiet. Three things changed as a result:

Recall went down. Review time went down more. That trade is the entire product.

Why publish the weakness next to the strength

Because the failure mode of this category is not bad models. It is a demo culture where every tool shows a screenshot of a planted bug being found, and nobody shows the false positive rate. If we publish only the 10/10 we become another screenshot.

There is also a selfish reason. A benchmark you are willing to describe — method written down, harness shipped as a shell script — is one other people can attack. Ours fetches a fix commit, reverts it, strips docs and tests, runs the CLI, and diffs against the files the fix touched. It is not pretty and it is not peer reviewed. That is precisely why it is written down: so the next version can be better, and so someone can tell us where it is wrong.

The next version of the page needs twenty ordinary pull requests and a false positive count. Until it has them, treat 10/10 as a floor on capability, not a rate of quality.

If you are evaluating a reviewer

A short list, whether you use ours or anyone else’s:

We published ours at 10/10 with four paragraphs explaining why that does not count for much. If you want to break it, the harness and the raw runs are in the repo, and I would genuinely like the counterexample.


The missing half is now published: twenty ordinary merged commits, four comments, read by hand.

The raw results, the reproduce steps, and the limitations are on the benchmark page. RevHive runs as a GitHub App, or in your own network with your own model key: install it here.