Does it catch the bug, or just talk?

Every AI review tool shows a demo where it finds something. The useful question is whether it finds your bug. So we took ten patches that are known to be dangerous, because they are the exact code maintainers had to fix, and asked RevHive to review them with no hints.

10/10
vulnerabilities localised
1
severity under-called (c02 rated low, fix was high)
too easy
see what this does not prove, below

Method

  1. Take a commit that fixed a real vulnerability in a widely used project.
  2. Revert it, so the working tree contains the vulnerable code and the diff looks like someone just introduced it.
  3. Strip documentation and test changes from the patch. Those files name the CVE, and a reviewer that reads them is reading the answer key, not the code.
  4. Run revhive review -d HEAD~1 with the security agent only. No repository context, no PR description, no hints.
  5. A case counts as caught when a finding names the vulnerable file and describes the right class of bug. Severity is recorded but not scored.

Ten cases, one pass each, security agent alone, small free model (SenseNova Flash). Run on 2026-11-20.

Results

ProjectVulnerability re-introducedSeverity reportedCaught
django/django Server-side request forgery through a GDAL spatial reference string
GDAL file/network access restrictions removed
high yes
django/django Password hash disclosure to view-only admin users
Flagged the exposure but rated it low; the real fix rated it high
low yes
django/django Cache key collision between absent and empty Vary headers medium yes
django/django Log injection through unescaped widget input medium yes
django/django Memory exhaustion through unbounded multipart uploads high yes
django/django Forged actions in model formsets high yes
django/django Memory exhaustion in language tag parsing high yes
django/django Stored XSS through an unvalidated URL scheme in admin high yes
axios/axios SSRF and credential leak through a redirect high yes
lodash/lodash Command injection through the template variable option critical yes

Two of the ten cases had to be re-run: the first attempt left the release notes in the patch, and the reviewer simply quoted the CVE from them. Those runs were discarded.

What this does not prove

A perfect score here is a sign that the benchmark is weak, not that the tool is perfect. Specifically:

Reproduce it

The harness is a shell script: fetch the fix commit, revert it, strip docs and tests, run the CLI, compare against the files the fix touched. It is not pretty and it is not peer reviewed, which is exactly why the method is written down here.

Source: github.com/jovian-zhibai/RevHive. Raw JSON for all ten runs on request. Longer writeup, including what real pull requests showed: evaluation notes.