Does it catch the bug, or just talk?
Every AI review tool shows a demo where it finds something. The useful question is whether it finds your bug. So we took ten patches that are known to be dangerous, because they are the exact code maintainers had to fix, and asked RevHive to review them with no hints.
Method
- Take a commit that fixed a real vulnerability in a widely used project.
- Revert it, so the working tree contains the vulnerable code and the diff looks like someone just introduced it.
- Strip documentation and test changes from the patch. Those files name the CVE, and a reviewer that reads them is reading the answer key, not the code.
- Run
revhive review -d HEAD~1with the security agent only. No repository context, no PR description, no hints. - A case counts as caught when a finding names the vulnerable file and describes the right class of bug. Severity is recorded but not scored.
Ten cases, one pass each, security agent alone, small free model (SenseNova Flash). Run on 2026-11-20.
Results
| Project | Vulnerability re-introduced | Severity reported | Caught |
|---|---|---|---|
| django/django | Server-side request forgery through a GDAL spatial reference string GDAL file/network access restrictions removed |
high | yes |
| django/django | Password hash disclosure to view-only admin users Flagged the exposure but rated it low; the real fix rated it high |
low | yes |
| django/django | Cache key collision between absent and empty Vary headers | medium | yes |
| django/django | Log injection through unescaped widget input | medium | yes |
| django/django | Memory exhaustion through unbounded multipart uploads | high | yes |
| django/django | Forged actions in model formsets | high | yes |
| django/django | Memory exhaustion in language tag parsing | high | yes |
| django/django | Stored XSS through an unvalidated URL scheme in admin | high | yes |
| axios/axios | SSRF and credential leak through a redirect | high | yes |
| lodash/lodash | Command injection through the template variable option | critical | yes |
Two of the ten cases had to be re-run: the first attempt left the release notes in the patch, and the reviewer simply quoted the CVE from them. Those runs were discarded.
What this does not prove
A perfect score here is a sign that the benchmark is weak, not that the tool is perfect. Specifically:
- A reverted fix is suspicious by construction. The patch removes a length check, a validation call, a sanitizer. Almost any reviewer that looks at the diff will notice. Real bugs are introduced by accident, inside a 400-line feature, and that is much harder.
- It measures recall, never precision. Nothing here says how often it flags clean code. We ran the other half separately: twenty ordinary merged commits, seventeen with no comment, four comments in total. Every one is written up in the noise measurement.
- Ten cases is an anecdote with a table. Adding or removing two cases moves the number by twenty points.
- Severity is unreliable. The password-hash disclosure was called low; Django shipped it as a security release. Read the finding, not just the colour.
Reproduce it
The harness is a shell script: fetch the fix commit, revert it, strip docs and tests, run the CLI, compare against the files the fix touched. It is not pretty and it is not peer reviewed, which is exactly why the method is written down here.
Source: github.com/jovian-zhibai/RevHive. Raw JSON for all ten runs on request. Longer writeup, including what real pull requests showed: evaluation notes.