Last week I published a benchmark where our reviewer caught 10 out of 10 re-introduced vulnerabilities, and then spent four paragraphs explaining why that number is close to worthless. The main reason: it measured recall and never precision. A tool that finds every planted bug and also comments on clean code is not half-good, it is unusable.
So here is the missing half. Twenty ordinary commits, nothing planted, and every comment read by hand.
The samples
Twenty commits from psf/requests, filtered to those touching src/requests/**.py and excluding pure type-annotation and CI churn. All twenty were merged by maintainers. None re-introduces a vulnerability.
For each one: check out the commit, run the security agent against HEAD~1 with no PR description and no hints, record what it says. One pass each, twenty out of twenty completed.
The result
| Commit | Change | Comments | Verdict |
|---|---|---|---|
| a044b02 | DigestAuth hashes: add usedforsecurity=False (5 lines) | 2 (high, medium) | noise |
| fd62809 | Preserve leading slashes in path_url (2 lines) | 1 (high) | overstated |
| a4f9a59 | Port bpo-39057 (proxy bypass fix) | 1 (low) | real but marginal |
| 17 others | small refactors, typing, parsing fixes | 0 | silent |
Seventeen of twenty got complete silence. That is the part that matters most: on most real changes the reviewer has nothing to say, which is the correct answer.
What it said when it did speak
a044b02 — two comments, both noise
The commit adds usedforsecurity=False to the hashlib calls in HTTP Digest auth. That flag exists so the code keeps working under FIPS restrictions, where MD5 and SHA1 are otherwise blocked. It is a deliberate annotation meaning this hash is part of a wire protocol, not a security primitive.
The reviewer reported a high finding titled “FIPS security bypass via usedforsecurity=False”, plus a medium one about weak hashing for cnonce generation. Both restart from “MD5 is weak”, which has been true since 2004, and neither engages with what the diff does. The maintainers added the flag because the hash is not a security control. The reviewer read the flag as a security control being switched off.
fd62809 — one comment, overstated
Reported high: removing the //-to-/ normalization may enable request smuggling, since some proxies treat //evil.com/path as a network-path reference. There is a real debate in that area and the finding is not stupid. But the change was made on purpose, after discussion, and “may enable request smuggling” is a much stronger claim than two lines of diff support.
a4f9a59 — one comment, real but marginal
If no_proxy contains an entry consisting only of dots, host.lstrip(".") produces an empty string and the suffix match degenerates. Genuine edge case, rated low, will never happen in a real deployment.
The failure mode is one thing, repeated
All three false positives have the same shape: the diff contains something that looks like a security control being removed or disabled, and the reviewer reacts to the shape without recovering the intent.
usedforsecurity=False reads as “we are telling the crypto library to skip security”. Removing // normalization reads as “we stopped sanitising the path”. Both are locally true about the text and both are wrong about the change, because the reason lives outside the diff: in the FIPS requirement, in the linked issue, in the discussion that decided the trade-off.
This is the mirror image of the recall problem. A reverted security fix is easy to catch precisely because the reason is visible in the diff: a guard clause disappeared and nothing replaced it. Real commits that touch security surfaces are hard because the reason is usually absent and the shape looks identical to a regression.
An LLM reviewer with no PR description is being asked to tell those two apart from the patch alone. Most of the time it cannot, so the honest design goal is not “understand intent”, it is “say less when intent is missing”.
What this does and does not support
Supports: on ordinary changes this reviewer is quiet. Fifteen percent of commits got a comment, and the absolute cost is low — one extra thing to read per five PRs, not a wall of noise.
Does not support: a precision rate. Four comments is not a sample, and I graded “noise” myself, knowing what each diff was supposed to do — which a reviewer in CI does not.
- One project, one language.
requestsis Python, small, and unusually well reviewed. A codebase with more security surface will produce more comments, not fewer. - No PR description, by design. That is the harshest setting. In CI the reviewer usually gets a title and description, which should reduce these false positives — and could increase them, if the description hands the model a thesis to confirm.
- Security agent only. A full review is noisier, and I have no honest number for it yet.
What changed in the product
These findings are why the reviewer now has a confidence floor and only reports findings anchored to a specific line with a concrete failure path. “MD5 is weak” has no failure path, so it does not get to speak. “This path is used for auth, therefore disabling FIPS checks on it is a bypass” has no failure path either, because the premise is wrong.
Recall went down when we added that. On the twenty commits it cost nothing, since none of the four comments was actionable. On the ten vulnerability cases it cost nothing either. That is the trade I want: wrong less often, at the price of being quiet more often.
The next version needs a hundred commits across three or four projects, PR descriptions included, and two people labelling comments independently. Four comments is an anecdote with a table, and I would rather say so than round it into a percentage.
The recall side, and why it is too easy: 10/10 and why it does not count. Run it yourself: GitHub App.