Original research conducted internally and with clients and partners, to drive product innovation and share knowledge and best practices.
LLM-based scanners are good at finding potential vulnerabilities but struggle to identify which ones are, in fact, real.
To rule out the noise, each vulnerability needs to be retested, with findings reproduced on live targets. This process is typically very difficult to automate, and is expensive, especially when dealing with large volumes of potential false positives.
At FAZE Labs, we handed that exact job to 13 frontier and open-weight models across 82 real vulnerabilities and 1,066 agent runs.
The findings were striking: the best validator isn't the biggest or the priciest model. Cheap models were worse than nothing. And reliability, not raw accuracy, is what separated the leaders from the rest.
This benchmark is the engine that sits behind FAZE's autonomous retest platform.
Automatic scanners surface potential vulnerabilities but rarely, if at all, validate them. Retesting to reproduce the findings on live targets is what determines whether a vulnerability is real, or simply noise.
Our research question was direct: which LLMs can perform the retest job reliably. And if so, at what cost?
This was a quantitative, operational benchmark.
Rather than scoring models on a static answer key, we measured their performance as autonomous agents doing live validation work, and reduced each model's behavior to comparable numeric metrics: verification rate, false-positive rate, error rate, latency, and cost
per finding.
Each model received a raw security finding plus live target access, and operated as an autonomous agent: issuing requests, reading responses, and reasoning across multiple turns until it returned a verdict of verified or false positive with supporting evidence.
All 13 models ran on an identical harness with identical prompts, served through AWS Bedrock.
For every run we captured: the verdict, every tool call, wall-clock time, token spend, and whether or not the run completed.
We tested 13 frontier and open-weight models against 82 real vulnerability findings, spanning security headers, GraphQL abuse, exposure, injection (SQLi, XSS, SSTI, SSRF, RCE, XXE, LFI/RFI), and misconfiguration.
Each model validated the full finding set, producing 1,066 agent runs
in total.
Results were aggregated into a single operational score that blends verification accuracy, false-positive restraint, reliability, latency
and cost.
We deliberately chose not to prioritize raw accuracy alone, as speed, cost, and likelihood of crashing are critical variables that need to be factored in to decide the usefulness of any model.
Statistics were deterministic and run consistency across the field was 0.98, meaning the gaps between models reflect genuine capability differences rather than run-to-run variance.
The benchmark used no external ground-truth labels.
All verdicts are the result of each models' own determinations as our goal was to uncover operational usefulness, instead of grading correctness against a known key.
Full per-vulnerability heatmaps, per-model run logs, and raw exports are available on request for independent inspection.
Nemotron-super leads on verification (82.9%) and restraint (14.6% false positives) at ~$0.02 per finding and 40 seconds.
It beats models costing 6× more, earning the top operational score of 0.793.
Nemotron-nano is the cheapest model tested ($0.0047) and never errors. However, it flagged 86.6% of findings as false positive and verified only 13.4%—meaning it was burying real vulnerabilities.
It beats models costing 6× more, earning the top operational score of 0.793.
Kimi-k2.5 posts the second-best verified rate (81.7%) and the lowest false-positive rate (11.0%) — but charges $0.12 per finding and 80 seconds to do it, dropping it to 8th on operational score.
In autonomous validation, an error is a silent miss. The leader errored on just 2.4% of runs.
Deepseek-v3 failed to complete 20.7% of its runs, with three other models hovering near 13%.
llama-405b did the least work (2.1 turns, 1 tool call) and finished last.
Kimi-k2.5 did the most (6.4 turns, 5.3 tools) and was accurate, but slow.
The leader hit the sweet spot at 3.7 turns. Which is enough to be sure, but not so much that it stalls.
The results flipped common intuition on its head, clearly proving that bigger or pricier models do not necessarily validate better than their less expensive counterparts.
nemotron-super topped the leaderboard, verifying 82.9% of findings with evidence at roughly $0.02 per finding and 40 seconds—beating models that cost 6× more.
In addition, the most expensive model guaranteed accuracy, but at a steep cost.
Yet running the cheapest model (nemotron-nano) was worse than doing nothing as it flagged 86.6% of findings as false positives.
The practical takeaway: choosing a validation model is an operational decision, and shouldn’t be based on benchmark accuracy alone.
The best validator is the one that gets the verdict right, proves it with evidence, finishes every time, and does so at a cost that’s affordable enough to run on every finding. Sometimes, running a combination of models works out best across multiple tasks.
The full 15-page report includes per-vulnerability heatmaps,
per-model run logs, and the complete raw exports.
Enter your email address now to see the report.
