Elevated Retesting - Now at Machine Speed
Finding a vulnerability is the easy part. The expensive part is proving, over and over, whether it is still there.
That is retesting - and at scale it is where budgets spill over.
At FAZE we automate this step with our native retesting module. It drives an autonomous agent that takes a previously reported finding, goes back to the live target, re-runs the exact checks, gathers fresh evidence, and returns a verdict: VERIFIED (still exploitable) or FALSE_POSITIVE (gone), with receipts attached. An agent is only as good as the model behind it - so we designed a research study to find the best one. This post, written together with NVIDIA, explains our findings.
FAZE is Offensive Security Orchestration — an AI-powered agentic red team that runs continuously against your attack surfaces, fusing proprietary deterministic and probabilistic capabilities. It attacks and validates real, exploitable risk, then does the real heavy lifting: rating exposure against your actual environment, prioritizing against your business and orchestrating remediation across teams. This removes the friction inherent across the: finding an issue → filing a ticket → triaging → prioritizing → patching and verifying closure process.
The result: not more findings — less risk, delivered through an enterprise level workflow designed to remove it, at scale.
The methodology
We ran 13 models against 82 real vulnerabilities for a total of 1,066 retest runs. To keep it honest, everything was held constant: every model ran on Amazon Bedrock, received the identical prompts, the identical evidence and target data, and the same tools. Scoring is deterministic, and the models were never handed the ground truth - they had to earn each verdict.Crucially, this was not a toy sample of one or two bug classes. The 82 findings span a broad cross-section of vulnerability families, including:
- Injection & exploitation - SQLi (time-based and boolean-blind), RCE, XXE, SSTI, SSRF, RFI, LFI, Open Redirect, Reflected and DOM XSS
- Missing security headers - Clickjacking / X-Frame-Options, CSP, X-XSS-Protection, Permissions-Policy, Content-Type, Subresource Integrity
- GraphQL abuse - introspection, field suggestions, alias overloading, 500-field batching, GraphiQL exposure, GET-method queries, non-JSON POST
- Information disclosure - exposed .git repos, dev config files, source maps, Node.js source disclosure, directory listing, internal IP leaks, improper error handling, exposed API schemas
- Cloud misconfiguration - public S3 buckets and permissive bucket ACLs
If a model got lucky on headers, it still had to survive time-based SQL injection. Breadth was the point.
The results
We rank by operational_score - a blend of how often a model verifies a true finding with evidence, how rarely it cries wolf, and what it costs in money and time. Here is the leaderboard:
The winner, comfortably, is the NVIDIA Nemotron 3 Super (120B). It verified 82.9% of findings - the highest true-detection rate of any model in the field - while keeping false positives to 14.6% and erroring out on just 2.4% of runs. It did that at $0.0208 per retest and about 40 seconds a run.
Why accuracy alone doesn't win
Look closely and you'll spot the plot twist. Kimi K2.5 posted a gorgeous 81.7% verified rate with the lowest false-positive rate in the whole test (11.0%). On a pure accuracy chart it looks like a co-winner. So why is it sitting in 8th place?
Because it costs $0.1192 a run - roughly 5.7× more than Nemotron Super - and takes twice as long (79.7s vs 40.2s), burning 6.4 conversational turns to Super's 3.7. Nemotron Super delivers slightly better detection than Kimi for about one-sixth of the cost and half the latency. In retesting, where you re-run the same findings again and again, that ratio is the entire ballgame.
Second place went to gpt-oss-120b - cheap and quick, but its false-positive rate nearly doubles Super's (28.1% vs 14.6%), and in retesting every false positive is a human being re-opening a ticket that was already closed. That is exactly the friction our retesting module was designed to remove.
The Nano surprise
Now the fun one. Nemotron 3 Nano (30B) finished 10th overall, and on paper it looks rough: only 13.4% verified, with a high false-positive rate on ambiguous, judgment-heavy checks. A 30B model being cautious-to-a-fault on subtle findings is not shocking.
But read the fine print and Nano quietly flexes. It was the cheapest model in the entire test at $0.0047 per run, the fastest at 28.4 seconds, and the only model of all 13 with a 0.0% error rate - it never once crashed or timed out. And on high-signal, deterministic checks it doesn't just keep up, it wins: on “Exposed API Schema,” Nemotron 3 Nano returned a clean VERIFIED where Nemotron 3 Super logged a false positive. It also nailed exposed .git repositories, directory listings, public S3 buckets, dev-config leaks and Permissions-Policy gaps at a quarter of Super's cost.
The lesson isn't “bigger is always better.” It's that the Nemotron family gives you a tiered fleet: route the deterministic, high-volume checks to Nano for near-free instant answers, and escalate the ambiguous, high-stakes findings to Super. Same family, same tooling, two very different price-performance points.
The part we're quietly proud of: efficient, private and fully self-hostable
Here's the detail that's easy to skim past: every single model in this bake-off is open-weight and self-hostable. We ran the whole gauntlet on Amazon Bedrock, keeping the deployment and evaluation workflow within our own AWS environment.
Nemotron 3 Super, an open-weight model we control delivered the best true-detection rate in the field at about two cents a retest - the kind of elite red-team result that is more economical for repetitive security testing, produced by one you can run inside your own AWS estate. That means you retain control over infrastructure, deployment, and sensitive evidence.
For a security platform, open models aren’t justnice-to-have; they are often the whole compliance conversation.
What this means for FAZE - and anyone else
Let’s put the economics in plain numbers. Retesting our full 82-finding backlog end-to-end costs $1.70 on Nemotron 3 Super. The same backlog on Kimi K2.5 costs $9.78. Now scale it: a team running 10,000retests a month pays about $210 with Nemotron Super versus roughly $1,190 with the priciest model in our test - for equal-or-better accuracy. That is the difference between "retest continuously" and "retest once a quarter and hope.
"For any security organization, the takeaways are portable:
- Pick for false positives, not just detection. A model that verifies well but hallucinates findings just relocates the work back to your analysts.
- Price and latency are features. Retesting is repetitive by nature; a 5.7x cost gap compounds every single day.
- Tier your models. Nemotron 3 Nano for the easy, high-volume checks; Nemotron 3 Super for the ones that need judgment.
- Open beats closed for repetitive work. A self-hostable model like Nemotron avoids the per-token frontier tax of Opus / Sonnet / GPT-5-class APIs and keeps your evidence inside your own perimeter.
- Because Nemotron runs on Bedrock, it drops into an existing AWS estate with no new procurement overhead and with open weights you control.
Bottom line
Across 82 vulnerabilities and 1,066 runs, NVIDIA Nemotron 3 Super gave us the best true-detection rate of any model tested, a low false-positive rate, and did it for a fraction of the cost compared to other leading models - all from an open model we run ourselves, with Nemotron 3 Nano waiting in the wings to make the cheap work free. For RetestAI, that combination turns vulnerability re-validation from a quarterly chore into something we can run all day, every day.