Autonomous Security Research

How to Evaluate an Autonomous AI Security Tool: 8 Questions to Ask

Published Oct 10, 2026·7 min read·Vorthix Research Team

To evaluate an autonomous AI security tool, test trust rather than a demo: ask how it behaves over long unsupervised runs, what “safe” rests on, whether it can say it does not know, whether each claim has its own evidence, whether you can reproduce findings, how false positives are measured, who reviews results and where your code goes. Eight questions follow, with what a good answer sounds like.

Why a demo is not enough

Most modern language models can find a bug in a demo with a person guiding them. What separates tools is what happens next: hours into an unsupervised run, when context is full and the easy findings are gone, does the tool still tell the truth about what it knows? The checklist below comes from the failure modes we describe in why we built Vorthix XOR-1 and from the verification burden discussed in AI-generated vulnerability reports and bug bounties. Use it on any tool, including ours.

Eight evaluation questions with good answers and red flags
Question	A good answer	A red flag
1. What happens when it runs for hours without supervision?	The vendor can describe how progress is preserved and how repeated work is avoided over a long session.	Only short demos, or no description of how long runs behave.
2. What does “safe” mean when it says something is safe?	A conclusion tied to specific, retrievable evidence from your code.	Confident wording with no traceable evidence behind it.
3. Can it say “I could not establish this”?	“Insufficient evidence” is a defined outcome, reported openly.	Every investigation ends as vulnerable or safe.
4. Does each claim have its own evidence?	A conclusion that rests on several facts has evidence for each, and a real citation cannot vouch for a claim it does not prove.	One citation is accepted as support for the whole conclusion.
5. Can you reproduce a finding yourself?	A working proof of concept, a minimized reproducer and a runtime trace come with each finding.	A description or a confidence score instead of a reproducer.
6. What is the false-positive story, measured on whose data?	Claims come with a method you can inspect, ideally measured on your own code in a pilot.	A headline percentage with no method, or one taken from marketing material.
7. Who reviews findings before you see them?	Named human review, plus a clear disclosure process for third-party code.	Unreviewed output pushed straight into your backlog.
8. Where does your code go?	Isolated, ephemeral environments; no sharing with third-party model providers; no training on your code.	Vague data-handling language, or code retained by default.

How to run the evaluation

Pick a codebase your team knows well, ideally with a few known issues and a few known false alarms.
Ask each vendor the eight questions in writing and compare the specifics.
Run a pilot. Have an engineer reproduce each finding from the evidence supplied, without help.
Count three things: findings that reproduce, findings that do not, and investigations the tool honestly left open.
Check the data-handling terms against your policy before sending real code.

A tool that leaves some items open and tells you so is behaving better than one that closes everything. See how we apply this in the Vorthix XOR-1 evidence standard, and the definitions in the glossary.

Frequently Asked Questions

How do I compare AI vulnerability research tools?

Compare them on trust, not on a single demo: how they behave over long unsupervised runs, what their conclusions rest on, whether they can report uncertainty, whether findings are reproducible, how false positives are measured, who reviews results and how your code is handled.

Should I trust a vendor’s false-positive percentage?

Treat any vendor-reported percentage as a claim to test, not a benchmark. The reliable approach is a pilot on your own code with the reproduction of each finding checked by your engineers.

Does an AI security tool replace human review?

No. It changes where human time goes: toward reviewing evidence and fixing issues, and away from re-checking whether each finding is real. Look for tools where a person is accountable before findings are delivered.