What happened at curl
curl is one of the most widely used libraries on the internet, and its maintainers ran a paid bug bounty for years. According to maintainer Daniel Stenberg, the share of submissions that turned out to be confirmed vulnerabilities was above 15% before 2025 and fell below 5% afterwards, as AI-written reports with no real finding behind them increased. Over the life of the program curl confirmed 87 vulnerabilities and paid out more than $100,000. The paid bounty ended on January 31, 2026, with the maintainers concluding that the financial incentive was attracting more fabricated reports than real ones.
HackerOne’s Internet Bug Bounty pause
On March 27, 2026, HackerOne stopped accepting new submissions to its Internet Bug Bounty program, which funds rewards for vulnerabilities in widely used open-source software. Coverage at the time described a program strained by the volume of AI-assisted reporting and by the limited capacity of maintainers to triage and fix what arrived. The details differ by project; the pattern is the same. Reports became cheap to produce and remained costly to verify.
It is a verification problem, not a detection problem
A language model can write a convincing vulnerability report in seconds. Checking whether the report is true takes a maintainer real time: reading code, building the project, trying to reproduce. When reports arrive without reproducible evidence, the cost of verification lands on the people least able to absorb it. The same thing happens inside companies: teams that adopt AI scanners often spend their saved time re-checking whether each finding is real.
Long autonomous runs add a second effect. Suppose each reasoning step of an agent is 95% reliable. Over 10 steps the chance that every step is right is about 60%, and over 40 steps it is about 13%. These numbers are an illustration of compounding, not a measurement of any product, but they show why a flawless first step says little about hour three. The cure is not hoping for fewer errors; it is checking each step’s conclusion against evidence so that an early unproven assumption cannot quietly become a later “fact”.
What a trustworthy AI finding looks like
- It reproduces. A working proof of concept that someone else can run, ideally with a runtime trace.
- Each claim has its own evidence. A conclusion that depends on three technical facts needs evidence for all three, not one citation covering them.
- It can say “insufficient evidence”. A tool forced to pick “vulnerable” or “safe” will guess.
- A person is accountable. Findings are reviewed by researchers before they reach an engineering team or a maintainer.
- Disclosure is coordinated. Maintainers hear first, privately, with a reproducer and a suggested fix.
This is the standard behind Vorthix XOR-1, summarized on the evidence standard page and explained in why we built it. It is also the reason our published research includes the reproducer and trace with each write-up.
The category is growing, so trust is the question
Large vendors are now building AI vulnerability researchers. OpenAI, for example, announced Codex Security as a research preview on March 6, 2026. As raw ability to find bugs becomes more common, the useful question for a security team shifts from “can it find a bug” to “can I trust what it tells me after hours of unsupervised work”. Use the evaluation checklist to ask that of any tool, including ours.