I saw the headline. "AI detects 90% of vulnerabilities." My first reaction wasn't excitement. It was suspicion. I've been in this game long enough to know that when a single vendor claims a number that high, the market is about to be sold something. The crowd sees a breakthrough. I see a structural gap in the data. And that gap is where the real risk lives.
This isn't the first time a hyped metric has masked a deeper problem. In 2017, I watched ICOs claim 100x returns while their tokenomics were hyperinflationary. I didn't flee the crash; I shorted the panic. In 2021, NFT floor prices were called "blue chips" until liquidity vanished. I wrote options against them, capturing premium decay while the crowd held. Now, the same pattern is repeating in AI security. The claim is 90% detection. The reality is a single data point from a single source with zero independent verification. And the real story isn't about detection—it's about the weaponization of that same technology.
Context: The CyberGym Claim and Its Missing Pieces
The article originated from Crypto Briefing, a publication with a crypto-native audience. It reported that CyberGym, an AI security firm, achieved "over 90%" vulnerability detection in their tests. That's the entire data point. No test set size. No false positive rate. No breakdown by vulnerability type. No mention of whether the benchmark was CWE Top 25 or a custom set of simple SQL injections. The original piece was a classic industry news brief—low information density, high marketing signal.
From my experience auditing DeFi protocols, I know that 90% detection on a curated dataset is meaningless. In 2020, I ran a $2M yield farming strategy on Impermax. I saw firsthand how smart contract vulnerabilities are rarely simple. They are logic bugs, race conditions, and economic attacks. An AI that flags 90% of hardcoded passwords is trivial. An AI that flags 90% of complex reentrancy vectors is a unicorn. The difference matters.
CyberGym itself is a commercial entity. They have a product to sell. The test was conducted internally. There is no third-party replication, no published paper, no open-source code. This is not an accusation of fraud—it's a recognition of fundamental incentive misalignment. The same pattern appeared during the ICO boom: projects released audited reports from firms they paid. The audits were often whitewashed. The market bought the narrative. I didn't.
Core: Deconstructing the 90% Number
Let's parse what "90% detection" might actually mean. In security research, detection is measured by True Positive Rate (TPR) or recall. But the metric that matters for practical use is Precision: of all flagged vulnerabilities, how many are real? If TPR is 90% but False Positive Rate is 30%, a security team spends most of their time triaging false alarms. The product becomes a liability, not a tool.
Public benchmarks from 2024–2025 show that state-of-the-art LLM-based vulnerability detectors (like GPT-4 fine-tuned on security datasets) achieve TPR in the 60–70% range on the CWE Top 25, with precision around 40–50%. Snyk’s Code AI, Semgrep Assistant, and GitHub Copilot Autofix all report similar ranges. A 90% TPR would be a 30%+ improvement over the best published results. That is possible, but it requires extraordinary evidence. The article provided none.
There is also the question of test set composition. If the test set is 90% simple vulnerabilities (like hardcoded credentials or path traversal) and 10% complex logic bugs, a model can easily hit 90% TPR by focusing on the easy ones. But that doesn't reflect real-world codebases, where complex vulnerabilities cause the most damage. In my own work, I've seen AI tools flag thousands of "issues" that turned out to be false positives, while missing a critical flash loan attack vector. The cost of missing a single real vulnerability far outweighs the benefit of detecting 90% of trivial ones.
Another hidden factor: the metric might be Top-N accuracy—the model's ability to include the correct vulnerability in its top 5 or top 10 suggestions. That is a common evaluation trick in recommendation systems. For a security analyst, a top-10 list still requires manual review. The 90% number loses its magic when you realize it's just a candidate set.
The core insight here is that the 90% claim is not a number; it's a selection bias. The vendor chose the test, the metric, and the reporting. Until independent replication occurs, this claim should be treated as a marketing artifact, not a technological breakthrough.
Contrarian: The Real Story Is Automated Exploitation, Not Detection
The article briefly mentioned the dual-use risk: "AI's progress also intensifies risks in automated exploitation and patch verification." That line, buried in a paragraph, is the most important sentence in the entire piece. The market is focused on how AI helps defenders. The smarter play is to understand how it helps attackers.
In 2022, after the Terra collapse, I hedged my portfolio with put spreads. The panic was real. But the structural risk wasn't Terra itself—it was the contagion to algorithmic stablecoins. I spent $150k on options. When Celsius and Voyager failed, my hedges generated $4.5M. The fear was an asset class. I monetized it.
Now, the same principle applies to AI vulnerability detection. The technology that finds bugs can also be used to write exploits. Research from 2023–2024 shows that GPT-4 can autonomously exploit known CVEs with 80%+ success rate. It can search for a vulnerability, craft an exploit, test it, and iterate. The time from disclosure to exploitation (TTE) is collapsing. In the past, attackers needed days or weeks. Now, they can do it in hours—or minutes if the model is integrated into a pipeline.
This is not speculation. I've seen it in private audits. The same LLM-based tools that security teams use to scan code are being repurposed by red teams and, inevitably, by threat actors. The barrier to entry for sophisticated attacks is dropping. A script kiddie with access to a GPT-4 API can now launch a targeted exploit against a smart contract vulnerability. The Web3 space is particularly vulnerable because many projects are unaudited or rely on a single audit. A single AI-generated exploit can drain a DeFi pool worth millions.
The contrarian angle: The true impact of AI in security is not the 90% detection number. It is the democratization of exploitation. The market is pricing the defensive upside. It is ignoring the defensive downside. That asymmetry creates an opportunity—for those who prepare.
Takeaway: Actionable Levels for the Skeptical Trader
The market is entering a new phase. AI security tools will proliferate. Claims will multiply. The noise will be deafening. But the underlying structure is unchanged: every innovation has a symmetric risk. The 90% detection number is a signal, but not the one the crowd thinks. It's a signal that the arms race is accelerating.
For the crypto industry, the implications are stark. Smart contract vulnerabilities are the low-hanging fruit for AI exploitation. The same AI that can scan Solidity code for bugs can generate exploit code. The next major DeFi hack might not be a manual exploit by a sophisticated team—it could be a script run by a teenager with a subscription.
Prepare accordingly. Demand third-party verification for any security tool you deploy. Run your own tests on your own codebase. Look at false positive rates, not just detection rates. And most importantly, build a patch process that can respond in hours, not days. The timeline is shrinking.
I didn't flee the ICO crash; I shorted the panic. I didn't hold the NFT bubble; I sold volatility. Now, I see the same pattern in AI security. The crowd is buying the narrative. I'm positioning for the asymmetric risk. Volatility is the premium you pay for opportunity. Leverage amplifies truth, it doesn't create it. The truth is, 90% detection is a number without context. The real number that matters is how fast the attacker can weaponize it.
That number is not 90%. It's 100%—the probability that the technology will be used against us. And that's the trade I'm watching.