The ZKML Mirage: Why 'Verifiable AI' Proves Less Than You Think

A week ago, a project with $40 million in fresh Series A funding announced "zero-knowledge verified AI inference" for enterprise deployments. The press release was confident. The deck showed latency charts measured in milliseconds. The GitHub repository — which I spent a weekend dissecting instead of sleeping — tells a more complicated story. Buried in a commit message from four months ago was the caveat: the benchmark ran on a distilled model with 0.3% of the production parameter count. This project is far from alone. In this bull market, the ZK plus AI narrative has absorbed hundreds of millions in capital. Every week, another protocol claims to be building "verifiable intelligence." My job, as someone who has audited smart contract code since the ICO era, is to check whether the proofs actually prove anything worth trusting. The math whispers what the network shouts. And what the network shouts is loud indeed. The core claim of zero-knowledge machine learning (ZKML) is simple. A prover takes a neural network, converts its inference computations into a circuit, and generates a cryptographic proof that the computation was executed faithfully. A verifier — smart contract, enterprise, or individual — can check the proof in milliseconds without re-running the entire model. This is the technical basis for on-chain AI, trustless model markets, and "verifiable agents." Proving truth without revealing the secret itself. An elegant idea. But the gap between the theory and the deployed reality is measured in orders of magnitude. Here is what actually happens in production. First, circuit size. Take a moderately capable model — say, a 7-billion-parameter transformer. A single forward pass requires trillions of arithmetic operations: matrix multiplications, attention computations, and normalization layers. To express this as an arithmetic circuit over a finite field, you need hundreds of millions or billions of constraints. The current generation of proving systems, from Groth16 to PLONK to the newer recursive SNARKs, handles millions of constraints in reasonable time. Billions is a different world. Proving times stretch to days, not milliseconds. And this is for a single inference, not training. Then there is the choice of proving system itself. Groth16 requires a trusted setup ceremony — a moment where the entire security architecture rests on at least one participant honestly discarding toxic waste. PLONK and its recursive successors eliminate that ceremony but at substantial proving cost. The projects I review rarely discuss this trade-off in public materials because it undermines the one-line pitch. When a team boasts about twenty-second proving times, I ask to see the constraint count. I am still waiting for a response that includes one. The engineering reality forces additional compromises. The most common is quantization. Weights are compressed from 32-bit floating point to 8-bit integers to reduce circuit complexity. Quantization changes the model. The quantized model behaves differently from the trained one. Sometimes the difference is small; sometimes it is not. I have seen projects present accuracy metrics for their original model and then ship a quantized version that loses several percentage points — without, of course, updating the numbers in their documentation. When I raised this in an audit discussion last year, the response was: "The user just wants a proof that the computation ran." That is precisely the problem. A proof of a degraded model is still a valid proof. The proof mechanism is incorruptible; the pipeline around it is not. Second, model provenance. This is the issue that concerns me most because it is structural. A zero-knowledge proof of inference demonstrates that the computation followed the weights in the model. It does not demonstrate that those weights are sound. A model provider can deliberately train a model with backdoors — the adversarial machine learning literature documents these attacks extensively. A malicious actor can then generate a proof of inference for the backdoored execution and present it as verified, trustworthy AI output. Trust is not given; it is computed and verified. But the verification here only extends to computational integrity. The semantic content of the model is entirely unaddressed. This confusion is pervasive in marketing material. The word "verified" appears everywhere, yet no one specifies what property is being verified. We have built a mathematical infrastructure for proving tautologies. Third, data commitment. Inference is a function of two inputs: model weights and runtime data. A proof of inference shows that some input produced some output according to the model. But unless the data is committed to at the time of proof generation — cryptographically bound to the input of the circuit — the prover can select favorable data after the fact. A "verified" system for credit scoring, for example, could prove a favorable outcome for a cherry-picked input while failing for the actual user's data. The deployment incentives amplify this risk. In a bull market, the pressure to ship proof-of-concept demos is enormous. Investors hear "zero-knowledge proof" and assume a baseline of security and correctness. The technical term has become a trust signal, a badge, rather than a precise mathematical statement. From my experience reconstructing the Terra collapse — where I reverse-engineered the UST seigniorage mechanism to show my community exactly where the death spiral began — I have learned that market narratives and technical realities diverge most dramatically at market peaks. In 2020, it was liquidity mining yields that were mathematically unsustainable. In 2021, it was NFT metadata on centralized servers, where 30% of the high-value collections I audited with Taipei artists were one server shutdown away from losing their work forever. In 2025, it is ZKML that has captured the collective imagination. The contrarian angle is uncomfortable for the industry. We have spent years building a technology that creates mathematical certainty about computations. But mathematical certainty is not the same as trust. The proof system can be flawless while the underlying model, data, and incentives remain corrupt. The mechanism of verification has become the seductive story, while the things that actually need verification — training data, model behavior, deployment integrity — stay opaque. The computing asymmetry is the second overlooked vulnerability. Generating a ZK proof for a production model requires high-end proving infrastructure that most organizations cannot run. Look closely at the hosting. The proving services are almost always centralized. They are the single most powerful actors in the system. A prover with the ability to collude with model providers could generate valid proofs of malicious models indefinitely. We replaced a decentralized trust model with a centralized one precisely because decentralization is computationally expensive. In this configuration, ZKML recreates the very trust architecture it claims to dissolve. What I want to see is not more funding announcements but more granular discipline. A project should specify exactly what predicate its proof establishes. The proof statement should include the model hash, the data commitment, the quantization scheme, and the exact semantics being verified. Anything less is still marketing. Between the lines of every optimistic ZKML announcement, a quieter reality is emerging: the proof market is becoming a commodity, while the model market remains unregulated and opaque. When a verified inference produces a fraudulent loan decision or a biased hiring outcome, the mathematical infrastructure is unlikely to be the problem. The model pipeline will be. For decision-makers, the question is not whether the proof is valid, but whether the proof's object is the right object. An invalid proof is a technical bug. A valid proof of the wrong thing is a design catastrophe — and infinitely harder to detect. Proving truth without revealing the secret itself. That remains the mission. But before we can claim to prove truth, we need to be honest about the narrow slice of computation that our proofs actually cover. The rest — judgment, integrity, model quality — will need to be earned the old-fashioned way. Trust is not given; it is computed and verified. But only fully so when we trust our own definitions. The next time you see a "verified AI" announcement, ask one question: what exactly is being verified? The answer is the most revealing proof of all.