THE VERDICT
The coffee had gone cold. My screen showed five numbers, and the fifth one made me sit up: 31.1%.
That's the attack completion rate โ ACR โ clocked by a security research team for AI agents built on SmolAgents. Same model. Same prompts. Same attack payloads. The only difference between this and a "safe" baseline? The orchestration framework stitching the agent's brain to its tools.
The paper is called "The Orchestration Framework Is the Attack Surface." It dropped at DEF CON 34's AI Village. If you're building anything in the AI-agent corner of crypto โ a trading bot, a DeFi strategist, a wallet automator โ this is the most important security read this year.
TL;DR: the model isn't the risk. The glue is. Framework choice can triple your attack surface โ and every "model safety" benchmark you've seen is probably inflated by 4 to 6 times.
That's not a vibe. That's 5,119 evaluation rows of evidence.
WHY NOW
Hackers don't hack, they listen. And what this research hears is the conversation between an agent and its tools โ the tool calls, the context handoffs, the memory writes that happen outside the model's "attention."
SADF's methodology is clean. They fixed Claude Sonnet as the base model, set up a direct API call as the control group, then routed the same model through four separate frameworks: CrewAI, LangChain, AutoGen, and SmolAgents. Because the model is constant, any difference in how attacks land has to come from the framework layer.
The numbers: Direct API sits at 15.5% ACR. CrewAI undercuts it at 11.9%. LangChain climbs to 18.1%. AutoGen hits 20.0%. SmolAgents bleeds at 31.1%.
Read that again. Adding a framework can make your agent more dangerous than no framework at all. CrewAI's discrete task isolation actually reduced attack surface. SmolAgents' context-sharing design nearly tripled it. The 2.6x spread between CrewAI and SmolAgents, both running identical Claude Sonnet, is the single most important data point in agent security this year.
Why does this land in a blockchain news feed instead of a dedicated security outlet? Because crypto is where agents are shipped at maximum velocity and minimum review. Every "AI agent" token launch, every autonomous DeFi strategy bot, every LLM-powered wallet โ all running on frameworks like these, usually with default settings, usually holding real keys.
Based on my time auditing Web3 projects, that checks out. Teams obsess over jailbreak resistance. They benchmark prompt-injection defenses. They never once inspect the tool-calling layer โ how cross-tool data flows, how context is shared, where permissions actually live. The framework is the new smart contract. And smart contracts taught us exactly what happens when you skip the audit.
THE NUMBERS
Here's what makes this research credible โ and where the gaps are.
The methodology's self-correction loop is rare. The researchers discovered that naive substring-matching evaluation โ the standard way most teams score "did the attack succeed" โ overestimates Claude models by 4 to 6 times. Why? Because a refusal can still contain strings that look like a successful tool call. When they applied refusal-filtered scoring, Claude Sonnet's real ACR settled at 15.5%, and Claude Haiku dropped to 22.3%.
Let that sink in: most published AI-security numbers from the last 18 months used that naive scoring. The first thing SADF did was admit their own earlier data was wrong. That's not weakness โ it's the only reason to trust the corrected numbers.
The taxonomy is the real gold. SADF gives the industry a shared vocabulary with eight failure modes: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation. If you build agents, commit these to memory. If you audit them, start your checklist here.
The per-mode data is where it gets spicy. SmolAgents shows 64% Context Boundary Violation โ nearly two-thirds of attacks moved data across intended boundaries. That's a design indictment: a framework built around shared context must be treated as a high-alert security surface, not a convenience feature. RAG Poisoning in SmolAgents: 20%. In a RAG-heavy Web3 agent pulling price feeds and governance signals, that's a backdoor with a welcome mat.
The CVE grounding matters, too. Framework-level flaws are already being exploited in the wild. Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) are real, patched, and public. The CVE registry is the market's memory: framework bugs pay real money when they hit production.
Now the serious caveats.
The study runs in a SimulatedToolEnvironment. No real systems, no live credentials. Ethical, sure. But it also means the timing, latency, permission boundaries, and real-tool responses that define production behavior never enter the experiment. Some failure modes will morph in the wild. The ACR numbers are lower bounds, not ceilings.
The payload set is 32 curated attacks. That's a thin sample. Real-world attacker payloads are distributed differently โ rare-but-catastrophic paths can slip through a 32-item net untouched. And there's no transparency on whether those payloads were adversarially augmented or red-teamed.
The paper claims 8 architectures were covered, but only 5 ACR datasets are detailed. Three frameworks got a mention without public-facing numbers. Inconsistent? Incomparable? That "claim vs. coverage" gap matters if you're using this research to pick your stack.
And the biggest open question: does the ranking hold when the model changes? Every number here is Claude Sonnet. Swap in GPT-5.4, DeepSeek, or Llama, and the model ร framework interaction could reshuffle the entire table. We don't know yet.
THE TRAP
Here's what everyone will get wrong about this paper.
They'll read the table, see CrewAI at 11.9%, and declare it "the secure framework." That's a trap. Discrete task isolation doesn't eliminate attack surface โ it relocates it. A low ACR tells you why CrewAI is safe under these 32 payloads. It says nothing about configuration drift, human-agent delegation abuse, or malicious tools already inside the perimeter. The safest framework on paper can be the most dangerous in production, because the false confidence arrives first.
There's an even stranger detail: the research is being amplified through blockchain/Web3 channels. That's not random. Crypto is the deployed-experiment frontier for AI agents โ the place where un-audited frameworks get connected to real money faster than anywhere else. The channel mismatch is the hazard map. Enterprise CISOs will read the DEF CON slide deck. The people who actually need it โ crypto agent developers โ are seeing it here months after the hype cycle made them commit to a stack.
Contrarian prediction: within 12 months, "framework security score" becomes a line item in every agent audit RFP, and Security-Evaluation-as-a-Service becomes a product category. SADF just built the yardstick. Someone will monetize it. The question isn't whether audits become mandatory for production agents. It's whether the audit industry learns the smart contract audit lesson: a score is not a shield.
One more transparency note: the earlier version of this research โ the one claiming coordinated disclosure across 10 architectures โ now sits in a SUPERSEDED/ folder. Honest engineering. Also a reminder: always verify you're reading the current data, especially when a security claim is older than the codebase you're about to trust.
WHAT'S NEXT
The merge wasn't the last all-nighter I pulled for a protocol. The next one โ the market waking up to the fact that model safety scores mean nothing when the middleware leaks โ is already loading. Watch for framework-level CVE disclosures, CI/CD security gates that block low-ACR architectures, and the first benchmark suite that crosses models and frameworks.
And before you deploy your next crypto agent: ask which framework it's running, whether the attack surface was measured, and whether your 4-6x-inflated confidence is the only thing standing between a tool-call hijack and your treasury.
Hackers don't hack, they listen. Right now, they're listening to your middleware.
