Loading...
Deepfakes are now indistinguishable from real voices, and voice AI agents are about to face adversarial traffic. Here's what network-layer trust looks like.
At Mobile World Congress 2026 in Barcelona, Resemble AI ran a simple test. 140 attendees listened to audio clips and tried to tell which were real and which were AI-generated. The highest score was 8 out of 10. The average was roughly 50/50. A coin flip.
If trained listeners standing in a Resemble booth at the largest telecoms event in the world cannot reliably tell a deepfake from a real recording, the implication for every voice AI deployment exposed to public traffic is straightforward: the security model needs to start from the assumption that the audio coming in might not be what it claims to be.
Related articles
During the webinar that followed MWC, Resemble AI CEO Zohaib Ahmed played two audio clips of Angela Merkel speaking German, back to back. One was a real recording. The other was generated from five seconds of public YouTube footage, run through an open-source model. Most listeners cannot tell which is which, even when explicitly listening for the seams.
Five seconds of source audio. An open-source model. A public figure with hours of footage online, but the threshold is five seconds, which means the same attack works against anyone with a podcast appearance, a conference talk, or a voicemail greeting.
This is the technological baseline. The defensive architecture has to assume it.
In 2024, attackers used a deepfake video conference to extort roughly $25 million from Arup, the engineering firm, in Hong Kong. The attack was real-time and conversational, not a pre-recorded voicemail scam. A finance employee joined what they thought was a video call with the CFO and several colleagues. Every other participant was synthetic.
Europe was initially insulated by the English-only nature of voice models; the multilingual quality wasn't there, so the attacks weren't either. That window has closed. Spanish, Italian, German, and French quality have caught up over the last twelve months, which means the same playbook now travels.
Then there is the layer that is only just starting to show up in production traffic: agent-to-agent calls. Zohaib's framing during the webinar was that when you publish any new endpoint on the internet, the first traffic is bots, not humans. Voice agents are the same. The first inbound voice traffic to a new agent is often other agents, some legitimate (a customer's AI assistant calling on their behalf), some malicious (a probing attacker testing the agent's behavior). The traffic mix on a public voice agent is going to look more like the traffic mix on a public API than the traffic mix on a telephone line.
Two principles fall out of this for any voice AI deployment exposed to public traffic.
Assume the LLM can be prompt-injected - Critical flows like PIN authentication, identity verification, and payment authorization should not pass through the LLM. Route them through deterministic call control instead. The LLM is good at conversation; it is not a gatekeeper for high-stakes actions, and treating it as one creates exactly the attack surface that adversarial agent traffic will probe first.
Establish chain of custody at the network layer - Watermarking signs the source. Within seconds of audio hitting the carrier interconnect, deepfake detection flags AI-generated impersonation. Origin validation catches spoofed caller IDs by checking the actual routing path. None of this is achievable from the application layer alone, because by the time audio reaches it, it has already been transformed, and the trust signals have already been lost.
These are not nice-to-haves added on top of a working voice AI stack. They are the working stack in any deployment that takes adversarial traffic seriously.
Watermarking, the kind that survives real telephony, embeds inaudible signals directly into the audio stream. Resemble's watermarks are trained to sit below 8 kHz, deep in the low-frequency range where they resist filtering, and they have been tested against codec changes (G.711, EVS, modern telephony codecs), compression algorithms, replay attacks, and multiple transformation hops. The signal persists through all of them.
The point of watermarking is chain of custody. Screening is a separate capability. Each watermark identifies which AI model generated the content and signals the source. It establishes a verifiable record for every piece of audio flowing through the system, so that downstream, when something goes wrong, when a regulator asks, when a customer disputes a transaction, the audit trail exists.
Application-layer watermarks break under telephony. Audio reaches the application layer already transformed; a watermark applied there can be lost in the next codec hop. Infrastructure-layer watermarking, enforced at the carrier interconnect, is structurally stronger because it runs before the transformations rather than after them.
Detection is the other half of the loop. Telnyx and Resemble announced a deepfake detection feature that produces a webhook on every call, flagging AI-generated voice within four seconds of audio. It runs at the carrier layer, which means it sees the audio before any application-layer transformation and produces a signal the application can act on, drop the call, escalate to a human, or route to step-up authentication. Four seconds matter because real-time fraud cases like the Hong Kong extortion don't give you minutes. The underlying detection model is probabilistic, but it produces deterministic outputs for given inputs, and it has been trained on synthetic data spanning over 150 TTS architectures and 52 languages.
Origin validation is the capability that vendors operating purely at the application layer cannot match, because it requires sight of the carrier interconnect itself.
The example Zohaib used in the webinar: a caller claims to be on Vodafone, but the call isn't coming over the Vodafone interconnect. It's arriving via some VoIP carrier, which means the caller ID is spoofed and the routing is lying. The carrier interconnect is invisible to the application layer, so all the application sees is a phone number and an audio stream. From the carrier layer, the mismatch is immediate and flaggable.
This is what owning the stack at the network layer buys: trust signals that are simply not available higher up. Watermarking, detection, and origin validation aren't three separate features bolted onto a voice AI deployment; they're three views of the same underlying capability, which is having sight of the audio at the point it enters the network.
The way to read everything above is that the threat model for voice AI has moved from "occasional novelty deepfake of a CEO" to "the inbound traffic mix is partly adversarial agents, all the time." That shift changes what trust infrastructure has to do. It has to operate continuously, at the carrier layer, on every call, not as a forensic capability invoked after an incident, but as a real-time signal feeding the application's decision logic.
Article 50 of the EU AI Act formalizes some of this from 2 August 2026, with the provider-side watermarking deadline under Article 50(2) pushed to 2 December 2026 in the May 2026 Digital Omnibus agreement. Watermarking and detection become legal requirements, not just security best practices. But the architectural case for network-layer trust holds independent of the regulation. The deepfakes are good enough, the attack surface is wide enough, and the agent-to-agent traffic is already arriving.
If you're working through what trust infrastructure should look like in your voice AI stack, talk to our team. For the regulatory side of this, Article 50, DORA, and what compliant deployment actually requires, see our companion piece on EU AI Act compliance.
Build voice AI on infrastructure you can trust
Watermarking, detection, and origin validation at the carrier layer. Watch the webinar with Telnyx CEO David Casem and Resemble AI CEO Zohaib Ahmed, or talk to our team.
Watch the webinarTalk to us