On this page
.png?width=2040&format=webp)
Ultradoc tried Retell AI, then ElevenLabs with Twilio. The phone agents had ridiculously long response times. "I would say something and get a response a second later. That's terrible. It should be instant," said Vincent Sajkowski, CEO and co-founder. For a healthcare platform built around real-time data, a one-second lag wasn't acceptable. Patients would hang up and call the front desk anyway.
For Ultradoc, adding a phone agent wasn't a standalone bet. The platform already captured rich, continuously-updated patient data, from progress notes to CPT-coded encounters, and the natural next step was to let patients interact with that data directly over the phone. A prescription refill, an , or a status update could be handled by an AI agent and automatically attached to the correct patient record, saving clinical staff from hours of repetitive call handling.
Ultradoc discovered Telnyx while investigating ElevenLabs' supported phone providers. Sajkowski found that Telnyx wasn't just a phone number provider but a full Voice AI stack, with inference hosted on the same infrastructure as the telephony layer.
"I saw you had your own voice AI implementation, and that you host the models locally, everything on one infrastructure. I tried that and was like, wow, this thing responds so fast. That was one of the main things that sold me."
Every other option required a conversation turn to hop between a telephony vendor, a speech-to-text vendor, an LLM vendor, and a text-to-speech vendor, each adding latency. Telnyx collapsed that onto one platform. The speed was structural, not incremental.
Beyond latency, the commercial fit was compelling. Telnyx's phone numbers were , and critically for a healthcare technology company, Telnyx was willing to execute the Business Associate Agreement Ultradoc needed to operate under HIPAA. "Telnyx was able to work out a BAA with us, which was really nice," Sajkowski said. "That allowed us to be HIPAA compliant."
As a developer-led organization, Ultradoc cared as much about the build experience as the runtime performance.
The build was fast. "It was pretty easy. The documentation is great," Sajkowski said. "You have a Node.js package, which is really awesome. That made the implementation even easier. I didn't have to make any custom calls. The webhooks are very simple and comprehensive."
The SDK was clean enough that Sajkowski handed portions of the integration to an AI coding assistant. "Claude Code was able to build the Telnyx integration super easily because of your Node package. That simplified it."
The phone agent sits inside the Ultradoc platform with immediate access to the same patient records, tasks, and clinical summaries that clinicians work with. A patient calls in, authenticates, and the AI acts on their record in real time: creating tasks, submitting refill requests, or updating notes that surface instantly in the clinical dashboard.
Ultradoc's first production phone agent was a prescription refill assistant. "It asks for name, date of birth, things like that, and then where you get your prescription filled," Sajkowski explained. "It's so easy to throw in tool calls. Do you want to send it to the pharmacy you last used? You pull up these things so fast. That's all through the tool calls within the phone agent, and then it calls a webhook to our server, which returns the data in real time."
In healthcare, voice AI raises the stakes. A poorly designed phone agent can leak protected health information with a single careless response. Ultradoc addressed this with a deliberately restrictive authentication and data-handling model built on top of Telnyx.
"We built an authentication system where if the Voice AI tries to make a patient tool call and they're not authenticated, it'll say, 'please authenticate the patient,'" Sajkowski said. "The patient provides their first name, last name, and date of birth. It looks it up, makes sure the patient actually exists in the database, and if it does, all the tool calls are unlocked."
Even after authentication, the agent is designed to be write-heavy rather than read-heavy, so it doesn't casually recite protected information back to the caller. "The AI is never reading patient data back out to the user. So if someone tried to impersonate that user, it's not going to say, 'okay, when you did this.' It's mostly going to write data. And if it is going to read something out, it's something that's not confidential at all."
"Their clinicians were mostly impressed by the speed, which I was very happy about," Sajkowski said.
Tasks like prescription refills and status updates collapse from live human interactions into structured text that shows up in the app, already attached to the correct patient. "It just allows your existing data to flow so much easier, and it frees up tasks for the people in the office. They don't have to do repetitive tasks anymore. A prescription refill shows up as text in the app, completely handled. You don't even have to think about it."
Sajkowski sees this trajectory ending in near-complete automation of the 90% of inbound calls that are well-defined and repetitive. "Now you can save humans for tasks that actually require a bunch of thought power, things that are more meticulous than the automated 90% of calls for these places."
And because Telnyx hosts inference models and co-locates them with telephony PoPs, model upgrades flow through to Ultradoc's agents without re-architecture. As open-source models improve, the phone agent improves with them. "As these open-source models get smarter, the Telnyx stack will get smarter because you can just upgrade it."
There are two main takeaways worth pulling out of Ultradoc's build.
First, if your Voice AI agent is going to interact with patients in real time, sub-second response time isn't optional. It's the difference between a patient who engages and one who hangs up. Ultradoc evaluated multiple vendors and found that 'Frankenstack'-type architectures with separate telephony, STT, LLM, and TTS providers consistently added too much latency for live conversation. A unified stack on a single network was the only architecture that met the bar.
Second, a voice agent that can read data out is a liability. Ultradoc's write-heavy, auth-gated approach is a model worth replicating: authenticate before acting, write more than you read, and never volunteer protected information unprompted. HIPAA compliance isn't just about signing a BAA. It's about how your agent is architecturally prevented from leaking data under adversarial conditions.
Ultradoc is a modern clinical workflow platform for physicians: real-time patient rounding, AI-powered billing, voice dictation, fax analysis, and EHR integration, all unified in a single source of truth. When the team set out to add a Voice AI phone agent as a new module, they evaluated several of the most visible names in the space before landing on Telnyx, and built a AI Agent that turns inbound patient calls into structured, patient-matched tasks inside Ultradoc.
But that vision had one non-negotiable prerequisite: the agent had to feel fast enough that patients didn't notice it was AI. That was harder to find than it sounds.
After evaluating several providers, Ultradoc found that the existing options couldn't meet their latency requirements. "I started with Retell AI, which was a wrapper on top of ElevenLabs and Twilio. Then I tried using ElevenLabs and Twilio directly," said Vincent Sajkowski, CEO and co-founder of Ultradoc.
The problem showed up immediately. "The phone agents had ridiculously long response times. I would say something and get a response a second later. That's terrible. It should be instant."
For a platform whose whole identity is built around real-time sync across every device a clinician touches, a one-second lag on a phone call was a dealbreaker.
What makes the architecture distinctly Ultradoc is that customers don't pick from a catalog of pre-built agents. They describe what they want in natural language, and Ultradoc's platform generates the phone agent, wires up the right tool calls, writes the prompt, and assigns a Telnyx phone number, automatically.
"You tell an AI that builds the AI," Sajkowski said. "You say, 'I want this phone agent to do this,' and it automatically adds tool calls and builds its prompt. Call the phone agent, test it out, and it just works."
The result is a phone agent that can take high-value actions on a patient's behalf, without becoming a vector for social-engineering attacks on their record.
