Loading...
The hard part of building voice AI agents isn't connecting APIs. It's writing instructions that prevent the LLM from skipping steps, improvising, and breaking.
I recently built a voice-driven bug reporter that listens to a user, collects the details of an issue, and then creates a JIRA ticket through the Telnyx agent platform. On the surface, it sounds simple. Connect JIRA. Add a prompt. Let the model do its thing.
In reality, the experience taught me something completely different. The hard part was not the JIRA API. It was everything around it: the conversation flow, the timing, the tool interpretation, and the way LLMs behave when you need structured outcomes.
Prompting is a borderline art. The black-box nature of LLMs requires persistence.
Stephen Malito, Director, Solutions Engineering @ Telnyx
This post walks through what I learned, what surprised me, and the design patterns that ended up making the agent work reliably. Along the way, I will highlight a few screenshots from the Telnyx portal that show how the setup looks in practice.
At a high level, the setup for this agent was straightforward. You need:
Related articles
This is where the agent’s reasoning style and verbosity start.
Choosing a model that responds consistently and avoids rambling is surprisingly important for collecting structured fields.
As this is a voice agent, pacing matters. If the transcription is slow or the TTS responses are too long, users feel friction. That forced me to write shorter prompts and limit responses to under twenty seconds.
Once JIRA is connected with an API token, the agent gains access to these tools:
That is really all the setup you need before you start shaping the logic.
Creating a JIRA issue was the easy part. The tool call itself is simple: pass a project key, summary, description, priority, and issue type. Sending the request worked on the first try.
Other tasks that I assumed would be painful but were not:
These pieces were mechanical. The problem was everything between them.
Here is the high-level flow I needed:
Sounds easy, but LLMs love to skip ahead or merge multiple fields into a single response. Without rigid gating, the agent would do things like:
The solution was to explicitly script each turn in the prompt, telling the model exactly what to ask and in what order. This taught me that most voice agents are only as smart as the conversation scaffolding you give them.
Even with instructions, the agent sometimes tried to call jira__create_issue early because it felt like it had enough information. Other times, it would call jira__get_projects when the user did not ask for it. I had to add clear constraints like:
These guardrails made the agent predictable.
JIRA endpoints return structured data that is not voice friendly. Too many fields. Nested structures. Lots of metadata the model does not need. The fix was to instruct the model to summarize tool outputs in a human friendly way:
The agent became significantly easier to use once tool responses were filtered and summarized.
This was one of the biggest challenges. The agent must collect:
If you do not explicitly force one question per field, the model starts improvising. It might combine fields, misinterpret the priority, or treat part of the description as a summary. This reinforced a pattern that applies to all real-world agents:
LLMs do not necessarily handle slot filling. You must enforce it.
Before creating a ticket, the agent says:
“Let me confirm: project is X, summary is Y, description is Z, priority is P. Should I create this?”
This checkpoint mattered more than I expected. Without it, the agent would occasionally move forward with a wrong or incomplete field. With it, mistakes became catchable and fixable.
This is essential any time an agent writes to a system of record.
Tool calls fail sometimes. Or they return empty results. Or latency causes confusion. In my prompt, I added a rule:
“If something does not work, acknowledge it and move on.”
Without this, the agent would retry forever or get stuck in an apology loop. In production, graceful degradation matters as much as correctness.
The Telnyx analysis tab became my best friend during this build. Insert screenshot of analysis tab showing conversation logs, tool calls, responses. I used it to see:
Watching the agent in real time revealed issues that were not visible from the prompt alone. It also helped validate that the model was respecting the order I intended.
Here are the design patterns that emerged and that I would reuse for any agent:
The biggest lesson is that the complexity of agent building is not in the API integrations. It is in shaping how the model asks questions, manages state, handles failures, interprets tools, and interacts with humans.
In other words, the JIRA tool was the easy part. The conversation was the hard part and the pattern applies to almost every real-world agent. If you get the flow, gating, and failure paths right, the agent feels reliable. If you do not, even the simplest task becomes unpredictable.
Building this agent was a great reminder that agents are not automagic. They need structure, good prompt architecture, error handling, and careful shaping of how they gather information from users. The Telnyx portal made it easy to iterate quickly, inspect calls, adjust prompts, and connect external tools. Once the agent logic was in place, the actual JIRA operations were trivial. Try building one yourself from here.
If you're curious what the final prompt looked like after all those iterations, here it is:
You are a friendly and efficient Bug Reporter assistant. Your job is to help users report bugs by voice and create properly formatted JIRA issues.
You have access to these JIRA tools:
"Which JIRA project should this bug go in? For example, you might have DR, PORTAL, VOICEAI, or DEVREL."
"What's the bug? Give me a short summary." "Can you describe what happened in detail?" "What's the priority? Low, Medium, Major, or Critical?"
"Let me confirm:
If user confirms, use the tool jira_createissue with: { "projectkey": "[PROJECT]", "summary": "[SUMMARY]", "description": "[DESCRIPTION]", "issuetype": "Bug", "priority": "[PRIORITY]" }
If it succeeds: "Perfect! Created issue [ISSUE KEY] in [PROJECT]." If it fails: "I tried to create the issue but ran into a problem. Your bug report was: [SUMMARY] in [PROJECT]."
"Would you like to report another bug?"
If no, say "Thanks for reporting!" and hangup.
IMPORTANT:
Building with Voice AI Instructions? Join our subreddit now.
