New Telnyx-Managed Insights for Agent Instruction Following and User Satisfaction

Two new Telnyx-managed insights, Agent Instruction Following and User Satisfaction, are now available for AI Assistants. These built-in insights measure assistant quality out of the box, with no prompt engineering or custom schema design required.

What's new

• Agent Instruction Following: Scores how well your assistant followed its system prompt and tool-use instructions during each conversation. Grades on a five-point scale: Excellent, Good, Fair, Poor, and N/A.
• User Satisfaction: Estimates how satisfied the caller was based on their responses, tone, and engagement signals. Same five-point scoring scale.
• Not auto-assigned to default insight groups: Because these insights are more compute-intensive than the existing Summary insight, they are not automatically added to the default Insight Group. Add them to any group of your choosing.
• Insights Over Time visualization: A new tab in the Analysis section of the Portal shows a 7-day stacked-bar chart of daily score counts, color-coded by grade (Poor through Excellent plus N/A).
• Version comparison: Toggle "Compare by assistant version" to split the over-time chart into small multiples, one per assistant version. Useful alongside traffic distribution for A/B testing.
• Aggregate API endpoint: Query aggregated insight counts programmatically with GET /ai/conversations/conversation-insights/aggregates. Group by score, metadata.assistant_id, metadata.assistant_version_id, or metadata.telnyx_conversation_channel. Filter by insight_id and created_at range to build per-day time series.
• On-network processing: Insights run inside Telnyx-hosted evaluation, keeping conversation analysis on the same private backbone as your inference and voice traffic.

Why it matters

• Building custom insight rubrics from scratch takes time and produces inconsistent scoring across teams. Telnyx-managed insights provide maintained, consistent scoring rubrics available to every account.
• Agent Instruction Following catches prompt adherence issues that are hard to spot manually, especially in complex assistants with many tool instructions or compliance-sensitive flows.
• User Satisfaction gives you a caller-experience signal without setting up sentiment analysis pipelines, directly correlating voice AI quality to business outcomes.
• The over-time chart and version comparison let you regression-test assistant changes before promoting a new version. If "Poor" or "Fair" segments grow after a prompt update, the chart shows it within days.
• The aggregate API endpoint lets you pipe insight data into your own dashboards, alerting systems, or CI/CD pipelines to block bad versions from promotion.

Example use cases

• Customer support teams tracking User Satisfaction across assistant versions to catch regressions before promoting a new prompt to production.
• Compliance-sensitive flows using Agent Instruction Following to verify that disclosure scripts and tool-use rules are followed on every call.
• A/B testing two assistant versions with traffic distribution, then comparing side-by-side insight charts to decide which version to promote.
• Engineering teams piping aggregate insight counts into internal dashboards to monitor assistant quality alongside deployment metrics.

Getting started

1. In Mission Control, navigate to AI > Assistants and select your assistant.
2. Go to the Analysis tab.
3. Click on the Insights Over Time sub-tab.
4. Use the Insight dropdown to select Agent Instruction Following or User Satisfaction.
5. The 7-day stacked-bar chart loads, showing daily score counts color-coded by grade (Poor through Excellent plus N/A).
6. Toggle Compare by assistant version above the chart to split the data into separate charts per assistant version.

To enable these insights on an assistant, add them to an Insight Group first: AI > Insights > AI Insight Groups tab > create or edit a group > search for and add Agent Instruction Following and User Satisfaction > save. Then assign the group to your assistant under Analysis > Insights sub-tab.

To query aggregated insight counts via API:

curl -X GET "https://api.telnyx.com/v2/ai/conversations/conversation-insights/aggregates?group_by=score&group_by=metadata.assistant_version_id&metadata.assistant_id=eq.<your-assistant-id>&insight_id=<insight-id>&created_at=gte.2025-07-01T00:00:00Z&created_at=lt.2025-07-08T00:00:00Z" \
  -H "Authorization: Bearer $TELNYX_API_KEY"

Response:

{
  "data": [
    { "score": "excellent", "metadata": { "assistant_version_id": "v1" }, "record_count": 142 },
    { "score": "good", "metadata": { "assistant_version_id": "v1" }, "record_count": 38 },
    { "score": "excellent", "metadata": { "assistant_version_id": "v2" }, "record_count": 98 },
    { "score": "good", "metadata": { "assistant_version_id": "v2" }, "record_count": 71 }
  ]
}

Learn more in the Telnyx-managed insights guide, the Insight Groups docs, or the Aggregate Conversation Insights API reference.