Client-Side Tools Now Available for Telnyx AI Assistants

AI Assistants on Telnyx can now call functions that run directly in the browser during a voice or chat conversation. Client-side tools give the assistant access to browser state, local data, and any API the page is already authenticated to, without a webhook or backend round-trip.

What's new

• Client-side tool type: A new tool type that executes JavaScript on the client instead of sending an HTTP request to your backend. The assistant calls the tool, your handler runs in the browser, and the return value is sent back to the assistant to continue the conversation.
• Browser-state access: Tools can read shopping carts, user sessions, page context, or call APIs authenticated with the user's browser session, all without exposing that data through a webhook.
• UI action triggers: Tools can trigger frontend actions like opening a modal, navigating to a page, or playing media, so the assistant can drive the UI alongside the conversation.
• Runtime registration: Register tools at construction time or add and remove them dynamically with registerClientTool and unregisterClientTool. Per-tool timeout is configurable via clientToolTimeoutMs.
• Built-in error handling: The library returns safe error outputs for unknown tools, invalid arguments, handler exceptions, and timeouts, so the conversation never hangs on a failed tool call.

Why it matters

• Webhook tools require a backend server to receive and respond to tool calls. Client-side tools eliminate that infrastructure for browser-based interactions, reducing integration surface and deployment complexity.
• Conversational agents that can act on page state, not just backend data, can do things webhook tools cannot: read a shopping cart, navigate the page, or call an API with the user's existing session token.
• The Voice SDK Proxy WebSocket relays tool invocations and results between the assistant and client code, so no additional HTTP endpoint is needed.

Example use cases

• An e-commerce assistant reads the user's shopping cart from page state and answers questions about order totals, item availability, or shipping options without a backend lookup.
• A support assistant on a SaaS dashboard navigates the user to the relevant settings page or opens a help modal based on what the caller asks about.
• A booking assistant calls an API authenticated with the user's browser session to check availability and confirm reservations without building a separate webhook endpoint.

Getting started

1. Open AI Assistants in the Telnyx Portal and select an assistant.
2. Scroll to the Tools section, click Add Tool, and select Client-Side Tool.
3. Configure the tool name, description, parameters (JSON Schema), and timeout.
4. Install the Telnyx AI Agent library: npm install @telnyx/ai-agent-lib (requires version 0.5.0 or later).
5. Register a handler in your client code that matches the tool name:

import { TelnyxAIAgent } from '@telnyx/ai-agent-lib';

const agent = new TelnyxAIAgent({
  agentId: 'your-agent-id',
  clientTools: {
    lookup_order: async (args) => {
      const { orderId } = args as { orderId: string };
      const response = await fetch(`/api/orders/${orderId}`);
      const order = await response.json();
      return { status: 'found', orderId: order.id, total: order.total };
    },
  },
});

Client-side tools are available for WebRTC-based conversations (voice and chat) using the @telnyx/ai-agent-lib library. They are not available for SIP or phone-call-based conversations.

Learn more in the client-side tools guide or the AI Agent Lib docs.