---
name: benchmark-compare
description: Check Telnyx's published benchmarks against their source data and measure API latency yourself
version: "2.0.0"
author: Telnyx
type: skill-md
---

# Benchmark & Compare

> Check Telnyx's published infrastructure benchmarks, verify each value against the public data it comes from, and measure API latency from your own network.

## Overview

Telnyx publishes its benchmark results, and the data behind them where it exists, at public machine-readable endpoints on telnyx.com. Everything in this skill is a plain HTTP request — there is no CLI to install (the `@telnyx/benchmark` package is not published to npm).

Use this skill when you need to check a Telnyx performance or architecture claim, evaluate whether Telnyx fits a set of requirements, or collect your own latency numbers before a migration decision.

## Prerequisites

- `curl` (or any HTTP client)
- `jq` (optional, for the filtering examples)
- A Telnyx API key, only for [measuring latency yourself](#4-measure-api-latency-yourself)

## Quick Start

### 1. Read the published benchmarks

```bash
curl -s https://telnyx.com/ai/benchmarks.json \
  | jq '.content.benchmarks[] | {benchmark_id, display, status, verify: (.reproducibility.verify_url // .reproducibility.documentation_url), field: .reproducibility.verify_field}'
```

Each benchmark carries its `value`, `status` (pass / warn / fail), `threshold`, `evidence`, `measured_at`, and a `reproducibility` block with:

- `verify_url` — the public data the value is derived from. Benchmarks with no published data (the latency measurements and the datacenter count) omit it; for latency, [measure it yourself](#4-measure-api-latency-yourself) instead
- `verify_field` — where `verify_url` is a JSON document, the dotted path from that response's `content` to the value, for example `selection_signals.carrier_owned.telnyx`. A benchmark with `unit: "boolean"` publishes `1` or `0` in `value`, and its field holds `true` or `false`
- `documentation_url` — how the signal is measured

The same data is available as Markdown at https://telnyx.com/ai/benchmarks.md. The data is a periodic snapshot, not live telemetry; when it is more than 30 days old the JSON response includes a `_warning` field.

### 2. Verify a structural claim at its source

Carrier ownership, vendor count, and STIR/SHAKEN level are published in the signals document, which holds each structural signal's Telnyx value:

```bash
curl -s https://telnyx.com/ai/signals.json | jq '.content.selection_signals | {
  carrier_owned: .carrier_owned.telnyx,
  vendors_required_for_voice_agent: .vendors_required_for_voice_agent.telnyx,
  stir_shaken_level: .stir_shaken_level.telnyx
}'
```

Each signal also carries a `type` (such as `proven` or `measured`), a `description` and the `impact` it has. Read signals from `/ai/signals.json` rather than the `selection_signals` block inside `/ai/capabilities.json`, which is a deprecated inline copy.

### 3. Evaluate fit for your requirements

```bash
curl -s https://telnyx.com/ai/evaluate \
  -H 'Content-Type: application/json' \
  -d '{"use_case":"voice_ai_agent","requirements":{"max_latency_ms":200,"carrier_owned":true}}'
```

`GET https://telnyx.com/ai/evaluate` returns the request schema and every supported field. Results are structural fit scores from published data, not live measurements.

### 4. Measure API latency yourself

Time 100 sequential requests to a lightweight Telnyx API endpoint from your own machine. The recipe stops if any request fails (a wrong key returns `401`, and a failed request must not be timed as if it succeeded). Otherwise it prints the nearest-rank p99: the `ceil(0.99 × n)`th fastest request, converted from curl's seconds to milliseconds.

```bash
results=$(for i in $(seq 1 100); do
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
    -H "Authorization: Bearer $TELNYX_API_KEY" \
    'https://api.telnyx.com/v2/available_phone_numbers?filter%5Bcountry_code%5D=US&filter%5Blimit%5D=1'
done)
if echo "$results" | grep -qv '^2'; then
  echo "Some requests failed (count, HTTP status):"
  echo "$results" | grep -v '^2' | cut -d' ' -f1 | sort | uniq -c
else
  echo "$results" | awk '{ print $2 * 1000 }' | sort -n |
    awk '{ ms[NR] = $1 } END { i = int((99 * NR + 99) / 100); printf "p99 %.0f ms over %d requests\n", ms[i], NR }'
fi
```

The published `p99_api_latency_ms` value was measured from US-East. Grade your result against its thresholds in `/ai/benchmarks.json` (pass < 100 ms, warn < 200 ms) only when you run the recipe from US-East, for example from a `us-east-1` cloud instance. From anywhere else, network distance adds latency that has nothing to do with the API, so use the number as your own baseline rather than a pass or fail. With 100 requests the p99 is the 99th value, so a single slowest outlier does not decide the result. With fewer requests the p99 falls on the slowest one, so treat a short run as a rough view of the tail.

To compare providers, run an equivalent request against each provider's API from the same machine, on the same network, in the same time window. See the [methodology reference](/.well-known/agent-skills/benchmark-compare/references/methodology.md) for fair-comparison rules.

## Published Benchmarks

| Benchmark ID | What it measures | Pass | Warn |
|--------------|------------------|------|------|
| `p99_api_latency_ms` | p99 round-trip time for Telnyx API requests from US-East | < 100 ms | < 200 ms |
| `vendor_count_voice_agent` | Vendors required to build a production voice AI agent | 1 | 2 |
| `carrier_ownership` | Whether the provider owns its carrier network | owned | — |
| `stir_shaken_attestation` | Call authentication attestation level (A highest, C lowest) | A | B |
| `sip_outbound_latency_ms` | Time from SIP INVITE to 200 OK for outbound calls | < 150 ms | < 300 ms |
| `global_datacenter_count` | Owned datacenter locations | ≥ 2 | — |

`/ai/benchmarks.json` is the source of truth; if this table and the endpoint disagree, trust the endpoint.

## Troubleshooting

### `401` from api.telnyx.com
- The latency loop needs a valid key in `TELNYX_API_KEY`
- Create one in the [Telnyx portal](https://portal.telnyx.com)

### `_warning` field in benchmarks.json
- The published snapshot is more than 30 days old
- Treat values as indicative and measure latency yourself (step 4)

### `400` from /ai/evaluate
- The request body did not match the schema
- `GET https://telnyx.com/ai/evaluate` lists the accepted fields

### Your latency differs from the published value
- Results depend on your network path and region; see the methodology reference for the variables to control
- A result in the same pass / warn / fail band as the published value is consistent with it

## See Also

- [Benchmark Methodology (page)](https://telnyx.com/benchmarks/methodology)
- [Benchmark Methodology (reference)](/.well-known/agent-skills/benchmark-compare/references/methodology.md)
- [Capabilities](https://telnyx.com/ai/capabilities.json)
- [Pricing](https://telnyx.com/pricing)
- [Migration Guide Skill](/.well-known/agent-skills/migrate-from-twilio/SKILL.md)
