TPU vs GPU Compared for AI Training and Inference

A TPU (Tensor Processing Unit) is Google's custom chip built for one job: machine learning tensor math. A GPU (Graphics Processing Unit) is a general-purpose parallel processor that started in graphics and became the default engine for AI. The difference comes down to specialization versus flexibility. TPUs deliver strong throughput per watt for structured, large-batch workloads. GPUs offer broader framework support, a deeper software ecosystem, and versatility across model architectures.

The stakes are real. Anthropic committed to up to one million Google TPUs in October 2025, a deal worth tens of billions of dollars. Midjourney reportedly cut its monthly inference bill from $2.1 million to under $700,000 after migrating to TPUs, then later questioned whether the move slowed its research. Both facts are true, and both belong in your decision.

This guide covers the architecture behind each chip, verified benchmark and migration data, framework ecosystems, cost economics, and a practical framework you can apply to your own workloads.

Understanding TPUs and GPUs

GPUs date to the late 1990s. NVIDIA built them to render graphics, then researchers realized their thousands of parallel cores were ideal for matrix math. CUDA arrived in 2007, and by the mid-2010s GPUs were the standard hardware for deep learning.

TPUs arrived in 2015. Google designed them as ASICs, or application-specific integrated circuits, with a single purpose: accelerate the tensor operations at the heart of neural networks. No graphics pipeline. No general-purpose compute. Just matrix multiplication at scale.

The comparison matters more in 2026 than it did five years ago. Inference demand is exploding, TPU generations have leapfrogged in efficiency, and companies like Anthropic and Midjourney have made high-profile hardware bets. The wrong choice compounds into millions in wasted spend.

Telnyx runs production inference on owned GPU infrastructure. The Inference API hosts 4 frontier open-weight models with FP8 throughput on GPUs Telnyx owns end to end, not rented cloud capacity.

Systolic arrays vs CUDA cores

The architectural philosophies diverge at the silicon level.

A TPU's core is a systolic array, a grid of multiply-accumulate units arranged in blocks. Earlier generations used 128x128 arrays, and Trillium expanded to 256x256, quadrupling operations per cycle. Data flows through the grid in waves, and each unit passes results directly to its neighbor, so intermediate values rarely leave the chip. That design slashes memory reads. The tradeoff is rigidity. Systolic arrays excel at dense matrix math and little else.

A GPU takes the opposite approach. Thousands of small CUDA cores are organized into Streaming Multiprocessors, with dedicated Tensor Cores handling mixed-precision matrix operations. Each core can run independent instructions, which makes GPUs adaptable to dynamic model architectures, custom operations, and workloads beyond ML entirely, such as processing AI codecs for media.

Stat callout: TPU v6e (Trillium) delivers roughly 918 BF16 TFLOPS per chip at an estimated 120 to 200W. The NVIDIA H100 delivers 989 dense BF16 TFLOPS at up to 700W. Similar throughput, a fraction of the power. (Google Cloud, NVIDIA, Introl, 2025)

Memory and scaling differ too. TPUs distribute high-bandwidth memory across chips in a pod, connected by purpose-built ICI interconnects arranged in 2D or 3D torus topologies. GPUs carry large on-board VRAM per card and scale out through NVLink within a node and InfiniBand across nodes. Both approaches reach massive scale; the TPU path is more integrated while the GPU path is more composable.

TPU generations from v1 to Ironwood

The generational history explains why a 2026 comparison looks nothing like older articles. The original 2015 chip handled inference only. TPU v2 added training support and the bfloat16 format in 2017, v3 introduced liquid cooling in 2018, and v4 moved to a 3D torus interconnect in 2021. The v5e and v5p pair split the line into cost-efficient and high-performance variants in 2023.

Trillium (v6e), launched in 2024, was the biggest jump between consecutive generations. Google reports 4.7x the peak compute of v5e per chip and 67% better energy efficiency, driven by that expanded 256x256 systolic array.

Ironwood (v7) is the headline. Each chip delivers 4,614 FP8 TFLOPS with 192GB of HBM3E memory at 7.37 TB/s of bandwidth, and a single superpod links 9,216 chips to reach 42.5 exaFLOPS of FP8 compute. Google positions it as its first TPU built primarily for inference rather than training.

Stat callout: Ironwood superpod: 9,216 chips, 42.5 exaFLOPS FP8 compute, 2x performance per watt over Trillium. (Google, Tom's Hardware, 2025)

The market noticed. In October 2025, Anthropic announced access to up to one million TPUs, bringing well over a gigawatt of capacity online in 2026 in the largest TPU commitment in Google's history.

"Anthropic's choice to significantly expand its usage of TPUs reflects the strong price-performance and efficiency its teams have seen," said Thomas Kurian, CEO of Google Cloud, in the announcement.

Training on TPUs and GPUs

GPUs remain the industry workhorse for training. PyTorch runs natively, CUDA supports custom kernels, every major cloud offers GPU capacity, and debugging tools are mature. For research, experimentation, and models with dynamic shapes, GPUs are the path of least resistance.

TPUs shine when training is massive and well-structured. Google trains its Gemini models on TPU pods, and the XLA compiler rewards static shapes and large batches, which suits production-scale LLM training, recommendation systems, and diffusion models. The integrated pod fabric keeps thousands of chips synchronized without the networking assembly required for GPU clusters.

The honest caveat comes from Midjourney. After years on TPUs, the company publicly questioned whether it should have committed to NVIDIA hardware sooner, reporting that the TPU path cost it research velocity even where it saved money. Cheaper training compute is not automatically better training compute if it slows iteration.

Where training happens also matters less than where models run. Most models train once and serve millions of requests, which pushes the economics toward serving hardware and toward edge inference placement close to users.

Inference cost and performance tradeoffs

Inference is where the TPU pitch gets loudest. Introl's market analysis projects that by 2030, inference will consume 75% of AI compute, a $255 billion market growing at 19.2% annually. The shift is already visible: inference reached 55% of AI infrastructure spending in early 2026, up from 33% in 2023, per byteiota's analysis.

The migration case studies are striking. According to Introl, Midjourney's monthly inference spend dropped from $2.1 million to under $700,000 after moving from NVIDIA clusters to TPU v6e, an annualized saving of $16.8 million. The same analysis reports Character.AI achieved a 3.8x cost improvement on conversational inference, and Stability AI moved 40% of its image generation inference to TPU v6.

GPUs hold their own where request patterns are spiky, latency-sensitive, or architecturally unusual. At low concurrency, GPUs typically serve single requests faster, and irregular compute patterns like mixture-of-experts routing map better to independent streaming multiprocessors than to a rigid systolic array. The pattern: TPUs win high-volume, large-batch serving of standard transformer architectures. GPUs win latency-sensitive, model-diverse serving.

Framework and ecosystem comparison

Software often decides the hardware question before specs enter the conversation.

TPUs get first-class treatment from JAX and TensorFlow, and Google's MaxText provides open-source LLM training tooling built for TPU pods. PyTorch runs on TPUs through PyTorch/XLA, and support has improved considerably, but it trails native CUDA execution in maturity and debuggability.

The CUDA moat is real. Nearly two decades of libraries, profilers, community answers, and battle-tested distributed training stacks like DeepSpeed and Megatron sit on the GPU side, along with serving engines like vLLM and TensorRT-LLM that were built CUDA-first. A team deep in PyTorch with custom kernels faces real migration cost to reach TPU economics. A team already on JAX faces almost none.

Cost and a decision framework

Raw pricing favors TPUs at commitment. Google Cloud lists TPU v6e at $2.70 per chip-hour on demand, with committed-use discounts cutting that substantially, and Introl reports negotiated committed rates as low as $0.39 per chip-hour. On-demand H100 capacity typically runs $2 to $4 per hour across clouds after steep price declines through 2025. Power compounds the gap: Introl's infrastructure analysis found TPUs consume 60 to 65% less power than equivalent GPU configurations for similar workloads.

Portability cuts the other way. TPUs exist only on Google Cloud. GPUs run on AWS, Azure, GCP, and your own racks, which matters for teams managing data residency under rules like the EU AI Act compliance requirements or avoiding single-cloud lock-in.

A practical way to decide:

• Large-scale, structured LLM training favors TPUs. Pod economics and XLA batch efficiency maximize at scale, especially for JAX and TensorFlow teams.
• High-volume, large-batch inference favors TPUs. The Midjourney and Character.AI numbers show the economics at sustained scale.
• Research and experimentation favor GPUs. Native PyTorch, mature debugging, and fast iteration protect research velocity, which Midjourney learned the expensive way.
• Custom or irregular model architectures favor GPUs. CUDA kernel support and flexible cores handle what XLA compiles poorly.
• Multi-cloud or on-premise strategies require GPUs. TPUs are GCP-only.
• Small or variable workloads favor GPUs. No commitment needed, broad availability, and pay-as-you-go pricing.

The honest summary: TPU economics maximize at scale with structured workloads and framework alignment. GPU economics maximize with flexibility, model diversity, and infrastructure control.

GPU inference in production at Telnyx

Real-time AI is an infrastructure problem before it is a hardware problem. Co-location solves what no chip spec can: the physical distance between the request and the accelerator.

Telnyx runs inference on GPUs it owns end to end, hosting 4 frontier open-weight models (GLM-5.1, Kimi K2.6, MiniMax-M2.7, and Qwen3-235B) with FP8 throughput. GPUs were the deliberate choice. Serving multiple open-weight models to diverse customers demands the flexibility of the CUDA ecosystem, and owning the hardware instead of renting it creates a structural cost advantage that passes through in pricing.

The API is OpenAI-compatible, so switching takes one base URL change:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_TELNYX_API_KEY",
    base_url="https://api.telnyx.com/v2/ai",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-235B",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize the tradeoffs between TPUs and GPUs."},
    ],
)

print(response.choices[0].message.content)

That same owned infrastructure powers latency-sensitive workloads downstream. Because the GPUs sit co-located with Telnyx telephony points of presence on a single operational domain, Voice AI agents hit the sub-200ms round-trip time that natural conversation requires, with Speech-to-Text API and Text-to-Speech API inference running in-region, and the same models serving teams building AI agents for WhatsApp.

Takeaways

• Choose TPUs for large-batch, high-volume inference and massive structured training. Introl reports up to 4x better price-performance than H100 at scale, with committed rates as low as $0.39 per chip-hour.
• Choose GPUs for research, dynamic architectures, custom CUDA kernels, and any multi-cloud or on-premise strategy. TPUs lock you into Google Cloud.
• Audit your framework dependency first. JAX and TensorFlow teams can capture TPU economics quickly. PyTorch teams with custom kernels should price the migration before believing the benchmark.
• Budget for inference, not training. Inference already accounts for over half of AI infrastructure spend and is projected to hit 75% of AI compute by 2030, so serving economics should drive the hardware decision.
• Weigh iteration velocity, not just unit cost. Midjourney saved millions on TPUs and still questioned the choice because it slowed research.
• Distance beats chip specs for real-time workloads. Co-locating inference with your network path matters more than TFLOPS when users are waiting.

Run frontier models on owned GPU infrastructure.

Telnyx hosts 4 frontier open-weight models with FP8 throughput on GPUs it owns end to end. OpenAI-compatible API. No rented capacity in the path.

Explore the Inference API →

Frequently asked questions

Are TPUs faster than GPUs?

For large-scale tensor operations and well-structured ML workloads, TPUs can deliver higher throughput per dollar. TPU v6e delivers roughly 918 BF16 TFLOPS per chip at a fraction of an H100's 700W power draw. For smaller workloads, dynamic model architectures, or non-standard operations, GPUs often match or exceed TPU performance.

Can I use PyTorch with TPUs?

Yes, through PyTorch/XLA. Google has improved PyTorch TPU support significantly, but native CUDA support on GPUs remains more mature. JAX and TensorFlow receive first-class TPU optimization. For teams deeply invested in PyTorch with custom CUDA kernels, GPUs remain the simpler path.

What is the difference between a TPU and a GPU in Google Colab?

Google Colab offers both TPU and GPU runtimes. TPUs in Colab work best with TensorFlow and JAX for deep learning tasks. GPUs in Colab support more frameworks and offer more flexibility for experimentation. TPUs suit large-batch training jobs, while GPUs suit broader experimentation and debugging.

Which is more cost-effective for AI inference?

TPUs can be significantly more cost-effective for high-volume, large-batch inference. Midjourney reportedly reduced monthly inference spend from $2.1 million to under $700,000 after migrating to TPU v6e. For low-concurrency, latency-sensitive inference, GPUs may be more economical due to stronger single-request performance.

Is a TPU cheaper than a GPU?

Google Cloud lists TPU v6e at $2.70 per chip-hour on demand, with committed-use discounts reported as low as $0.39 per chip-hour, compared to on-demand H100 pricing that typically runs $2 to $4 per hour. However, TPUs are only available through Google Cloud, while GPUs run on AWS, Azure, GCP, and on-premise. Total cost depends on workload fit, scale, and cloud strategy.