IR4 Leaders

Google technical article 011

Google Tensor Processing Units

How Google’s eighth-generation TPU 8t and TPU 8i separate large-scale training from latency-sensitive serving through different memory, compute and interconnect designs.

01

What it is

A Google Tensor Processing Unit is a custom accelerator for neural-network computation in Google’s data centres and Cloud AI Hypercomputer. The current eighth generation is split into TPU 8t for large-scale pre-training and TPU 8i for sampling, serving, reasoning, reinforcement learning and mixture-of-experts workloads.

Both are complete systems, not isolated chips. Each chip combines matrix, vector, on-chip memory and high-bandwidth memory resources, then uses an inter-chip interconnect to form pods. Arm Axion CPUs host both variants. As of this article’s verification time, TPU 8t and 8i are announced for upcoming customer availability; seventh-generation Ironwood remains generally available.

GenerationEighthTPU 8t and TPU 8i
Training systemTPU 8t3D torus; 9,600-chip superpod
Serving systemTPU 8iBoardfly; up to 1,024 active chips
SoftwareXLA + PathwaysJAX, PyTorch, Keras and vLLM
02

Why Google built two TPUs

Training and serving use the same model mathematics but stress hardware differently. Training applies large, regular matrix operations across huge batches and exchanges gradients or parameters at high throughput. Autoregressive serving repeatedly generates a small amount of output, maintains a growing KV cache and may route each token to different experts, making memory capacity and collective latency more important.

Google therefore stopped treating one topology as optimal for both. TPU 8t retains a large 3D torus and SparseCore for training and embedding traffic. TPU 8i increases HBM and on-chip SRAM, adds a Collectives Acceleration Engine and replaces the torus with a lower-diameter Boardfly network.

PressureTPU 8t responseTPU 8i response
Dense matrix throughput12.6 PFLOPS peak FP4; native FP4 MXU path.10.1 PFLOPS peak FP4, balanced for serving.
Embedding lookupsSparseCore handles irregular access and selected collectives.CAE replaces prior SparseCores.
KV cache and decoding state128 MB on-chip Vmem; 216 GB HBM.384 MB on-chip Vmem; 288 GB HBM.
All-to-all communicationLarge 3D torus optimised for training scale.Boardfly cuts maximum 1,024-chip path from 16 to seven hops.
03

Chip and memory architecture

Google TPU 8t and 8i architecture comparisonBoth TPU systems contain matrix and vector execution, on-chip Vmem and HBM; TPU 8t adds SparseCore and a torus while TPU 8i adds CAE and Boardfly.TPU 8t · TRAININGMXU + VPUMatrix and vector executionSparseCore + decoder engineEmbeddings and irregular collectives128 MB Vmem + 216 GB HBM6,528 GB/s HBM bandwidth3D torus ICITPU 8i · SERVING2 × Tensor Core diesMatrix and vector executionCollectives Acceleration EngineLow-latency reductions and synchronisation384 MB Vmem + 288 GB HBM8,601 GB/s HBM bandwidthBoardfly ICI
The eighth generation specialises both the silicon surrounding matrix units and the network connecting chips.
VideoGoogle Cloud · TPU 8t and TPU 8iYouTube ↗
VideoGoogle Cloud · Introducing eighth-generation TPUsYouTube ↗
VideoGoogle Cloud Tech · Scaling AI with TPUsYouTube ↗
Videoscrollypedia · Why Google split TPU 8t and 8iYouTube ↗
04

How a model layer executes

  1. Compile the graph.XLA maps model operations to TPU matrix, vector, memory and collective primitives.
  2. Shard model and data.JAX or Pathways partitions tensors across chips according to the selected parallelism strategy.
  3. Stage operands.Weights and activations move from HBM into on-chip Vmem close to execution units.
  4. Run arithmetic.MXUs process matrix multiplications while VPUs overlap quantisation, softmax, normalisation and other vector work.
  5. Handle specialised traffic.TPU 8t uses SparseCore for embedding-heavy access; TPU 8i uses CAE for reductions and synchronisation.
  6. Exchange partitions.ICI carries scale-up collectives inside the pod; the data-centre network and Pathways connect work beyond one pod.
05

From chip to distributed system

LevelTPU 8tTPU 8i
ChipTraining-oriented tensor, vector and SparseCore resources.Two Tensor Core dies plus one CAE chiplet die.
Local building blockChips connect into the 3D torus.Four-chip ring with 16 external links.
Pod / superpod9,600 chips; 121 exaflops aggregate; about 2 PB shared HBM.Eight boards per group; 36 groups; up to 1,024 active chips through optical circuit switches.
Scale-outVirgo links more than 134,000 chips with up to 47 Pb/s non-blocking bisection bandwidth.Cloud network connects serving pools and surrounding services.
Host and softwareArm Axion CPU; XLA, JAX and Pathways.Arm Axion CPU; XLA, JAX, PyTorch preview and vLLM.
Generation context. Ironwood is the seventh-generation TPU and remains generally available in 256- or 9,216-chip pods. TPU 8t and TPU 8i are the announced eighth-generation flagships, but Google stated that customer availability was coming rather than already general at the cited April 2026 launch.
06

TPU 8t and TPU 8i specifications

AreaTPU 8tTPU 8i
Primary workloadLarge-scale pre-training and embedding-heavy modelsSampling, serving, reasoning, RL and MoE
Peak FP412.6 PFLOPS per chip10.1 PFLOPS per chip
HBM capacity216 GB per chip288 GB per chip
HBM bandwidth6,528 GB/s8,601 GB/s
On-chip Vmem128 MB384 MB
Specialised blockSparseCore and LLM Decoder EngineCollectives Acceleration Engine
ICI topology3D torus; twice Ironwood scale-up bandwidthHierarchical Boardfly; 19.2 Tb/s ICI
Published pod scale9,600-chip superpodUp to 1,024 active chips; 1,152 physical positions cited
CPU hostArm AxionArm Axion
07

Mechanisms that change utilisation

MechanismFunctionWhy it matters
Native FP4Processes four-bit floating-point values directly in TPU 8t MXUs.Reduces operand size and doubles the cited MXU rate relative to a wider path.
VPU/MXU overlapRuns vector operations alongside matrix work.Avoids serial quantisation, softmax or normalisation stalls.
SparseCoreHandles embedding lookups and irregular gather/collective patterns.Prevents sparse data access from idling dense matrix hardware.
CAEAccelerates reductions and synchronisation on TPU 8i.Google reports fivefold lower on-chip collective latency.
BoardflyUses fully connected local groups and optical links between groups.Reduces maximum 1,024-chip diameter from 16 to seven hops.
TPUDirect RDMA / StorageMoves data between HBM, NICs and managed storage without host-DRAM staging.Removes CPU and host-memory copies from large transfer paths.
08

Terms

TPU
Google’s custom Tensor Processing Unit accelerator.
MXU
Matrix Multiply Unit for dense tensor arithmetic.
VPU
Vector Processing Unit for element-wise and reduction work.
Vmem
Fast on-chip SRAM used near the execution units.
ICI
Inter-chip interconnect forming the TPU scale-up network.
CAE
TPU 8i engine for collective communication operations.
MoE
Mixture of experts, routing tokens to selected model sub-networks.
XLA
Google’s Accelerated Linear Algebra compiler.
09

Limits of the evidence

  • TPU 8t and 8i are announced systems; Google stated customer availability was upcoming at launch.
  • Peak FP4 rates depend on supported operations and numeric format and are not sustained model throughput.
  • Pod and fabric figures are aggregate design maxima; application scaling depends on partitioning and communication.
  • The cited 1,152-chip Boardfly physical scale and 1,024 active-chip architecture describe different boundaries and are retained separately.
  • Google’s comparative performance, cost and energy claims are excluded.
  • Detailed cache, clock, power and die-manufacturing specifications are not fully published.
  • The embedded videos are explanatory and launch material, not evidence for the specification tables.
  • Price and promotional material are excluded.
10

Sources

  1. TPU 8t and TPU 8i technical deep diveChip blocks, memory, compute, topology, direct I/O and software.
  2. Google Cloud AI infrastructure at Next ’26Eighth-generation launch, system scale and availability wording.
  3. Introducing Virgo NetworkTPU 8t scale-out topology and bisection bandwidth.
  4. Ironwood TPU general availabilitySeventh-generation deployment context and pod sizes.
  5. TPU cluster reliabilityDistributed checkpointing, resilience and large-cluster operation.
  6. Cloud TPU system architectureHost, accelerator, network and TPU VM programming model.
  7. Google TPU 8t and TPU 8i: Purpose-built for the Agentic EraOfficial Google Cloud overview supplied for this article.
  8. Introducing eighth-generation TPUsOfficial Google Cloud launch video supplied for this article.
  9. Scaling AI with Google Cloud’s TPUsGoogle Cloud Tech system-level explainer supplied for this article.
  10. TPU 8t vs 8i: Why Google Split Its Chip in TwoIndependent explainer supplied for additional context.