ORBiS TREE · ENGLISH DOCUMENT
Infrastructure →Tensor Processing Unit (TPU)
Google's specialized machine-learning accelerator, explained through matrix computation, software compatibility and workload-level evaluation.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Overview
A Tensor Processing Unit (TPU) is a family of specialized integrated circuits designed by Google to accelerate machine learning. Cloud TPU documentation describes matrix computation, memory and accelerator interconnection as important parts of the platform.[1] Whether a TPU helps a particular workload depends on the computation and its software execution path.
Computation and software
Many neural-network operations can be expressed as matrix multiplication. Faster matrix computation does not eliminate delays in data preparation or transfer. Cloud TPU documentation also describes the role of XLA compilation.[1] Compare supported operations, framework behavior and compilation requirements alongside hardware specifications.
Comparing TPU and GPU
- Workload: how much time is spent on matrix operations, and how are batches organized?
- Software: can the required code and operations run in the target environment?
- Memory: do weights, intermediate activations and inputs fit?
- Data supply: can preprocessing and storage keep the accelerator busy?
- Operations: what are the availability, migration and ongoing operating costs?
Neither GPU nor TPU is universally cheaper or faster. Compare complete runtime and cost under the same model, data, batch and quality conditions. These are ORBiS Tree's evaluation questions, not published benchmark results.
Worked example: an accelerator waiting for data
This example is hypothetical. A training job spends more time waiting for preprocessing than performing device computation. Adding accelerators may provide limited improvement if that bottleneck remains. Record preparation, transfer, compilation and computation separately, adjust the input pipeline and batches, then measure again under comparable conditions.
Generations and evidence
The original 2017 TPU paper evaluates an early data-center inference design.[2] Its results cannot be carried over unchanged to every later generation, training task or Cloud price. When a generation name such as Trillium appears, check that generation's documented configuration and workload assumptions.
Reading infrastructure news
Project Suncatcher is an example of researching a new computing environment. A successful hardware test and sustained data-center-scale operation are different claims. For inference, measure first-output latency and total throughput; for training, compare time and cost to the target quality.
Worked example: overlapping data preparation and compute
Suppose a fictional batch needs 70 ms of data preparation and 50 ms of device compute. Sequential execution takes 120 ms. If preparation and computation for different batches overlap completely with no additional bottleneck, steady-state batch spacing is max(70, 50) = 70 ms, excluding startup and drain. Faster 25 ms compute still gives max(70, 25) = 70 ms.
These are not measured TPU timings or a guarantee that overlap is possible. Measure preparation, transfer and compute along the actual timeline, including data-starvation waits. Cloud TPU documentation ↗ describes compute, host execution and compilation.[1] Include the same input pipeline when comparing with other accelerators.
Count compilation separately for short jobs
If a fictional initial compilation takes two seconds and each later batch takes 20 ms, 100 batches require 2 + 100×0.02 = 4 seconds. Five batches require 2 + 5×0.02 = 2.1 seconds. Warmed-up 20 ms batch timings alone do not describe either full job.
Record shapes, data types, compilation cache, framework and versions. Verify shape-change support and recompilation behavior in the actual execution path. The original TPU paper ↗ evaluates an early data-center inference design, not every current training or compilation environment.[2] A generation name alone does not establish code compatibility or short-job benefits.
Distribution example: four devices need not give fourfold speed
A fictional 100 ms job has 20 ms of serial work and 80 ms that can be divided perfectly. Four devices give 20 + 80/4 = 40 ms, a 2.5-fold speedup. Adding 10 ms communication gives 50 ms, a twofold speedup. These assumptions about communication, batches and synchronization do not measure distributed TPU performance.
Record serial work, inter-device movement, synchronization and quality changes after altering batches. With distributed experts, reduced active compute and fast overall communication are different properties. Compare scaling under the same target quality and data conditions.
Fix success criteria before comparing hardware
For training, compare total time to the same evaluation quality, including failures and restarts. For inference, compare actual completion, quality and latency. Specify whether compilation, input supply, transfer and compute are included. Numbers with different measurement boundaries cannot simply be subtracted. Higher throughput can accompany longer individual waits.
Retain quality conditions from the model card and deployment configuration from the system card. Do not infer undisclosed specifications or current Cloud prices from older research. This document's numbers teach principles rather than prescribe hardware purchases.
SOURCES
Sources
-
Introduction to Cloud TPUGoogle CloudOpen source ↗
-
In-Datacenter Performance Analysis of a Tensor Processing UnitJouppi et al. / ISCAOpen source ↗
KNOWLEDGE LINKS
Continue from here
BACKLINKS