The torch.addmm call in Code 20 does not actually run on the CUDA cores we discussed in The streaming multiprocessor. It runs on Tensor Cores, hardware specialized for matrix multiply-accumulate (MMA), the sum += a * b operation at the heart of matrix multiplication.

As covered in The streaming multiprocessor, CUDA cores are like CPU cores but simpler. Because they support many different operations, each instruction follows the full fetch-execute cycle: fetch, decode, read operands, execute, and write back. This overhead exists to provide flexibility for general-purpose computation.

For matrix multiply-accumulate, none of that flexibility is needed. The operation is always the same, and the data always flows the same way. A Tensor Core hard-codes all of this, eliminating the fetch, decode, and writeback overhead entirely. Hardware specialized to do exactly one thing is called an application-specific integrated circuit (ASIC). The throughput advantage over a CUDA core is roughly 100×.

A Tensor Core is structured as a grid of multiply-accumulate units, as shown in the following figure.

Each circle in the figure is one unit. It receives data flowing in from the left and from above, passes it along to the right and downward, and computes one multiply-accumulate on the way. There is no fetch, no decode, no writeback: the operation and the data path are both fixed in silicon. Because all units in the grid operate in parallel, the entire tile is processed at once. An animated version of this is available on the home page.

Tensor Cores are not the only ASIC on a GPU. Special Function Units (SFUs) handle transcendental functions like exp and log, which appear in activation functions such as softmax and GeLU. Knowing which operations run on which unit, and how fast each one is, helps you reason about where the bottleneck in your workload actually lies.

Because Tensor Cores are so much faster than CUDA cores for MMA, matmul almost always bottlenecks on memory bandwidth rather than compute, which is exactly why the memory access optimizations throughout this chapter matter as much as they do.

You rarely invoke Tensor Cores directly. For standard operators like torch.addmm, PyTorch routes the computation to Tensor Cores automatically. For custom kernels written in Triton or Helion, the compiler handles the dispatch as well.

One detail we have glossed over is that the performance of Tensor Cores depends on the data type. Different formats represent numbers with different amounts of precision, trading accuracy for speed. Understanding how numbers are represented at the bit level, and what these tradeoffs look like in practice, is the subject of the next chapter.