For machine learning workloads, compute is the gold: we want the machine to calculate values, not to move data. Moving data is just the logistics needed to keep the compute running.

For a given ML job, we would like to measure its speed. This measurement tells us whether the job is well optimized and whether we are fully utilizing the hardware's capability. This speed metric is called throughput.

Intuitively, you can think of the GPU as a factory, and the processing unit as the worker sitting at the assembly line producing the product. The worker's speed is the throughput: how many products it can assemble per unit time.

However, there is a caveat. Assembling one product may require several different premade parts as input, and several assembled products may be packaged into a single box before shipping. Just like the matmul op, where the number of input floats and output floats differ, this makes speed hard to define. Should we measure inputs per second? Outputs per second? Neither.

We should measure how fast the worker is actually working, not rely on indirect estimators like the number of inputs or outputs. We measure it by how many floating-point operations, like adding or multiplying two numbers, the processing unit performs in one second. This metric is called FLOPs/s, where FLOPs stands for floating-point operations, the additions and multiplications the worker performs, and /s means we count how many happen every second.

For example, a matmul between two 1024 × 1024 matrices reads 2 × 1024² input floats but writes only 1024² output floats, so neither inputs per second nor outputs per second gives one well-defined number. What we measure instead is the worker's throughput: about 2 × 1024³ ≈ 2.1 billion FLOPs (each of the 1024² output elements needs 1024 multiplications and 1024 additions). If this matmul finishes in 1 millisecond, its throughput is 2.1 × 10⁹ / 0.001 ≈ 2.1 × 10¹² FLOPs/s, or 2.1 TFLOPS.

This needs some disambiguation. So far, we have described what is more precisely called compute throughput: how many FLOPs the processing unit executes per second. This is what people usually mean by throughput in MLSys, and it is what we use throughout this book unless noted otherwise.

However, when optimizing a specific kernel, you will also encounter memory throughput (GB/s), which measures how fast data moves through the memory bus, introduced in Bandwidth. A byte read and a byte written count the same toward the GB/s budget, so memory throughput behaves like a pipe with a maximum flow rate, regardless of which direction the data is moving.

When we say throughput without qualification, we mean achieved throughput: how fast a specific op or job actually runs, which depends on how well it is implemented. A poorly optimized kernel, for example, executes far fewer FLOPs per second than the hardware is capable of. What the hardware is capable of is called hardware peak throughput: a hardware property fixed by the chip's design, regardless of what code runs on it, that no op can exceed. The NVIDIA H100 SXM5, for example, has a peak throughput of about 67 TFLOPS for FP32.

The following figure shows an example with illustrative achieved values for a hypothetical kernel on an H100 SXM5.

We will see later in this chapter how a hardware's two peak numbers, together with an op's arithmetic intensity, determine the boundary between memory-bound and compute-bound.