Throughput is the rate at which a system processes work. The key word is processes: it measures what happens inside, not what goes in or comes out. Think of a factory: the production rate tells you how many units the factory completes per hour, regardless of how many raw materials arrive or how many boxes ship out. Similarly, hardware throughput tells you how fast the hardware can process, independent of the specific data flowing through.

This distinction matters because memory and compute throughput measure fundamentally different things:

Memory throughput (GB/s) measures how fast data moves through the memory bus, introduced in Bandwidth. Here the input and output are the same data (bytes transferred in, bytes transferred out), so it behaves like a pipe with a maximum flow rate.

Compute throughput (FLOPs/s) measures how fast the processing units execute floating-point operations. This is the transformation rate: how many multiplications and additions the hardware can perform per second, independent of how many bytes go in or come out.

Both come in two flavors. Peak throughput is a hardware property: the ceiling no op can exceed. The NVIDIA H100 SXM5 has a peak compute throughput of about 67 TFLOPS for FP32 and a memory bandwidth of about 3.35 TB/s. Achieved throughput is an op property: how fast a specific kernel actually runs. A kernel with redundant memory accesses will achieve far less than the memory peak. Arithmetic intensity tells us how efficiently a kernel balances the two: a higher arithmetic intensity means more compute per byte moved, pushing the bottleneck from memory toward compute.

The following figure shows an example with illustrative achieved values for a hypothetical kernel on an H100 SXM5.

We will see in Roofline model how these two numbers together determine the boundary between memory-bound and compute-bound.