The roofline model is a tool for setting a performance ceiling on any op. Given an op's arithmetic intensity and the hardware's two peak throughput numbers (memory bandwidth and compute throughput), the model tells you the maximum achievable performance for that op on that hardware. No matter how well you optimize the implementation, you cannot exceed this ceiling.

The model is shown in the following figure. It is a 2D plot where the x-axis is arithmetic intensity (FLOPs/byte) and the y-axis is achievable performance (FLOPs/s). One roofline is associated with one piece of hardware, for example, the NVIDIA H100 GPU. Every op can be represented as a point on this plot based on its arithmetic intensity and achieved throughput. The point always falls at or below the roofline; a point above it would require performance beyond what the hardware can deliver.

The roofline itself is shaped like a roof: a slanted line on the left and a flat line on the right, meeting at the ridge point. The slanted line represents the memory bandwidth limit: performance scales with arithmetic intensity because doing more compute per byte makes better use of the memory bandwidth available. The flat line represents the peak compute limit: no matter how high the arithmetic intensity, the hardware cannot execute FLOPs faster than its peak. The ridge point is where the two limits are perfectly balanced. On the H100 SXM5, this works out to about 20 FLOPs/byte, as derived in Throughput.

The roofline answers three questions for any op:

How much room is there to improve? A point deep below the roofline means the implementation is far from the hardware ceiling, and basic optimizations like parallelization and pipelining may help. A point close to the roofline means the implementation is near-optimal and further gains will be hard to come by.

What kind of optimization will help? If the op is to the left of the ridge point (memory-bound), increasing arithmetic intensity moves the ceiling up, so reducing memory accesses pays off. If it is to the right (compute-bound, as illustrated in Figure 18), the ceiling is already flat, so increasing arithmetic intensity changes nothing, and the focus should shift to reducing instruction latency or improving parallelism.

What is the performance target? Before writing any code, you can compute the arithmetic intensity of your algorithm, look up the hardware specs, and immediately know the best performance you could ever achieve. This gives you a concrete target to optimize toward.

With these tools in hand, we can now diagnose and reason about the performance of any op. In the next chapter, we will apply them to optimize matrix multiplication.