The roofline model shows whether an implementation is close to the hardware limit or suboptimal. When a point falls well below the ceiling, we know there is room for improvement, but the model does not tell us what is causing the slowdown or where to focus our efforts. To find what is holding the performance back and where the code can be potentially further optimized, we use profiling.

Profiling is the process of measuring what the software and hardware actually do while a program runs. The most common form of profiling tracks how much time each function, operation, or process takes to execute. Consider the following Python program:

import time

def load_data():
    time.sleep(0.5)

def compute():
    time.sleep(0.2)

def step():
    load_data()
    compute()

def main():
    for _ in range(3):
        step()

main()

When we run a profiler on this program, it records the start and end time of every function call and visualizes them on a timeline called a trace, as shown in the following figure. In a trace, the horizontal axis represents time, and each row shows when a function is active. The width of each rectangle indicates how long that function call takes, and nested calls appear on rows below their caller.

In this trace, each step takes 0.7s because data loading (0.5s) and computation (0.2s) do not overlap at all. Over three steps, the program takes 2.1s.

We can overlap data loading and computation using pipelining, as introduced in Pipelining. By running the next step's load_data in parallel with computation instead of waiting for each step to finish, the total time drops to 1.7s, as shown in the following figure.

Even with pipelining, the trace reveals that the program is memory-bound, as introduced in Memory-bound: load_data runs back-to-back without pause, while compute finishes in 0.2s and sits idle for 0.3s every step waiting for the next batch of data. To eliminate this bottleneck, we can apply op fusion, introduced in Op fusion, to increase the arithmetic intensity. Reducing load_data from 0.5s to 0.2s balances data loading and computation, bringing the total time down to 0.8s, as shown in the following figure.

We can also view these two optimization steps on the roofline plot, as shown in the following figure. The initial sequential program starts at the bottom dot, well below the slanted memory-bound roofline, because neither memory nor compute is kept busy. Pipelining does not change the amount of data loaded or the amount of computation performed, so the arithmetic intensity stays the same. Instead, it moves the program straight up toward the slanted roofline: load_data now runs continuously, approaching the memory bandwidth ceiling.

Once the program is close to the slanted roofline, pipelining alone cannot push performance any higher. Op fusion reduces the amount of data transferred for the same computation, which increases the arithmetic intensity. This moves the program up and to the right along the slanted roofline toward the ridge point, where data loading and computation are balanced.

Beyond execution time, profilers can also track other metrics, such as memory usage and hardware utilization, for diagnosing out-of-memory errors and underutilized hardware.

With these tools in hand, we can now diagnose and reason about the performance of any op. In the next chapter, we will apply them to optimize matrix multiplication.