As we talked about in the previous section, FLOPs/s is the gold. We do not want to underutilize the compute capacity of our hardware because the data is slow to load from memory. Using the same analogy, we want the data to flow fast enough, so that the worker always has enough to compute. We do not want the worker to sit idle, waiting for the next batch of data to do its job. In other words, we want the jobs to be compute-bound, not memory-bound (introduced in Compute-bound and Memory-bound).

Therefore, we want a metric that describes how data-hungry a job is. Basically, we need to know how fast the data should be fed into the processing unit to keep it working at the hardware's peak throughput. And this metric is called arithmetic intensity. The actual measurement is more nuanced. It describes the memory and compute pattern of a job, where the job can be an entire machine learning model or a single op. As long as it uses memory and does some compute, we can measure it with this metric.

Arithmetic intensity measures the ratio between the amount of compute a job performs and the amount of data it moves across the memory hierarchy. If the arithmetic intensity is high, the job performs a large amount of compute relative to the small amount of data it moves in and out of memory. In such cases, the job is more likely to be compute-bound, meaning it can fully utilize the compute capacity of the hardware.

Conversely, if the arithmetic intensity is low, the job moves a large amount of data while performing relatively little compute on it. The job is likely to be memory-bound, meaning that the compute capacity is not fully utilized due to a slow feed of data.

The following figure gives an intuitive picture of arithmetic intensity.

In the next section, we derive arithmetic intensity's formal unit and compute it for a concrete example.