To objectively measure how well we optimize the memory access, we need a metric. One simple metric is to count how many compute operations we do per data access. This metric is known as arithmetic intensity. The more ops we perform for a single data access, the higher the arithmetic intensity is.

For example, in Code 18 (inefficient example), we have 40 data accesses (20 reads and 20 writes) and 20 compute operations (10 doublings and 10 additions). The two loops each touch all 10 elements, so each element is read and written twice. So, the compute per data access is 0.5 = (20 compute ops / 40 memory accesses).

In Code 19 (efficient example), we have 20 data accesses (10 reads and 10 writes) and 20 compute operations (10 doublings and 10 additions). Each element is read and written exactly once. So, the compute per data access is 1 = (20 compute ops / 20 memory accesses).

Actually, the above are over-simplified examples. The numbers are not meaningful without proper units. In practice, we need to measure data access and compute in unified metrics that are comparable across different ops and hardware.

Most commonly, we use bytes to measure how much data an op accesses. For compute, we use FLOPs (floating-point operations), where a multiply or an addition each counts as one FLOP. So, the unit for arithmetic intensity is FLOPs/byte.

We can formally write it as:

In Code 19, each memory access is actually reading/writing a floating point number. Each floating point number takes 32 bits (4 bytes) in memory. A multiply and an addition are two FLOPs. So, the actual arithmetic intensity for Code 19 is not 1 = (20 compute ops / 20 memory accesses), but 0.25 FLOPs/byte = (20 FLOPs / 80 bytes) = (20 compute ops × 1 FLOP/op) / (20 memory accesses × 4 bytes/memory access).

To increase the arithmetic intensity, we can reduce the number of memory accesses like in Code 19, or we can even reduce the number of bytes for each memory access. For example, we can use 16-bit (2 bytes) floating point numbers instead of 32-bit (4 bytes) floating point numbers to reduce the number of bytes for each memory access by half, which increases the arithmetic intensity by a factor of 2. We will dive more into how the different data types work later in the book.