Now, let's see some examples. In Code 17 (inefficient example, introduced in Op fusion), reproduced below, we have 40 data accesses (20 reads and 20 writes) and 20 compute operations (10 doublings and 10 additions). The two loops each touch all 10 elements, so each element is read and written twice. So, the compute per data access is 0.5 = (20 compute ops / 40 memory accesses).

for i in range(len(data)):
    val = read_from_memory(i)
    write_to_memory(i, val * 2)

for i in range(len(data)):
    val = read_from_memory(i)
    write_to_memory(i, val + 1)

In Code 18 (efficient example, introduced in Op fusion), reproduced below, we have 20 data accesses (10 reads and 10 writes) and 20 compute operations (10 doublings and 10 additions). Each element is read and written exactly once. So, the compute per data access is 1 = (20 compute ops / 20 memory accesses).

for i in range(len(data)):
    val = read_from_memory(i)
    val = val * 2
    val = val + 1
    write_to_memory(i, val)

Actually, the above are over-simplified examples. The numbers are not meaningful without proper units. In practice, we need to measure data access and compute in unified metrics that are comparable across different ops and hardware.

Most commonly, we use bytes to measure how much data an op accesses. For compute, we use FLOPs (floating-point operations), where a multiply or an addition each counts as one FLOP. So, the unit for arithmetic intensity is FLOPs/byte.

We can formally write it as:

In Code 18, each memory access is actually reading/writing a floating-point number. Each floating-point number takes 32 bits (4 bytes) in memory. A multiply and an addition are two FLOPs. So, the actual arithmetic intensity for Code 18 is not 1 = (20 compute ops / 20 memory accesses), but 0.25 FLOPs/byte = (20 FLOPs / 80 bytes) = (20 compute ops × 1 FLOP/op) / (20 memory accesses × 4 bytes/memory access).

The same correction applies to Code 17: its naive ratio of 0.5 becomes 0.125 FLOPs/byte (20 FLOPs / 160 bytes), since it makes twice as many memory accesses for the same amount of compute. Fusing the two loops into one, as Code 18 does, doubles the arithmetic intensity from 0.125 to 0.25 FLOPs/byte by halving the number of memory accesses.

Beyond reducing the number of memory accesses through fusion, we can also reduce the number of bytes each access moves. For example, using 16-bit (2 bytes) floating-point numbers instead of 32-bit (4 bytes) ones cuts the bytes per access in half, doubling the arithmetic intensity again. We will dive deeper into how different data types work later in the book.