Bandwidth does not tell the whole story of memory speed. We are not always transferring data at the full capacity of the memory bandwidth. If the memory bandwidth is 10 GB/s, it does not mean we can get 1 byte of data in 0.1 nanoseconds (1 byte / 10 GB * 1 second = 0.1 nanoseconds). This is because of the delay before the data starts to be transferred.

That is why we need another metric, latency, to measure the delay of memory access. Intuitively, latency is similar to the concept of "time to first byte". We request a very small amount of data from memory, and measure how long it takes before we get the data back. We can also formally define memory latency as the delay between when a processor requests data and when that data is actually available for use.

Latency is the total time across all three stages. The dominant part is the "Wait" stage, where the processor sits idle while the memory looks up the address, prepares the data, and gets ready to send it. Once the wait is over, the data travels as an electrical signal at close to the speed of light. The distances inside a computer are only a few centimeters, so it arrives in a fraction of a nanosecond. This is why latency feels like a fixed delay: no matter how small the data is, you still pay that waiting cost upfront.

Unlike bandwidth, latency is measured in cycles or time units like nanoseconds. You can convert between the two if you know the clock frequency. For example, a memory with 200 cycles of latency at 3.5 GHz has a latency of about 200 cycles / (3.5 billion cycles / 1 sec) = 57 nanoseconds.

Bandwidth and latency are independent of each other. A memory can have high bandwidth but also high latency.

With capacity, bandwidth, and latency in hand, we now have the vocabulary to reason about memory performance precisely. In the next chapter, we will look at how a GPU organizes its memory into a hierarchy, and see what the bandwidth and latency actually look like at each level.