GPU memory hierarchy

MLSYS ENGINEERING

5.4. GPU memory hierarchy

Each level of the hardware hierarchy has a corresponding memory. Together they form the GPU memory hierarchy.

Figure 15. GPU memory hierarchy.

The table below shows the sizes, scope, bandwidth, and latency of each level.

Table 1. GPU memory hierarchy (H100)

Memory Type	Scope	Size	Bandwidth	Latency
Global Memory	Public to all	80 GB	3 TB/s	400 cycles
Shared Memory	Private to block	228 KB per SM	33 TB/s	20 cycles
Registers	Private to thread	255 per thread	33 TB/s	1 cycle

The GPU also has an L2 cache between global memory and the SMs. It is not directly visible to the programmer, but it can significantly affect performance by caching frequently accessed global memory data.

Given the 10x difference in bandwidth and 20x difference in latency between global memory and shared memory, we often need to reduce the data transfer between them to avoid hitting the memory wall. When transferring data between global and shared memory, global memory bandwidth is the bottleneck.

The 20x difference in latency between shared memory and registers makes registers perfect for storing local variables, which are usually scalar values used repeatedly in a computation.

With this picture of the GPU in mind, we have everything we need to talk about performance. In the next chapter, we will look at how to measure it and where the bottlenecks come from.