Ideally, we want the memory to be large enough to hold all the data we need, and fast enough to feed the data to the CPU/GPU without delay. However, in reality, it is impossible to have both at a reasonable cost. Therefore, there is always a trade-off between speed, size, and cost of memory.

So, we can only design memory to be large and slow, or small and fast, or anywhere in between. In modern computer architectures, we usually have multiple memory types. Some are large and slow, like main memory. Some are small and fast, like caches.

These different types of memory form a memory hierarchy. Whether it is a GPU or a CPU, the memory always has this hierarchical structure.

We will look at the specific memory hierarchy of a GPU in detail in the next chapter.