5.1. The GPU
An NVIDIA GPU consists of multiple streaming multiprocessors (SMs). Each SM is an independent processor with its own fast memory that can run many threads in parallel. The computing power of a GPU comes from all these SMs working together. For example, the NVIDIA H100 SXM5 has 132 SMs, as shown below.
Figure 13. H100 GPU with 132 SMs and global memory.
The GPU also has a large pool of memory called global memory, also known as high bandwidth memory (HBM) or device memory. It is shared by all SMs and holds all the data the GPU is working with.