Inside each SM, there are multiple CUDA cores. A CUDA core is similar to a CPU core but simpler and smaller, which is what allows a GPU to have so many of them. Each CUDA core executes one thread at a time. In the H100, each SM has 128 CUDA cores, as shown below.

The 128 CUDA cores in an SM do not run 128 independent threads. Each truly independent thread would need its own logic circuit, which is expensive in chip area. Instead, threads are grouped into warps of 32, and all 32 threads in a warp share the same control logic, executing the same instruction at the same time. With 128 cores and 32 threads per warp, an SM runs 4 warps simultaneously at any given clock cycle.

However, an SM can hold far more warps at once and juggle between them. When a warp stalls waiting for data from memory, the SM instantly switches to another ready warp at no cost.

Each SM also has its own fast memory called shared memory. It is much faster than global memory but also much smaller, and only the threads in the block assigned to that SM can access it.

At the finest level, each thread has its own private storage called registers. Registers hold the local variables a thread is actively working with. They are the fastest memory on the GPU, with essentially no wait time.