For example, we have two 4 by 4 matrices A and B, and we want to compute C = AB. We have 48 floating-point numbers (3 matrices * 4 rows * 4 columns) in total. Let's say the global memory can host all of them, but the shared memory can only host 24 floating-point numbers, which is half of them. Note that we are using small numbers here for simplicity. In reality, the matrices and the shared memory are much larger.

The SM can only process data that is already in shared memory. Since not everything fits at once, we need to load the data in multiple batches. The transfer between global memory and shared memory is slow, so the goal is to minimize the total traffic between them to maximize arithmetic intensity.

The general strategy is as follows. First, we slice matrices A and B into small pieces. Second, each SM picks some slices and loads them from the global memory to shared memory. Third, the SMs compute in parallel to get different slices of the resulting matrix C. Fourth, SMs write the slices of C back to global memory to get the full matrix C.

This strategy of breaking matrices into small tiles is known as tiling. The main question is how we should slice the matrices and load them to maximize the arithmetic intensity.

Let's start with a naive strategy for slicing. As shown in the following figure, we have P1 to P4 representing four different processes. They all run in parallel on different SMs. For each process, we load one row of A (4 floats) and the entire B (16 floats) to shared memory. The inner product of that row with every column of B gives one row of C (4 floats), which is stored in shared memory and written back to global memory.

For this strategy, we have exactly 24 (= 4 + 16 + 4) floats in the shared memory with each of them transferred exactly once between the global memory and the shared memory. So, there are exactly 24 floats transferred per process.