The tiling strategy we introduced in the previous section still depends on the number of columns of matrix A and the number of rows of matrix B, since we are loading entire rows of A and entire columns of B into shared memory at once. For a large matrix whose rows are too long to fit into shared memory, we cannot even perform the tiling.

There is an additional constraint. To overlap computation with loading using pipelining, as introduced in Pipelining, we need to fit two sets of tiles in shared memory simultaneously: one being computed on while the other is loading. This makes the size constraint even tighter.

Even when the rows do fit, the shared memory may not be fully used. If we can fit 1.7 rows' worth of data, we can only load 1 row, leaving the remaining space empty. The following figure illustrates the difference between high and low shared memory utilization.

This is called low shared memory utilization, a negative signal in performance tuning. Low utilization hurts in two ways.

First, loading less data per batch means more batches, and each batch pays a fixed overhead cost regardless of how much data it transfers. This overhead includes setting up the memory address and waiting for the data to be ready, as introduced in Latency. The following figure shows how loading the same total data in two batches costs more time than loading it in one.

Second, smaller batches give less compute time to overlap with each load. This makes it harder to hide the loading latency with pipelining.

Note that in both cases, the total data transferred is the same, so the arithmetic intensity is identical. Yet the low-utilization case is slower. This shows that arithmetic intensity alone does not fully predict performance: two strategies with equal arithmetic intensity can differ in speed, depending on the number of batches and the opportunity to hide loading latency with pipelining.

To avoid this, we want tile sizes that fill shared memory well regardless of how long the matrix rows are. This calls for a more flexible tiling strategy where tile size is a free choice, independent of the matrix dimensions, so that we can handle matrices of any shape.