Every time we run an op on a tensor, we first need to copy the data from global memory to shared memory. Then, we perform the computation using the data in shared memory. Finally, we write the result back to global memory. For simplicity, we call these three steps read, compute, and write respectively.

Imagine we have three such ops to run. Each of the steps, like read, compute, and write, takes exactly 1 second. If we run them one by one, it will take 3 ops * 3 steps * 1 second = 9 seconds in total.

There is clearly room for optimization. There is a lot of idle time for the GPU processors. They are only busy during the compute step. During the read and write steps, they are idle. It is the same for the load unit, which is responsible for the read step, copying data from global memory to shared memory, and the store unit, which is responsible for the write step, copying data from shared memory back to global memory.

We can use a technique called pipelining to improve the utilization of all these units to reduce idle time. The idea is very simple. When the first op is computing, we start the read step of the second op in parallel instead of waiting for the first op to finish.

The read, compute, and write units are like three workers in a factory assembly pipeline. The ops are the products being assembled flowing down the conveyor belt. At any given time, each worker should be busy working on one of the ops. To simplify the mental model, one worker cannot work on multiple ops at the same time.

The following figure illustrates the pipelining process. It shows how the read, compute, and write steps of three ops are overlapped. The total time is reduced to 5 seconds. It also shows that the three workers are all busy most of the time.

This technique is also known as overlapping computation with memory access and hiding memory latency. GPUs also do this at the hardware level: when a warp stalls waiting for data, the SM instantly switches to another ready warp. The programmer does not need to do anything. The hardware juggles the warps automatically. We introduced warps and SMs in The streaming multiprocessor.