Chapter 7.
Optimizing matmul

In the previous chapters, we implemented a naive matmul and then parallelized it across CPU threads, as covered in Multi-threading. That implementation runs on the CPU with main memory, where the hardware manages the memory hierarchy for us. Now that we understand how the GPU is organized and how its memory hierarchy works, we are ready to bring matmul to the GPU.

On the GPU, performance depends on how data moves between global memory and shared memory. In this chapter, we apply the tools from the previous chapter to understand how to control that, and gradually improve the memory access pattern to increase arithmetic intensity. We finish by looking at what a simple GPU kernel implementation looks like.