11.4. Traditional vs. ML compilers
Now we can see how the stages of an ML compiler compare to those of a traditional compiler.
The frontend is very different. A traditional compiler parses text source files such as .cc and .h into an intermediate representation. In contrast, an ML compiler typically uses tracing to build a computation graph as its intermediate representation.
The optimizations are very similar. In an ML compiler, the optimizer inspects the computation graph, finds that matmul connects directly to add, and fuses them into a single addmm operation. Similarly, in Code 27, the optimizer swapped mul with shl. In both examples, the optimization relies on pattern matching: finding specific sequences in the intermediate representation and replacing them with faster ones.
The backend is more nuanced. A traditional compiler translates its intermediate representation into CPU machine instructions. An ML compiler does something similar: it generates Triton or CUDA kernel code from its computation graph, and compiles that code to produce the GPU kernel. However, writing new code is not its only option. There are already a lot of pre-compiled kernels shipped with the framework. Because the ML compiler integrates directly with the deep learning framework, it can skip generating new code and dispatch directly to an existing pre-compiled kernel, such as a vendor library like cuBLAS.
The following figure summarizes how each stage of a traditional compiler compares to an ML compiler.