11.2. What is an ML compiler?
Unlike traditional compilers that optimize general-purpose programming languages, an ML compiler is specialized to optimize neural networks.
The way we use an ML compiler is also very different. A C++ compiler must run before you run your code, translating your source files into an executable. In contrast, an ML compiler is typically triggered while you run your code. Wrapping a function with torch.compile prepares it for compilation, and calling the function for the first time actually triggers the compiler to generate the optimized code.
The following code shows a concrete example using torch.compile to optimize a linear layer written with matrix multiplication and addition.
import torch
def linear_layer(x, w, b):
return x @ w + b
# Compile the function
compiled_linear = torch.compile(linear_layer)
# Run the compiled function
output = compiled_linear(
torch.randn(1024, 1024, device="cuda"),
torch.randn(1024, 1024, device="cuda"),
torch.randn(1024, device="cuda"),
)
Just like the C example in Code 27, let's see how our linear layer transforms through each stage of the ML compiler pipeline to produce the compiled function.
// 1. Model code
y = x @ w + b
// 2. Frontend output
z = matmul(x, w)
y = add(z, b)
// 3. Optimizer output
y = addmm(b, x, w)
// 4. Backend output
for each tile of y:
y_tile = b_tile + matmul(x_tile, w_tile)
The kernel we wrote in Matmul GPU Kernel also used torch.addmm to perform matrix multiplication and accumulation in a single call. Here, the ML compiler uses the same strategy to fuse the bias addition into the matrix multiplication: even though we defined the linear layer as two separate operations (x @ w and + b), the compiler combines them into a single kernel. This avoids the overhead of writing intermediate matrices to slow global memory, keeping all intermediate data in fast on-chip memory.
The snippet above demonstrates how an ML compiler follows the exact same modular design introduced in What is a compiler?. It is decoupled into distinct, sequential stages: a frontend that translates high-level code into an intermediate representation, an optimizer that rewrites the IR into a faster version, and a backend that translates the optimized IR into executable kernel code.
Because it adopts this modular architecture, modern ML compilers can directly use traditional programming language compiler frameworks like LLVM. Instead of building code generation and machine-level optimizations from scratch, an ML compiler can optimize high-level neural network operations and then hand the resulting IR off to LLVM to generate fast machine instructions for the target hardware.