MLSYS ENGINEERING

11.5. JIT vs. AOT

When working with compilers, you will often hear two terms: Ahead-of-Time (AOT) and Just-in-Time (JIT). These terms describe when compilation actually takes place.

In traditional computing, an AOT compiler finishes all its work before you run the program. Languages like C++ and Rust use this approach. If a function can accept different data types, the compiler uses templates or generics, as shown in the following snippet.

Code 33. A C++ template function called with different types.
template <typename T>
T multiply(T x, T y) {
    return x * y;
}

multiply(2, 4);       // Uses int
multiply(2.5f, 4.5f); // Uses float

An AOT compiler inspects all the call sites in your source files ahead of time. Because it sees calls with both integers and floating-point numbers, it generates both compiled versions directly into the binary file before you ever launch the program. When you run the application, the computer executes the pre-compiled machine code immediately with zero compilation delay.

In contrast, a JIT compiler postpones compilation until the program is already running. It does not generate compiled code in advance. Instead, it compiles on demand: when the program executes a call with integers, the JIT compiler compiles the integer version right then and caches it in memory for future calls. Many general-purpose JIT compilers go further: they start by interpreting code and only compile functions that run frequently.

You will often see the words static and dynamic used to describe these two styles. In computing, "static" describes things that are fixed and decided before running your code, while "dynamic" describes things that happen on the fly while your code is running. For this reason, AOT compilation is often called static compilation, and JIT compilation is called dynamic compilation.

In machine learning, compilers usually operate as JIT compilers, especially during training and research. This choice is driven by how modern deep learning frameworks define and construct models.

In practice, a model architecture is rarely hard-coded into the source code. Instead, properties like the hidden layer dimensions, the number of attention heads, and the vocabulary size are defined in external configuration files, such as a config.json file. When you run your training or inference script, Python loads this JSON file from disk and builds the model layers based on those configuration values.

An ahead-of-time compiler analyzing raw Python source code cannot know the contents of that configuration file before execution. As a result, it cannot know the shapes of the weight tensors or the input and output tensor shapes for each operation in the model.

An ML compiler needs these exact tensor shapes to generate fast machine code. For example, knowing the shape of an operation's input tensor allows the compiler to choose optimal thread block tile sizes for GPU kernels.

A JIT compiler fits this workflow naturally. Tools such as torch.compile wait until the program runs, loads the configuration, and instantiates the model. When the model runs for the first time, the JIT compiler inspects the actual tensor shapes flowing through each operation. With this concrete model information available, it compiles optimized GPU kernels tailored specifically to the model's architecture.

Once the compiler finishes generating and compiling these specialized kernels, it caches them in memory. On subsequent executions, the framework runs the compiled kernels directly with zero overhead. By combining tracing, graph optimizations, and just-in-time compilation, machine learning compilers deliver hardware performance that raw, line-by-line Python execution could never achieve.