Chapter 9.
Precision

Every number in a neural network has to be stored and computed in some format. Computers represent all numbers in binary, as sequences of bits, and the format determines how many bits are used and what they mean. For example, the integer 42 stored as an 8-bit integer is 00101010 in binary. Weights, activations, and gradients are all numbers.

The choice of format affects how much memory a model uses, how fast it runs, and how accurately it computes. The core tradeoff is simple: more bits per number means more precision, but also more memory and slower compute. Fewer bits are faster and cheaper, but lose precision, which can make the model less accurate.

In this chapter, we start with how floating-point numbers represent values and handle special cases like overflow. We then survey the common data types used in ML, explore how floating-point arithmetic works, and examine how precision loss and non-associativity arise in hardware.