Now that we have a good understanding of the decimal format as a mental model, we are ready to see how the binary format works. This is the true format used by the computer. It is almost the same as the decimal format, but with a few optimizations. From now on, we will use the subscript 2 for all binary numbers to avoid confusion between binary and decimal. For example, writing the decimal 2 in the binary format would be 10₂.

We will use BF16 (short for bfloat16), one of the most widely used formats for training large models, as an example before we generalize to other formats. It has 16 bits in total. It also consists of three parts: sign, exponent, and mantissa. It roughly looks like the figure below. Let's see how each part works in the binary format.

The sign is a single bit, where 0 means positive, 1 means negative. It is very similar to the decimal format in the previous section.

The exponent has 8 bits. 8 bits can represent 2⁸ different numbers, ranging from 0 (00000000₂) to 255 (11111111₂). Similar to the decimal example, we shift this range of 0 to 255 left to cover negative values by subtracting 127 from the stored number. However, two extreme values are reserved: 0 is reserved to represent zero (and some extremely tiny numbers), and 255 is reserved to represent infinity (inf) and NaN (not a number). So, normal exponents use the stored numbers from 1 to 254. After subtracting 127, the actual exponent ranges from -126 (1 - 127) to 127 (254 - 127).

The mantissa has 7 bits. 7 bits can represent 2⁷ different numbers, ranging from 0 (0000000₂) to 127 (1111111₂). In the mantissa representation, we use a smart trick that does not apply to the decimal format. Let's see how it works.

For a decimal mantissa in the scientific representation, the leading digit can be anything from 1 to 9. For example, the leading digit of 4.7 is 4. It cannot be 0 though since 0.47 × 10^-2 is obviously an incorrect way to write it in scientific representation. The correct way to write it is 4.7 × 10^-3. Similarly, in the binary format scientific representation, the leading digit cannot be 0 either. For example, we should never write 0.11₂ × 2³ but 1.1₂ × 2². For any binary digit, it can only be either 0₂ or 1₂. So, now, the leading digit of the mantissa can only be 1₂.

Since it is always led by 1₂, we do not need to waste a bit on it. We only need to store the remaining bits of the mantissa in the binary format. For example, to represent 1.1000001₂, we can store it as 1000001₂. In this way, we can represent an 8-bit mantissa with only 7 bits. The figure below shows a few more examples of this.

Another important difference is the base of the exponent is 2 instead of 10. So, 4.7× 10^-2 is converted to some number × 2^-5. Since the multiplier changed from 1/100 to 1/32, the mantissa would not be the same either. And we use the binary digits to try our best to represent that number.

Finally, what happens if a number is too large to fit in this range? In BF16, a stored exponent of 255 (all exponent bits set to 1) with all zero mantissa bits is reserved to represent infinity (inf), indicating that a value has overflowed the maximum representable magnitude.

Setting the sign bit gives negative infinity (-inf), and a nonzero mantissa gives NaN. Not every format keeps these special values. Some 8-bit and 4-bit formats give up inf, or even NaN, to have more bit patterns for ordinary numbers. These rules come from the IEEE 754 standard, which BF16 follows even though it is not officially part of the standard.