Let's see how a floating-point number is stored in a computer. Take the value -0.047, which could very well be a real neural network weight value. We will first try to convert it to an imaginary decimal format, which is a useful mental model when thinking about floating-point numbers. Then, we will show how the binary format is similar or different. To be clear, hardware used for machine learning does not store numbers in decimal. This example serves purely for educational purposes.

We first write it in scientific notation, which is -4.7 × 10^-2. Then, we extract the three fields: a sign (-), a mantissa (47), and an exponent (-2). Those are the exact 3 parts we need to write it in the decimal format. See the figure below. You may have noticed that we are using 48 to represent -2. We will get right to it.

In the figure, you can see the first part is the sign. It can be either positive or negative. This is quite straightforward.

The second part is the exponent, which has two decimal digits. It should be able to represent numbers from 0 to 99. However, as the exponent, we may want both negative and positive values, so that we can represent a wider range of numbers, as small as 1.0 × 10^-50 and as large as 9.9 × 10⁴⁹. So, we use a simple trick to translate the range from [0, 99] to [-50, 49]. In our format, we always subtract 50 from the stored number to get the actual exponent. So, 0 represents -50, and 99 represents 49. Every value in between has a corresponding value to represent. In our case of 10^-2, we want to represent -2. So, x - 50 = -2, which means x = 48. We use 48 to represent -2.

The third part is the mantissa, which is also very simple. We use 0 to 99 to represent 0.0 to 9.9.

This three-part representation has two main limitations on what numbers it can represent. First, it cannot represent any number that is out of its range. For example, it cannot represent 1.2 × 10⁶⁰ since it is out of the maximum range. Second, it cannot effectively represent numbers with higher precision. For example, there is no way to represent 4.75 × 10^-2 since we only have two digits to represent the mantissa, but we need three digits. We are running out of precision.