When adding a small number to a big one, the small number can be lost completely. For example, in our two-digit mantissa format, consider adding 100 and 4:

100 + 4 = 1.04 × 10² ≈ 1.0 × 10² = 100

Because our format only stores two mantissa digits, 1.04 rounds back down to 1.0. The small number is completely lost. This effect is known as swamping.

This kind of total loss does not happen in multiplication, as long as the result stays within range. Multiplying 100 and 4 gives (1.0 × 10²) × (4.0 × 10⁰) = (1.0 × 4.0) × 10²⁺⁰ = 4.0 × 10² = 400. Because the hardware multiplies mantissas and adds exponents directly without aligning them first, the small number is never ignored; at most, there is only a tiny rounding error in the last digit of the product.

Another key difference is how operations are chained. Inside a matrix multiplication, multiplications are not chained in long sequences like a × b × c × d; numbers are multiplied only in isolated pairs (a_i × b_i). Additions, by contrast, are chained into a long running total: (a₁ b₁) + (a₂ b₂) + (a₃ b₃) + .... As this sum grows, additions repeat thousands of times in a row, causing rounding errors and swamping to accumulate with every step. This makes addition far more sensitive to low precision than multiplication.

The common solution is to multiply in lower precision and accumulate in higher precision. For example, machine learning accelerators often take inputs in BF16 and accumulate the running sum in FP32, greatly reducing swamping and rounding errors as partial products are added together. The product of two BF16 numbers always fits exactly in FP32, so no precision is lost in the multiplication. We can see this in how a dot product is implemented:

import torch

def dot_product(a, b):
    total = torch.tensor(0.0, dtype=torch.float32)  # FP32 accumulator
    for x, y in zip(a, b):
        prod = x.float() * y.float()                # BF16 inputs, exact product
        total += prod                               # Accumulated in FP32
    return total

a = torch.tensor([1.0, 2.0, 3.0], dtype=torch.bfloat16)
b = torch.tensor([0.5, 1.5, 2.5], dtype=torch.bfloat16)
result = dot_product(a, b)

Swamping also causes a surprising result when adding multiple numbers together. For example, consider adding 100, 4, and 4 in two different orders:

If we add the numbers from left to right as (100 + 4) + 4, the first addition 100 + 4 = 104 rounds back down to 100. Adding the second 4 also rounds down to 100, swamping both small numbers completely. But if we add the two small numbers first as 100 + (4 + 4), their sum 8 is large enough to survive when added to 100, rounding 1.08 × 10² up to 1.1 × 10² (110).

Because the results differ depending on which pair is added first, floating-point addition is non-associative: (a + b) + c is not always equal to a + (b + c).

This non-associativity is also why running the same model on the same inputs can produce slightly different results in different runs. Some GPU kernels split a large sum across many cores and add each core's partial sum to the total as soon as that core finishes. Which core finishes first changes from run to run, so the addition order changes, and so does the result. Changing the batch size can also change the result, because the GPU may pick a different kernel that adds in a different order. To get the same result every time, frameworks provide special kernels that always add in the same order.

Floating-point representations allow modern hardware to cover an enormous range of numbers using only a handful of bits. But as we have seen, this efficiency comes with compromises. Hardware must constantly balance memory footprint and compute speed against rounding errors, swamping, and non-determinism. Understanding these trade-offs is essential for training deep neural networks stably and pushing models into lower precisions like FP8 and FP4 without breaking convergence.