Floating-Point Arithmetic: Why Computers Make Rounding Errors

Anyone who has opened a browser console and typed 0.1 + 0.2 has seen the infamous result: 0.30000000000000004. Far from being a software bug, this is a direct consequence of how hardware implements binary floating-point numbers under the IEEE-754 standard — the same standard used by JavaScript, Python, Java, and essentially every programming language.

The Binary Fraction Problem

Just as 1/3 cannot be written finitely in base-10 (it becomes 0.3333...), fractions whose denominators contain prime factors other than 2 cannot be written finitely in base-2. In binary, the decimal value 0.1 becomes an infinite repeating sequence:

The computer must truncate this sequence at a fixed bit boundary, introducing a rounding error of roughly 10^-17 before the calculation even starts. Add 0.2 (which has its own tiny error), and the sum lands just above 0.3 — printing as 0.30000000000000004.

The IEEE-754 Double-Precision Format

In standard 64-bit floating point, every number is packed into three bitfields:

• 1 sign bit — positive or negative
• 11 exponent bits — the scale (roughly 10^-308 to 10³⁰⁸)
• 52 mantissa (fraction) bits — about 15 to 17 significant decimal digits

The implied leading 1 means only the fraction is stored, and the 53rd significand bit position is where rounding decisions occur. Two numbers that are distinct in decimal can map to the same 64-bit pattern — for example, any dollar amount above $2⁵³ (~$9 quadrillion) can no longer be represented exactly, which is why financial systems track money in integer cents rather than floating-point dollars.

Where the Errors Come From

Three operations introduce error, and they compound:

1. Representation error — the decimal input is not exactly representable in binary (the 0.1 problem).
2. Cancellation error — subtracting nearly equal numbers discards significant digits (e.g., computing small differences in large coordinates).
3. Accumulation error — summing millions of values in a loop drifts unless the order is controlled.

How Calculators Compensate

Production calculators avoid displaying raw binary artifacts using three techniques:

• Epsilon comparisons — Math.abs(a - b) < Number.EPSILON (2.22 × 10^-16) instead of ===.
• Decimal rounding at display time — results are rounded to a sensible precision (2 decimals for currency, 6-10 for scientific work) exactly when shown, never stored pre-rounded for further math.
• Kahan summation — when adding long series (amortization schedules, statistics), a compensation term tracks the lost low-order bits so cumulative error stays near machine epsilon.

Calculator Archive applies epsilon comparisons plus decimal normalization so results render as clean mathematical values — 0.3 shows as 0.3, and currency rows total exactly to the cent.