CSC108H1/Unit 10
Unit 10 · Intro to Programming

Floating Point & Numerical Computation

The unit that explains why 0.1 + 0.2 is not 0.3. Floats are not real numbers — they are a finite approximation, and once you know the shape of the approximation, the surprises stop being surprises.

By the end you can
  • Explain why some decimals cannot be stored exactly.
  • Compare floats correctly with a tolerance.
  • Recognise catastrophic cancellation and avoid it.
  • Choose between float, int and Decimal.
Why it matters & where it comes from
Why it exists

A float has $64$ bits and there are infinitely many real numbers between any two, so almost every real must be rounded to the nearest storable value. Every arithmetic result is rounded again, and the errors accumulate.

In the real world

A Patriot missile battery failed in 1991 after $100$ hours of uptime: its clock accumulated a tiny rounding error per tick, drifting by a third of a second and missing an incoming target. Financial systems avoid floats entirely for money for the same reason.

Story & open questions

The IEEE 754 standard of 1985, led by William Kahan, made floating point behave identically across machines — before it, the same program gave different answers on different hardware. Kahan won the Turing Award for it.

Why 0.1 is not 0.1

Rule Binary fractions

A float stores a binary fraction: sums of $1/2$, $1/4$, $1/8$ and so on. A decimal is exactly representable only when its value is such a sum.

$0.5$ is $1/2$ — exact.
$0.25$ is $1/4$ — exact.
$0.1$ is $1/10$, and $10$ has a factor of $5$, which is not a power of two. In binary it repeats forever, exactly as $1/3$ does in decimal, and must be cut off.
Example See the stored value
python
0.1 + 0.2            # 0.30000000000000004
0.1 + 0.2 == 0.3     # False

format(0.1, '.20f')  # '0.10000000000000000555'

Python usually prints the shortest string that round-trips, which is why 0.1 displays as 0.1. Asking for twenty decimal places shows what is really stored.

Never compare floats with ==
python
# wrong
if total == 0.3:
    ...

# right - compare within a tolerance
if abs(total - 0.3) < 1e-9:
    ...

The same rule made assertAlmostEqual necessary in Unit 8, and it is why a while loop testing a float for equality can run forever.

Accumulated error

Example Adding a tenth, ten times
python
total = 0.0
for _ in range(10):
    total = total + 0.1

total          # 0.9999999999999999
total == 1.0   # False

Each addition rounds, and ten roundings do not cancel. Where possible, count in integers and divide once at the end: total = count / 10.

Rule Catastrophic cancellation

Subtracting two nearly equal floats destroys precision. The leading digits agree and cancel, leaving only the noisy trailing digits, which are then scaled up by whatever comes next.

Example Losing everything at once
python
a = 1.0000000001
b = 1.0000000000
(a - b) * 1e10       # about 1.000000082740371, not 1.0
Both inputs carry about sixteen good digits, but the first ten cancel, so the difference has only about six left.
Multiplying magnifies the remaining error. Rearranging a formula to avoid the subtraction is the standard fix — the quadratic formula is the textbook case.
Example Order of operations matters
python
1e16 + 1 - 1e16      # 0.0   - the 1 was lost to rounding
1e16 - 1e16 + 1      # 1.0   - same operations, different order

Floating point addition is not associative. Adding a tiny number to a huge one can change nothing at all, because the result rounds back to the huge number.

Choosing a numeric type

Rule Which type when
int — counts, indices, and money in cents. Python integers are exact and unbounded, so there is no reason to avoid them.
float — measurements, averages, anything physical. Approximation is acceptable because the input was approximate too.
Decimal — money and anything where decimal rounding rules are specified. It stores base-ten digits, so 0.1 is exact.
Example Money three ways
python
# wrong: cents go missing over many transactions
total = 0.0
total = total + 0.10

# simple and exact: work in cents as integers
total_cents = 0
total_cents = total_cents + 10

# exact with decimal points, when the format matters
from decimal import Decimal
total = Decimal('0.00') + Decimal('0.10')   # Decimal('0.10')

Note Decimal('0.1') takes a string. Writing Decimal(0.1) passes an already-imprecise float and preserves the error you were trying to avoid.

Example Rounding is not what you expect
python
round(2.5)     # 2  - banker's rounding, ties go to even
round(3.5)     # 4
round(2.675, 2)# 2.67 - because 2.675 is really 2.67499999...

Ties rounding to even is deliberate: it avoids the upward bias of always rounding half up when summing many values.

Common pitfalls

• Comparing floats with ==.
• Using floats for money.
• A while loop testing a float for exact equality — step past the target and it never ends.
• Assuming round rounds half up.
• Writing Decimal(0.1) rather than Decimal('0.1').

Practice

Predict each answer before running it. Original exercises written for this guide in the style of the course's weekly sets.

Practice 1 · Exact or not

Which of $0.5$, $0.25$, $0.2$, $0.125$ are exactly representable?

Answer
$0.5$, $0.25$ and $0.125$ — all powers of two. $0.2$ is $1/5$ and is not.
Practice 2 · Safe comparison

Rewrite if x == 0.3: safely.

Answer
if abs(x - 0.3) < 1e-9:. Choose the tolerance to suit the magnitudes involved.
Practice 3 · Infinite loop

Why might this never stop?

python
x = 0.0
while x != 1.0:
    x = x + 0.1
Answer
After ten additions x is 0.9999999999999999 and steps straight past 1.0. Use while x < 1.0:, or better, loop over range(10) with integers.
Practice 4 · Money

Why is 0.1 + 0.2 == 0.3 a problem for a shop?

Answer
Totals drift by fractions of a cent and stop reconciling. Store cents as integers, or use Decimal.
Practice 5 · Cancellation

Why does (1e16 + 1) - 1e16 give 0.0?

Answer
At that magnitude the gap between consecutive representable floats exceeds $1$, so adding 1 rounds straight back to 1e16. The information is gone before the subtraction happens.
Practice 6 · Averaging safely

How should you average a long list of floats?

Answer
Sum then divide once, rather than accumulating a running mean. For very long or wide-ranging lists, math.fsum tracks the error and gives a correctly rounded total.
Practice 7 · round

What is round(0.5) and round(1.5)?

Answer
0 and 2. Ties go to the even neighbour, so half the ties round down and the bias cancels over many values.
Practice 8 · Nearly equal

Write a helper testing whether two floats are close.

Solution
python
def close(a: float, b: float, tol: float = 1e-9) -> bool:
    """Return whether a and b agree to within tol.

    >>> close(0.1 + 0.2, 0.3)
    True
    """
    return abs(a - b) < tol
Python's own math.isclose does this with a relative tolerance, which behaves better across very different magnitudes.

Expanded topic checklist

Course coverage. This unit is part of the complete CSC108H1 topic map. It should cover the following items before you move on:

  • Representation. Binary fractions and why some decimals repeat forever.
  • Comparison. Tolerance instead of equality.
  • Error growth. Accumulation, cancellation, and the failure of associativity.
  • Type choice. int, float and Decimal, and money in particular.
Textbook/source alignment

The calendar names "floating-point numbers and numerical computation" as the final topic. It is the one most often skipped by students and most often examined by a tracing question.

Core idea map

Finite bits force rounding; rounding accumulates; subtraction of near-equal values magnifies what is left. Every rule in this unit follows from those three sentences.

Study routine. Whenever a float appears in a condition, stop. Either compare with a tolerance or switch to integers. That single habit prevents most numerical bugs you will meet in this course.

Deep practice extensions

Use these after the main practice set. They push on definitions and edge cases rather than routine calculation.

Extension 1 · Why 0.1 is inexact

Explain in one sentence.

Guide
$1/10$ is not a finite sum of powers of two, so its binary expansion repeats forever and has to be cut off.
Extension 2 · Comparison

How should floats be compared?

Guide
Within a tolerance: abs(a - b) < tol, or math.isclose for a relative test.
Extension 3 · Cancellation

When does subtraction lose precision?

Guide
When the two values are nearly equal. Their agreeing leading digits cancel and only the noisy tail survives.
Extension 4 · Money

What should represent currency?

Guide
Integer cents, or Decimal constructed from a string. Never a float.
← Unit 09Searching, sorting & complexity Back to IndexAll chapters