Skip to content

The law of large numbers and the central limit theorem

Probability · level 8, lesson 5 of 7

A question to start

Roll a die once: the result could be anything from 1 to 6. Roll it 1000 times and take the average: it will be very close to 3.5, almost certainly between 3.4 and 3.6. Why? Each roll is just as wild as the first one. Nothing "corrects" the die. So where does the steadiness of the average come from?

And there is a second surprise. The result of one die has a flat distribution, all six values equally likely. But the average of many dice has a bell-shaped distribution, the normal curve of lesson 3, even though nothing bell-shaped went in. Where does the bell come from?

These two facts, the law of large numbers and the central limit theorem, are the two great theorems of probability. This lesson is a first look at both, with a real proof of the first.

The idea

Averages of independent copies

Let \(X_1, X_2, \dots, X_n\) be independent random variables with the same distribution, each with mean \(\mu\) and standard deviation \(\sigma\): \(n\) rolls of a die, \(n\) measurements of the same thing, \(n\) people chosen at random. Their sample mean is

\[\bar X = \frac{X_1 + X_2 + \dots + X_n}{n}.\]

(Read \(\bar X\) as "X bar".) Using the rules from level 7:

  • the sum has mean \(n\mu\) and, since the \(X_i\) are independent, variance \(n\sigma^2\);
  • dividing by \(n\) divides the mean by \(n\) and the variance by \(n^2\).

So

\[E(\bar X) = \mu, \qquad \operatorname{Var}(\bar X) = \frac{\sigma^2}{n}, \qquad \operatorname{SD}(\bar X) = \frac{\sigma}{\sqrt n}.\]

This is the heart of everything in this lesson. The average is centred on the true mean, and its spread shrinks like \(\tfrac{1}{\sqrt n}\). For a die, \(\sigma = \sqrt{35/12} \approx 1.71\). The average of 100 rolls has standard deviation \(\tfrac{1.71}{10} = 0.171\); of 10 000 rolls, 0.0171. To make the spread 10 times smaller you need 100 times as many rolls.

A simulation

A computer rolled a die 1000 times and kept a running average. The first ten rolls were 2, 1, 5, 6, 1, 2, 2, 1, 1, 6 (average 2.7, well below 3.5). After 100 rolls the average was 3.30; after 500, 3.572; after 1000, 3.512.

1 2 3 4 5 6 3.5 1 10 100 1000 number of rolls (each mark ten times the last)

(The horizontal scale is stretched at the start so you can see the early swings: each labelled mark is ten times the one before.) The average wanders a lot at first, then settles, more and more tightly, near the dashed line at 3.5. Another run would wander differently, but it would settle near 3.5 too.

Why it settles: Chebyshev's inequality

How unlikely is a value far from the mean? Here is a bound that works for every distribution.

Chebyshev's inequality. For any random variable with mean \(\mu\) and variance \(\sigma^2\), and any \(c > 0\),

\[P\big(|X - \mu| \ge c\big) \le \frac{\sigma^2}{c^2}.\]

Proof. The variance is \(\sigma^2 = \sum (x - \mu)^2 P(x)\), a sum of terms that are never negative. Throw away all the terms with \(|x - \mu| < c\); the sum can only get smaller. In each remaining term, \((x - \mu)^2 \ge c^2\). So

\[\sigma^2 \ge \sum_{|x - \mu| \ge c} (x - \mu)^2 P(x) \ge c^2 \sum_{|x - \mu| \ge c} P(x) = c^2\, P\big(|X - \mu| \ge c\big).\]

Divide by \(c^2\). \(\square\) (The same proof works for continuous variables with integrals.)

With \(c = k\sigma\), it says at most \(\tfrac{1}{k^2}\) of the probability lies \(k\) or more standard deviations from the mean: at most \(\tfrac14\) beyond 2 standard deviations, at most \(\tfrac19\) beyond 3. For a normal distribution the true figures are much smaller (0.046 and 0.003), but Chebyshev's bound holds for any shape, which is what we need.

The law of large numbers

Apply Chebyshev to \(\bar X\), which has variance \(\tfrac{\sigma^2}{n}\). For any small distance \(\varepsilon > 0\) (Greek "epsilon"),

\[P\big(|\bar X - \mu| \ge \varepsilon\big) \le \frac{\sigma^2}{n\varepsilon^2}.\]

As \(n\) grows, the right side goes to 0. So, however small a distance \(\varepsilon\) you choose, the chance that the average is further than \(\varepsilon\) from \(\mu\) can be made as small as you like by taking \(n\) large enough. That is the (weak) law of large numbers:

\[P\big(|\bar X - \mu| \ge \varepsilon\big) \to 0 \quad \text{as } n \to \infty.\]

This is the promise made back in level 3 ("relative frequency settles down") and in level 7 ("expected value is the long-run average"), now proved. Applied to an indicator (1 if an event happens, 0 if not), \(\bar X\) is the relative frequency of the event and \(\mu\) is its probability, so relative frequencies settle down to probabilities.

What the law does not say

It does not say the die "evens out" by correcting itself. After ten rolls averaging 2.7, the die is not more likely to roll high. The early shortfall is not cancelled; it is diluted. The first 10 rolls were 8 below the expected total of 35. After 1000 rolls that same shortfall of 8, spread over 1000, moves the average by only 0.008. New rolls swamp the old ones. This is why the law of large numbers and the gambler's fallacy of level 5 do not contradict each other.

The central limit theorem

The law of large numbers says where \(\bar X\) ends up. The central limit theorem (CLT) describes the shape of its wobbles along the way:

For large \(n\), the sample mean \(\bar X\) of \(n\) independent copies of any random variable with mean \(\mu\) and standard deviation \(\sigma\) is approximately normal:

\[\bar X \approx \operatorname{N}\!\left(\mu, \frac{\sigma^2}{n}\right), \qquad X_1 + \dots + X_n \approx \operatorname{N}(n\mu, n\sigma^2).\]

The original distribution can be flat, skewed or lumpy; it does not matter. Here are the exact distributions of the average of 1, 2 and 5 dice:

1 6 average of 1 die 1 6 average of 2 dice 1 6 average of 5 dice

One die is flat; two dice make a triangle (lesson 1 of level 7); five dice already look like a bell. The proof needs tools beyond this level (it is met in university probability courses), but the pictures make it believable, and you can test it yourself by simulation.

This is why the normal curve is everywhere. A measurement error is the sum of many small independent disturbances. A binomial count is a sum of indicators (lesson 4). A person's height results from many genetic and environmental contributions. Whenever a quantity is a sum of many small independent parts, the CLT says to expect a bell.

How large must \(n\) be? For a symmetric starting distribution like a die, \(n = 5\) to 10 already looks good; for a very skewed one (like the geometric) you may need 30, 50 or more.

Worked examples

1. The average of 100 dice. Find \(P(3.3 \le \bar X \le 3.7)\).

\(E(\bar X) = 3.5\) and \(\operatorname{SD}(\bar X) = \tfrac{1.708}{10} = 0.1708\). By the CLT, \(z = \tfrac{0.2}{0.1708} \approx 1.17\), so \(P \approx \Phi(1.17) - \Phi(-1.17) \approx 0.76\).

(More precisely, the sum is a whole number from 330 to 370, so with the continuity correction of lesson 4 the bounds are 329.5 and 370.5 for a sum with standard deviation 17.08: \(z = \pm 1.20\), giving 0.770. The exact answer, from all \(6^{100}\) outcomes by computer, is 0.770.)

2. The sum of 30 dice. Chance the total is at least 120? The sum has mean \(30 \times 3.5 = 105\) and standard deviation \(\sqrt{30 \times \tfrac{35}{12}} = \sqrt{87.5} \approx 9.35\). With continuity correction, \(z = \tfrac{119.5 - 105}{9.35} \approx 1.55\), so \(P \approx 1 - 0.9394 = 0.061\). (Exact: 0.0607.)

3. Chebyshev versus the CLT. How likely is the average of 1000 die rolls to be within 0.1 of 3.5?

  • Chebyshev guarantees: \(P(|\bar X - 3.5| \ge 0.1) \le \tfrac{35/12}{1000 \times 0.01} \approx 0.29\), so at least 71% within 0.1.
  • The CLT estimates: \(\operatorname{SD}(\bar X) = \tfrac{1.708}{\sqrt{1000}} \approx 0.054\), \(z = \tfrac{0.1}{0.054} \approx 1.85\), so about 94% within 0.1.

Chebyshev is a guarantee that holds for every distribution, so it is cautious; the CLT is an approximation that uses the bell shape, so it is sharper.

4. A lift. A lift's safe limit is 1300 kg. Suppose people's masses have mean 75 kg and standard deviation 12 kg (no need to assume a normal shape). Sixteen people get in. Their total has mean \(16 \times 75 = 1200\) kg and standard deviation \(\sqrt{16} \times 12 = 48\) kg, and by the CLT it is roughly normal. \(P(\text{total} > 1300) \approx P(Z > 2.08) \approx 0.019\), about 1 in 50.

Try it

  1. A random variable has \(\mu = 10\) and \(\sigma = 4\). Find the mean and standard deviation of the average of 25 independent copies, and of 400 copies.
  2. Use Chebyshev to give a bound on the chance that any random variable is 3 or more standard deviations from its mean. What is the chance for a normal distribution?
  3. A fair coin is tossed \(n\) times. Use Chebyshev to find an \(n\) that guarantees the proportion of heads is within 0.05 of 0.5 with probability at least 0.95. (The proportion has variance \(\tfrac{1}{4n}\).)
  4. For the coin in question 3, use the CLT instead. How many tosses does it suggest?
  5. After 10 coin tosses you have 8 heads. Explain why, after 10 000 tosses, the proportion of heads is still expected to be very close to 0.5, even though the coin does not "make up" for the extra heads.
  6. Waiting times for a bus are uniform on 0 to 12 minutes (lesson 2). You wait for the bus on 36 different days. Find the approximate chance that your average wait is more than 7 minutes.
  7. Explain why the sum of 2 dice is not normal but the sum of 50 dice is very nearly normal.
  8. A casino's roulette bet has expected profit \(-\tfrac{1}{37}\) dollar for the player and standard deviation about 1 dollar per 1-dollar bet. In one night players make 100 000 such bets. Estimate the casino's mean profit and the chance it loses money that night.
Answers
  1. 25 copies: mean 10, SD \(\tfrac{4}{5} = 0.8\). 400 copies: mean 10, SD \(\tfrac{4}{20} = 0.2\).
  2. At most \(\tfrac{1}{9} \approx 0.111\). For a normal it is about 0.0027, far less.
  3. \(P(|\bar X - 0.5| \ge 0.05) \le \tfrac{1/(4n)}{0.05^2} = \tfrac{100}{n}\). For this to be at most 0.05 we need \(n \ge 2000\).
  4. \(\operatorname{SD} = \tfrac{1}{2\sqrt n}\), and we want \(1.96 \times \tfrac{1}{2\sqrt n} \le 0.05\), so \(\sqrt n \ge 19.6\) and \(n \ge 384.16\): about 385 tosses. Much fewer than Chebyshev's 2000, because the CLT uses the bell shape.
  5. The extra 3 heads (8 instead of the expected 5) stay. But in 10 000 tosses they are diluted: they shift the proportion by only \(\tfrac{3}{10\,000} = 0.0003\), while the other 9990 tosses give a proportion close to 0.5 by the law of large numbers.
  6. One wait: \(\mu = 6\), \(\sigma = \tfrac{12}{\sqrt{12}} = \sqrt{12} \approx 3.46\). The average of 36: SD \(= \tfrac{3.46}{6} \approx 0.577\). \(z = \tfrac{7 - 6}{0.577} \approx 1.73\), \(P \approx 0.042\).
  7. Two dice give a triangle with only 11 values, far from a smooth bell. Fifty dice give a sum of 50 independent parts, and the CLT says such sums are close to normal; the exact distribution, drawn, is indistinguishable from a bell at ordinary scales.
  8. Mean player total: \(100\,000 \times \left(-\tfrac{1}{37}\right) \approx -2703\) dollars, so the casino expects to win about 2703 dollars. SD of the total: \(\sqrt{100\,000} \times 1 \approx 316\) dollars. The casino loses if the players' total is above 0: \(z = \tfrac{2703}{316} \approx 8.5\), a probability so small it is effectively 0. The law of large numbers is the casino's business model.

Common mistake

Thinking the law of large numbers works by correction. Chance has no memory. Averages settle because new results swamp old ones, not because the past is balanced out.

Applying the CLT to a single observation. The CLT is about sums and averages of many independent parts. One roll of a die is still uniform, however many dice you rolled yesterday; one skewed measurement is still skewed.

Where it leads