Mathematical Details of the Law of Large Numbers

Setup

Let

$X_1,X_2,X_3,\dots$ be a sequence of independent and identically distributed random variables.
Let

$\mu =E[X]$ be the expected value of each

$X$.
Let

${\sigma }^2=\text{Var}(X)$ be the variance of each

$X$.

Define the sample mean:

$${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i$$

Weak Law of Large Numbers (WLLN)

For any

$\epsilon >0$:

$$\underset{n\to \mathrm{\infty }}{lim}P(|{\overline{X}}_n-\mu |<\epsilon )=1$$

This means

${\overline{X}}_n$ converges in probability to

$\mu$, written as:

$${\overline{X}}_n\stackrel{p}{\to }\mu$$

Strong Law of Large Numbers (SLLN)

$$P(\underset{n\to \mathrm{\infty }}{lim}{\overline{X}}_n=\mu )=1$$

This means

${\overline{X}}_n$ converges almost surely to

$\mu$, written as:

$${\overline{X}}_n\stackrel{a.s.}{\to }\mu$$

Implications

1. The sample mean is a consistent estimator of the population mean.
2. As sample size increases, the probability of large deviations from the true mean decreases.
3. In the limit, the sample mean will converge to the true population mean.
4. This applies to any distribution with a finite mean, not just normal distributions.

Note on Convergence

• WLLN: Convergence in probability
• SLLN: Almost sure convergence (stronger than convergence in probability)

The LLN doesn't specify the rate of convergence. This is addressed by other theorems, such as the Central Limit Theorem.

---

Central Limit Theorem (CLT)

Setup

Let

$X_1,X_2,X_3,\dots ,X_n$ be a sequence of independent and identically distributed random variables.
Let

$\mu =E[X]$ be the expected value of each

$X$.
Let

${\sigma }^2=\text{Var}(X)$ be the variance of each

$X$.

Define the sample mean:

$${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i$$

Statement of the Central Limit Theorem

As

$n$ approaches infinity, the distribution of the standardized sample mean:

$$\frac{{\overline{X}}_n-\mu }{\sigma /\sqrt{n}}$$

converges in distribution to a standard normal random variable

$Z\sim N(0,1)$.

In other words:

$$\sqrt{n}({\overline{X}}_n-\mu )/\sigma \stackrel{d}{\to }N(0,1)\text{ as }n\to \mathrm{\infty }$$

Implications

1. For large

   $n$,
   ${\overline{X}}_n$ is approximately normally distributed with:
   
   $E[{\overline{X}}_n]=\mu$
   
   $\text{Var}({\overline{X}}_n)={\sigma }^2/n$

2. The distribution of

   ${\overline{X}}_n$ can be approximated by:
   
   $${\overline{X}}_n\sim N(\mu ,{\sigma }^2/n)$$

3. The standard error of the mean is given by:
   

   $$SE({\overline{X}}_n)=\sigma /\sqrt{n}$$

Key Points

• Applies regardless of the shape of the original distribution (as long as
  $\mu$ and
  ${\sigma }^2$ are finite).
• "Sufficiently large"
  $n$ depends on the original distribution, but
  $n\ge 30$ is often used as a rule of thumb.
• Enables the use of z-scores and t-scores in hypothesis testing and confidence intervals.

Comparison with Law of Large Numbers (LLN)

• LLN:
  ${\overline{X}}_n\stackrel{p}{\to }\mu$ (WLLN) or
  ${\overline{X}}_n\stackrel{a.s.}{\to }\mu$ (SLLN)
• CLT:
  $\sqrt{n}({\overline{X}}_n-\mu )/\sigma \stackrel{d}{\to }N(0,1)$

CLT provides information about the distribution and spread of

${\overline{X}}_n$, while LLN only addresses its convergence to

$\mu$.

Applications

1. Hypothesis testing
2. Construction of confidence intervals
3. Approximation of binomial distribution by normal distribution
4. Quality control in manufacturing
5. Sampling theory in surveys and polls

The CLT is a cornerstone of statistical inference, allowing us to make probabilistic statements about sample means even when we don't know the underlying population distribution.