$$

Covariance estimation

Question

Given a set of samples

$\mathbf{x}=(x_1,\dots ,x_N)$. What is its covariacne? It is agreeable that the mean is

$$\bar{\mathbf{x}}=\frac{1}{N}\sum _{i=1}^N{x}_i.$$

For covariance, maybe you have heard of

$${\sigma }^2=\frac{1}{N}\sum _{i=1}^N(x_i-\bar{\mathbf{x}}{)}^2\text{ and }{s}^2=\frac{1}{N-1}\sum _{i=1}^N(x_i-\bar{\mathbf{x}}{)}^2.$$

Which formula for covariance is correct?

Experiments

You need: handout, 5 dice per group

1. Review that the covariance of a fair dice is
   $2.916\cdots$.
2. Roll 5 dice at once. Record their numbers, calculate the mean, and the value
   ${\tau }^2$.
3. Calculate the mean for the column of
   ${\sigma }^2$. Check if it is close to
   $2.916\cdots$.
4. If there are several groups, we may combine the data together.

Intuition

In probability theory, we assume we know the details of the probability distribution

$X$, where the event

$X=x$ happens with probability

$p_x$. Here is an example of a fair dice.

value	1	2	3	4	5	6
probability	1/6	1/6	1/6	1/6	1/6	1/6

There are fomal definitions for the mean and the covariance of

$X$

$$\mathbb{E}[X]=\sum _x{p}_x\cdot x\text{ and }\mathrm{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X]{)}^2].$$

This covariance is usually called the population covariance , indicating that you know the details of the whole population.

However, we never know if a dice is fair or not. We can only get some samples and use these data to estimate the population covariance. Here comes the problem. You obtain some samples

$\mathbf{x}=(x_1,\dots ,x_N)$. When we want to estimate the covariance

$\mathrm{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X]{)}^2]$, we do not know the mean

$\mathbb{E}[X]$ as well. The only thing we can do is to replace it by the sample mean

$$\bar{\mathbf{x}}=\frac{1}{N}\sum _{i=1}^N{x}_i.$$

Then calculate the sample covariance

$$s^2=\frac{1}{N-1}\sum _{i=1}^N(x_i-\bar{\mathbf{x}}{)}^2.$$

The point is that

$s^2$ is itself a random variable, depending on your samples. Ideally, when you run the experiment several times, the average of

$s^2$ should be close to the real anser

$\mathrm{Var}(X)$. As you have seen in the experiment, setting the denominator as

$N-1$ suprisingly did the job!

More questions to think about

1. Calculate the covariance of a fair coin of two sides
   $0$ and
   $1$.
2. Consider a random variable
   $\bar{\mathbf{x}}$ as the mean of five dice. Describe its probability distribution.
3. Consider a random variable
   $s^2$ as the sample variance of five dice. Describe its probability distribution.

Resources

1. YouTube: Why Sample Variance is Divided by n-1 by Krish Naik