Basics about probability distribution and Gaussians

written by @marc_lelarge

Probability recap

We start with real random variables (r.v.).

1- Why is variance positive?

Recall that

$\text{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X{]}^2$ so

$\text{Var}(X)\ge 0$ means that

$\mathbb{E}[X^2]\ge \mathbb{E}[X{]}^2$.

answer: Start with

$$\begin{array}{rcl}\mathbb{E}[(X-\mathbb{E}[X]{)}^2]& =& \mathbb{E}[X^2-2X\mathbb{E}[X]+\mathbb{E}[X{]}^2]\\ & =& \mathbb{E}[X^2]-2\mathbb{E}[X]\mathbb{E}[X]+\mathbb{E}[X{]}^2\\ & =& \mathbb{E}[X^2]-\mathbb{E}[X{]}^2\\ & =& \text{Var}(X).\end{array}$$

Similarly, we have for the Covariance of the random variables

$X$ and

$Y$:

$$\begin{array}{r}\text{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y]]=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y].\end{array}$$

Note that

$\text{Var}(X)=\text{Cov}(X,X)$.
We have for

$a,b\in \mathbb{R}$,

$\text{Var}(aX+b)=a^2\text{Var}(X)$, note that we use the standard notation where capital letters denote random variables and lowercase letters for constants or parameters.

We have

$$\begin{array}{r}\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)+\text{Cov}(X,Y).\end{array}$$

2- How to compute moments?

We start with a remark: if a random variable has a symmetric density function, i.e.

$p(X=x)=p(X=-x)$ for all

$x\in \mathbb{R}$, then its odd moments are zero:

$\mathbb{E}[X^{2k+1}]=0$.

answer thanks to the moment generating function:

$M_X(t)=\mathbb{E}\left[e^{tX}\right]$ so that we get:

$$\begin{array}{r}\mathbb{E}\left[X^k\right]={\left[\frac{d^k{M}_X(t)}{d{t}^k}\right]}_{t=0}\end{array}$$

To understand why this is true, we can write :

$e^{tX}=\sum _{k\ge 0}\frac{(tX{)}^k}{k!}=1+tX+\frac{(tX{)}^2}{2}+\frac{(tX{)}^3}{6}+\dots$ so that we have:

$$\begin{array}{rcl}\frac{d}{dt}{e}^{tX}& =& X+t{X}^2+\frac{t^2{X}^3}{2}+\dots \\ \frac{d^2}{d{t}^2}{e}^{tX}& =& X^2+t{X}^3+\dots \\ & \dots & \end{array}$$

Let's apply this method for the normalized Gaussian random variable

$Z$. We have

$$\begin{array}{rcl}{M}_Z(t)& =& \mathbb{E}\left[e^{tZ}\right]\\ & =& {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tz}\frac{e^{-z^2/2}}{\sqrt{2\pi }}dz\\ & =& {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\frac{e^{-(z^2-2tz+t^2)/2}}{\sqrt{2\pi }}{e}^{t^2/2}dz\\ & =& e^{t^2/2}.\end{array}$$

In particular, we have

$$\begin{array}{rcl}{M}_Z^{\prime }(t)& =& t{e}^{t^2/2}\\ M_Z^{″}(t)& =& (1+t^2)e^{t^2/2}\\ M_Z^{(3)}(t)& =& (3t+t^3)e^{t^2/2}\\ M_Z^{(4)}(t)& =& (3+6t^2+t^4)e^{t^2/2}...\end{array}$$
so that we have

$\mathbb{E}[Z]=0$,

$\mathbb{E}[Z^2]=1$,

$\mathbb{E}[Z^3]=0$,

$\mathbb{E}[Z^4]=3$…

Note that we already knew that the odd moments are zero but if you need the fourth moment, you need to compute the fourth derivative anyway…

Note that for simple distributions, the direct computation might be easier. For example, for the uniform distribution on the interval

$[a,b]$ with

$a<b$, we have for

$U\sim \text{Unif}(a,b)$:

$$\begin{array}{r}{M}_U(t)=\frac{e^{tb}-e^{ta}}{t(b-a)},\end{array}$$
and

$$\begin{array}{rcl}\mathbb{E}[U]& =& {\int }_a^{b}x\frac{dx}{b-a}=\frac{b^2-a^2}{2(b-a)}=\frac{a+b}{2},\\ \mathbb{E}[U^2]& =& {\int }_a^b{x}^2\frac{dx}{b-a}=\frac{a^2+ab+b^2}{3},\end{array}$$
so that

$\text{Var}(U)=\frac{(b-a{)}^2}{12}$.

3- Independence implies null covariance but null covariance does not imply independence!

Here is a simple example: consider the random variables

$(X,Y)$ equal to

$(-1,1)$,

$(0,-2)$ or

$(1,1)$ with equal probability.

We clearly have

$\mathbb{E}[X]=\mathbb{E}[Y]=0$ and

$\text{Cov}(X,Y)=\mathbb{E}[XY]=\frac{-1}{3}+\frac{0}{3}+\frac{1}{3}=0$ but

$X$ and

$Y$ are not independent as knowing

$X$ determines

$Y$. More formally, we have for example

$\mathbb{E}[X^2]=2/3$ and

$\mathbb{E}[X^{2}Y]=2/3\ne \mathbb{E}[X^2]\mathbb{E}[Y]$.

Gaussian random variables

4- If

$X$ is a Gaussian r.v. and

$Y$ is another Gaussian r.v. such that

$X\perp \perp Y$ then

$(X,Y)$ is a Gaussian vector.

5- If

$(X,Y)$ is a Gaussian vector then

$X\perp \perp Y$ is equivalent to

$\text{Cov}(X,Y)=0$.

But even if

$X\sim \mathcal{N}(0,1)$,

$Y\sim \mathcal{N}(0,1)$ and

$\text{Cov}(X,Y)=0$, this does not imply that

$(X,Y)$ is a Gaussian vector. Here is a simple counter-example: take

$X\sim \mathcal{N}(0,1)$ and define for

$a>0$:

$$\begin{array}{r}Y=\{\begin{array}{cc}X& \text{if }|X|>a\\ -X& \text{if }|X|\le a\end{array}\end{array}$$
It is easy to see that

$Y\sim \mathcal{N}(0,1)$, moreover, we have

$$\begin{array}{rcl}\text{Cov}(X,Y)& =& \mathbb{E}[XY]\\ & =& \mathbb{E}[X^2\mathbf{1}(|X|>a)]-\mathbb{E}[X^2\mathbf{1}(|X|\le a)].\end{array}$$
We see that for

$a\to 0$, we have

$\text{Cov}(X,Y)\to 1$ and when

$a\to \mathrm{\infty }$, we have

$\text{Cov}(X,Y)\to -1$, so there exists a value of

$a>0$ for which

$\text{Cov}(X,Y)=0$.
But

$X+Y$ is never a Gaussain r.v. as

$$\begin{array}{r}X+Y=\{\begin{array}{cc}2X& \text{if }|X|>a\\ 0& \text{if }|X|\le a\end{array}\end{array}$$

6- Moments of Gaussian r.v.

We have for

$X\sim \mathcal{N}(0,1)$

$$\begin{array}{rcl}{M}_X(t)& =& \mathbb{E}[e^{tX}]=e^{\frac{t^2}{2}}\\ & =& \int e^{tx}\frac{e^{-x^2/2}}{\sqrt{2\pi }}dx\\ & =& \int \frac{e^{-(x^2-2tx+t^2)/2}}{\sqrt{2\pi }}dx{e}^{t^2/2}\\ & =& e^{t^2/2}\int \frac{e^{-(x-t{)}^2/2}}{\sqrt{2\pi }}dx.\end{array}$$

Hence the moments for

$X\sim \mathcal{N}(0,1)$ are given by

$\mathbb{E}[X^{2m+1}]=0$ and

$$\mathbb{E}[X^{2m}]=\frac{2m!}{2^{m}m!}.$$
In general we have

$$\mathbb{E}[(\mu +\sigma X{)}^k]=\sum _{m=0}^k(\genfrac{}{}{0ex}{}{k}{m}){\mu }^m{\sigma }^{k-m}\mathbb{E}[X^{k-m}].$$

Gaussian vectors

7- Partititoned Gaussians

We consider now a Gaussian vector

$\mathbf{x}\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$ that we decompose as

$\mathbf{x}=(\genfrac{}{}{0ex}{}{{\mathbf{x}}_a}{{\mathbf{x}}_b})$. We consider the same decomposition for the parameters:

$$\begin{array}{r}\mu =\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right),\mathrm{\Sigma }=\left(\begin{array}{cc}{\mathrm{\Sigma }}_{aa}& {\mathrm{\Sigma }}_{ab}\\ {\mathrm{\Sigma }}_{ba}& {\mathrm{\Sigma }}_{bb}\end{array}\right).\end{array}$$
Note that

${\mathrm{\Sigma }}_{ba}={\mathrm{\Sigma }}_{ab}^T$.
We also introduce the precision matrix

$\mathrm{\Lambda }={\mathrm{\Sigma }}^{-1}$ and decompose it as:

$$\begin{array}{r}\mathrm{\Lambda }=\left(\begin{array}{cc}{\mathrm{\Lambda }}_{aa}& {\mathrm{\Lambda }}_{ab}\\ {\mathrm{\Lambda }}_{ba}& {\mathrm{\Lambda }}_{bb}\end{array}\right).\end{array}$$
Note that

${\mathrm{\Lambda }}_{aa}\ne {\mathrm{\Sigma }}_{aa}^{-1}$, indeed we can use the following formula for the inverse of a partitioned matrix:

$$\begin{array}{r}{\left(\begin{array}{cc}A& B\\ C& D\end{array}\right)}^{-1}=\left(\begin{array}{cc}M& -MB{D}^{-1}\\ -D^{-1}CM& D^{-1}+D^{-1}CMB{D}^{-1}\end{array}\right),\end{array}$$
where

$M={(A-B{D}^{-1}C)}^{-1}$.

Hence we see that

${\mathrm{\Lambda }}_{aa}={({\mathrm{\Sigma }}_{aa}-{\mathrm{\Sigma }}_{ab}{\mathrm{\Sigma }}_{bb}^{-1}{\mathrm{\Sigma }}_{ba})}^{-1}$.

Conditional distributions:

$$\begin{array}{r}p(x_a|x_b)=\mathcal{N}(x_a|{\mu }_{a|b},{\mathrm{\Lambda }}_{aa}^{-1}),\end{array}$$
with

${\mu }_{a|b}={\mu }_a-{\mathrm{\Lambda }}_{aa}^{-1}{\mathrm{\Lambda }}_{ab}(x_b-{\mu }_b)$

Marginal distribution

$$\begin{array}{r}p(x_a)=\mathcal{N}(x_a|{\mu }_a,{\mathrm{\Sigma }}_{aa}).\end{array}$$

8- Marginal and Conditional Gaussians
Consider a Gaussian vector:

$$\begin{array}{r}p(x)=\mathcal{N}(x|\mu ,{\mathrm{\Lambda }}^{-1})\end{array}$$
and a linear Gaussian model:

$$\begin{array}{r}p(y|x)=\mathcal{N}(y|Ax+b,L^{-1}),\end{array}$$
where

$A,b,\mu$ are parameters governing the means, and

$\mathrm{\Lambda }$ and

$L$ are precision matrices. Then

$z=(\genfrac{}{}{0ex}{}{x}{y})$ is a Gaussian vector and we have

$$\begin{array}{rc}p(y)& =\mathcal{N}(y|A\mu +b,L^{-1}+A{\mathrm{\Lambda }}^{-1}{A}^T)\\ p(x|y)& =\mathcal{N}(x|\mathrm{\Sigma }\{A^{T}L(y-b)+\mathrm{\Lambda }\mu \},\mathrm{\Sigma }),\\ & \text{ with }\mathrm{\Sigma }={(\mathrm{\Lambda }+A^{T}LA)}^{-1}\end{array}$$

tags: public tutorials