---
# System prepended metadata

title: mathstat_ch3_Expectation_3.1-3.4
tags: [mathematic statistics]

---

# Introduction  
這篇主要在講我們的期望值  



# Expectation of a Random Variable  
這些名詞所代表的東西是一樣的，都是表示期望值 :  
* expected value  
* mean  
* first moment  

直觀的來看，就是代表這個函式的中心  

如果用數學表示 :  
$\mu_x = \mathbb{E}(X) = \int x \ dF(x) \ \begin{cases} \Sigma_x \ xf(X),\ \text{X is discrete} \\  \\ \int \ xf(x) \ dx,\ \text{X is continuous}   \end{cases}$  


## 常見 distribution 的 $EX$  
接下來就用常見的 distribution 當作例子 :  
* $X \sim Bernoulli(p) \Rightarrow x \in \left\{ 0, 1 \right\}$  
$EX = 0 \times (1-p) + 1 \times p = p$  

* $X \sim BIN(n, p) \Rightarrow x \in \left\{ 0, 1, ......, n \right\}$  
$EX = \Sigma_{x= 0}^{n}\ xC^{n}_{x}p^x(1-p)^{n-x} = np$  

* $X \sim UNIF(p)$  
$EX = \int_a^b xf(x)dx = \cfrac{a+b}{2}$  

* $X \sim POI(\lambda)$  
$EX =...... \ \text{by def} = \lambda$  

* $X \sim EXP(\beta)$  
$EX =...... \ \text{by def} = \beta$  

* $X \sim Gamma(\alpha, \beta)$  
$EX =...... \ \text{by def} = \alpha\beta$  


## 基礎公式  
先令 $Y = r(X),\ Z = r(X,\ Y)$
* $EY = E(r(x)) = \int r(x) dF_X(x)$  
* $EZ = E(r(X,\ Y)) = \int \int r(x,\ y) f(x,\ y) \ dxdy$  

證明過程如下 :  
:::spoiler  
$EY$ :  
我們在已知 $X$ 的分布的情況下要求出 $EY$  
$\text{Let } Y = r(X)$  
$\Rightarrow \underbrace{E(Y)}_{=\int y dF_Y(y) \Rightarrow 找\ y \ 的分佈} = E(r(X)) = \int r(x) d\underbrace{F_X(x)}_{直接使用\ x \ 的分佈}$  
:::  



## Examples  
### **Question 1**  
**question** : $X \sim UNIF(0,\ 1),\ Y =r(X) = e^X$, 求出 $EY$  

**solution** :  
首先把題目做解析  
從 $X$ 可以看出 : $f_X = \begin{cases} 1,\ 0 \le x \le 1 \\ \\ 0 ,\ \text{otherwise} \end{cases}$  
從 $Y$ 可以看出 : $x = r^{-1}(y) = ln(y)$  

然後有 2 種解題方式 :  
* method 1 : 找出 $y$ 的分佈  
$f_Y(y) = f_X(ln(y))\ \cfrac{d \ ln(y)}{dy} = 1 \times \cfrac{1}{y} = \cfrac{1}{y},\ 1 \le y \le e$  
$EY = \int_{1}^{e} y f_Y(y)dy = \int_1^e y \cdot \cfrac{1}{y} dy = \left[ \ y \ \right]^{e}_{1} = e - 1$  

* method 2 : lazy, by fomula  
$\underbrace{EY}_{對\ y\ 的分佈做} = \underbrace{Er(X)}_{對\ x\ 的分佈做} = \int_{\color{red}{0}}^{\color{red}{1}} r(x)f_{\color{red}{x}} \color{red}{(x)dx} = \int^1_0 e^x \times 1 = \left[ \ e^x \ \right]^{1}_{0} = e - 1$  


### **Question 2**  
**question** : $X \sim UNIF(0,\ 1),\ Y = max(x,\ 1-x)$, 求出 $EY$  

**solution** :  
有 2 種解題方式 :  
* method 1 : 找出 $y$ 的分佈  
$F_Y(y) = P(Y \le y) = P(max(x,\ 1-x) \le y) = P(x \le y \ \And\ (1-x) \le y) = P((1-y) \le x \le y)$  
$= \int^y_{1-y}f_X(x)dx = \left[ \ x \ \right]^{y}_{1-y} = 2y-1,\ \cfrac{1}{2} \le y \le 1$  
因為上面做出來的是 $cdf$，我們要得知 $EY$ 是要用 $pdf$，因此要進行微分  
$f_Y(y) = \cfrac{d}{dy}F_Y(y) = y \Rightarrow EY = \int^1_{\frac{1}{2}} y \cdot f_Y(y)dy = \int^1_{\frac{1}{2}}2y \ dy = \cfrac{3}{4}$  


* method 2 : lazy, by fomula  
從題目可以先得知可能會需要拆開來看，因此我們先做表格確認一下  
	| $x$            | $y = r(x)$                                  |
	|:-------------- |:------------------------------------------- |
	| $\cfrac{1}{3}$ | $\cfrac{2}{3}\Rightarrow\color{green}{1-x}$ |
	| $\cfrac{2}{3}$ | $\cfrac{2}{3}\Rightarrow\color{green}{x}$   |  
	
	從表格可以看出我們會需要拆成 $2$ 段來計算  
	$\begin{cases} 0 < x < \cfrac{1}{2} : 1-x  \\ \cfrac{1}{2} < x < 1 : x \end{cases}$  
	
	$EY = Er(X) = \int^{\color{purple}{1}}_{\color{purple}{0}} \color{purple}{r(x)} f_X(x)dx$  
	$= \int^{\frac{1}{2}}_{0} (1-x)f_X(x)dx + \int^{1}_{\frac{1}{2}} (x)f_X(x)dx$  
	$= \int^{\frac{1}{2}}_{0} (1-x)dx + \int^{1}_{\frac{1}{2}} (x)dx = \cfrac{3}{4}$


### **Question 3**  
**question** : $(X,\ Y) \sim \underbrace{\text{uniform distribution}}_{\color{purple}{f(x,\ y) = 1}} \underbrace{\text{ on the unit square}}_{\color{blue}{[0,\ 1] \times [0,\ 1]}},\  Z = r(x,\ y) = X^2 + Y^2,$ 求 $EZ$  

**solution** :  
有 2 種解題方式 :  
* method 1 : 找出 $z$ 的分佈  
$...同上,\ 省略...$  

* method 2 : lazy, by fomula  
$EZ = Er(X,\ Y) = \int^1_0 \int^1_0 (x^2 + y^2) f(x,\ y) \ dxdy$  
$= \int^1_0 \int^1_0 (x^2 + y^2) \ dxdy = \cfrac{2}{3}$  




# Properties of Expectation  
## 基礎公式  
$X_1,\ ...,\ X_n$ 是 random variables，且 $a_1,\ ...,\ a_n$ 是常數  
這時 $E(\Sigma_i a_i X_i) = \Sigma_i a_i E(X_i)$  

證明過程如下 :  
:::spoiler  
$E(a_1 x_1 + a_2 x_2) = \int \int (a_1 x_1 + a_2 x_2) f(x_1,\ x_2) dx_1 dx_2$  
$= \int \int \color{orange}{(a_1 x_1)} \cdot f(x_1,\ x_2) \ dx_2 \color{orange}{dx_1} + \int \int \color{orange}{(a_2 x_2)} \cdot f(x_1,\ x_2) \ dx_1 \color{orange}{dx_2}$  
$= \int (a_1 x_1) \cdot  \int f(x_1,\ x_2) \ dx_2 \ dx_1 + \int (a_2 x_2) \cdot \int f(x_1,\ x_2) \ dx_1 \ dx_2$  
$= \int a_1 \left[ \ \underline{x_1 \cdot  f_1(x_1) \ dx_1} \right]_{=EX_1} + \int a_2 \left[ \ \underline{x_2 \cdot  f_2(x_2) \ dx_2} \right]_{=EX_2}$  
$= a_1 EX_1 + a_2 EX_2$  
:::  

---

$X_1,\ ...,\ X_n$ 是 independent random variables  
這時 $E(\Pi_i^n X_i) = \Pi_i^n E(X_i)$  

證明過程如下 :  
:::spoiler  
$E(x_1 x_2) = \int \int ( x_1 x_2) \ \underbrace{f(x_1,\ x_2)}_{\color{red}{\text{independent}}} \ dx_1 dx_2$  
$= \int \int x_1 \color{orange}{\underline{x_2}} f_1(x_1) \color{orange}{\underline{f_2(x_2)}} \ dx_1 dx_2 \stackrel{\text{把橘色部分丟到外面}}{\Longrightarrow}$  
$= \int x_2 f_2(x_2) \cdot  \left[ \ \underline{\int x_1 f_1(x_1) \ dx_1} \right]_{=EX_1} dx_2$  
$= \int x_2 f_2(x_2) \cdot EX_1 \ dx_2 = EX_1 \left[ \ \underline{\int x_2 \cdot  f_2(x_2) \ dx_2} \right]_{=EX_2}$  
$= EX_1 \cdot EX_2$  
:::  




# Variances and Covariance  
## 基礎公式  
先令 $X$ 是一個 random variable，並且它的 mean 跟 variance 分別為 $\mu,\ \sigma^2(也可用 \sigma^2_X,\ \mathbb{V}(X),\ \mathbb{V}X 代稱)$  
並且 $\sigma^2$ 的定義為 :  
$\sigma^2 = \mathbb{E}(X - \mu)^{\color{red}{2}} = \int (x-\mu)^2 dF(x)$  

另外 variance 會遵循以下特性 :  
1. $\mathbb{V}(X) = \mathbb{E}(X^2) - \mu^2$  
2. $\mathbb{V}(aX - b) = a^2\mathbb{V}(X),\ \text{for } a,b \in \text{constant}$  
3. $\mathbb{V}(\Sigma_{i=1}^n \ a_i X_i) = \Sigma_{i=1}^n \ \color{red}{a_i^2} \mathbb{V}(X_i),\ \text{for } X_1,\ ...,\ X_n \text{ are independent, and }a \in \text{constant}$  

證明過程如下 :  
:::spoiler  
1. $VX = E(x - \mu)^2 = E(x^2 -2x\mu + \mu^2) = ...... = EX^2-\mu^2$  
2. $V(ax+b) = E \left[ \ (ax+b)-E(ax+b) \ \right]^2 = ...... = a^2VX \longrightarrow$ 數學上  
$V(ax+b) = V(aX) = a^2VX \longrightarrow$ 直觀上  
3. $V(a_1x_1 + a_2x_2) = E \left[ \ a_1x_1 + a_2x_2 - E(a_1x_1 + a_2x_2) \ \right]^2 = ...(\text{related to } EY_1Y_2=EY_1EY_2)...$  
:::  

---

上述說的都是母體的情況，接下來這部分要講 sample mean & sample variance  
先令 $X_i,\ ...,\ X_n$ 都是相同且相互獨立的分佈  
並且令 $\mu = \mathbb{E}(X_i),\ \sigma^2 = \mathbb{V}(X_i)$  
這時 :  
$\mathbb{E}(\overline{X_n}) = \mu,\ \mathbb{V}(X) = \cfrac{\sigma^2}{n},\ \mathbb{E}(S_n^2) = \sigma^2$  

證明過程如下 :  
:::spoiler  
$E \overline{X_n} = E(\cfrac{1}{n}\Sigma_{i=1}^{n} \ X_i) = ...... = \mu$  
$V \overline{X_n} = E\left[ \ (\cfrac{1}{n}\Sigma_{i=1}^{n} \ X_i)-E(\overline{X_n}) \ \right]^2 = \cfrac{\sigma^2}{n}$  
**important :**  
$ES_n^2 = E\left[ \ \cfrac{1}{n-1} \Sigma(X_i - \overline{X_n})^2 \ \right] =E\left[ \ \cfrac{1}{n-1} \Sigma(X_i^2 - 2X_i \overline{X_n} + \overline{X_n}^2) \ \right] = ...... = \sigma^2$
:::  

---

接著我們來看 covariance & correlation  
先令 $X_i,\ ...,\ X_n$ 和 $Y_i,\ ...,\ Y_n$ 都是 random variables，並且 means 跟 standard deviation 分別為 $\mu_X,\ \sigma_X,\ \mu_Y,\ \sigma_Y$  
這時 :  
* **covariance :**  
$Cov(X,\ Y) = \mathbb{E}((X - \mu_X)(Y - \mu_Y)) = \underbrace{E(XY) - E(X)E(Y)}_{= E \left[ \ (X - EX) (Y - EY) \ \right]}$  
小補充 : $Cov$ 的大小代表線性關係的強度，$Cov$ 的正負代表線性關係的方向  

* **correlation :**  
$\rho = \rho_{X,\ Y} = \rho(X,\ Y) = \cfrac{Cov(X,\ Y)}{\color{red}{\rho_X \ \rho_Y}},\ -1 \le \rho(X,\ Y) \le 1$  
假設 $Y = aX + b,\ a,\ b \in \text{constant}$  
這時 : $\begin{cases} \rho (X,\ Y) = 1,\ a > 0 \\  \rho (X,\ Y) = -1,\ a < 0   \end{cases}$ (下面有附上證明)  
另外，如果 $X$ 跟 $Y$ 是獨立的(independent) $\Rightarrow Cov(X,\ Y) = \rho = 0$  
證明過程如下 :  
	:::spoiler  
	已知 $Y = aX +b ,\ VY = a^2VX$  
	$\Rightarrow cov(X,\ Y) = cov(X,\ aX +b) = E\left[ \ X(aX+b) \ \right] - (EX)(aEX + b)$  
	$= E(aX^2+bX) - (EX)(aEX + b) = ...... = \underbrace{a\sigma^2}_{corr(X,\ Y) = 1 \text{ or } -1 \text{ depending on }a} = \underbrace{(a\sigma)}_{sd(Y)} \underbrace{\sigma}_{sd(X)}$  
	:::  

----

接著我們來看 $2$ 個變數的 variance  
$\begin{cases} 
V(X+Y) = V(X) + V(Y) + 2 \cdot Cov(X,\ Y)  \\
V(X-Y) = V(X) + V(Y) - 2 \cdot Cov(X,\ Y)
\end{cases}$  
廣義上來說，也可以轉換成這樣 :  
$V(\Sigma_i \ a_i X_i) = \Sigma_i \ a_i^2 V(X_i) + 2 \Sigma\Sigma_{i<j} \ a_i a_j \cdot Cov(X_i,\ X_j)$  




## Examples  
### **Question 1**  
**question** : $X \sim BIN(n,\ p),\ 求出\ EX,\ VX$  

**solution** :  
然後有 2 種解題方式 :  
* method 1 : by basic def  
$EX = \Sigma_X \ xf(x)$  
$VX = \Sigma_X \ (x-\mu)^2 = EX^2 - \mu$  

* method 2 : by scratch  
$X = \Sigma^n_{i=1} \ Z_i,\ Z_i \overset{iid}{\sim} Bernoulli(p) \Rightarrow EZ_i = p,\ VZ_i = p(1-p)$  
$EX = E\ \Sigma^n_{i=1} \ Z_i = \Sigma^n_{i=1} \ EZ_i = np \longleftarrow \color{purple}{不需要 \ Z_i \ 的條件}$  
$VX = V\ \Sigma^n_{i=1} \ Z_i = \Sigma^n_{i=1} \ VZ_i = np(1-p) \longleftarrow \color{purple}{需要 \ Z_i \ 的條件}$  

### **Question 2**  
**question** : $X \sim Gamma(\alpha,\ \beta),\ 求出\ EX,\ VX$  

**solution** :  
然後有 2 種解題方式 :  
* method 1 : by basic def  
$......$  

* method 2 : by scratch  
$X = \Sigma^n_{i=1} \ Z_i,\ Z_i \overset{iid}{\sim} exp(\beta) \Rightarrow EZ_i = \beta,\ VZ_i = \beta^2$  
$EX = E\ \Sigma^{\alpha}_{i=1} \ Z_i = \Sigma^{\alpha}_{i=1} \ EZ_i = \alpha \beta$  
$VX = V\ \Sigma^{\alpha}_{i=1} \ Z_i = \Sigma^{\alpha}_{i=1} \ VZ_i = \alpha \beta^2$  



# Expectation and Variances of Important Random Variables  
**粗體的部分是一定要記住的!!!**  
| Distribution                         | Mean                       | Variance                                            |
|:------------------------------------ |:-------------------------- |:--------------------------------------------------- |
| **Point mass at $a$**                | $a$                        | $0$                                                 |
| **Bernoulli $(p)$**                  | $p$                        | $p(1-p)$                                            |
| **Binomial $(n,\ p)$**               | $np$                       | $np(1-p)$                                           |
| Geometric $(p)$                      | $1/p$                      | $(1-p)/p^2$                                         |
| **Poisson $(\lambda)$**              | $\lambda$                  | $\lambda$                                           |
| **Uniform $(a,\ b)$**                | $(a+b)/2$                  | $(b-a)^2/12$                                        |
| **Normal $(\mu,\ \sigma^2)$**        | $\mu$                      | $\sigma^2$                                          |
| **Exponential $(\beta)$**            | $\beta$                    | $\beta^2$                                           |
| **Gamma $(\alpha,\ \beta)$**         | $\alpha \beta$             | $\alpha \beta^2$                                    |
| Beta $(\alpha,\ \beta)$              | $\alpha/ (\alpha + \beta)$ | $\alpha \beta / ((\alpha+\beta)^2(\alpha+\beta+1))$ |
| $t_{\nu}$                            | $0,\ \text{if } \nu>1$     | $\nu /(\nu-2),\ \text{if }\nu>2$                    |
| $\chi^2_{p}$                         | $p$                        | $2p$                                                |
| Multinomial $(n,\ p)$                | $np$                       | $\text{see below}$                                  |
| Multivariate Normal $(\mu,\ \Sigma)$ | $\mu$                      | $\Sigma$                                            |

































