機率 - 2 (Expectation ~ Conti R.V.)

Expectation

• ex.
  • Suppose
    $X\sim \text{Poisson}(\lambda =15,T=1)$. What is the PMF of
    $X$? How to define the expected value of
    $X$?
  • sol
    • $p_{{}_X}(x)=\{\begin{array}{l}{\displaystyle \frac{e^{\lambda T}(\lambda T{)}^x}{x!}},x=0,1,\dots \\ 0,\text{otherwise}\end{array}$
      
      $E(X)={\displaystyle \sum _{x=0}^{\mathrm{\infty }}x{\displaystyle \frac{e^{\lambda T}(\lambda T{)}^x}{x!}}={\displaystyle \sum _{x=0}^{\mathrm{\infty }}{\displaystyle \frac{e^{\lambda T}(\lambda T{)}^x}{(x-1)!}}=\lambda T{\displaystyle \sum _{x=0}^{\mathrm{\infty }}x{\displaystyle \frac{e^{\lambda T}(\lambda T{)}^{x-1}}{(x-1)!}}=\lambda T}}}$
      
      ${\displaystyle \sum _{x=0}^{\mathrm{\infty }}x{\displaystyle \frac{e^{\lambda T}(\lambda T{)}^{x-1}}{(x-1)!}}=1}$ since it look like the sum of PMF of another Poission R.V and the sum of the probability is 1
• Def (Expect Value / Mean / Expectation)
  $E[X]:={\displaystyle \sum _{x\in S}x\cdot p_{{}_X}(x)}$
  • $S$: the set of possible value
  • another notation:
    ${\mu }_{{}_X}=E[X]$
• Def (Expectation using CDF)
  $E[X]={\displaystyle \sum _{i=1}^{\mathrm{\infty }}(x_i-x_{i-1})\cdot (1-F_X(x_{i-1}))}$
  • Denote
    $F_X(x_0)=0$
• ex.
  • Suppose
    $X$ is a discrete R.V. and the set of possible value
    $A=\{2,4,\dots \}$. The CDF is
    $F_X(t)=1-{\displaystyle \frac{1}{t^2}}$. What is
    $E[X]$?
  • sol
    • $E[X]={\displaystyle \sum _{i=1}^{\mathrm{\infty }}(x_i-x_{i-1})(1-F_X(x_{i-1}))=2\cdot 1+{\displaystyle \sum _{\mathbf{i}\mathbf{=}\mathbf{2}}^{\mathrm{\infty }}2\cdot {\displaystyle \frac{1}{(i-1{)}^2}}=2+{\displaystyle \frac{{\pi }^2}{3}}}}$
• Thm (Expectation of a Function of r.v.) Let
  $X$ be a discrete R.V. with the set of possible value
  $S$ and PMF
  $p_{{}_X}(x)$. And let
  $g(\cdot )$ be a real valued func. Then
  $E[g(X)]={\displaystyle \sum _{x\in S}g(x)p_{{}_X}(x)}$
  • Also called Law of the unconscious statistician (LOTUS)

  pf
  Define a R.V

  $Y:=g(Y)$
  

  $E[g(X)]=E[Y]={\displaystyle \sum _{\begin{array}{c}y:y=g(x)\\ x\in S\end{array}}y\cdot p_{{}_Y}(y)=\sum _{\begin{array}{c}y:y=g(x)\\ x\in S\end{array}}y\sum _{x:g(x)=y}{p}_{{}_X}(x)}$
  

  $={\displaystyle \sum _{\begin{array}{c}(x,y):g(x)=y\\ x\in S\end{array}}y{p}_{{}_X}(x)=\sum _{x\in S}g(x)p_{{}_X}(x)}$

• Prop
  $E[g(X)+h(X)]=E[g(X)]+Eh(X)$

  pf
  

  $E[g(X)+h(X)]=\sum (g(x)+h(x))p_{{}_X}(x)=\sum g(x)p_{{}_X}(x)+\sum h(x)p_{{}_X}(x)$
  

  $=E[g(X)]+E[h(X)]$

Moment and Variable

• moments

  function	expectation
  $g(X)=X^2$	2-nd moment
  $g(X)=X^n$	n-th moment
  $g(X)=(X-{\mu }_{{}_X}{)}^2$	2-nd central moment
  $g(X)=(X-{\mu }_{{}_X}{)}^n$	n-th central moment
  $g(X)=e^{tX}$	moment generating func.

• Def (Variance)
  $\text{Var}[X]:=E[(X-{\mu }_{{}_X}{)}^2]$
  • also denote as
    ${\sigma }_{{}_X}^2$
• ex
  • Suppose we are given a random variable
    $X$. We need to output a prediction of
    $X$ (denoted by
    $z$). The penalty of prediction is
    $(X-z{)}^2$. What is the minimum expected penalty?
  • sol
    • $g(z):=E[(X-z{)}^2]=E[X^2-2zX+z^2]=E[X^2]-2zE[X]+z^2$
      
      $=(z-E[X]{)}^2+E[X^2]-(E[X]{)}^2\ge E[X^2]-(E[X]{)}^2=\text{Var}(X)$
  • Variance = minimum expected quadratic penalty
• Thm
  $\text{Var}[X]:=E[X^2]-(E[X]{)}^2$

  pf
  

  $\text{Var}[X]=E[(X-{\mu }_{{}_X}^2)]=E[X^2]-2{\mu }_{{}_X}E[X]+{\mu }_{{}_X}^2$
  

  $=E[X^2]-2(E[X{]}^2)+(E[X]{)}^2=E[X^2]-(E[X]{)}^2$

• Prop
  • $\text{Var}[X+c]=\text{Var}[X]$
  • $\text{Var}[aX]=a^2\text{Var}[X]$
  • $E[X^2]\ge (E[X]{)}^2$

    pf

    $\text{Var}[X]=E[X^2]-(E[X])\ge 0$

• Def (Existence) Let
  $X$ be a random variable. Then, the n-th moment of
  $X$ (i.e.
  $E[X]$) is said to exist if
  $E[|X^n|]<\mathrm{\infty }$
  • ex.
    • Let
      $z_n=(-1{)}^n\sqrt{n}$,
      $n=1,2,\dots$. Let
      $Z$ be a random variable with the set of possible values
      $\{z_n:n=1,2,\dots \}$and the PMF
      $p_{{}_Z}(z)$ as
      $p_{{}_Z}(z_n)={\displaystyle \frac{6}{(\pi n{)}^2}}$. What is Var[X]?
    • $\text{Var}[Z]=E[Z^2]-(E[Z]{)}^2$
      
      $E[|Z^2|]={\displaystyle \sum _{n=1}^{\mathrm{\infty }}(\sqrt{n}{)}^2{\displaystyle \frac{6}{(\pi n{)}^2}}=\mathrm{\infty }}$
      So
      $\text{Var}[Z]$ DNE.
• Thm If
  $E[|X{|}^{n+1}]<\mathrm{\infty }$, then
  $E[|X{|}^n]<\mathrm{\infty }$
  • pf
    • $E[|X^n|]={\displaystyle \sum _{x\in S}|X{|}^n{p}_{{}_X}(x)=\sum _{\begin{array}{c}x:|x|\le 1\\ x\in S\end{array}}|X{|}^n{p}_{{}_X}(x)+\sum _{\begin{array}{c}x:|x|>1\\ x\in S\end{array}}|X{|}^n{p}_{{}_X}(x)}$
      
      $<1+{\displaystyle \sum _{\begin{array}{c}x:|x|>1\\ x\in S\end{array}}|X{|}^{n+1}{p}_{{}_X}(x)<\mathrm{\infty }}$

$E[X]$ and

$\text{Var}[X]$ of Special Discrete R.V.

• Bernoulli R.V.:
  $X\sim$ Bernoulli
  $(p)$
  • $E[X]=p$
  • $E[X]=p(1-p)$
• Binomial R.V.:
  $X\sim$ Binomial
  $(n,p)$
  • $E[X]=np$
    • $E[X]={\displaystyle \sum _{k=0}^{n}k\cdot {\displaystyle (\genfrac{}{}{0ex}{}{n}{k})}{p}^k(1-p{)}^{n-k}=\sum _{k=0}^{n}k\cdot {\displaystyle \frac{n!}{k!(n-k)!}}{p}^k(1-p{)}^{n-k}}$
      
      $=np{\displaystyle \sum _{k=1}^n{\displaystyle \frac{(n-1)!}{(k-1)!(n-k)!}}{p}^{k-1}(1-p{)}^{n-k}=np}$
    • ${\displaystyle \frac{(n-1)!}{(k-1)!(n-k)!}}{p}^{k-1}(1-p{)}^{n-k}$ is the probability of Binomial(n-1, p), so the sum would be 1
  • $\text{Var}[X]=np(1-p)$
    • $E[X^2]={\displaystyle \sum _{k=0}^n{k}^2\cdot {\displaystyle \frac{n!}{k!(n-k)!}}{p}^k(1-p{)}^{n-k}}$
      
      $={\displaystyle \sum _{k=1}^n(k-1)\cdot {\displaystyle \frac{n!}{(k-1)!(n-k)!}}{p}^k(1-p{)}^{n-k}}$
      
      $+{\displaystyle \sum _{k=0}^n{\displaystyle \frac{n!}{(k-1)!(n-k)!}}{p}^k(1-p{)}^{n-k}}$
      
      $=n(n-1)p^2{\displaystyle \sum _{k=2}^n{\displaystyle (\genfrac{}{}{0ex}{}{n-2}{k-2})}{p}^{k-2}(1-p{)}^{n-k}+np=n(n-1)p^2+np}$
      
      $\text{Var}[X]=E[X^2]-(E[X]{)}^2=n(n-1)p^2+np-(np{)}^2=np(1-p)$

Continuous Random Variables

Probability Density Function (PDF)

• Def (PDF) Let
  $X$ be a random variable. Then,
  $f_{{}_X}(x)$ is the PDF of
  $X$ if for every subset
  $B$ of the real line, we have
  $P(X\in B)={\displaystyle {\int }_B{f}_{{}_X}(x)dx}$
  • $P(X\in \mathbb{R})=1$
  • $P(X\le t)={\displaystyle {\int }_{-\mathrm{\infty }}^t{f}_{{}_X}(x)dx}$
  • $P(a\le X\le b)={\displaystyle {\int }_a^b{f}_{{}_X}(x)dx}$
  • $P(a\le X<b)={\displaystyle {\int }_a^b{f}_{{}_X}(x)dx-P(X=b)={\displaystyle {\int }_a^b{f}_{{}_X}(x)dx}}$
• Check valid (3 axioms of probability)
  • $P(x\in \mathbb{R})=1\Rightarrow {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{{}_X}(x)dx=1}$
  • $P(X\in A)\ge 0\Rightarrow {\displaystyle {\int }_A{f}_{{}_X}(x)dx\ge 0}$
  • $P(X\in {\displaystyle \bigcup _{i\ge 1}{A}_i)=\sum _{i\ge 1}P(X\in A_i)\Rightarrow {\int }_{\cup A_i}{f}_{{}_X}(x)dx=\sum _{i\ge 1}{\int }_{A_i}{f}_{{}_X}(x)dx}$
• CDF & PDF
  • Let
    $X$ be a random variable with a CDF
    $F_X(\cdot )$ and a PDF
    $f_{{}_X}(\cdot )$
  • PDF -> CDF:
    $F_X(t)=P(X\le t)={\displaystyle {\int }_{-\mathrm{\infty }}^t{f}_{{}_X}(x)dx}$
  • CDF -> PDF:
    $f_{{}_X}(x_0)=F_X^{\prime }(x_0)$ when
    $f_{{}_X}(\cdot )$ is continuous at
    $x_0$
  • like Fundamental Thm. of Calculus
  • Given CDF, the derived PDF is not unique

Uniform Random Variables

• Def A random variable
  $X$ is uniform with parameters
  $a,b(a<b)$ if its PDF is
  $f_{{}_X}(x)=\{\begin{array}{l}\frac{1}{b-a},a<x<b\\ 0,\text{otherwise}\end{array}$
• ex
  • Let
    $X$ be a random variable with CDF
    $F(t)$. Define another random variable
    $Y=F(X)$. What type of random variable is
    $Y$
  • sol
    • $F_Y(t)=P(Y\le t)=P(F(X)\le t)$
      To simplify, let
      $t=0.5$. Then we are finding that the probability of
      $F(X)\le 0.5$, which is just 0.5
      So
      $P(F(X)\le t)=\{\begin{array}{l}1,t\ge 1\\ t,0<t<1\\ 0,t\le 0\end{array}$
• • 目的：對於一個已知 CDF 的分佈，我們如何從中 sample
  • pf
    • The CDF we generate is
      $P(X\le t)=P(F^{-1}(U)\le t)=P(F(F^{-1}(U))\le F(t))$
      
      $=P(U\le F(t))=F(t)$
  • 假設
    $X$ 為一個連續隨機變數，其 CDF 為
    $F_X$。此時，隨機變數
    $Y=F_X(X)\sim \text{Unif}(0,1)$。ITS 即是將該過程反過來進行：首先對於隨機變數
    $Y$，我們從 0~1 中隨機均勻抽取一個數
    $u$。之後，由於隨機變數
    $F_X^{-1}(Y)$ 與
    $X$ 有著相同的分布，
    $x=F_X^{-1}(u)$ 即可看作是從分布
    $F_X$ 中生成的隨機樣本。 Ref