Prerequisites (1/2)

• basic calculus, linear algebra
• discrete vs. continuous distributions
• pmf, pdf, cdf
• joint, marginal, conditional distributions
• expectation, variance

Prerequisites (2/2)

Bayes Theorem

\(\boxed{\color{darkorange}{P(H|E)} = \dfrac{\color{green}{P(E|H)}\color{red}{P(H)}}{\color{blue}{P(E)}}}\)

• \(\color{red}{P(H)}: \text{prior}\)
• \(\color{green}{P(E|H)}: \text{likelihood}\)
• \(\color{blue}{P(E)}: \text{marginal evidence}\)
• \(\color{darkorange}{P(H|E)}: \text{posterior}\)

What is Data Science? (1/2)

What is Data Science? (2/2)

What is Machine Learning?

The History of Topic Modeling

• Early 90s - LSA (Latent Semantic Analysis)
  • Analyze relationships between a set of documents and the terms they contain.
  • tf-idf matrix
  • Singular value decomposition (SVD)
• Late 90s - pLSA (probalistitc LSA)
• 2003 - LDA (Latent Dirichlet Allocation)

Model Intuitions (1/3)

Model Intuitions (2/3)

Model Intuitions (3/3)

Model Assumptions

1. The order of the words in the document does not matter.
2. The order of documents does not matter.
3. The number of topics is assumed known and fixed.

Generative Process (1/3)

Generative Process (2/3)

\(\theta_d \sim \text{Dirichlet}(\alpha)\)
\(z_{d,n} | \theta_d \sim \text{Categorical}(K, \theta_d)\)
\(\beta_{k} \sim \text{Dirichlet}(\eta)\)
\(w_{d,n} | \beta_{1:K}, z_{d,n} \sim \text{Categorical}(K, \beta_k)\)

Generative Process (3/3)

\[ \tiny{ \color{brown}{p(\beta_{1:K}, \theta_{1:D}, z_{1:D}, w_{1:D})} = \displaystyle \prod_{k=1}^K p(\beta_k) \displaystyle \prod_{d=1}^D \Bigg( p(\theta_d) \bigg( \displaystyle \prod_{n=1}^N p(z_{d,n} | \theta_d) p(w_{d,n} | \beta_{1:K}, z_{d,n}) \bigg) \Bigg) } \]

Bayesian Inference

\[ \scriptsize{ \color{darkorange}{p(\beta_{1:K}, \theta_{1:D}, z_{1:D} | w_{1:D})} = \dfrac{\color{green}{p(w_{1:D} | \beta_{1:K}, \theta_{1:D}, z_{1:D})} \color{red}{p(\beta_{1:K}, \theta_{1:D}, z_{1:D})}}{\color{blue}{p(w_{1:D})}} } \]

Two categories of topic modeling algorithms:

• Variational algorithms
• Sampling-based algorithms

Prior and Likelihood (1/8)

• The coin belongs to the magician. (prob. may far from 0.5)
• There’s nothing obviously strange about the coin. (probably a fair coin)

\(\color{red}{z} \sim \text{Beta}(\alpha = 3, \beta = 3)\)
\(\color{green}{x_n|z} \sim \text{Bernouli}(p = z) \quad \forall n = 1, \dots, N\)

Prior and Likelihood (2/8)

z = np.linspace(0, 1, 250)
prior_a, prior_b = 3, 3
# prior distribution: Beta[3, 3]
p_of_z = scs.beta(prior_a, prior_b).pdf(z)
 
plt.xlabel('z')
plt.ylabel('p(z)')
plt.plot(z, p_z)
plt.show()

Toss a Coin (3/8)

N = 30
true_prob = scs.uniform.rvs(size = 1)
x = scs.bernoulli.rvs(p = true_prob, size = N)
print("x =", x)
# output:
#     x = [0 0 0 1 0 0 0 1 0 0
#          0 0 0 0 0 0 0 0 1 0 
#          1 1 0 0 0 1 0 0 1 0]

Posterior (4/8)

\(\scriptsize{ \color{darkorange}{p(z | \boldsymbol{x})} = \dfrac{ \color{green}{p(\boldsymbol{x} | z)} \color{red}{p(z)} }{ \color{blue}{p(\boldsymbol{x})} } }\)

• Find the distribution \(\scriptsize{\color{yellow}{q^∗(z)}}\) that is the closest approximation of posterior \(\scriptsize{\color{darkorange}{p(z|\boldsymbol{x})}}\).
• Let \(\scriptsize{\color{yellow}{q(z)} \sim \text{Beta}(\alpha_q, \beta_q)}\)
• Measure some sort of distance between two distributions by using \(\scriptsize{D_{KL}(Q||P)}\) and our goal is to minimize it.
• \(\boxed{\tiny{D_{KL}\big( Q(x) || P(x) \big) = \mathbb{E}_{x \sim Q}\big[ \log \frac{Q(x)}{P(x)} \big]}}\)
  • non-negative but asymmetric.

Evidence Lower Bound (5/8)

\[ \scriptsize{ \begin{aligned} D_{KL}\Big( \color{yellow}{q(z)} \space || \space \color{darkorange}{p(z | \boldsymbol{x})} \Big) &= \mathbb{E}_{\color{yellow}{q}} \big[ \log \frac{\color{yellow}{q(z)}}{\color{darkorange}{p(z | \boldsymbol{x})}}\big] = \mathbb{E}_{\color{yellow}{q}}\Big[ \log \frac{\color{yellow}{q(z)} \color{blue}{p(\boldsymbol{x})}}{\color{green}{p(\boldsymbol{x}|z)} \color{red}{p(z)}} \Big] \\ &= \mathbb{E}_{\color{yellow}{q}}\Big[ \log \color{yellow}{q(z)} \Big] - \mathbb{E}_{\color{yellow}{q}} \Big[ \log \color{green}{p(\boldsymbol{x} | z)} \color{red}{p(z)} \Big] + \mathbb{E}_{\color{yellow}{q}} \Big[ \log \color{blue}{p(\boldsymbol{x})} \Big] \\ &= \mathbb{E}_{\color{yellow}{q}}\Big[ \log \color{yellow}{q(z)} \Big] - \mathbb{E}_{\color{yellow}{q}} \Big[ \log \color{brown}{p(\boldsymbol{x}, z)} \Big] + \log \color{blue}{p(\boldsymbol{x})} \end{aligned} } \]

Evidence Lower Bound (6/8)

\[ \scriptsize { \begin{aligned} \log \color{blue}{p(\boldsymbol{x})} &= D_{KL}\Big( \color{yellow}{q(z)} \space || \space \color{darkorange}{p(z | \boldsymbol{x})} \Big) - \mathbb{E}_{\color{yellow}{q}}\Big[ \log \color{yellow}{q(z)} \Big] + \mathbb{E}_{\color{yellow}{q}} \Big[ \log \color{brown}{p(\boldsymbol{x}, z)} \Big] \\ &\ge \mathbb{E}_{\color{yellow}{q}} \Big[ \log \color{brown}{p(\boldsymbol{x}, z)} \Big] - \mathbb{E}_{\color{yellow}{q}}\Big[ \log \color{yellow}{q(z)} \Big] = \mathcal{L}(\alpha_q, \beta_q) \end{aligned} } \]

Evidence Lower Bound (7/8)

\[ \boxed{ \mathcal{L}(\alpha_q, \beta_q) = \mathbb{E}_q \big[ \log p(\boldsymbol{x},z) \big] - \mathbb{E}_q \big[ \log q(z) \big] } \]

• \(\scriptsize{\mathcal{L}(\alpha_q, \beta_q)}\) is so called ELBO, evidence lower bound.
• Minimizing \(\scriptsize{D_{KL}\big(q(z) || p(z|\boldsymbol{x})\big)}\) is equivalent to maximizing \(\scriptsize{\mathcal{L}(\alpha_q, \beta_q)}\).

Evidence Lower Bound (8/8)