---
# System prepended metadata

title: A Theoretical Analysis of Contrastive Unsupervised Representation Learning

---

# A Theoretical Analysis of Contrastive Unsupervised Representation Learning

$x, x^+, x^-$ anchor, pos sample, neg sample

$f: \mathcal{X} \mapsto \mathbb{R}^d$ embedding function, with $||f(x)||^2 = 1$

$(x, x^+) \sim \mathcal{D}_\text{sim}$ joint distribution of anchor and pos sample
$p(x, x^+) = p(x)p(x^+)$

$x^- \sim \mathcal{D}_\text{neg}$ distribution of neg sample
$p(x^-|x) = \sum_{c\in\mathcal{C}} p(x^-,c|x) = \sum_{c\in\mathcal{C}}p(x^-|c)p(c|x)$

$p(c|x): c \sim \mathcal{D}_c(x)$ distribution of classes for a given $x$
$p(c): c \sim \rho$ prior on classes

In Bayes' terms:
$p(c|x) = \frac{p(x|c)p(c)}{\sum_{c\in\mathcal{C}}p(x|c)p(c)}$

$p(x,x^+) = \sum_{c\in\mathcal{C}}p(c)p(x|c)p(x^+|c)$

Supervised Task $\mathcal{T}\subseteq\mathcal{C}$:

$p(x,c|\mathcal{T}) = p(x|c)p(c|\mathcal{T})$

$g: \mathcal{X}\mapsto\mathbb{R}^k$, $k$ number of classes
$g(x) = Wf(x)$
$v = \left( g(x)_i - g(x)_j \right)_{j\neq i}$

Expected loss:
$L_\text{sup}(\mathcal{T}, g) = \sum_{c\in\mathcal{C}}\sum_{x\in\mathcal{X}}p(x|c)p(c)l(v)$

$L_\text{sup}(\mathcal{T}, f) = \inf_W L_\text{sup}(\mathcal{T}, Wf)$

What is an expectation?
$\mathbb{E}_{x\sim p(x)} [f(x)] = \sum_{x\in\mathcal{X}}p(x)f(x) \approx \frac{1}{|S|}\sum_{x\in S}f(x)$

The mean estimator:
$W^\mu_c = \sum_{x\in\mathcal{X}}p(x|c)f(x)$

Average Supervised Loss (for $k+1$ classes):
$L_\text{sup}(f) = \mathbb{E}_{\mathcal{T}\subseteq\mathcal{C}:|\mathcal{T}|=k+1 \sim p(\mathcal{T})}[L_\sup(\mathcal{T},f)]$
$p(\mathcal{T}) = \prod_{c\in\mathcal{T}}p(c)$

$\hat{f} = \arg\min_{f\in\mathcal{F}}\hat{L}_\text{un}(f)$