# Inverse Transform Sampling

The goal is to sample $x$ from some distribution $X$ with CDF $F_X$. This is useful when $X$ is not the standard distributions (uniform, normal...) that we can easily sample from. The idea is to use what we can sample from to generate samples of $x$.

To do this, we need to know $F^{-1}_X$, the inverse CDF of $X$. To generate a random sample of $X$, we can simply do $u \sim U(0,1)$ then we get a sanmple $x = F^{-1}_X(u)$.

To transform a random variable $Y$ into $X$, we will need its distribution function $F_Y$. Then, we have that

$$F^{-1}_X(F_Y(y)) \sim X \quad \text{for} \quad y \sim Y.$$

Note that intuitively this checks out as the value of $F_Y(y)$ is always between 0 and 1. Therefore, $F^{-1}_X$ simply inverts that CDF value into whatever domain that is supported by $X$.

## Method

1. Sample $y \sim Y$.
2. Generate $x$ with $x=F^{-1}_X(F_Y(y)).$

## Example: Uniform to Beta

The Beta distribution has 2 shape parameters $a$ and $b$. Unfortunately, there's no analytic form for general $a$ and $b$, so we will focus on some [special cases](https://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function).

- When $b = 1$, then CDF is $u = F(x;a,b=1) = x^a$ and the inverse CDF is $F^{-1}(u) = u^{1/a}.$
- When $a = 1$, then CDF is $u = F(x;a=1) = 1-(1-x)^b$ and the inverse CDF is $F^{-1}(u) = 1-(1-u)^{1/b}.$
- When $a=b=1$, then $F(x)=x$ and $F^{-1}(u)=u$.

Let's take the case of $b=1$ and $a=5$. We choose our simple distribution to be the uniform distribution $U(0,1)$. We sample $u \sim U(0,1)$ and since CDF $F_U(u) = u$ for this particular case, our Beta sample is $u^{1/a} = u^{1/5}$.

## Example: Whatever to Cauchy

We write the Cauchy CDF as $F_X(x) =F_X(x;x_0,\gamma)$ where $x_0$ and $\gamma$ are location and scale parameters. Let's just show the CDF and inverse CDF of Cauchy

$$ F_X(x) = \frac{1}{\pi} \arctan \Big(\frac{x-x_0}{\gamma} \Big) + \frac{1}{2},$$

$$ F_X^{-1}(u) = x_o + \gamma \tan \Big[\pi(u-\frac{1}{2}) \Big].$$

## What about truncated distributions?

Instead of using $U(0,1)$. We can simply use $U(F_X(a), F_X(b))$ to generate samples $x \sim X$ such that $x \in (a,b]$.

This is from Wikipedia too and I guess this means for a truncated distribution we can truncate $Y$ too. This means this method does not solve the issue of a truncated domain/support when it comes to MCMC.

## MCMC Sampling

For example, in MCMC sampling this can be useful as sampling distributions with fat tails (e.g. Cauchy) can be problematic -- we can sample something else and then transform it into a Cauchy.

However, for MCMC we want our ($d$-dimensional) target distribution to be supported on the $\mathbb{R}^d$. Therefore, using the uniform distribution as $Y$ does not work because it is only supported on an interval. A natural solution is to use the normal distribution, where we require the computation of its the CDF.