Quantile Regression

A simple method to estimate uncertainty in Machine Learning

Why estimate uncertainty?

• Get bounds for the data.
• Estimate the distribution of the output.
• Reduce Risk.

Problem

Problem

1. It is not normally distributed.
2. Noise it not symetric.
3. Its variance is not constant.

Solution

Estimate uncertainty by predicting the
quantiles of \(y\) given \(x\).

Quantile Loss

\[ \begin{aligned} E &= y - f(x) \\ L_q &= \begin{cases} q E, & E \gt 0 \\ (1 - q) (-E), & E \lt 0 \end{cases} \end{aligned} \]

Quantile Loss

\[ \begin{aligned} E &= y - f(x) \\ L_q &= \max \begin{cases} q E \\ (q - 1) E \end{cases} \end{aligned} \]

JAX Implementation

    def quantile_loss(q, y_true, y_pred):
        e = y_true - y_pred
        return jnp.maximum(q * e, (q - 1.0) * e)

Loss landscape for a continous sequence of y_true values between [10, 20].

    class QuantileRegression(elegy.Module):
        def __init__(self, n_quantiles: int):
            super().__init__()
            self.n_quantiles = n_quantiles

        def call(self, x):
            x = elegy.nn.Linear(128)(x)
            x = jax.nn.relu(x)
            x = elegy.nn.Linear(64)(x)
            x = jax.nn.relu(x)
            x = elegy.nn.Linear(self.n_quantiles)(x)
            
            return x

Recap

• Quantile Regression: simple and effective.
• Use when risk management is needed.
• Neural Networks are an efficient way to predict multiple quantiles.
• With sufficient quantiles you can approximate the density function.

Next Steps

• Check out the blog and repo
  • Blog: BLOG_URL
  • Repo: cgarciae/quantile-regression
• Take a look at Mixture Density Networks.
• Learn more about jax and elegy.