# 4.4 Validating a model

## Introduction

Understanding your model's reality is only one step in determining the usefulness of a model.

How well does your model's reality align to the data's reality? Does the model explain the key patterns in the data, or is there variability in the data that the model does not capture well?

Recall why we are modelling. We want to learn something that we can general _beyond the observation space_ (i.e. the data) to the _true phenomenon_.

> **Fitting is easy, Prediction is hard** (Mcelreath)

We have fitted to the observation space. We want to predict about the true phenomenon.

This important notion underlies most model evaluation.

Model evaluation has two stages: assessing the **fit**, and assessing the usefulness of the **predictions**.

### Assessing the fit

First we assess the extent to which the model is fit to the data. Essentially how we do this is to predict the dataset and assess how well the model predictions match up to the real-values.

- Residuals
- Q-Q plots.

Your model has difference sources of uncertainty:
- known (and maybe modelled) measurement uncertainty (i.e. expected spread of residuals)
- parameter fitting uncertainty
- in multi-level models you also have further between group uncertainty.

### Assessing Predictions

In the previous section when assessing the fit we are not necessarily looking for 100% replication of the dataset. There is a balance to be struck between _model complexity_ vs _out-of-sample prediction_.

The more complex a model is the more flexiblity it has to mould itself to data. However, the higher the risk that you may be fitting to the pecularities of that particular sample rather than fitting to the data generating process of the _true phenomenon_.

In machine learning this trade-off is called the bias-variance trade-off. High bias is when your model is too simple and does not capture important information in your data. High variance is when your model is too complex and fitting to your data set, so the parameter values vary across different datasets.

#### Regularization

A key concept in overfitting is regularization. Regularization is an umbrella term for a bunch of different ways to dampen the sensitivity of your parameters to your data set. Common ways of regularizing is to add an additional penalising term into your cost function that increases cost when there are multiple parameters each iwth high-valued coefficients.

<!-- Note on regularising priors -->

#### Assessing out-of-sample error

- Cross validation
- Simulations