4.2 Building a simple model

Introduction

For the remainder of this module we will use the the European Quality of Life Time Series (EQLTS) dataset to ilustrate the steps and considerations to have in mind when building a simple module. More details about this dataset can be found in here.

As discussed in Module 1, we want to investigate the contribution of material, occupational, and psychosocial factors on the self reported health (SRH) across different European countries. In this particular excersice we will focus on the data from the UK taken for Wave 3.

In the survey self reported health was assessed using the following question: “In general, would you say your health is …” and response categories were “very good”, “good”, “fair”, and “bad” and "very bad". For simplicity, we will follow the strategy from (Aldabe et al.) of dicotomising the variable as “good” (“very good”, “good”, “fair”) health versus “bad” health (“bad” and “very bad”).

In figure X you can see the frequency distribution of the answers that individuals from the UK provided Wave 3 of the survey.

TODO: DEPENDING IN HOW THE FIGURE LOOK ADD TEXT ON THE ADVANTAGE OF DICHOTOMISING.

Models

To the goal of our stury is to capture the relationships that certain socialfactors have with our target variable, the self reported health. A straight forward way of doing this is to think in
a multiple linear regression which attempts to model the relationship between two or more explanatory variables and a response variable by fitting a linear equation to observed data.

In a multiple linear regression every value of the \(n\) independent variables is associated with a value of the dependent variable in the following way:

\[ Y = \beta_0 + \beta_1 x_{1} + \beta_2 x_{2} + ...+ \beta_q x_{q} + \epsilon \]

where \(Y\) is the value of the dependent variable, \(\beta_0\) is the intercept, the coefficients \(\beta_1\), \(\beta_2\), \(\beta_3\) and \(\beta_q\) describe the mathematical relationship between each independent variable and the dependent variable are to be estimated and \(\epsilon\) is used to account for error and is a normal random variables with mean \(0\) and variance \(\sigma^2\).

In our particular case our target variable is an ordered categorical variable that we decided to dichotomise, but we we'll get to this later when we introduce logistic regression. For now, let's use a simple toy dataset to explore further how to fit a MLR module and how to assess its quality.

A toy example: Polynomial Curve Fitting from Bishops

A python demo of bishops that we can use for this section https://mantas.info/wp/wp-content/uploads/learn/curve-over-fitting.html

• high-level overview of how models are fitted (e.g. wrt some cost function).
  • Cost of error function
  • choosing the order M of the polynomial, and as we shall see this will turn out to be an example of an important concept called model comparison or model selection.
  • Model complexity and over fitting
  • By adopting a Bayesian approach, the over-fitting problem can be avoided?
  • Colinearity

From linear regression to logistic regression

The previous sections have given you an idea of the fundamental concepts of building and testing a linear model regresion model, but the problem we are trying to solve with our research question is not a regression problem but a classification one. We want to predict a binary dependent variable that has 2 values: either 1 or 0.

\[ Y = \begin{cases} 1 & \text{good health} \\ 0 & \text{bad health} \end{cases} \]

This kind of binary choices are represented by a Bernoulli variable where the probabilities of an event belonging to a given class are bounded on both ends (they must be between 0 and 1). In this case we'll mostly focus on the case when \(Y = 1\) (good health), defined with the following way:

\[ p({\bf x}) = P[Y = 1 \mid {\bf X} = {\bf x}] \]

Our usual linear regression can deals continuous variables instead of Bernoulli variables, which means that the predictions are not sensible for classification since the true probability must fall between 0 and 1, but the prediction from linear regression it can take any continous values incluing larger than 1 or smaller than 0 as seeen in the Figure X.

Figure from here[https://www.javatpoint.com/linear-regression-vs-logistic-regression-in-machine-learning

The solution to this issue (and a way to continously using a linear, understandable model) is to transform the linear regression to a logistic regression curve. This means feeding the the predicted values into a sigmoid function, thus converting them into probability as shown next:

In order to see this more formaly lets define now the logistic regression model.

\[ \log\left(\frac{p({\bf x})}{1 - p({\bf x})}\right) = \beta_0 + \beta_1 x_1 + \ldots + \beta_{p - 1} x_{p - 1} \]

The left hand side is called the log odds, which is the log of the odds. The odds are the probability for a positive event \((Y = 1)\) divided by the probability of a negative event \((Y = 0)\). So when the odds are \(1\), the two events have equal probability. Odds greater than \(1\) favor a positive event. The opposite is true when the odds are less than \(1\).

Essentially, the log odds are the logit transform applied to \(p({\bf x})\).

\[ \text{logit}(\xi) = \log\left(\frac{\xi}{1 - \xi}\right) \]

The inverse logit, otherwise known as the "logistic" or sigmoid function is defined as follows:

\[ \text{logit}^{-1}(\xi) = \frac{e^\xi}{1 + e^{\xi}} = \frac{1}{1 + e^{-\xi}} \]

Note that for \(x \in (-\infty, \infty))\), this function outputs values between 0 and 1.

Where is the error term? Notice that the linear regression model with the error term,

\[ Y = \beta_0 + \beta_1x_1 + \ldots + \beta_qx_q + \epsilon, \ \ \epsilon \sim N(0, \sigma^2) \]
can instead be written as

\[ Y \mid {\bf X} = {\bf x} \sim N(\beta_0 + \beta_1x_1 + \ldots + \beta_qx_q, \ \sigma^2). \]

While our main focus is on estimating the mean, \(\beta_0 + \beta_1x_1 + \ldots + \beta_qx_q\), there is also another parameter, \(\sigma^2\) which needs to be estimated. With logistic regression, which uses the Bernoulli distribution, we only need to estimate the Bernoulli distribution's single parameter \(p({\bf x})\), which happens to be its mean.

\[ \log\left(\frac{p({\bf x})}{1 - p({\bf x})}\right) = \beta_0 + \beta_1 x_1 + \ldots + \beta_{q} x_{q} \]

So even though we introduced ordinary linear regression first, in some ways, logistic regression is actually simpler.

Note that applying the inverse logit transformation allow us to obtain an expression for \(p({\bf x})\).

\[ p({\bf x}) = P[Y = 1 \mid {\bf X} = {\bf x}] = \frac{e^{\beta_0 + \beta_1 x_{1} + \cdots + \beta_{p-1} x_{(p-1)}}}{1 + e^{\beta_0 + \beta_1 x_{1} + \cdots + \beta_{p-1} x_{(p-1)}}} \]

Practical tought session

Now that we have all the theoretical basis covered, we can start getting with our dataset ready for modeling. As discussed in Module 2, missing data is an important step that needs to solved before building a model.

A strategy to deal with missing data is to discard entire rows and/or columns containing missing values. However, this comes with a price os losing data and predictive power or further distorting our sample if the data is not missing completly at random (this could have some serious ethical implications e.g What gets counted counts TODO: Find others). A better strategy is to do data imputation of the missing values, i.e., to infer them from the known part of the data.

Data pipelines allow one to transform data from one representation to another through a series of steps. Pipelines allow one to apply and chain intermediate steps of transform to our data. For example, one can fill missing values, pass the output to cross validation and grid search and then fit the model in series of steps chained together where the output of one is the input to another.

Preparing our data for modeling

• Simple imputation of missing values
• Data wrangling step but has such an effect on modelling, and has ethical implications (e.g. persisting dominant norms but imputing the mean or most common values), that it should be considered a key part of the model.
  • Note that some models (bayesian) have the imputation step explicitly stated within it.

Fitting a logistic regression

• Turining our regression model to solve a classification problem.
• First fit of out data
• First tipical output of the fit: MSE and classification matrix for test sample.
• This move us to 4.3

Refereces

Aldabe, et al. (2011). Contribution of material, occupational, and psychosocial factors in the explanation of social inequalities in health in 28 countries in Europe. Journal of epidemiology and community health, 65(12), 1123–1131. https://doi.org/10.1136/jech.2009.102517

Logistic regresssion discussion from https://daviddalpiaz.github.io/appliedstats/logistic-regression.html#binary-response