{%hackmd 8nGPjMOiTy-0DWU2xy0q0Q %}

# Categorical Predictors

Resucre: [Stat 501](https://online.stat.psu.edu/stat501/lesson/8)

[Home Page](/_9v1g3C3TXmUfBbxkjnn0A)
[toc]

## Preparation

Category variables can’t directly use, need to be transformed.

### Binary predictor

Example: gender (male, female)

Use an indicator

$$
X_{1i}
= \begin{cases}
1 & \text{ if preson } i \text{ is male} \\
0 & \text{ if preson } i \text{ is female}
\end{cases}
$$

| Gender | | Gender |
| ------ |:--- | ------ |
| M | | 1 |
| F | | 0 |
| F | | 0 |
| ... | | ... |
| M | | 1 |

### Multinomial predictor

Example: educate level (primary school, junior high school, university)

Use indicators, or called *one-hot encoding*

$$
\begin{align*}
X_{1i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from primary school} \\
0 & \text{ otherwise}
\end{cases} \\
X_{2i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from junior high school} \\
0 & \text{ otherwise}
\end{cases} \\
X_{3i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from university} \\
0 & \text{ otherwise}
\end{cases}
\end{align*}
$$

| Educate level | | Primary school | Junior high school | University |
| ------------------ | --- | -------------- | ------------------ |:---------- |
| Primary school | | 1 | 0 | 0 |
| Junior high school | | 0 | 1 | 0 |
| University | | 0 | 0 | 1 |
| ... | | ... | ... | ... |
| Junior high school | | 0 | 1 | 0 |

### Why One-hot Encoding

Solve 3 problem

1. Not binary: educate level (Primary, Junior, University)
2. Not ratio: educate level (University - Junior $\ne$ Junior - Primary)
3. Not ordinal: color (red $\not >$ green)

Create 1 problem

1. Sparse

## **Additive Effects**

### Binary Example

Does education level affect salary?

- Salary $(y)$: dollar (numerical)
- Seniority $(x_1)$: years (numerical)
- University graduate $(x_2)$: true or false (binary)


Consider model:

$$
y_i = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \varepsilon_i
$$


Actually, $\beta_2$ just an intercept

$$
\begin{align*}
y_i
= \begin{cases}
\beta_1 x_{i1} + \beta_0 & \text{ if } x_{i2} = 0 \\
\beta_1 x_{i1} + \beta_0 + \beta_2 & \text{ if } x_{i2} = 1 \\
\end{cases}
\end{align*}
$$


### Multinomial Example

Does education level affect salary?

- Salary $(y)$: dollar (numerical)
- Seniority $(x_1)$: years (numerical)
- Educate level $(x_2)$: university, junior or primary (multinomial)


Make one-hot encoding

$$
\begin{align*}
X_{3i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from primary school} \\
0 & \text{ otherwise}
\end{cases} \\
X_{4i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from junior high school} \\
0 & \text{ otherwise}
\end{cases} \\
X_{5i}
& = \begin{cases}
1 & \text{ if preson } i \text{ graduating from university} \\
0 & \text{ otherwise}
\end{cases}
\end{align*}
$$

Note that,

$$
X_{3i} + X_{4i} + X_{5i} = 1
$$

i.e. $X_5$ is a linear combination of $X_3$ and $X_4$

$$
X_{5i} = 1 - X_{3i} - X_{4i}
$$

Consider model:

$$
\begin{align*}
y_i
& = \beta_0 + \beta_1 x_{i1} + \beta_{3} x_{i3} + \beta_{4} x_{i4} + \varepsilon_i \\
& = \begin{cases}
\beta_1 x_{i1} + \beta_0 & \text{ if } x_{2i} = \text{"primary"} \\
\beta_1 x_{i1} + \beta_0 + \beta_3 & \text{ if } x_{2i} = \text{"junior"} \\
\beta_1 x_{i1} + \beta_0 + \beta_4 & \text{ if } x_{2i} = \text{"university"} \\
\end{cases}
\end{align*}
$$


## Interaction

Does education level affect salary?

- Salary $(y)$: dollar (numerical)
- Seniority $(x_1)$: years (numerical)
- University graduate $(x_2)$: true or false (binary)


Consider

$$
\begin{align*}
y_i
& = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \varepsilon_i \\
& = \begin{cases}
\beta_1 x_{i1} + \beta_0 & \text{ if } x_{i2} = 0 \\
\beta_1 x_{i1} + \beta_0 + \beta_2 & \text{ if } x_{i2} = 1 \\
\end{cases}
\end{align*}
$$


A slope parameter can no longer be interpreted as the change in the mean response.

$$
\begin{align*}
y_i
& = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \beta_{12} x_{i1} x_{i2} + \varepsilon_i \\
& = \begin{cases}
\beta_0 + \beta_1 x_{i1} & \text{ if } x_{i2} = 0 \\
(\beta_0 + \beta_2) + (\beta_1 + \beta_{12}) x_{i1} & \text{ if } x_{i2} = 1 \\
\end{cases}
\end{align*}
$$


### Interaction Interpret

Never interpret the main effect in the presence of an interaction.


## Hypothesis Test

### Limit and Benefit

Check normal assumption



Violate: constant variance



Benefit: sample size increase, easier to pass hypothesis test, and smaller confidence interval.


### One Parameter Test

In model

$$
y_i = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \varepsilon_i
$$


To test:

$$
H : \beta_2 = \beta_{2,0} \quad \text{against} \quad A : \beta_2 \ne \beta_{2,0}
$$

By

$$
t^* = \frac{\beta_2 - \beta_{2,0}}{s \{ \beta_2 \}} \sim t_{n - p}
$$

In significance level $\alpha \in (0, 1)$

- If $t^* \leq t_{n - p} (1 - \alpha)$ , conclude $H$
- If $t^* > t_{n - p} (1 - \alpha)$ , conclude $A$

### Multi Parameter Test

Seem $X_1 X_2 = X_3$ as a new predictor

$$
\begin{align*}
y_i
& = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \beta_{12} x_{i1} x_{i2} + \varepsilon_i \\
& = \beta_0 + \beta_1 x_{i1} + \beta_{2} x_{i2} + \beta_{12} x_{i3} + \varepsilon_i
\end{align*}
$$


To test:

$$
\begin{align*}
& H : \beta_2 = \beta_{12} = 0 \quad \text{(Redce model)} \\
& A : \text{ not all } \beta_i = 0 \quad \text{(Full model)}
\end{align*}
$$

By general linear approach

$$
F^*
= \frac{SSE_R - SSE_F}{df_R - df_F} / \frac{SSE_F}{df_F}
\sim F_{df_R - df_F, df_F}
$$

In significance level $\alpha \in (0, 1)$

- If $F^* \leq F_{df_R - df_F, df_F} (1 - \alpha)$ , conclude $H$
- If $F^* > F_{df_R - df_F, df_F} (1 - \alpha)$ , conclude $A$

## Piece Wise Linear Regression Models

In classification and regression trees (CART) sense, need to find a good separation.

### Intercept Piece Wise

Need to determined $c$

$$
x_{i2} = I (x_{i1} > c)
$$

Only consider intercept term

$$
\begin{align*}
y_i
& = \beta_0 + \beta_2 x_{i2} \\
& = \begin{cases}
\beta_0 & \text{ if } x_{i1} \leq c \\
(\beta_0 + \beta_2) & \text{ if } x_{i1} > c \\
\end{cases}
\end{align*}
$$


### Linear Piece Wise

Data is piecewise linear. Need determined $c$ that separate into two regression line

$$
x_{i2} = I (x_{i1} > c)
$$

Fitted by

$$
\begin{align*}
y_i
& = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_{12} x_{i1} x_{i2} \\
& = \begin{cases}
\beta_0 + \beta_1 x_{i1} & \text{ if } x_{i1} \leq c \\
(\beta_0 + \beta_2) + (\beta_1 + \beta_{12}) x_{i1} & \text{ if } x_{i1} > c \\
\end{cases}
\end{align*}
$$


[Home Page](/_9v1g3C3TXmUfBbxkjnn0A)
[toc]