---
# System prepended metadata

title: ISLR hw6
tags: [ISLR]

---

---
tags: ISLR
---

# ISLR hw6

Range: 「課本第三章習題：第2題、第14題、第15題、第四章習題：第3題、第5題」
          
## ch03 Q2

classifier search k nearest-neighbor "voted" to output the answer
regression search k nearest-neighbor "calculate mean" to output the anser

## Q14

```r
set.seed(1)
x1 <- runif(100)
x2 <- 0.5*x1 + rnorm(100) / 10
y <- 2 + 2*x1 + 0.3*x2 + rnorm(100)
```

### a

$y = \beta_0 + \beta_1 * x_1 + \beta_2 * x_2 + c$

Regression coefficients is $\beta_0 = 2$, $\beta_1 = 2$ and $\beta_2 = 0.3$

### b

```r
cor(x1, x2)

# [1] 0.8351212
```

![](https://i.imgur.com/QuSqNxl.png)

### c
```r
x1x2_lm <- lm(y ~ x1 + x2)
summary(x1x2_lm)

# Call:
# lm(formula = y ~ x1 + x2)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -2.8311 -0.7273 -0.0537  0.6338  2.3359 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   2.1305     0.2319   9.188 7.61e-15 ***
# x1            1.4396     0.7212   1.996   0.0487 *  
# x2            1.0097     1.1337   0.891   0.3754    
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 1.056 on 97 degrees of freedom
# Multiple R-squared:  0.2088,	Adjusted R-squared:  0.1925 
# F-statistic:  12.8 on 2 and 97 DF,  p-value: 1.164e-05
```

estimated $\beta_0$ is 2.1305, $\beta_1$ is 1.4396 and $\beta_2$ is 1.0097

some of they is close with true coefficients

We can reject null hypothesis of $\beta_1$
We cannot reject null hypothesis of $\beta_2$

### d
```r
x1_lm <- lm(y ~ x1)
summary(x1_lm)

# Call:
# lm(formula = y ~ x1)

# Residuals:
#      Min       1Q   Median       3Q      Max 
# -2.89495 -0.66874 -0.07785  0.59221  2.45560 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   2.1124     0.2307   9.155 8.27e-15 ***
# x1            1.9759     0.3963   4.986 2.66e-06 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 1.055 on 98 degrees of freedom
# Multiple R-squared:  0.2024,	Adjusted R-squared:  0.1942 
# F-statistic: 24.86 on 1 and 98 DF,  p-value: 2.661e-06
```
Yes, because p of x1 coef is < 0.05
We can reject null hypothesis of $\beta_1$

### e
```r
x2_lm <- lm(y ~ x2)
summary(x2_lm)

# Call:
# lm(formula = y ~ x2)

# Residuals:
#      Min       1Q   Median       3Q      Max 
# -2.62687 -0.75156 -0.03598  0.72383  2.44890 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   2.3899     0.1949   12.26  < 2e-16 ***
# x2            2.8996     0.6330    4.58 1.37e-05 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 1.072 on 98 degrees of freedom
# Multiple R-squared:  0.1763,	Adjusted R-squared:  0.1679 
# F-statistic: 20.98 on 1 and 98 DF,  p-value: 1.366e-05
```

Yes, because p of x2 coef is < 0.05
We can reject null hypothesis of $\beta_1$

### f

No. In c. and d. we notice there are high correlation between x1 and x2.

d. and e. show that x1 and x2 have statistically evidence to reject null hypothesis

c. shows that x1 has more confidence in statistic than x2

### g
```r
x1 <- c(x1, 0.1)
x2 <- c(x2, 0.8)
y <- c(y,6)
```

```r
# c plot
x1x2_lm = lm(y ~ x1 + x2)
summary(x1x2_lm)

# Call:
# lm(formula = y ~ x1 + x2)

# Residuals:
#      Min       1Q   Median       3Q      Max 
# -2.73348 -0.69318 -0.05263  0.66385  2.30619 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   2.2267     0.2314   9.624 7.91e-16 ***
# x1            0.5394     0.5922   0.911  0.36458    
# x2            2.5146     0.8977   2.801  0.00614 ** 
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 1.075 on 98 degrees of freedom
# Multiple R-squared:  0.2188,	Adjusted R-squared:  0.2029 
# F-statistic: 13.72 on 2 and 98 DF,  p-value: 5.564e-06
```

Now x2 is significant, x1 is not.

d plot and e plot keep the same(skip)

replot the `plot(x1, x2)`
![](https://i.imgur.com/n4pE9Kh.png)

run `plot(x1x2_lm)`
![](https://i.imgur.com/5haZ1dQ.png)


Yes, it looks like outlier and high-leverage point.

## Q15

### a
```r
colnames(Boston)
lapply(colnames(Boston), FUN = function(p) summary(lm(Boston$crim ~ Boston[,p])))

#  [1] "crim"    "zn"      "indus"   "chas"    "nox"     "rm"      "age"     "dis"     "rad"    
# [10] "tax"     "ptratio" "black"   "lstat"   "medv"   
# > lapply(colnames(Boston), FUN = function(p) summary(lm(Boston$crim ~ Boston[,p])))
# [[1]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#        Min         1Q     Median         3Q        Max 
# -3.074e-15 -2.120e-16 -7.000e-17  7.200e-17  4.023e-14 

# Coefficients:
#              Estimate Std. Error   t value Pr(>|t|)    
# (Intercept) 1.263e-15  8.895e-17 1.420e+01   <2e-16 ***
# Boston[, p] 1.000e+00  9.542e-18 1.048e+17   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 1.844e-15 on 504 degrees of freedom
# Multiple R-squared:      1,	Adjusted R-squared:      1 
# F-statistic: 1.098e+34 on 1 and 504 DF,  p-value: < 2.2e-16


# [[2]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -4.429 -4.222 -2.620  1.250 84.523 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  4.45369    0.41722  10.675  < 2e-16 ***
# Boston[, p] -0.07393    0.01609  -4.594 5.51e-06 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.435 on 504 degrees of freedom
# Multiple R-squared:  0.04019,	Adjusted R-squared:  0.03828 
# F-statistic:  21.1 on 1 and 504 DF,  p-value: 5.506e-06


# [[3]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -11.972  -2.698  -0.736   0.712  81.813 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -2.06374    0.66723  -3.093  0.00209 ** 
# Boston[, p]  0.50978    0.05102   9.991  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.866 on 504 degrees of freedom
# Multiple R-squared:  0.1653,	Adjusted R-squared:  0.1637 
# F-statistic: 99.82 on 1 and 504 DF,  p-value: < 2.2e-16


# [[4]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -3.738 -3.661 -3.435  0.018 85.232 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   3.7444     0.3961   9.453   <2e-16 ***
# Boston[, p]  -1.8928     1.5061  -1.257    0.209    
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.597 on 504 degrees of freedom
# Multiple R-squared:  0.003124,	Adjusted R-squared:  0.001146 
# F-statistic: 1.579 on 1 and 504 DF,  p-value: 0.2094


# [[5]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -12.371  -2.738  -0.974   0.559  81.728 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  -13.720      1.699  -8.073 5.08e-15 ***
# Boston[, p]   31.249      2.999  10.419  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.81 on 504 degrees of freedom
# Multiple R-squared:  0.1772,	Adjusted R-squared:  0.1756 
# F-statistic: 108.6 on 1 and 504 DF,  p-value: < 2.2e-16


# [[6]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -6.604 -3.952 -2.654  0.989 87.197 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   20.482      3.365   6.088 2.27e-09 ***
# Boston[, p]   -2.684      0.532  -5.045 6.35e-07 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.401 on 504 degrees of freedom
# Multiple R-squared:  0.04807,	Adjusted R-squared:  0.04618 
# F-statistic: 25.45 on 1 and 504 DF,  p-value: 6.347e-07


# [[7]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -6.789 -4.257 -1.230  1.527 82.849 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -3.77791    0.94398  -4.002 7.22e-05 ***
# Boston[, p]  0.10779    0.01274   8.463 2.85e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.057 on 504 degrees of freedom
# Multiple R-squared:  0.1244,	Adjusted R-squared:  0.1227 
# F-statistic: 71.62 on 1 and 504 DF,  p-value: 2.855e-16


# [[8]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -6.708 -4.134 -1.527  1.516 81.674 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   9.4993     0.7304  13.006   <2e-16 ***
# Boston[, p]  -1.5509     0.1683  -9.213   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.965 on 504 degrees of freedom
# Multiple R-squared:  0.1441,	Adjusted R-squared:  0.1425 
# F-statistic: 84.89 on 1 and 504 DF,  p-value: < 2.2e-16


# [[9]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -10.164  -1.381  -0.141   0.660  76.433 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -2.28716    0.44348  -5.157 3.61e-07 ***
# Boston[, p]  0.61791    0.03433  17.998  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 6.718 on 504 degrees of freedom
# Multiple R-squared:  0.3913,	Adjusted R-squared:   0.39 
# F-statistic: 323.9 on 1 and 504 DF,  p-value: < 2.2e-16


# [[10]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -12.513  -2.738  -0.194   1.065  77.696 

# Coefficients:
#              Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -8.528369   0.815809  -10.45   <2e-16 ***
# Boston[, p]  0.029742   0.001847   16.10   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 6.997 on 504 degrees of freedom
# Multiple R-squared:  0.3396,	Adjusted R-squared:  0.3383 
# F-statistic: 259.2 on 1 and 504 DF,  p-value: < 2.2e-16


# [[11]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -7.654 -3.985 -1.912  1.825 83.353 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -17.6469     3.1473  -5.607 3.40e-08 ***
# Boston[, p]   1.1520     0.1694   6.801 2.94e-11 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.24 on 504 degrees of freedom
# Multiple R-squared:  0.08407,	Adjusted R-squared:  0.08225 
# F-statistic: 46.26 on 1 and 504 DF,  p-value: 2.943e-11


# [[12]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -13.756  -2.299  -2.095  -1.296  86.822 

# Coefficients:
#              Estimate Std. Error t value Pr(>|t|)    
# (Intercept) 16.553529   1.425903  11.609   <2e-16 ***
# Boston[, p] -0.036280   0.003873  -9.367   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.946 on 504 degrees of freedom
# Multiple R-squared:  0.1483,	Adjusted R-squared:  0.1466 
# F-statistic: 87.74 on 1 and 504 DF,  p-value: < 2.2e-16


# [[13]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -13.925  -2.822  -0.664   1.079  82.862 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -3.33054    0.69376  -4.801 2.09e-06 ***
# Boston[, p]  0.54880    0.04776  11.491  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.664 on 504 degrees of freedom
# Multiple R-squared:  0.2076,	Adjusted R-squared:  0.206 
# F-statistic:   132 on 1 and 504 DF,  p-value: < 2.2e-16


# [[14]]

# Call:
# lm(formula = Boston$crim ~ Boston[, p])

# Residuals:
#    Min     1Q Median     3Q    Max 
# -9.071 -4.022 -2.343  1.298 80.957 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) 11.79654    0.93419   12.63   <2e-16 ***
# Boston[, p] -0.36316    0.03839   -9.46   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.934 on 504 degrees of freedom
# Multiple R-squared:  0.1508,	Adjusted R-squared:  0.1491 
# F-statistic: 89.49 on 1 and 504 DF,  p-value: < 2.2e-16
```

all parameters except chas has significant with crime.rate 

### b
```r
summary(lm(crim ~ ., data = Boston))

# Call:
# lm(formula = crim ~ ., data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -9.924 -2.120 -0.353  1.019 75.051 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  17.033228   7.234903   2.354 0.018949 *  
# zn            0.044855   0.018734   2.394 0.017025 *  
# indus        -0.063855   0.083407  -0.766 0.444294    
# chas         -0.749134   1.180147  -0.635 0.525867    
# nox         -10.313535   5.275536  -1.955 0.051152 .  
# rm            0.430131   0.612830   0.702 0.483089    
# age           0.001452   0.017925   0.081 0.935488    
# dis          -0.987176   0.281817  -3.503 0.000502 ***
# rad           0.588209   0.088049   6.680 6.46e-11 ***
# tax          -0.003780   0.005156  -0.733 0.463793    
# ptratio      -0.271081   0.186450  -1.454 0.146611    
# black        -0.007538   0.003673  -2.052 0.040702 *  
# lstat         0.126211   0.075725   1.667 0.096208 .  
# medv         -0.198887   0.060516  -3.287 0.001087 ** 
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 6.439 on 492 degrees of freedom
# Multiple R-squared:  0.454,	Adjusted R-squared:  0.4396 
# F-statistic: 31.47 on 13 and 492 DF,  p-value: < 2.2e-16
```

We can reject null hypothesis with zn, dis, rad, black and medv

### c
```r
uni_lm <- lapply(colnames(Boston)[-1], FUN = function(p) lm(Boston$crim ~ Boston[,p])$coefficients[2])
mul_lm <- lm(crim ~ ., data = Boston)$coefficients[-1] # remove beta_0
plot(uni_lm, mul_lm)
```
![](https://i.imgur.com/bdyDYjv.png)


### d

```r
lm_formula <- function(p) {
  crim_formula <- as.formula(paste("crim ~ ", p, " + I(", p, "^2)", " + I(", p, "^3)"))
  return(summary(lm(crim_formula, data = Boston)))
}
lapply(colnames(Boston), FUN = lm_formula)

# [[1]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -12.420  -1.403  -1.249   1.071  13.763 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  1.4813187  0.1170849   12.65   <2e-16 ***
# I(crim^2)    0.0421933  0.0009366   45.05   <2e-16 ***
# I(crim^3)   -0.0003695  0.0000122  -30.30   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 2.519 on 503 degrees of freedom
# Multiple R-squared:  0.9146,	Adjusted R-squared:  0.9143 
# F-statistic:  2694 on 2 and 503 DF,  p-value: < 2.2e-16


# [[2]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -4.821 -4.614 -1.294  0.473 84.130 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  4.846e+00  4.330e-01  11.192  < 2e-16 ***
# zn          -3.322e-01  1.098e-01  -3.025  0.00261 ** 
# I(zn^2)      6.483e-03  3.861e-03   1.679  0.09375 .  
# I(zn^3)     -3.776e-05  3.139e-05  -1.203  0.22954    
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.372 on 502 degrees of freedom
# Multiple R-squared:  0.05824,	Adjusted R-squared:  0.05261 
# F-statistic: 10.35 on 3 and 502 DF,  p-value: 1.281e-06


# [[3]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -8.278 -2.514  0.054  0.764 79.713 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  3.6625683  1.5739833   2.327   0.0204 *  
# indus       -1.9652129  0.4819901  -4.077 5.30e-05 ***
# I(indus^2)   0.2519373  0.0393221   6.407 3.42e-10 ***
# I(indus^3)  -0.0069760  0.0009567  -7.292 1.20e-12 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.423 on 502 degrees of freedom
# Multiple R-squared:  0.2597,	Adjusted R-squared:  0.2552 
# F-statistic: 58.69 on 3 and 502 DF,  p-value: < 2.2e-16


# [[4]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -3.738 -3.661 -3.435  0.018 85.232 

# Coefficients: (2 not defined because of singularities)
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   3.7444     0.3961   9.453   <2e-16 ***
# chas         -1.8928     1.5061  -1.257    0.209    
# I(chas^2)         NA         NA      NA       NA    
# I(chas^3)         NA         NA      NA       NA    
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.597 on 504 degrees of freedom
# Multiple R-squared:  0.003124,	Adjusted R-squared:  0.001146 
# F-statistic: 1.579 on 1 and 504 DF,  p-value: 0.2094


# [[5]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -9.110 -2.068 -0.255  0.739 78.302 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)   233.09      33.64   6.928 1.31e-11 ***
# nox         -1279.37     170.40  -7.508 2.76e-13 ***
# I(nox^2)     2248.54     279.90   8.033 6.81e-15 ***
# I(nox^3)    -1245.70     149.28  -8.345 6.96e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.234 on 502 degrees of freedom
# Multiple R-squared:  0.297,	Adjusted R-squared:  0.2928 
# F-statistic: 70.69 on 3 and 502 DF,  p-value: < 2.2e-16


# [[6]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -18.485  -3.468  -2.221  -0.015  87.219 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)  
# (Intercept) 112.6246    64.5172   1.746   0.0815 .
# rm          -39.1501    31.3115  -1.250   0.2118  
# I(rm^2)       4.5509     5.0099   0.908   0.3641  
# I(rm^3)      -0.1745     0.2637  -0.662   0.5086  
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.33 on 502 degrees of freedom
# Multiple R-squared:  0.06779,	Adjusted R-squared:  0.06222 
# F-statistic: 12.17 on 3 and 502 DF,  p-value: 1.067e-07


# [[7]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -9.762 -2.673 -0.516  0.019 82.842 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)   
# (Intercept) -2.549e+00  2.769e+00  -0.920  0.35780   
# age          2.737e-01  1.864e-01   1.468  0.14266   
# I(age^2)    -7.230e-03  3.637e-03  -1.988  0.04738 * 
# I(age^3)     5.745e-05  2.109e-05   2.724  0.00668 **
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.84 on 502 degrees of freedom
# Multiple R-squared:  0.1742,	Adjusted R-squared:  0.1693 
# F-statistic: 35.31 on 3 and 502 DF,  p-value: < 2.2e-16


# [[8]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -10.757  -2.588   0.031   1.267  76.378 

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  30.0476     2.4459  12.285  < 2e-16 ***
# dis         -15.5543     1.7360  -8.960  < 2e-16 ***
# I(dis^2)      2.4521     0.3464   7.078 4.94e-12 ***
# I(dis^3)     -0.1186     0.0204  -5.814 1.09e-08 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.331 on 502 degrees of freedom
# Multiple R-squared:  0.2778,	Adjusted R-squared:  0.2735 
# F-statistic: 64.37 on 3 and 502 DF,  p-value: < 2.2e-16


# [[9]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -10.381  -0.412  -0.269   0.179  76.217 

# Coefficients:
#              Estimate Std. Error t value Pr(>|t|)
# (Intercept) -0.605545   2.050108  -0.295    0.768
# rad          0.512736   1.043597   0.491    0.623
# I(rad^2)    -0.075177   0.148543  -0.506    0.613
# I(rad^3)     0.003209   0.004564   0.703    0.482

# Residual standard error: 6.682 on 502 degrees of freedom
# Multiple R-squared:    0.4,	Adjusted R-squared:  0.3965 
# F-statistic: 111.6 on 3 and 502 DF,  p-value: < 2.2e-16


# [[10]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -13.273  -1.389   0.046   0.536  76.950 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)
# (Intercept)  1.918e+01  1.180e+01   1.626    0.105
# tax         -1.533e-01  9.568e-02  -1.602    0.110
# I(tax^2)     3.608e-04  2.425e-04   1.488    0.137
# I(tax^3)    -2.204e-07  1.889e-07  -1.167    0.244

# Residual standard error: 6.854 on 502 degrees of freedom
# Multiple R-squared:  0.3689,	Adjusted R-squared:  0.3651 
# F-statistic:  97.8 on 3 and 502 DF,  p-value: < 2.2e-16


# [[11]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#    Min     1Q Median     3Q    Max 
# -6.833 -4.146 -1.655  1.408 82.697 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)   
# (Intercept)  477.18405  156.79498   3.043  0.00246 **
# ptratio      -82.36054   27.64394  -2.979  0.00303 **
# I(ptratio^2)   4.63535    1.60832   2.882  0.00412 **
# I(ptratio^3)  -0.08476    0.03090  -2.743  0.00630 **
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 8.122 on 502 degrees of freedom
# Multiple R-squared:  0.1138,	Adjusted R-squared:  0.1085 
# F-statistic: 21.48 on 3 and 502 DF,  p-value: 4.171e-13


# [[12]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -13.096  -2.343  -2.128  -1.439  86.790 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  1.826e+01  2.305e+00   7.924  1.5e-14 ***
# black       -8.356e-02  5.633e-02  -1.483    0.139    
# I(black^2)   2.137e-04  2.984e-04   0.716    0.474    
# I(black^3)  -2.652e-07  4.364e-07  -0.608    0.544    
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.955 on 502 degrees of freedom
# Multiple R-squared:  0.1498,	Adjusted R-squared:  0.1448 
# F-statistic: 29.49 on 3 and 502 DF,  p-value: < 2.2e-16


# [[13]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -15.234  -2.151  -0.486   0.066  83.353 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)  
# (Intercept)  1.2009656  2.0286452   0.592   0.5541  
# lstat       -0.4490656  0.4648911  -0.966   0.3345  
# I(lstat^2)   0.0557794  0.0301156   1.852   0.0646 .
# I(lstat^3)  -0.0008574  0.0005652  -1.517   0.1299  
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 7.629 on 502 degrees of freedom
# Multiple R-squared:  0.2179,	Adjusted R-squared:  0.2133 
# F-statistic: 46.63 on 3 and 502 DF,  p-value: < 2.2e-16


# [[14]]

# Call:
# lm(formula = crim_formula, data = Boston)

# Residuals:
#     Min      1Q  Median      3Q     Max 
# -24.427  -1.976  -0.437   0.439  73.655 

# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept) 53.1655381  3.3563105  15.840  < 2e-16 ***
# medv        -5.0948305  0.4338321 -11.744  < 2e-16 ***
# I(medv^2)    0.1554965  0.0171904   9.046  < 2e-16 ***
# I(medv^3)   -0.0014901  0.0002038  -7.312 1.05e-12 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Residual standard error: 6.569 on 502 degrees of freedom
# Multiple R-squared:  0.4202,	Adjusted R-squared:  0.4167 
# F-statistic: 121.3 on 3 and 502 DF,  p-value: < 2.2e-16
```

There are statistically siginificent in indus, nox, age, dis, pratio and medv.


## ch04 Q03

from 4.11
$$
p_k(x)=\frac{\pi_k\frac{1}{\sqrt{2\pi}\delta_k}exp(-\frac{1}{2\delta^2}(x-\mu_k)^2)}{\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2\delta_k^2}(x-\mu_l)^2)}
$$
LaTeX refrence from website

then we apply log into it

$$
p1 = log(\pi_k)log(\frac{1}{\sqrt{2\pi}\delta_k})-\frac{1}{2\delta^2}(x-\mu_k)^2 \\
p2 = log(\sum_{l=1}^K \pi_l \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2\delta_k^2}(x-\mu_l)^2)) \\
log(p_k(x)) = p1 / p2
$$
This LaTeX from myself

because p2 in always a constant, so it could be ignored

$$
log(p_k(x)) = log(\pi_k)log(\frac{1}{\sqrt{2\pi}\delta_k})-\frac{1}{2\delta^2}(x-\mu_k)^2
$$

The last x^2 is quadratic, so the Bayes classifier is non-linear

## Q5

### a
QDA is better on train set, LDA is better on test set.

### b
QDA is better in the both set.

### c
Bigger size n will reduces the variance. QDA will get benefit from that.

### d
False
QDA have higher variance. LDA will perform better if the boundary is linear(means model is not flexible)

