---
# System prepended metadata

title: Machine Learning Share (2018/04/24)
tags: [technical, shared]

---

# Machine Learning Share (2018/04/24)
###### tags: `shared` `technical`

[TOC]

## Curse of dimensionality
+ Think of distances between samples measured using features
+ **High dimensional feature $\rightarrow$ far away**
+ All the samples are ALL far away
+ No clue for prediction/clustering
+ Need an efficient way to select useful/informative features

## Feature selection v.s. feature extraction
+ Feature **selection**
    + Original features "preserved"
    + Know which ones to **keep** and which ones to **drop**
    + e.g., Recursive Feature Elimination (RFE)
+ Feature **extraction**
    + Original features not "preserved"
    + Features are **transformed** into scores or indicies
    + e.g., Principal Component Analysis (PCA)

## Three categories for feature selection
+ Filter methods
    + Each feature is fed into a **scoring function**
    + Often select the top-scoring features as final feature set
    + Scoring function is **independent** of the following classifier
    + e.g., majority vote with weights being p-values, SNRs, variances, etc.
+ Wrapper methods
    + Often adopted together with a classifier
    + The scoring function is "specialized" for the following classifier
    + e.g., RFE with SVM (SVM-RFE)
+ Embedded methods
    + Feature selection and classification are closely combined

## Examples of feature extraction
+ Restricted Boltzmann Machine (RBM)
    + Visible layer v.s. hidden layer
    + Visible $\rightarrow$ hidden & hidden $\rightarrow$ visible
    + Trained with VB
    + **Hidden layer $\rightarrow$ new features**
    + Reference (Wiki): https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine
+ Autoencoder (AE)
    + Latent code as new features
    + Many variants (with various objective functions)
    + Unsupervised self-learning
    + Can better utilize unlabeled samples 
+ **Conditional variational autoencoder (CVAE)**
    + Detailed explanation: https://hackmd.io/s/Hyn2oZpdz
    + "Pre-defined" latent feature distribution
    + Easy to incorporate auxilliary information
+ **AIC/BIC/$C_p$**
    + Mallow's $C_p$ and Akaike's Information Criterion (AIC)
    + Bayesian Information Criterion (BIC)
    + Consider MSE and **model complexity** at the same time
    + Forward/Backward (stepwise) feature selection v.s. best subset selection
+ Other commonly-seen approaches
    + **Principal component analysis (PCA)**
    + SVM-RFE
    + Many filter methods
+ **LASSO (Least Absolute Shrinkage and Selection Operator)**
    + Objective function + penalty term
    + Regularized: $l_1-norm$
    + Compared to ridge regression: $l_2-norm$
    + $l_1$ has much more probability to make regression coefficients zeros
