HW1 Conceptual: Beras Pt. 1

Conceptual questions due Friday, February 9, 2024 at 6:00 PM EST
Programming assignment due Monday, February 12, 2024 at 6:00 PM EST

Answer the following questions, showing your work where necessary. Please explain your answers and work.

We encourage the use of

$LATEX$ to typeset your answers. A non-editable homework template is linked, so copy the .tex file into your own Overleaf project and go from there!
Note: make sure to select the template file hw1.tex from the sidebar on the left of overleaf.

Latex Template

Do NOT include your name anywhere within this submission. Points will be deducted if you do so.

Theme

Haters will say that this handsome gentleman is fake.

Conceptual Questions

1. Consider Mean-Square Error (MSE) loss. Why do we square the error? Please provide two reasons. (2-3 sentences)
2. We covered a lemonade stand example in the class (Intro to Machine Learning), which was a regression task.
   • How would you convert it into a classification task?
   • What are the benefits of keeping it a regression task? What are the benefits of converting it to a classification task?
   • Give another example of a classification and a regression task each.

3. Consider the following network:
   
   Let the input, weight, bias, and expected hypothesis matrix
   $o_{expected}$ be the following:
   
   $$x=\left[\begin{array}{c}1\\ 2\end{array}\right]w=\left[\begin{array}{ccc}0.4& 0.7& 0.1\\ 0.2& 0.4& 0.5\end{array}\right]b=\left[\begin{array}{c}0.2\\ 0\\ 0.4\end{array}\right]o_{expected}=\left[\begin{array}{c}1\\ 2\\ 0\end{array}\right]$$
   Answer the following questions, write out equations as necessary, and show your work for every step.
   • Complete a manual forward pass (i.e. find the output matrix
     $o$ AKA
     $\hat{y})$. Assume the network uses a linear function for which the above matrices are compatible.
   • Using Mean Squared Error as your loss metric, compute the loss relative to
     $o_{expected}$.
   • Perform a single manual backward pass assuming that the the expected output
     $o_{expected}$, AKA the ground truth value
     $y$, is given. Calculate the partial derivative of the loss with respect to each parameter.
   • Give the updated weight and bias matrices using the result of your gradient calculations. Use the stochastic gradient descent algorithm with learning rate = 0.1.

HINTS:

• Check Lab 1 for the formula to get the partial derivative of MSE loss with respect to the weights and bias.
• To invoke the chain rule, consider computing
  $\frac{\partial L_{MSE}}{\partial o}$ as an upstream gradient and then composing it with the local gradients relating
  $o$ with the parameters.
• During the backwards pass, pay close attention to what dimensions the gradients have to be. That might help you figure out what operations/matrix configurations you'll need to use.
• If you're doing it right, you should expect relatively simple and repetitive numbers.

4. Empirical risk
   $\hat{R}$ for another machine learning model called logistic regression (used widely for classification) is:
   
   $${\hat{R}}_{\log }(\mathbf{w})=\frac{1}{n}\sum _{i=1}^n\ln (1+\exp (-y_i{\mathbf{w}}^{\mathrm{\top }}{\mathbf{x}}_i))$$
   This loss is also known as log-loss. Take the gradient of this empirical risk w.r.t
   $\mathbf{w}\in {\mathbb{R}}^d$. Your answer may need to include parameters
   $\mathbf{w}$, number of examples
   $n$, labels
   $y_i\in \mathbb{R}$, and training examples
   ${\mathbf{x}}_i\in {\mathbb{R}}^d$. Show your work for every step!

2470-only Questions

The following are questions that are only required by students enrolled in CSCI2470. For each question, include a 2-4 sentence argument for why this mathematical detail may be considered interesting and useful.

1. You have two interdependent random variables
   $x,y\sim X,Y$ and you'd like to learn a mapping
   $f:X\to Y$. The best function you can get is
   $f(x)=\mathbb{E}[Y|X=x]$. This does not account for noise associated with random events. Let
   $\xi$ be the noise such that
   $y=f(x)+\xi$. Prove that
   $\mathbb{E}[\xi ]=0$.

For Question 1, the following information might be relevant:

• Linearity of Expectation states that
  $\mathbb{E}[a+b]=\mathbb{E}[a]+\mathbb{E}[b]$
• According to the law of total expectation,
  $\mathbb{E}\mathbb{[}\mathbb{E}[B|A]]=\mathbb{E}[B]$

2. Let
   $p_{\mathbf{w}}$ conditionally parameterize a normal distribution such that
   $p_{\mathbf{w}}(y|\mathbf{x})\sim \mathcal{N}({\mathbf{x}}^T\mathbf{w},1)$. Show that the maximizing likelihood objective can be written by using
   $\underset{\mathbf{w}}{\arg min}$ and the squared error between
   ${\mathbf{x}}^T\mathbf{w}$ and
   $y$. Intuitively, this is a probabilistic interpretation of linear regression. We are interested in the joint probability of how the behaviour of the output
   $y$ is conditional on the values of the input vector
   $\mathbf{x}$ , as well as any parameters of the model
   $\mathbf{w}$. The model is represented as a Gaussian distribution (prior) with the linear function output as the mean. We assume that the variance in the model is fixed (
   $=1$) (i.e. that it doesn't depend on
   $\mathbf{x}$).

For Questions 2 and 3, the following information might be relevant:

• Weights
  $\mathbf{w}\in {\mathbb{R}}^{\mathbb{d}}$ parameterize a model
  $p_{\mathbf{w}}(y|\mathbf{x})$ to predict
  $p(y)$ given
  $\mathbf{x}$.
• The maximum likelihood objective for a dataset
  $({\mathbf{x}}_i,y_i{)}^n\in ({\mathbb{R}}^d,\mathbb{R}{)}^n$ involves choosing
  $\mathbf{w}$ such that the following two objectives (which are equivalent) are satisfied:
  
  $$\begin{array}{}\text{(1)}& \underset{\mathbf{w}}{\arg max}\prod _{i=1}^n{p}_{\mathbf{w}}(y_i|{\mathbf{x}}_i)=\underset{\mathbf{w}}{\arg min}\sum _{i=1}^n\ln \left({p_{\mathbf{w}}(y_i|{\mathbf{x}}_i)}^{-1}\right)\end{array}$$
• (To convert the multiplicative function to additive we take log, so now we are trying to learn weights such that the log-liklihood of
  $y|x$ is maximized)
• An i.i.d. vector
  $\mathbf{x}$ - where each value is pulled from the same distribution and does not depend on any of the other values - abides by
  $\mathbf{P}(\mathbf{x})=\prod _i\mathbf{P}(x_i)$.
• Recall that the normal (Gaussian) probability density function (PDF) is:
  
  $$f(x)=\frac{1}{\sigma \sqrt{(2\pi )}}\exp (-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2)$$

3. Assume instead that each element
   $w_j\sim \mathcal{N}(0,\frac{1}{\lambda })$ and
   $p_{\mathbf{w}}(y|(\mathbf{x},\mathbf{w}))\sim \mathcal{N}({\mathbf{w}}^T\mathbf{x},1)$. Show that the following must be true (
   $P(\mathbf{w}$) is the probability density of the vector
   $\mathbf{w}$, see hints above):
   
   $$\underset{\mathbf{w}}{\arg max}\{P(\mathbf{w})\prod _{i=1}^n{p}_{\mathbf{w}}(y_i|({\mathbf{x}}_i,\mathbf{w}))\}=\underset{\mathbf{w}}{\arg min}\{\frac{1}{2}\sum _{i=1}^n({\mathbf{w}}^T{\mathbf{x}}_i-y_i{)}^2+\frac{\lambda }{2}\sum _{j=1}^d{w}_j^2\}$$ This is a regularized version of linear regression (called ridge regression) where the second term tries to push the weight values that are least influential to 0.
   $\lambda$ is a hyperparameter (selected by user).