HW3 Beras

Assignment due October 29th at 6 pm EST on gradescope

Task 1.1 [Tensor]: We will begin completing the construction of the Tensor class at the top of the file. Note that it subclasses the np.ndarray datatype, you can find out more about what that means here.

The only TODO is to pass in the data to the a kwarg in np.asarray(a=???) in the __new__ method.

Task 1.2 [Callable]: There are no TODOs in Callable but it is important to familiarize yourself with this class. Callable simply allows its subclasses to use self() and self.forward() interchangably. More importantly, if a class subclasses Callable it will have a forward method that returns a Tensor and so, we can and will use these subclasses when constructing layers and models later.

Task 1.3 [Weighted]: There are 4 methods in Weighted for you to fill out: trainable_variables, non_trainable_variables, trainable, trainable (setter). Each method has a description and return type (if needed) in the stencil code. Be sure to follow the typing exactly or it's unlikely to pass the autograder.

Task 1.4 [Diffable.__call__] There are no coding TODOs in Diffable.__call__ but it is critical that you spend some time to familize yourself with what it is doing. Understanding this method will help clear up later parts of the assignment.

Task 1 [Dense.forward]: To begin, fill in the Dense.forward method. The parameter x represents our input. Remember from class that our Dense layer performs the following to get its output:

\[ f(\bf{x}) = \bf{x}\bf{W} + \bf{b} \]

Keep in mind that x has shape (num_samples, input_size).

Task 2 [Dense.get_input_gradients]: Refer to the formula you wrote in Dense.forward to compute \(\frac{\partial f}{\partial x}\). Be sure to return the gradient Tensor as a list, this will come in handy when you write back propagation in beras/gradient_tape.py later in this assignment.

Task 3 [Dense.get_weight_gradients]: Compute both \(\frac{\partial f}{\partial w}\) and \(\frac{\partial f}{\partial b}\) and return both Tensors in a list, like you did in Dense.get_input_gradients.

Task 4 [Dense._initialize_weight]: Initialize the dense layer’s weight values. By default, return 0 for all weights (usually a bad idea). You are also required to allow for more sophisticated options by allowing for the following:

• Normal: Passing normal causes the weights to be initialized with a unit normal distribution \(\mathcal{N}(0,1)\).
• Xavier Normal: Passing xavier causes the weights to be initialized in the same way as keras.GlorotNormal.
• Kaiming He Normal: Passing kaiming causes the weights to be initialized in the same way as keras.HeNormal.

Explicit definitions for each of these initializers can be found in the tensorflow docs

Note: _initialize_weight returns the weights and biases and does not set the weight attributes directly.

Task 3.1 [LeakyReLU]: Fill out the forward pass and input gradients computation for LeakyReLU. You'll notice these are the same methods we implemented in layers.py, this is by design.

Task 3.2 [Sigmoid]: Complete the forward and get_input_gradients methods for Sigmoid.

Task 3.3 [Softmax]: Write the forward pass and gradient computation w.r.t inputs for Softmax.

Task 1 [MeanSquaredError.forward]: Implement the forward pass for MeanSquaredError. We want y_pred - y_true, and not the other way around. Don't forget that we expect to take in batches of examples at a time so we will need to take the mean over the batch as well as the mean for each individual example. In short, the output should be the mean of means.

Don't forget that Tensors are a subclass of np.ndarrays so we can use numpy methods!

Task 2 [MeanSquaredError.get_input_gradients]: Just as we did for our dense layer, compute the gradient with respect to inputs in MeanSquaredError. It's important to rememeber that there are two inputs, y_pred and y_true. Since y_true comes from our database and is not dependent on our params, you should treat it like a constant vector. On the other hand, compute the gradient with respect to y_pred exactly as you did in Dense. Remember to return them both as a list!

Task 3 [CategoricalCrossEntropy.forward]: Implement the forward pass of CategoricalCrossEntropy. Make sure to find the per-sample average of the CCE Loss! You may run into trouble with values very close to 0 or 1, you may find np.clip of use…

Task 4 [CategoricalCrossEntropy.get_input_gradients]: Get input gradients for CategoricalCrossEntropy.

Task 1 [CategoricalAccuracy.forward]: Fill in the forward method. Note that our input probs represents the probability of each class as predicted by the model and labels is a one hot encoded vector representing the true model class.

Task 1 [OneHotEncoder.fit]: In HW2 you were able to use tf.onehot now you get to build it yourself! In OneHotEncoder.fit you will take in a 1d vector of labels and you should construct a dictionary that maps each unique label to a one hot vector. This method doesn't return anything.

Note: you should only associate a one hot vector to labels present in labels!

Task 2 [OneHotEncoder.forward]: Fill in the OneHotEncoder.forward method to transform the given 1d array data into a one-hot-encoded version of the data. This method should return a 2d np.ndarray.

Task 3 [OneHotEncoder.inverse]: OneHotEncoder.inverse should be an exact inverse of OneHotEncoder.forward such that OneHotEncoder.inverse(OneHotEncoder.forward(data)) = data.

Task 1 [BasicOptimizer]: Write the apply_gradients method for the BasicOptimizer.

For any given trainable_param, \(w[i]\), and learning_rate, \(r\), the optimization formula is given by \[w[i] = w[i] - \frac{\partial \mathcal{L}}{\partial w[i]}*r\]

Task 2 [RMSProp]: Write the apply_gradients method for the RMSProp.

In RMSProp there are two new hyperparams, \(\beta\) and \(\epsilon\).

\(\beta\) is referred to as the decay rate and typically defaults to .9. This decay rate has the effect of lowering the learning rate as the model trains. Intuitively, as our loss decreases we are closer to a minimum and should take smaller steps towards optimization to ensure we don't optimize past the minimum.

\(\epsilon\) is a small constant to prevent division by 0.

In addition to our hyperparams there is another term which we will call, v, which acts as the moving average of the gradients for each param. We update this value in addition to the trainable_params every time we apply the gradients.

For any given trainable_param, \(w[i]\), learning_rate, \(r\) the update is defined by \[v[i] = \beta*v[i] + (1-\beta)*\left(\frac{\partial \mathcal{L}}{\partial w[i]}\right)^2\] \[w[i] = w[i] - \frac{r}{\sqrt{v[i]} + \epsilon}*\frac{\partial \mathcal{L}}{\partial w[i]}\]

Hint: In our stencil code, we provide v as a dictionary which maps a key to a float. Keep in mind that we only need to store a single v value for each weight!

Task 3 [Adam]: Write the apply_gradients methods for the Adam.

At it's core Adam, is similar to RMSProp but it has more smoothing terms and computes an additional momentum term to further balance the learning rate as we train. This momentum term has it's own decay term, \(\beta_1\). Additionally, Adam keeps track of the number of optimization steps performed to further tweak the effective learning rate.

Here is what an optimization step with Adam looks like for trainable_param, \(w[i]\) and learning_rate, \(r\).

\[m[i] = m[i]*\beta_1 + (1-\beta_1)*\left(\frac{\partial \mathcal{L}}{\partial w[i]}\right)\] \[v[i] = v[i]*\beta_2 + (1-\beta_2)*\left(\frac{\partial \mathcal{L}}{\partial w[i]}\right)^2\] \[\hat{m} = m[i]/(1-\beta_1^t)\] \[\hat{v} = v[i]/(1-\beta_2^t)\] \[w[i] = w[i] - \frac{r*\hat{m}}{\sqrt{\hat{v}}+\epsilon}\]

Note: Don't forget the iterate time once when apply_gradients is called!

Task 1 [GradientTape.gradient]: Implement the gradient method to compute the gradient of the loss with respect to each of the trainable params. The output of this method should be a list of gradients, one for each of the trainable params.

Task 1 [Model.weights]: Construct a list of all weights in the model and return it.

Task 2 [Model.fit]: This method should train the model for the number of epochs given on the train and test data, x and y given with batch size, batch_size. Importantly, you want make sure you record the metrics throughout training and print stats out during the train so that you can watch the metrics as the model trains.

You can use the print_stats and update_metric_dict functions provided. Note that neither of these methods return any values, print_stats prints out the values directly and update_metric_dict(super_dict, sub_dict) updates super_dict with the mean metrics from sub_dict.

Note: You do not need to call the model here, you should instead use self.batch_step(...) which all child classes of Model will implement.

Task 3 [Model.evaluate]: This method should look very similar to Model.fit except we need to ensure the model does not train on the testing data. Additionally, we will test on the entirety of the test set one time, so there is no need for the epochs parameter from Model.fit.

Task 4 [SequentialModel.forward]: This method passes the input through each layer in self.layers sequentially.

Task 5 [SequentialModel.batch_step]: This method trains makes a model prediction and computes the loss inside of GradientTape just like you did in HW2. Be sure to use the training argument to adjust the weights of the model only when training is True. This method should return the loss and accuracy for the batch in a dictionary.

Note: you should have used the implict __call__ function for the layers in the forward method to ensure that each layer gets tracked to the GradientTape (i.e. layer(x)). However, because we don't define how to calculate gradients for a SequentialModel, make sure to use the SequentialModel's forward method in batch_step.

Task 1 [load_and_preprocess_data()]: We provide the code to load in the data, your job is to

1. Normalize the values so that they are between 0 and 1

2. Flatten the arrays such that they are of shape (number of examples, 28*28).

3. Convert the arrays to Tensors and return the train inputs, train labels, test inputs and test labels in that order.

4. You should NOT shuffle the data in this method or do any other transformations than what we describe in 1-3. Importantly, you should NOT return one_hot labels. You'll create those when training and testing.

Task 1 [Save predictions]: Once you have an architecture that works, use np.save("predictions.npy", arr: np.ndarry) to save your predictions for the test. Please name the file predictions.npy when you submit to gradescope or the autograder may not find it.

It might be helpful to change what's being returned by/done in batch_step and evaluate for this! Just make sure that batch_step only returns the metric dictionary when training=True, as that's what we test for.

Note: We have a number of safeguards in place to prevent folks from cheating, and the autograder will not tell you that your submission has been flagged. Instead, you'll get an email from Prof. Sun some time after you submit.