HW5 - Handwritten Assignment

LSTM Cell (1%)

In this exercise, we will simulate the forward pass of a simple LSTM cell. Figure.1 shows a single LSTM cell, where \(z\) is the cell input, \(z_i, z_f , z_o\) are the control inputs of the gates, \(c\) is the cell memory, and \(f, g, h\) are activation functions. Given an input \(x\), the cell input and the control inputs can be calculated by the following equations :

• \(z = w \cdot x + b\)
• \(z_i = w_i \cdot x + b_i\)
• \(z_f = w_f \cdot x + b_f\)
• \(z_o = w_o \cdot x + b_o\)

Where \(w, w_i, w_f, w_o\) are weights and \(b, b_i, b_f, b_o\) are biases. The final output can be calculated by

\(y = f(z_o)\ h(c′)\)

where the value stored in cell memory is updated by

\(c′=f(z_i)g(z)+cf(z_f)\)

Given an input sequence \(x^t\ ( t = 1, 2, ..., 8 )\), please derive the output sequence \(y_t\). The input sequence, the weights, and the activation functions are provided below. The initial value in cell memory is 0. Please note that your calculation process is required to receive full credit.

Word Embedding (1%)

Consider the Skip-Gram model below, let
\(h = W^{T}x\)
\(u = W^{'T}h\)
\(y = Softmax(u) = Softmax(W^{'T}W^{T}x)\)

\(Loss = L = -log\prod_{c\in{C}}P(w_{output, c}, w_{input})= -log\prod_{c\in{C}}\frac{exp(u_{c})}{\sum_{i\in{V}}{ exp(u_{i})}}\)
where \(C=\) the context words of the input word

Calculate \(\frac{\partial L}{\partial W^{T}_{ij}}\) and \(\frac{\partial L}{\partial W^{'T}_{ij}}\)

(Note: assume that softmax was performed on all output neuron, i.e., no negative sampling)