Recurrent Neural Networks

Architecture

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More โ†’

Forward Propagation

  • The hidden state
    ht
    depends on
    htโˆ’1
    and
    xt
    ie.
    ht=ฯƒ(Whtโˆ’1+Uxt+b)
  • Let us say, at timestep
    t
    • xt
      is the word embedding of size [e,1]
    • ht
      is the RNN hidden state of size [d,1]
  • Then the transition matrices,
    • U
      is of size [d,e]
    • W
      is of size [d,d]
  • Clearly,
    ht+1
    will also be of size [d,1].
  • In other words
    ht=RNN(htโˆ’1,xt)

    Where
    RNN=ฯƒ(Whtโˆ’1+Uxt+b)
  • There is also an optional output layer at each time step,
    yt

    yt=softmax(Vht+b)

    Where
    V
    is of size [d,V]