Try   HackMD

Long-Short Term Memory (LSTM)

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
摘要

{\boldsymbolx:xtt=1,,T}

LSTM Cell 結構

LSTM的架構主要由以下五個元件所組成:

  • 單元狀態(cell state):LSTM的內部單元狀態(
    c
    ),負責儲存先前的記憶
  • 隱藏狀態(hidden state):LSTM的外部隱藏狀態(
    h
    ),負責計算預測結果
  • 輸入閘(input gate):用來決定要把多少當期輸入(
    xt
    )送進當期單元狀態(
    ct
    )
    • 當期輸入(
      xt
      )會先被轉換成候選值(
      ct~
      )
  • 遺忘閘(forget gate):用來決定要把多少前期單元狀態(
    ct1
    )
    送進當期單元狀態(
    ct
    )
  • 輸出閘(output gate):用來決定要把多少當期單元狀態(
    ct
    )
    輸出到當期隱藏狀態(
    ht
    )

相關式子如下:

\colorredit=σ(\colorredWixxt+\colorredWihht1+\colorredbi)

\colorblueft=σ(\colorblueWfxxt+\colorblueWfhht1+\colorbluebf)

\colordarkorangeot=σ(\colordarkorangeWoxxt+\colordarkorangeWotht1+\colordarkorangebo)

\colorgreenct~=tanh(\colorgreenWcxxt+\colorgreenWchht1+\colorgreenbc)

σ()=11+ex

\colorpurplect=\colorblueft\colorpurplect1+\colorredit\colorgreenct~

\colorbrownht=\colordarkorangeottanh(\colorpurplect)

圖形表示如下:

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

改善表現的小技巧

  • 貪婪取樣(greedy sampling):尋找機率值最高的前 k 個候選項
  • 束搜尋(beam search):往下 m 個 timestep 尋找
  • 雙向LSTM
  • peehole connection:偷看前幾
tags: DL