---
# System prepended metadata

title: Sequence to Sequence Learning with Neural Networks
tags: [NLP]

---

---
title: Sequence to Sequence Learning with Neural Networks
date: 2020-04-22 15:12:00
comments: true
author: Darcy
categories:
- nlp study group
tags:
- NLP 
---
###### tags: `study` `paper` `DSMI lab`
paper: [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)

# Abstract and Introduction
* DNN cannot be used to map sequences to sequences because they can only work when dimensionalty of input/output is fixed and known
* Task: English to French translation task from the WMT’14 dataset
* Proposed method 的優點:
    * Minimal assumptions on the sequence structure: 甚麼樣的sequence 架構都可以處理
    * Sensitive to word order
    * Does well on long senetances
    
<!-- more -->
# The model
* Goal: given an input sentance $(x_1,x_2,...,x_T)$ and its corresponding output sentacne $(y_1, y_2, ...,y_{T'})$ (where $T$ need not equal to $T'$), want to estimate $p(y_1, y_2, ...,y_{T'}|x_1,x_2,...,x_T)$
* $p(y_1, y_2, ...,y_{T'}|x_1,x_2,...,x_T)=\Pi_{t=1}^{T'}p(y_{t}|v,y_1,...,y_{t-1})$, where $p(y_{t}|v,y_1,...,y_{t-1})$ is each distribution is represented with a softmax over all the words in the vocabulary
* 把 input sequence 倒過來餵進去效果比較好

![](https://i.imgur.com/kqvV9r8.png)


# Experiment 
* Dataset: WMT’14 English to French dataset
    * 12M sentences
    * Vocabulary: 160000 most frequently used English words and 80000 most frequently used Frech words
    * 沒有在vocabulary出現的字用"UNK"代替
    * test set (for evaluation): 1000-best lists generated by SMT system(baseline)

* Objective: maximizing the log probability of a correct translation $T$ given the source sentence $S$: $\hat{T}=\mathop{argmax}\limits_{T}p(T|S)$
* Left-to-Right beam search
* Reverse the source sentence:
    * Imporoves the performance, but they don't have a complete explanation XDD
    * 當input sentence 被反過來之後，input sentence 的前幾個字和output sentence的前幾個字更近了，有助於output sentence 在一開始就有更精準的生成，後面生成的也會比較準 (類似好的開始就是成功的一半的概念?)
* Trianing details
    * 1000 dimensional word embedding (但他沒有說word embedding是怎麼做的)
    * LSTM: 4 layers, 1000 cells in each layers
    * Parameter initialization from uniform distribution between -0.08 to 0.08
    * 平行化計算: 總共使用8個GPU，訓練10天
* Experimental Results
最好的結果是ensemble不同random initialization 的LSTM 所得到的
![](https://i.imgur.com/egkt2co.png)

* Model analysis:
    * 能夠分辨使用相同字但不同排序的句子，以及相同意思但使用不同文字表達的句子
    ![](https://i.imgur.com/2CgPoaY.png)
    * 對於長句的表現仍然良好(左圖: x軸是句子長度)
    * 句子中如果有出現很多不常用的字，表現也能維持一定的水準(右圖: x軸是句子裡面出現的字的詞頻在整個vocabulary中的排名的平均)
![](https://i.imgur.com/lZrQPq4.png)

# 補充
* SMT system: Statistical Machine translation
    * 通過對大量的平行語料進行統計分析，構建統計翻譯模型
    * 不需要依靠語法規則，所以容易推廣到不同語言的翻譯工作
    * Word-based translation: 一個一個字翻
    * Phrase-based translation: 視情況將幾個字組起來變成詞彙來翻
    * Syntax-based translation: 使用句法分析(例如parsing tree)作為翻譯的依據
    * Hierarchical phrase-based translation: a combination of pharse-based and syntax based
* Beam search: 演算法細節再[這裡](https://zhuanlan.zhihu.com/p/36029811?group_id=972420376412762112)