---
tags: paper_review, class_project
---

# Searching for Effective Neural Extractive Summarization: What Works and What’s Next

github: https://github.com/maszhongming/Effective_Extractive_Summarization

## Introduction

Test the performance on extractive summarization with different types of:
- Architectures
- Sentence Encoder
- Document Encoder
- Decoder
- External Transferable knowledge
- Learning schema

## Environment

### Architecture

- Sentence Encoder
- CNN (Kim, 2014)
- Kedzie et al. (2018) shows that different sentence encoder doesn't effect the final performance
- Document Encoder
- LSTM
- Transformer
- Decoder
- Extract subset of sentences from document
- Pointer Network
- Auto-regressive
- Simplified Attention mechanism
- Aware of previous prediction
- Vinyals et al., 2015
- Sequence Labeling
- Non-auto-regressive

### External Transferable Knowledge

Word embeddings:
- word2vec (baseline)
- Unsupervised
- Context independent
- GloVe
- Unsupervised
- Context independent
- BERT
- Unsupervised
- Contextualized
- Newsroom
- pre-train the model on the Newsroom dataset
- 1.3M article-summary data

### Learning schema

- Supervised
- Supervised + Reinforcement

### Evaluation
- Evaluate on cross-domain datasets:
- CNN/DailyMail
- Newsroom
- Scores: ROGUE-1, ROGUE-2, ROGUE-L
- Repetition Score:
- Evaluate on diversity.
- percentage of repeated n-grams in extracted summary to measure the word-level repetition
- $\text{REP}_n = \frac{\text{CountUniq}(ngram)}{\text{Count}(ngram)}$
- Positional Bias
- 描述標準答案在文章中的分佈均勻度
- Divide article into k parts and $p(i)$ denotes the probability that the first golden label is in part $i$ of the articles
- $\text{PosBias} = \sum_{i=1}^k -p(i) \log{p(i)}$
- Sentence Length
- 計算排名第k個的句子的平均長度
- Sentence Shuffling
- 探討句字在文章中的位置對模型的影響

## Pointer Network

Pointer Network is seq2seq network based on attention mechanism that tackles tasks where the output sequence is heavily dependent on the input sequence.
Pointer Network是一個Seq2Seq的架構，用於處理output sequence極度依賴於input sequence的資料，例如在這次的extractive summerization的任務中，output sequence是input sequence的index。

### Attention Network

In the original attention seq2seq network proposed by Bahdanau et al. 2015, in each step $i$, a context vector $c_i$ is calculated by multiplying the attention vector $a_{ij}$ and each encoding hidden states $e_j$.

在原本的attention seq2seq network中，計算出attention vector之後，它會與encoding state相乘，得到context vector。

$$ \begin{aligned}
& u_{ij} = V^{\top}tanh(W_1 e_j + W_2 d_i) & j \in (1, \ldots, n) \\
& a_i = \text{softmax}(u_i) \\
& c_i = \sum_{j=1}^n a_{ij} e_j
\end{aligned}
$$

$V$, $W_1$ and $W_2$ are trainable weights. The next decoding hidden state $d_{i+1}$ is then calculated with $d_i$ and $c_i$.

### Pointer Netowrk

Pointer network sees the values in the attention vector $a_{ij}$ as a conditional probability of selecting the $P_j$ from the encoding input sequence $\mathcal{P} = (P_1, \ldots, P_n)$ in decoding step $i$. The selected index in decoding step $i$ is denoted as $C_i$. The selected token $P_{C_i}$ is then feeded into the decoding input.

而在Pointer network，計算出的attention vector被看作是從input sequence中選擇某一的機率。在這次的extractive summerization任務中，我們會選擇attention vector數值最大的index作為這次的output。

encoding input: $\mathcal{P} = (P_1, \ldots, P_n)$
decoding output: $C_1, C_2 \ldots, C_i$

$$ \begin{aligned}
& u_{ij} = V^{\top}tanh(W_1 e_j + W_2 d_i) \qquad j \in (1, \ldots, n) \\
& p(C_i | C_1, \ldots, C_{i-1}, \mathcal{P}) = \text{softmax}(u_i)
\end{aligned}
$$

## Experiment

### Datasets

Training/Testing Dataset:
- CNN/DailyMail
- NYTimes
- WashingtonPost
- FoxNews
- TheGuardian
- NYDailyNews
- WSJ
- USAToday

Training Dataset (for external transferable knowledge):
- Newsroom

### Decoders

Pointer Network decoder在六個domain上有最好的表現。作者檢視:
- Repetition
- Decoder使用Pointer Network的模型，Repetition明顯高於其他的
- Positional Bias
- Pointer Network與SeqLab模型的平均分數差異($\Delta R$)會隨著positional bias一起增加
- Sentence length
- SeqLab tend to extract sentences with similar length.
- Pointer Network tend to choose longer sentences as the first sentence and greatly reduce the length of the sentence in the subsequent extractions

### Encoders

觀察:
1. LSTM在不同domain需要使用不同的參數才會有與Transformer有相近的表現。Transformer不需要改變參數。
2. When equipped with SeqLab decoder, Transformer always obtains a better performance compared with LSTM

Shuffled Testing: (放圖)

Disentangling Testing: (放圖)

### Transferable Knowledge

使用BERT的模型表現得比其他的都好一截。

BERT的兩種使用方式:
1. 句子分開餵進BERT，得到sentence representation
2. 將整篇文章餵進BERT，得到各個token的representation後，用mean pooling算出sentence representation

第二種的表現比第一種好

### Learning Schema and Complementarity

Base model: LSTM-Pointer and LSTM-Pointer+BERT

沒有說怎麼做的。

## Conclusion

"Our detailed observations can provide more hints for the follow-up researchers to design more powerful learning frameworks"