CONT: Contrastive Neural Text Generation

NIPS 2022

---

Outline

Intoduction

Method

Experiment

Conclusion

---

Introduction

Previous methods using contrastive learning in neural text generation usually lead to inferior performance

• If we simply use from-batch positive-negative samples following simCLR, and adopt the InfoNCE loss which ignores the difference between negative samples (Naive CL).
• Previous work attempts to build better contrastive samples by disturbing the ground truth in the discrete space or the continuous embedding space

Contrastive Neural Text generation (CONT) addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects:

• the construction of contrastive examples. (through beam search algorithm)
• the choice of the contrastive loss. (N-pairs)
• the strategy in decoding. (the learned sequence similarity score)

We validate CONT on five generation tasks with ten benchmarks:

• machine translation
• summarization
• code comment generation
• data-to-text generation
• commonsense generation.

---

Method

Architecture

Contrastive Examples from Predictions

We use the diverse beam search algorithm to create contrastive examples from the top-K list of the model’s lastest predictions and then append them to the from-batch samples to form the contrastive examples.

A warm-up stage where the model is only supervised by

${\mathcal{L}}_{\text{NLL}}$ is recommended as it guarantees the quality of the examples from the model’s prediction.

These self-generated contrastive examples alleviate the model’s exposure bias.

N-Pairs Contrastive Loss

We first rank all the contrastive examples based on an oracle function

$o(·,y)$, which computes a sequence-level score with the ground truth

$y$.

$${\mathcal{L}}_{\text{N-Pairs }}=\sum _{(y^{+},{\mathit{y}}^{-})\in \mathcal{P}}\mathcal{L}({\mathit{y}}^{+},{\mathit{y}}^{-})=\sum _{({\mathit{y}}^{+},{\mathit{y}}^{-})\in \mathcal{P}}max\{0,\cos ({\mathbf{z}}_{\mathit{x}},{\mathbf{z}}_{{\mathit{y}}^{-}})-\cos ({\mathbf{z}}_{\mathit{x}},{\mathbf{z}}_{{\mathit{y}}^{+}})+\xi \}$$

$$\mathcal{P}\text{ contains }{C}_K^2\text{ pairs constructed from }\mathcal{B}\text{, ground truth }y\text{, and from-batch examples. }$$

$\xi =\gamma \ast (\mathrm{rank}\left({\mathit{y}}^{-}\right)-\mathrm{rank}\left({\mathit{y}}^{+}\right))$ reflect the quality difference in these pairs.

Inference with Learned Similarity Function

$${\mathit{y}}^{\ast }=\arg \underset{\hat{\mathit{y}}}{max}\{\alpha \cdot \cos ({\mathbf{z}}_{\mathit{x}},{\mathbf{z}}_{\hat{\mathit{y}}})+(1-\alpha )\prod _{t=0}^{n}p({\hat{y}}_t\mid \mathit{x},{\hat{\mathit{y}}}_{<t})\}$$

---

Experiment

---

Conclusion

It models an additional contrastive learning objective to provide a sequence-level supervision for auto-regressive neural text generation models.

We explore three shortcomings that limit the development of contrastive learning on text generation tasks.

Speeding up the training stage without losing accuracy is the next important step to improve CONT.

---

Appendix

batch samples

The physical meaning of batch samples in this paper is that they represent a group of text sequences or data points that are fed into the neural network simultaneously for processing.

N-Pairs Contrastive Loss Algorithm

Background

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{NCE}}=-\log \frac{\exp (\cos ({\mathbf{z}}_{\mathit{x}},{\mathbf{z}}_{\mathit{y}})/\tau )}{\sum _{{\mathit{y}}^{\prime }\in \mathcal{B}}\exp (\cos ({\mathbf{z}}_{\mathit{x}},{\mathbf{z}}_{{\mathit{y}}^{\prime }})/\tau )}\\ {\mathcal{L}}_{\mathrm{NLL}}=-\sum _{t=1}^N\log p_{\theta }(y_t\mid \mathit{x},{\mathit{y}}_{<t}),\end{array}$$