<style>
img {
display: block;
margin-left: auto;
margin-right: auto;
}
</style>

> [Paper link](https://ojs.aaai.org/index.php/AAAI/article/view/26595) | [Code link](https://github.com/ssshddd/ConvNTM) | AAAI 2023

:::success
**Thoughts**

They propose the first Conversational Neural Topic Model (ConvNTM) specifically for the conversation scenario.

While in the conversational scenario, topic modeling is even more complicated with the following two unique properties to discover topics:
1. A conversation session generally consists of multiple turns of short-text utterances
2. There are multiple roles within a conversation session, speakers and addressees

For downstream task:
- Classification: utilize topic labels for each utterance when modeling speaker turns
- Generation: topical hierarchical recurrent framework for multi-turn response generation
:::

## Abstract

In this paper, they propose a Conversational Neural Topic Model (ConvNTM) designed in particular for the conversational scenario.

Comprehensive experimental results based on the benchmark datasets demonstrate that their proposed ConvNTM achieves the best performance in
- Topic modeling
- Downstream tasks within conversational research
- Dialogue act classification
- Dialogue response generation

## Introduction

Topic models are used to discover abstract topics in a series of documents to understand the latent semantics of a text corpus.

With the recent development of neural networks and generative models, various neural topic models (NTMs) have been proposed and applied in document classification, retrieval, semantic analysis, etc.

While in the conversational scenario, topic modeling is even more complicated with the following two unique properties to discover topics:
1. A conversation session generally consists of multiple turns of short-text utterances
2. There are multiple roles within a conversation session, speakers and addressees

---

This paper called Conversational Neural Topic Model (ConvNTM):
1. A hierarchical conversation encoder to capture the **multi-turn dialogue structure**.
2. A sequence encoder is utilized to model the conversation contexts and extract **utterance-level representations for the role modeling of speakers and addressees**.
3. A multi-role interaction graph to model speaker/addressee information from two perspectives.

## Related Work

### Topic Model

- Latent Dirichlet Allocation (LDA)
- Neural topic models (NTMs)
- Variational Autoencoder (VAE)
- GSM
- ProdLDA
- ETM
- GNTM: adds the document graph into the generative process of topic modeling
- Application-oriented NTMs
- LeadLDA
- ForumLDA

### Multi-Turn Dialogue

**The simple concatenation of multi-turn dialogue contexts performs poorly since it makes the latent dialogue structure ignored.**

- Multi-Turn
- Devise the **hierarchical LSTM** to encode the structure and generate responses
- Views the dialogue as a dynamic flow and designs **three objectives** to capture the information dynamics
- Speaker feature
- Incorporate the turn changes among speakers to capture the fine-grained semantics of dialogue
- Speaker-aware disentanglement strategy to tackle the entangled dialogues and improve the performance of multi-turn dialogue response selection
- Topic-aware models
- Two topic-aware contrastive learning objectives to handle information scattering challenges for the dialogue summarization task
- Topic-driven knowledge-aware Transformer to deal with the emotion detection in dialogue

## Conversational Neural Topic Model


### Hierarchical Conversation Encoder

- Conversation sequence encoder
- Multi-role graph encoder

#### Conversation sequence encoder

They employ a sequence encoder that models the conversation contexts from word level to utterance level.

A conversation sequence $c$ has $J$ speakers, and the speaker $j$ has $n_j$ utterances: $\{ u_1^{(j)}, u_2^{(j)}, \dots, u_{n_j}^{(j)} \}$.

The encoding process for the sequence encoder can be formulated as:

$$
\tag{1} e_k^{(j)} = f_e (u_k^{(j)})
$$

$$
\tag{2} s_k^{(j)} = f_{trm} (e_k^{(j)})_{[\text{CLS}]}
$$

$$
\tag{3} \tilde{s}_k^{(j)} = f_{rnn} (s_1^{(j)}, s_2^{(j)}, \dots, s_{n_j}^{(j)})_k
$$

$$
\tag{4} h_k^{(j)} = f_{attn} (\tilde{s}_1^{(j)}, \tilde{s}_2^{(j)}, \dots, \tilde{s}_{n_j}^{(j)})_k
$$

The words in the $k$-th utterance $u_k^{(j)}$ are first encoded as $e_k^{(j)}$ through an embedding layer $f_e$.

And the utterance-level representation $s_k^{(j)}$ from the $[\text{CLS}]$ token.

#### Multi-role graph encoder

They denote each utterance representation $h_k^{(j)}$ as a node and the two types of edges between nodes reflect the intra-speaker and inter-speaker dependencies.

They consider **intra-speaker** dependency to keep the topic consistency and avoid contradictions:
1. They add a bidirectional edge between $h_{k_1}^{(j)}$ and $h_{k_2}^{(j)}$ only if $| k_1 - k_2 | \le K_s$, where $K_s$ indicates the window size for aggregating contextual utterances from the same speaker.
2. A speaker will give feedback on the utterance conents from other speakers, and then decide whether to keep or shift the current topic.

It is also necessary to construct the **inter-speaker** dependency in the graph to simulate the dynamic interactions, where $|k_{j_1} - k_{j_2}| \le K_c$ and $K_c$ for two utterances in the conversation.

They utilize a GCN $f_{gcn}$ to update the utterance representations under the multi-role interaction relations.

$$
\tag{5} \tilde{h}_k^{(j)} = f_{gcn} (h_k^{(j)})
$$

### Topic Modeling

They assume that each speaker $j$ in the conversation session $c$ holds a global topic information $\theta_c^{(j)}$, and each utterance $k$ has local topic information $\theta_k^{(j)}$ which is fused with the corresponding global topic to determine the eventual topic distribution $\tilde{\theta}_k^{(j)}$.

**NTM framework**

They process the $n_j$ utterances of each speaker $j$ into bag-of-words (BoW) representations: $\{ x_1^{(j)}, x_2^{(j)}, \dots, x_{n_j}^{(j)} \}$, where $x_k^{(j)}$ is a $|V|$-dimensional multi-hot encoded vector.

To extract the representation $\tilde{x}_k^{(j)}$:

$$
\tag{6} \tilde{x}_k^{(j)} = g_x (\frac{x_k^{(j)}}{\sum_{v=1}^{|V|}(x_k^{(j)})_v})
$$

where $g_\ast$ mentioned represents a multilayer perceptron (MLP).

To introduce multi-role interactions into topic modeling:

$$
\tag{7} \theta_k^{(j)} = g_s (\tilde{x}_k^{(j)} \oplus \tilde{h}_k^{(j)} )
$$

All the utterances of each speaker $j$ are integrated to derive the global speaker-aware representation $h_c^{(j)}$.

$$
\tag{8} h_c^{(j)} = \tanh \left( \sum_{k=1}^{n_j} g_c (\tilde{x}_k^{(j)} \oplus \tilde{h}_k^{(j)}) \cdot \theta_k^{(j)} \right)
$$

And they use it to estimate the prior variables $\mu_c^{(j)}$ and $\log \sigma_c^{(j)}$ via two separate networks $g_\mu$ and $g_\sigma$:

$$
\tag{9} \mu_c^{(j)} = g_\mu (h_c^{(j)}), \ \ \ \log \sigma_c^{(j)} = g_\sigma (h_c^{(j)})
$$

Such that they can sample a latent variable $z_c^{(j)} \sim \mathcal{N} (\mu_c^{(j)}, \sigma_c^{(j)})$.

The global topic distribution $\theta_c^{(j)}$:

$$
\tag{10} \theta_c^{(j)} = \text{softmax} (g_\theta(z_c^{(j)}))
$$

Finally, thet can fuse local and global topic information:

$$
\tag{11} \tilde{\theta}_k^{(j)} = g_f (\theta_k^{(j)} \oplus \theta_c^{(j)})
$$

The reconstructed utterance BoW can be derived as:

$$
\tag{12} \hat{x}_k^{(j)} = \text{softmax} (\tilde{\theta}_k^{(j)} \beta)
$$

where $\beta \in \mathbb{R}^{K \times |V|}$ is a weight matrix that represent $K$ topic-word distributions.

### The Joint Training Objective

**Neural variational inference objective**

The marginal likelihood of the conversation session $c$ is decomposed as:

$$
\tag{13} p(c \mid \mu, \sigma, \beta) = \prod_{j=1}^J \int_{\theta_c^{(j)}} p(\theta_c^{(j)} \mid \mu_c^{(j)}, \sigma_c^{(j)}) \cdot \left( \prod_{k=1}^{n_j} \prod_w p(w \mid \beta, \theta_c^{(j)})\right) d \theta_c^{(j)}
$$

This part of the training loss can be formulated as:

$$
\tag{14} \mathcal{L}_c^{(j)} = - \mathbb{E}_{q(\theta_c^{(j)} \mid \mu_c^{(j)}, \sigma_c^{(j)})} \left( \sum_{k=1}^{n_j} \sum_w \log p(w \mid \theta_c^{(j)}, \beta)\right) + w_{kl} \cdot D_{KL} (q(\theta_c^{(j)} \mid \mu_c^{(j)}, \sigma_c^{(j)}) \| p(\theta_c^{(j)}))
$$

where $w_{kl}$ is the hyper-parameter for the weight of the KL term.

In the equation above:
1. Minimize the cross entropy between the input normalized BoW and reconstructed BoW.
2. Minimize the distance between the variational posterior $q(\theta_c^{(j)} \mid \mu_c^{(j)}, \sigma_c^{(j)})$ and true posterior of latent variables $p(\theta_c^{(j)})$.

**Controllable word co-occurrence objective**

They leverage the word co-occurrence information of the training corpus to improve the topic quality:

$$
\tag{15} \mathcal{L}_{co} = - \sum_{w_1 = 1}^{|V|} \sum_{w_2 = 1}^{|V|} M_{w_1, w_2} \log (\beta^\top \beta)_{w_1, w_2}
$$

where $M \in \mathbb{R}^{|V| \times |V|}$ is a co-occurrence matrix.

Suppose that there are $C$ conversations in the training set, the overall training loss of ConvNTM is given by:

$$
\tag{16} \mathcal{L} = (1 - w_{co}) \sum_{c=1}^C \sum_{j=1}^J \mathcal{L}_c^{(j)} + w_{co} \mathcal{L}_{co}
$$

The controllable factor $w_{c o}$ is dynamically adjusted as:

$$
\tag{17} w_{c o}= \begin{cases}0, & \mathcal{L}_{c o} \leq d_{c o}, \\ \min \left(1, \frac{\mathcal{L}_{c o}-d_{c o}}{W_{c o}}\right), & \mathcal{L}_{c o}>d_{c o},\end{cases}
$$

where $W_{c o}$ is another hyper-parameter of the correcting factor for the proportional signal and target co-occurrence distance as $d_{co}$.

## Experiments

### Experimental Setup

**Datasets**

- DailyDialog
- EmpatheticDialogues

**Evaluation metrics**

- Topic coherence (TC)
- Normalized pointwise mutual information (NPMI)
- Content Vector-based coherence metric (CV)
- Topic diversity (TD)

Topic quality score (TQ) as the product of TC and TD.

$$
\tag{18} \text{NPMI}(w_i, w_j) = \frac{\log\frac{p(w_i, w_j) + \epsilon}{P(w_i)P(w_j)}}{- \log (p(w_i, w_j) + \epsilon)}
$$

### Main Results


### Ablation Study



#### Downstream Tasks

The essence of topic modeling is an unsupervised learning process for latent semantic structure.

They take dialogue act classification and response generation as examples to verify that ConvNTM is helpful for improving both **classification** and **generation** tasks.

For dialogue act classification, they borrow the framework of STM, which utilized topic labels for each utterance when modeling speaker turns.


For dialogue response generation, they borrow the framework of THERD, which proposes a topical hierarchical recurrent framework for multi-turn response generation. THERD utilizes LDA to extract the top 100 topic words for each conversation.


## Conclusion

In this work, they propose the first Conversational Neural Topic Model (ConvNTM) specifically for the conversation scenario.