---
# System prepended metadata

title: Dialogue Comparision

---

# Dialogue Comparision

## Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

### Problem 
* **GP SARSA** provides an **explicit estimate of uncertainty**, but are **computational intensive** and preclude the use for **large action spaces**.
* **Deep Q network** models **scale nicely** with data and are **computationally less expensive** than GPs, but do not provide an **estimate of uncertainty** in vanilla form.

### Reason
* **Efficient exploration** is key to **successful policy optimisation**.
* **Q-function** can be augmented with an **estimate of its uncertainty** to **guide exploration** to achieve higher performance and efficient learning


### Solutions 
* get uncertainty info from **Bayesian neural networks**. Explore to area where the model is not confident
* **Five algorithms** to extract uncertainty estimates from deep Q-Networks
* **Four** of them can be casted within the **variational inference framework**
1. Bayes By Backprop 
 -- **All weights** are represented by **probability distributions** over possible values given observed dialogues.
 -- We use **sampling-based variational inference**. The intractable posterior is approximated with **variational posterior**.
2. Dropout 
--  Multiply the weight matrix in a given layer by some random noise.
3. Concrete dropout 
-- Continuously relax the dropout’s discrete masks and **optimize the dropout probability** using gradient methods. 
4. Alpha-divergence
-- measures the similarity between two distributions
5. Bootstrapped DQN
--  Several neural networks are randomly initialized which predict in ensemble uncertainty estimates.

### Result
![](https://i.imgur.com/2WJKurY.png)

state-of-the-art : GPSARSA
Bayes-by-Backprop algorithm achieves the best performance among the neural networks approaches

## BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems

### Problem 
* Deep Q-network (DQN) agents typically explore via the **$\epsilon$-greedy** heuristic, but when **rewards are sparse** and **action spaces are large** (as in dialogue systems), this strategy tends to **fail**.

### Solution
* improves the **efficiency of exploration** for deep Q-learning agents in dialogue systems.
* Bayes-by-Backprop captures **uncertainty information** from neural networks by maintaining a probability distribution over the weights in the network.

![](https://i.imgur.com/NgE84rs.png =75%x75%)
