# Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

*Adyasha Patra, Ankan Sarkar, Govind Kumar, Harsh Poonia*
## Introduction
In the era of transformative advances in artificial intelligence, we've witnessed lots of feats accomplished by large-scale generative models like GPT and DALL-E. These technologies have not only surprised us with their ability to generate lifelike text and captivating images, but have left us in awe with their remarkable ability to generalize on tasks they have not even been trained on.

---
*Yet, amidst these triumphs, there's a lingering question: **What about speech?***

---

Introducing Voicebox[^1], a groundbreaking innovation poised to bridge the gap in speech technology - **the first generative AI model for speech to generalize across tasks with state-of-the-art performance**. Similar to GPT, it can perform a myriad of tasks through in-context learning, but with added flexibility as it can also condition on future context. From mono to cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, to diverse sample generation, Voicebox excels. It has a non-autoregressive flow-matching architecture, trained on an extensive corpus of over 60,000 hours of English audiobooks and 50,000 hours of multilingual audiobooks across six languages. If all this sounds like too much, don't worry we are going to walk you through everything!

---

## Related Work

Voicebox emerges as a culmination of inspiration drawn from existing work in Generative speech models and Large scale in-context learning models.

Speech generative models are very task specific. Tasks include:

---
* *Audio style conversion* where specific attributes are modified while preserving others; challenges persist in finding datasets and annotations.
* *Controllable text-to-speech synthesis* aims to generate speech in a desired style from text input, but struggles with aspects like prosody control, that is rythmn in poetry or speech. Most existing models like [YourTTS](https://arxiv.org/abs/2112.02418) use annotated labels or embeds which are again not always available.
* *Infilling* which is predicting speech from context, encounters scalability issues due to deterministic mappings from text and context to target speech.

---

Advancements in Large scale in-context learning has led to the development of token-based language models for generating speech. One approach, called the GSLM-family built upon [HuBERT](https://arxiv.org/abs/2106.07447)[^2] units, focuses on generating speech without using text but struggles to preserve the voice of the prompt. Another model, [VALL-E](https://arxiv.org/abs/2301.02111)[^3], demonstrated the state-of-the-art (SOTA) zero-shot TTS performance through in-context learning, where speech of the desired style is used as prompt, but it requires a lot of computational steps.

Voicebox caters to both these family of models. On one hand,it can infill speech of any length, can be trained on in-the-wild datasets with rich variation, and provide a general solution that subsumes many tasks in a text-guided fashion.

On the other hand, it can generate speech more efficiently and with better control. It can use information from both the past and future to generate speech, making it useful for editing. It can generate speech faster than VALL-E and offers finer control over the details of the speech.

Now that we are aware of all the challenges at hand, let's dive into how Voicebox solves all of this!

----
## Method

Let us first give a bit of background on the underlying principles. Say we want to learn some the distribution $q(\cdot)$ in some high dimensional vector space. Continuous Normalizing Flows (CNFs) are a family of generative models that learn the transformation from a simple prior distribution $p_0$ (e.g., normal distribution) to the data distribution $p_1 ≈ q$. At each point in time, a CNF induces a vector field $v_t(x; \theta)$ (parameterised by $\theta$), which gives the direction of flow for every single point in our vector space. This vector field, nudging each point in some direction to eventually take points in the prior to some vector from the target distribution, creates a *flow* $\phi_t(\cdot)$, which essentially tells us where the starting point sampled from the prior ends up moving with the flow. If we sample a point $x_0$ from the prior $p_0$, we can estimate the density at some time $t$, also called the *probability path* as $p_t(x = \phi_t(x_0))$. We estimate the probability path using a mixture of simpler *conditional paths* whose vector fields can be easily computed. To get an unbiased estimate for target distribution, we can then sum over all the conditional paths $\left(\displaystyle \sum_{x_1 \sim q}p_t(x | x_1)\right)$.

Naturally, the next question is how to choose a conditional flow. A flow defines trajectories, which dictates how each point moves between $p_0$ and $p_1$. Intuitively, a simpler trajectory (e.g., a straight line) can be learned faster and the ODE governing the flow can be solved more accurately and efficiently. *Optimal transport path* is a conditional flow where points move with a constant speed and direction. We adopt it for Voicebox.

***We define our problem as follows:***
Given a dataset of transcribed speech, the goal is to build a single model that can perform many text-guided speech generation tasks through in-context learning. We propose to train such a generative model on the text-guided speech infilling task, which predicts a segment of masked speech given its surrounding audio and the complete text transcript. This masked prediction is similar to encoder-only, non-autoregressive pretraining models like BERT.

---

<figure>
<img src="https://hackmd.io/_uploads/ByXoxwal0.png"
alt="Voicebox Model">
<figcaption>Voicebox audio model. The lower half illustrates how inputs are created during training.</figcaption>
</figure>

---

Now that we have discussed the structure of the model, the next big thing to ask is how to train it. VoiceBox is decoupled into two major components, *an audio model and a duration model*. Lets delve towards the **audio model** first.

Notice that the distribution of the audio features is highly stochastic (especially when there's a large temporal span) and, thus, there's a need to parametrize the audio with a CNF and train it using the flow matching objective with the optimal transport path. This paves the way for employing a transformer model which is used to parametrize the conditional vector field, $v_t$. By concatenating the noisy speech, masked speech and the phonetic embedding, we form the input to the Transformer model after adding a sinusoidal positional encoding along the time dimension. The objective function is simply a MSE loss applied only on the masked frames.

---
> *This sums up the audio model and, now, lets divert our attention to the duration model.*

---

The **duration model** regresses the masked duration, $l_{mis}$, given the context duration, $l_{ctx}$ and phonetic transcript $y$. The Transformer model can be reused but we need to note that now there are only two input sequences instead of three, and that the time embedding is not required. The loss function employs an $L1$ regression instead of MSE. This completes the discussion on the model training and we head forward to inference.

Lets summarize the proposed model. Given the learned audio distribution, a noise, $x_0$, is first sampled from the prior distribution, $p_0$, and an ODE solver is being incorporated to evaluate the value of $\phi_1(x_0)$ given the derivative of the flow function. The ODE solver aims to tackle the problem using numerical integration. A metric used in this case is *number of function evaluation (NFE)* which calculates the number of times we calculated the value of the derivative. A higher NFE yields accurate solution but at the cost of higher runtimes. Thus, the model gives a lot of flexibility for users to decide the trade-off between speed and accuracy.

---
## Applications

Now dont worry if you don't get the idea completely, that's perfectly fine. Anyway we know the only question you have in your mind is "In what ways will this model help me?" *(just kidding :))*. Well guess what, we are going to discuss that right away now.

- **Zero-shot TTS**: Providing the target text and a transcribed reference audio, VoiceBox aims to synthesize unseen audio style speech by treating the reference audio and the masked target speech as one single utterance.
- **Transient noise removal**: Ever heard a speech so noisy that one can hardly make out the words? Well, VoiceBox for the win! With VoiceBox one can just edit the problematic segment by re-generating the noise corrupted segment given the original transcript and the surrounding clean audio.

---
<figure>
<img src="https://hackmd.io/_uploads/HkEanBWbR.png"
alt="Voicebox Model">
<figcaption>Task generalization via in context learning</figcaption>
</figure>

---
- **Content editing and alignment preservation**: Ever thought about uploading Vlogs but stepped back being unable to generate quality content on the first go? Well, you know what to do now. Using VoiceBox, one can create the edited frame-level phone transcript simply by passing the edited transcript and the duration of existing phones. It can also convert the audio style while preserving its alignment making it perfect for audio-editing that is synchronized with other modalities such as video.
- **Diverse speech sampling**: VoiceBox can generate diverse speech samples by infilling the whole utterance along with audio-style shuffling while at the same time preserving the alignment. Pretty cool, huh!

---

## Results

Now, let's crunch some numbers. The metrics we will use to compare our model are:
- **Correctness**: Our classic Word Error Rate *(WER)* on synthesized speech using a
speech-to-text model *(HuBERT-L/Whisper)*
- **Coherence**: Similarity *(SIM-o)* between embedding of context and generated speech
- **Fréchet Speech Distance** *(Diversity)*, **QMOS** *(Quality)*, **SMOS** *(Similarity)*

<!-- ---
<figure>
<img src="https://hackmd.io/_uploads/BJFuqIbWR.png"
alt="Table 1">
<figcaption>English zero-shot TTS results on filtered LS
test-clean</figcaption>
</figure>

--- -->

---

| Model | WER | SIM-o | SIM-r | QMOS | SMOS |
| ------------ | --- | ----- | ----- | ---- | ---- |
| Ground truth | 2.2 | 0.754 | n/a | 3.98 | 4.01 |
| cross sentence | | | | | |
| A3T | 63.3 | 0.046 | 0.146 |-| - |
| YourTTS | 7.7 | 0.337 | n/a | 3.27 | 3.19 |
| VALL-E | 5.9 | - | 0.580 | - | - |
| VB-En | 1.9 | 0.662 | 0.681 | 3.78 | 3.71 |
| continuation | | | | | |
| A3T | 18.7 | 0.058 | 0.144 | - | - |
| VALL-E | 3.8 | 0.451 | 0.508 | - | - |
| VB-En $(α = 0.7)$ | 2.0 | 0.593 | 0.616 | - | - |

***English zero-shot TTS results on filtered LS*** *(QMOS and SMOS values are approximate)*

-----

In *monolingual zero-shot TTS*, Voicebox outperforms the current SOTA VALL-E, achieving lower WERs $(5.9\% vs. 1.9\%)$ and higher similarity scores $(0.580 vs. 0.681)$. It also outperforms all other models in Mean Opinion Score *(MOS)* studies, offering effective style transfer and better quality than the YourTTS model.

In *cross-lingual zero-shot TTS*, Voicebox also exceeds YourTTS’s performance in every area, including in languages such as English, French, and Portuguese, reducing average WERs from $10.9\%$ to $5.2\%$ and improving average audio similarity from $0.335$ to $0.481$. On average, Voicebox yielded higher MOS for audio similarity and quality.

In terms of noise removal tests, the WERs for Voicebox are again lower than A3T or Demucs. Even in difficult noise environments, Voicebox fared well. It was demonstrated to produce samples that were higher quality than those from the other models, A3T and Demucs, and more comprehensible, and had greater resembling in the clear segments of the audio.

---
<div align="center">

| Model | WER | SIM-o | QMOS |
| ----------------- |:----:| ----- | ---- |
| Clean speech | 2.2 | 0.687 | 4.07 |
| Noisy speech | 41.2 | 0.287 | 2.50 |
| Demucs | 32.5 | 0.368 | 2.86 |
| A3T | 11.5 | 0.148 | 3.10 |
| VB-En $(α = 0.7)$ | 2.0 | 0.612 | 3.87 |

**Transient noise removal where noise overlaps with $50\%$ of the speech at a $-10dB$ $SNR$** *(QMOS values are within 0.15 of the values given in the table)*

</div>

---
<!-- ---
<figure>
<img src="https://hackmd.io/_uploads/H1mvsIZW0.png"
alt="Table 2">
<figcaption>Transient noise removal where noise
overlaps with 50% of the speech at a -10dB
SNR.</figcaption>
</figure>

--- -->

<figure>
<img src="https://hackmd.io/_uploads/HymN9vWZR.png"
alt="Fig 3">
<figcaption></figcaption>
</figure>

**Trade-off between NFE and different metrics**

---

The inference time on the lowest setting $(NFE = 2)$ is around $20x$ faster than VALL-E. On $NFE=64$, Voicebox is only $4\%$ slower than VALL-E.

In the cross-sentence setup, Voicebox has better WER than VALL-E. The WER remain stable around $2$. In the case of $SIM-r$, lower classifier guidance strength values $(α = 0$ $or$ $0.3)$ resulted in higher speaker similarity in a lower NFE regime. However, as NFE increased beyond $8$, higher classifier guidance strength improved speaker similarity.

Examining FSD by generating samples for **Librispeech** test-other text, it was found that lower classifier guidance strength produces lower FSD scores and more diverse samples. Through FSD and subjective listening, it was discovered that a lower NFE leads to generating less diverse samples especially when the guidance weight is lower. Although those samples are of lower quality, they are easier for the ASR model to recognize because they tend not contain extreme audio styles like whispering or high background noise. As a result, WERs are lower.

---
## Future Scope

Voicebox was trained on audiobooks, which may not transfer well to conversational speech which is more casual and contains more non-verbal sounds such as laughing and backchanneling like *"um-hmm"*.

On the other hand, Voicebox depends on a phonemizer and a forced aligner to produce frame-level phonetic transcript. Most phonemizers are word-based, which do not take into account neighboring words when predicting pronunciation. This becomes a problem in languages like French where the pronunciation is context-dependent.

Lastly, while Voicebox yields impressive results on transferring audio style *(voice, speaking style, emotion, and acoustic condition)*, the model does not allow independent control of each attribute. In other words, one cannot ask the model to generate speech that resembles voice of one sample while resembling the emotion of another sample.

With high fidelity speech generation models like Voicebox, it brings the potential for misuse and unintended harm. The paper presented a highly effective classifier that can distinguish between authentic speech and audio generated with Voicebox to mitigate these possible future risks. Future directions would include using diverse types of speech such as conversations, and exploring disentangled prompting for different attributes.

[^1]: [Matthew Le et al, "Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale," in Thirty-seventh Conference on Neural Information Processing Systems, 2023.](https://openreview.net/forum?id=gzCS252hCO)
[^2]: [Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, & Abdelrahman Mohamed. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units.](https://arxiv.org/abs/2106.07447)
[^3]:[Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, & Furu Wei. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers.](https://arxiv.org/abs/2301.02111)