###### tags: `PaperReview`
[Paper Link](https://research.facebook.com/file/6486163864750413/Scaling-Speech-Technology-to-1,000+-Languages.pdf)

> Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli
> Meta AI

## Introduction
- Speech technology has made much progress over the past decade and has been integrated into many consumer products, such as home assistants and smartphones.
- To address this, a new dataset comprising a moderate amount of labeled data for 1,107 languages and another dataset of unlabeled speech in 3,809 languages was build.
- The Massively Multilingual Speech (MMS) project aims to expand speech technology to many more people and we hope that it can be a small contribution to preserving the languages diversity of this world.

## Dataset Creation
- leverages two new datasets to expand the language coverage of speech technology.
- **speech audio** paired with **corresponding text** in **1,107 languages** (MMS-lab; 44.7K hours)
- **audio** recordings and **no corresponding text** in **3,809 languages** (MMS-unlab; 7.7K total hours).
- unlabeled version of MMS-lab is also used for pretraining and language identification, creating a new dataset (MMS-lab-U; 1,362 languages; 55K Hours)

### Paired Data for 1,107 languages (MMS-lab)
- Obtain speech data and transcriptions for 1,107 languages by aligning New Testament texts obtained from online sources using the following steps:
1. **Download and preprocess** both the speech audio and the text data.
2. **Apply a scalable alignment algorithm**.
3. Initial Data Alignment: **train an initial alignment model** using existing multilingual speech dataset and **use the model to align data for all languages**.
4. Improved Data Alignment: **train a second alignment model** on the newly aligned data for which the original alignment model has high confidence and generate the alignments again.
5. Final data filtering: **filter the low-quality samples** of each language based on a **cross-validation procedure**. Train a monolingual ASR model for each language with half of the aligned data to transcribe the other half.
6. **Partition** the data into train, development, and test.

#### Data Source
- Dataset was based on recordings of people reading the New Testament in different languages.
- Specifically, Obtain data from *Faith Comes By Hearing*, *goto.bible* and *bible.com*. This includes the original text data as well as the corresponding audio recording.
- Basic Data Characteristic:
- **MMS-lab-U** (**1,362** languages; **55K** Hours) and **MMS-lab** (**1,107** languages; **44.7K** hours)
- For 99 languages, there exist multiple recordings with each recording is on average 6.7 minutes long but there is significant variance, depending on the language and chapter.
- Most of recording are **single speaker** thus unwanted bias may be introduced.
- Recordings with Background Music:
- Some of the recordings contain background music and will be referred as **drama recordings**.
- In MMS-lab, 38% of language are represented solely by a drama recording, and 11% have both drama and non-drama.
- Apply pre-processing to remove background music.

#### Data Pre-processing
- Text Normalization
- First, perform NFKS normalization and lower-case all character.
- Remove HTML tags (">", "nbsp;") and punctuations.
- Also remove brackets and the text within since by observation, most of the time the bracket was not spoken.

#### Scalable Forced Alignment
- Recording from the data can be up to 43 minutes long with cannot be directly used to train so segmentation is needed to make the data smaller.
- Segment the data into individual verses which are typically a single sentence but can sometimes contain several sentences. The average duration of a verse is about 12 seconds.

##### Generating Posterior Probabilities
- Forced alignment requires posterior probabilities from an acoustic model which will be used for alignment.
- Chunk the audio files into 15 second segments, generate posterior probabilities for each audio frame using the alignment model, and then concatenate these posterior probabilities into a single matrix again.
- The acoustic model is trained with Connectionist Temporal Classification (CTC)

##### Forced Alignment using CTC
- Perform forced alignment which finds the most likely path in the posterior probabilities for a given input audio sequence of length T and a text transcription of length L.
- Compute path using Viterbi algorithm.
- Requires $\mathcal{O}(T x L)$ memory which is slow to run.

##### Efficient Forced Alignment on GPU
- Implemented a GPU version that computes the Viterbi path memory in a memory efficient way.
- Only store forward values for the current and the previous time-step and regularly transfer the computed backtracking matrices to CPU memory.
- Reduce required memory to $\mathcal{O}(L)$

##### Robust Alignment for Noisy Transcripts
- For many recordings, speakers introduce the chapter name and the version of the New Testament before reading the first verse, however, the corresponding text does not contain this information.
- Also numbers are generally spelled as digits in the text whereas our alignment model is trained on existing corpora which follow common practice of spelling numbers out fully.
- Introduce a star token **⟨∗⟩** to tackle this problem.
- Insert **⟨∗⟩** at the beginning of the text data and replace numerical digits with **⟨∗⟩**.
- Set the posterior probability for this token to one.
- After alignment, add back the original digits and the subsequent data filtering often removes segments where the audio contains additional information not present in the aligned text.

#### Initial Data Alignment
- Train a multilingual acoustic model on FLEURS and CommonVoice 8.0
- The model is based on fine-tuning XLS-R using total of 8K hours of data covering 127 languages.
- The text data is represented using the uroman transliteration tool which maps different writing scripts to common Latin scripts representation.
- Lowercase all the letters of the uroman output and retain only a to z characters as well as the apostrophe character.

#### Improved Data Alignment
- Use a subset of good-quality samples to train a new alignment model.
- Samples are selected based on the following formula
$$
\frac{1}{T} \log P\left(Y^{\text {aligned }} \mid X\right)-\log P\left(Y^{\text {greedy }} \mid X\right)
$$
- The score can range from (-inf, 0]. And select the sample with score $\geq -0.2$.

#### Final Data Filtering
- Observation shows that some recordings are not entirely faithful to the text and speakers sometimes add their own interpretation or paraphrase parts of the text.
- So monolingual ASR models are trained on half of the the aligned samples of each recording, measure performance on the remaining half and remove samples which have a character error rate (CER) exceed 10%.
- This removes about 1.7% of all samples across all languages.

#### Creating train/dev/test splits
- From MMS-lab-U, 36.8K hours will be used for training (82.3%), 3.5K hours for development (7.8%) and 4.4K hours for testing (9.9%).
- For each language, the train split contains an average of 32 hours (stddev=19), the dev split contains an average of 3.1 hours (stddev=1.8) and test split an average of 3.9 hours (stddev=2.3).

### Unpaired Data for 3,809 Languages (MMS-unlab)
#### Data Source
- From *Global Recordings Network* which provides recordings of Bible stories, evangelistic messages, scripture readings, and songs in more than 6,255 languages and dialects.
- group the data by language, combining dialects of the same language resulting in a total of 3,860 languages and 9,345 hours of audio.

#### Pre-processing
- Convert audio files into single channel 16kHZ.
- use inaSpeechSegmenter to identify segments of speech, music, noise, and silence in the audio.
- If two segments of speech are separated by intermediate segments containing music or noise, consider joining these segments if the intermediate segment is no longer than 20% of all segments together to build samples that are of longer duration.
- The remaining non-speech segments are discarded.
- Randomly split the speech segments into portions of between 5.5 and 30 seconds.

#### Dataset Split
- Split the samples of each language randomly into 80% training data, 10% development data, and 10% test data.
- Remove 51 languages for which training data is less than 5 minutes.
- Total of 7.7K hours in 3,809 languages.
- The training portion is 6.2K hours and there are 770 hours for the valid and test sets each.
- For each language, the train set contains an average of 97 minutes (stddev=177.4), and the dev/test sets contain an average of 12.1 minutes (stddev=22.3).

### Comparison to Existing Broad Coverage Approaches and Other Datasets

#### CMU Wilderness Dataset
- The most comparable prior work is the CMU Wilderness project which used data from similar sources.
- Improvement between 2.1% - 4.7% CER depending on the language.
- With this method, they also retains much larger amount of dataset.
- for Telugu, there are 26.5 hours of data, MMS-lab retains 26.2 hours compared to 11.1 hours for the CMU Wilderness process. For English, starts with 17.3 hours, MMS-lab retains 17 hours vs. 10.6 hours for CMU Wilderness.

#### ASR-2K
- ASR model that covers 1,909 languages.
- The monolingual models trained on MMS-lab dataset obtain an average CER of 9.6 on 22 languages. ASR-2K reports CER 50.9 on 34 languages.
- While not a like for like comparison, this difference suggest that MMS-lab enables higher quality ASR models.

#### Other Existing Dataset
- Models trained on CommonVoice perform better on 18 languages of FLEURS (average CER 9.3 vs. 12.2) but the models MMS-lab still enable good performance.
- this result suggests that the quality of the MMS-lab data can enable high quality speech systems for a large number of other languages.

## Cross-Lingual Self-Supervised Speech Representation Learning
- First, train a self-supervised model of speech representation.
- Use wav2vec 2.0 for pre-training on unlabeled data which later will be used as the basis for several downstream speech tasks.
- The resulting models were pre-trained on 1,406 languages.
- The increased language coverage results in better performance for both ASR and LID compared to XLS-R which covered 128 languages and is publicly available.

### Method: wav2vec 2.0 and XLS-R
- Pretrain wav2vec 2.0 models on data from multiple languages.
- During training the feature encoder representations are discretized to $q_1, \ldots, q_T$ with a quantization module $\mathcal{Z} \mapsto \mathcal{Q}$ to represent the targets in the objective.
- The quantization module uses a Gumbel softmax to choose entries form the codebooks and the chosen entries are concatenated.
- The model is trained by solving a contrastive task over masked feature encoder outputs.
- The objective is augmented by a codebook diversity penalty to encourage the model to use all codebook entries.
- In order to balance the training data, two data sampling steps are performed:
- Sample the data for the different languages L from a distribution $p_l \sim (\frac{n_l}{N})^{\beta_L}$ where ${\beta_L}$ is the upsampling factor which controls the trade-off between high- and low-resource languages during pretraining.
- balance the different datasets by treating each dataset as a language in the above sampling scheme with a sampling parameter ${\beta_D}$.

### Pre-training Setup
#### Data
- In total of 491K hours in 1,406 languages

#### Comparison to XLS-R
- Figure above shows that MMS performs better.


- MMS pre-trains on over ten times the number of languages of XLS-R and this improves performance, particularly on low resource languages but small degradation for high-resource languages.

## Automatic Speech Recognition
### Modeling and Training Approach
- Fine-tuning MMS (1B) with labeled data and add linear layer on top to maps to an output vocabulary.
- Fine-tune the entire model with CTC criterion.

#### Language-specific Adapters, Head, and Fine-Tuning (LSAH)
- Introduce adapters at every block of the transformer, and the adapter is added after the last feed-forward block.
- The adapter module consists of a LayerNorm, a downward linear projection followed by a ReLU activation and an upward linear projection; the inner dimension of the projections is 16.

### Scaling Multilingual ASR to 1,107 Languages
- This shows that scaling multilingual ASR models to over one thousand languages is feasible and that there is little performance degradation when coupled with language-specific parameters.

### Comparison to Other Works
#### Whisper
- MMS reduces the word error rate of Whisper by a relative 58% while supporting over 11 times the number of languages.

#### Google USM
- MMS performs very competitively compared to USM.

### Robust Multilingual ASR Models
- multi-domain model (MMS-lab+FL+CV+VP+MLS) can perform very competitively in several settings.

### Evaluation on 1,107 Languages
- MMS meets the CER quality threshold for 96% of the 1,107 languages.

## Language Identification
- Language Identification (LID) is the task of determining the language which is spoken in a given utterance.

### Training Setup
- First fine-tune MMS (1B) for LID and stack linear classifier to classify the possible languages for a particular task.
- Then fine tune whole parameter including the pre-trained model.

### Comparison to Existing Datasets
- MMS-lab-U+unlab enables LID with slightly lower performance compared to systems trained on existing datasets when both are evaluated out-of-domain.

### Scaling Language Identification to 4,017 Languges
- MMS models scale very well; increasing the number of languages from 126 to 4,017 results in a modest performance drop of just 0.3% on FLEURS and no drop on VoxLingua-107.
- Out-of-domain has a drop of 3.6% on BABEL and 0.2% on VoxPopuli.

## Speech Synthesis
- Speech synthesis or text-to-speech (TTS) where models output speech for a corresponding input text.

### Text-To-Speech Model
- Based on VITS and will be scaled to 1,107 languages.
- Train separate VITS models for each language.

### Text and Speech Data Pre-processing
#### Data Selection
- Using the MMS-lab dataset to train the TTS model.
- For 99 languages which have multiple recordings, choose the recording with lowest CER when fed to a trained ASR model.

#### Text Representation
- Represent the input text as individual letters for languages with a small vocabulary. For languages with 200 or more characters, we use a uroman encoding.

#### Speech Data Pre-processing
- Remove background music if there is any to enhance the quality of TTS models using denoiser model.
- Remove multi speaker utterances by removing top 15% of the utterances with highest pitch variance.

### Evaluation Methodology
- Mel-Ceptral Distortion (MCD)
- measures the closeness of synthesized speech to a human utterance for the same text in terms of the warping distance of mel frequency cepstral coefficients.
- Automatic Speech Recognition (ASR)
- Transcribing the synthesized speech with automatic speech recognition and then measuring the error rate.
- Mean Opinion Score (MOS)
- Ask raters to judge the fidelity of the synthesized samples together with how natural the speech sounds.

### Evaluation of Design Choices

#### Training Setup
- Reduced setup (row 5) generally performs less well than the highly optimized setup of VITS for LJSpeech (row 2; Kim et al. 2021) and each design choice (fewer training updates, MMS-lab training data and character inputs) leads to a reduction in quality
- Note that the CER of almost all models on FLEURS data is lower than for the corresponding natural speech and that the MOS scores of the human reference utterances is also lower than for other datasets.

#### Data with Background Music
- Train a model on an English recording with background music before and after the pre-processing steps and compare this to a model trained on another English recording that does not contain any background music to start with.
- Both denoising and removing samples with multi-speaker utterances results in performance improvements compared to the original data with background music.

### Out-of-Domain Evaluation
- MMS-lab data is from a particular narrow domain which poses the question of whether TTS models trained on this data will generalize to other domains.
- Train speech synthesis models and evaluate their quality on both in-domain and out-of-domain data.
- MMS models are robust to domain shift: the CER of synthesized speech (TTS) is only slightly higher out-of-domain compared to in-domain and MOS scores for the synthesized samples are nearly identical.
- TTS models trained on MMS-lab data perform well out-of-domain.

### Evaluation on 1,107 Languages
- Measure MCD and ASR CER on the MMS-lab test sets.
- about 85% of the 1,107 languages meet the CER quality threshold.

## Bias Analysis and Ethical Consideration
### Gender Bias
- Most speakers in MMS-lab dataset appear to be male and this bears the risk of machine learning models trained on this data performing better for male speakers.
- The average character error rate over these 27 languages is very similar, both for the MMS model and the model trained on FLEURS data.
- MMS exhibit similar gender bias to models trained on FLEURS data which is general domain data.

### Language Bias
#### Methodology
- Analysis is to identify a set of biased words in each language and to measure the rate at which biased words are produced by ASR models.
- The rate of biased words is much lower in the outputs of MMS models compared to the training data.

## Conclusion and Open Problems
- They propose first study which scaled speech technology to over one thousand languages.
- They presented how they collected datasets, pretrained models and then built models for automatic speech recognition, language identification and speech synthesis.
- Future works:
- Scaling to even more languages and dialects
- Multi-task models
- train a single model for several downsteam tasks.
- Tackling more speech tasks
- such as speech translation, keyword spotting, intent classification, etc.

## Ablation