# MusicLM Study Guide

MusicLM ([Paper](https://arxiv.org/pdf/2301.11325.pdf), [Examples](https://google-research.github.io/seanet/musiclm/examples/))

Components:

- SoundStream: E2E neural audio codec
- [Paper](https://arxiv.org/pdf/2107.03312.pdf)
- [Talk](https://www.youtube.com/watch?v=V4jj-yhiclk) by first author Neil Zeghidour
- Residual Vector Quantizer (RVQ)
- Extension of [VQ-VAE](https://arxiv.org/abs/1711.00937)
- Inspired by [multistage vector quantization (MSVQ)](https://www.researchgate.net/publication/26501392_Multiple-Description_Multistage_Vector_Quantization)
- Basis/inspiration for AudioLM
- Generative audio model via sequence modeling of SoundStream (acoustic) and w2v-BERT (semantic) tokens
- [Blog](https://ai.googleblog.com/2022/10/audiolm-language-modeling-approach-to.html)
- [Paper](https://arxiv.org/pdf/2209.03143.pdf)
- [Examples](https://google-research.github.io/seanet/audiolm/examples/)
- w2v-BERT
- [Paper](https://arxiv.org/abs/2108.06209)
- wave2vec 2.0
- [Paper](https://arxiv.org/pdf/2006.11477.pdf)
- [Lecture](https://www.youtube.com/watch?v=8Kpowre6yyk) from CU Boulder
- BERT
- [Paper](https://arxiv.org/pdf/1810.04805.pdf)
- [Yannic Kilcher overview](https://www.youtube.com/watch?v=-9evrZnBorM)
- MuLAN: A Joint Embedding of Music Audio and Natural Langauge
- [Paper](http://www.joonseok.net/papers/mulan.pdf)