## General Response to AC and Reviewer
(R1 = R-w2yt, R2 = R-yc4a, R3 = R-ZbF5, R4 = R-CLFG, R5 = R-D6DJ)

We sincerely thank the reviewers for their thoughtful and constructive feedback. We appreciate that all reviewers acknowledge the novelty of the proposed probabilistic residual vector quantization devised in a variational framework which enables direct residual code modeling without the need for cascaded modeling (R1, R2, R4, R5) and that our paper is clear and straightforward (R1, R3, R5), and conducts comprehensive experiments to demonstrate the efficacy of the proposed method (R3, R5).

As for the concerns/questions raised, we believe that we successfully addressed all of them sufficiently and replied in line with each review. We respond to some high-level comments here in the general response.

**Criticism 1: Effects of codeword emitting rate on the system**

We do appreciate the constructive suggestion of adding the experimental results to verify the effects of varying codeword rates on the proposed approach to further clarify our claim (R2, R4, R5).<!-- that our main model operates at a sweet spot in a trade-off (R2, R4, R5). -->

By following the suggestion, we conducted additional ablation studies to analyze the impact of varying codeword rates in the proposed framework. The results demonstrate a trade-off: *reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance.* This finding shows that we chose the codeword rate that operates at a sweet spot in this trade-off.
<!-- that reducing the code emitting frequency leads to the trade-off: *it degrades the quality of reconstructed audios from Mel-VAE, while increasing text-to-code language model performances.* This finding shows that we chose the codeword rate that operates at a sweet spot in this trade-off. -->

**The quality of reconstructed audios from Mel-VAE**: First, we compared the performances of Mel-VAEs trained with different codeword rates. The codeword rate in each MelVAE was adjusted according to the downsampling factor when generating latent representations from mel-spectrograms. We employed downsampling factors: 16, 8 (ours), and 4; and measured PESQ and ViSQOL of the reconstructed audio from each model. The below table shows that the audio reconstruction quality of the Mel-VAEs deteriorates as the downsampling factor increases.

| Model | PESQ | ViSQOL |
| -------- | -------- | -------- |
| Mel-VAE-df8 (default) | 2.95 | 4.66|
| Mel-VAE-df4 | 3.10 | 4.74|
| Mel-VAE-df16 | 2.42 | 4.35|

**Text-to-code language model performances**: Next, we trained identical latent language models to generate codes for each of the Mel-VAEs and measured WER, CER, and speaker similarity. Similar to the trend in reconstructed audio quality, the below table shows that a higher code emitting frequency tends to increase speaker similarity. However, intelligibility, measured by WER and CER, performed better with a 16-fold compression compared to a 4-fold compression. This suggests that the longer the code length predicted, the more challenging it is for latent language models. Moreover, the default setting of an 8-fold compression shows the best intelligibility, indicating that our setting finds a sweet spot in balancing audio quality and latent language model prediction in the trade-off.
| Model | WER | CER | SIM-o |
| -------- | -------- | -------- | -------- |
| ByT5-base-df8 (default) | 2.79 | 1.00| 0.3879|
| ByT5-base-df4 | 4.56 | 2.32| 0.4117|
| ByT5-base-df16 | 3.20 | 1.19| 0.3629|

**Inference time**: We also reported the end-to-end inference time of 10s speech, varying with frequency.
<!-- We included this ablation study in the revised manuscript. -->
| Model | Inference time |
| -------- | -------- |
| ByT5-base-df8 (default) | 3.46s |
| ByT5-base-df4 | 6.87s |
| ByT5-base-df16 | 1.88s |

We do believe this analysis not only strengthens our understanding of the system's performance under different conditions but also validates our choice of the default setting.

**Criticism 2: The performance of the proposed method compared to other methods**

We are grateful for the insightful observations made by the reviewers (R2, R3) concerning the comparative performance of our proposed method against other methods. The reviewers pointed out that Voicebox outperforms our method, and noted that the incorporation of phoneme and duration information may not be a significant factor in explaining this performance gap.

In light of these comments, we would like to clarify that our model can easily incorporate duration to improve the performance. We have validated and discussed several methods for performance enhancement in the subsequent paragraphs. Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

<!-- Nonetheless, we would like to reiterate why, unlike other approaches that employ these methods, we have chosen to rely solely on plain text input. Our model directly employs pretrained language models, enabling us to seamlessly integrate them, which have been trained on a diverse array of texts and tasks, in a plug-and-play fashion. -->

We incorporated the alignment search method from YourTTS to apply identified input durations and then evaluated its performance. The training involved adding the results of applying strided convolution to expanded character embeddings according to duration, to our decoder input embeddings. It is important to note that the only additional parameters in the resulting model are 164 character embeddings and a strided convolutional layer. The performance in terms of WER, CER, and speaker similarity is shown in the table below. When alignment from the alignment search was provided, we observed no quality gain in the continuation task.

| Model | WER | CER | SIM-o | SIM-r |
| ---------------- | ---- | ---- | ------ | ------ |
| CLaM-en | 2.36 | 0.79 | 0.4767 | 0.5128 |
| CLaM-en (w/ duration) | 2.53 | 0.86 | 0.4876 | 0.5203 |

However, we can observe a noticeable performance improvement in all metrics for the cross-sentence task, where our method exhibits greater degradation relative to other models and models are required to generate outputs from discontinuous texts and partially truncated audio prompts.
<!-- and our latent language model being trained to generate continuous and natural utterances from a sequence of naturally connected sentences using a pretrained text encoder and a transformer decoder. -->
| Model | WER | CER | SIM-o | SIM-r |
| ---------------- | ---- | ---- | ------ | ------ |
| CLaM-en | 5.11 | 2.87 | 0.4951 | 0.5382 |
| CLaM-en (w/ duration) | 3.59 | 1.55 | 0.5122 | 0.5484 |

Note that even with only negligible additional parameters, significant performance enhancement occurs in our method when duration is provided, particularly in cross-sentence cases where the input text is concatenated with partially truncated sentences. This also explains the limitations of our model's performance in the cross-sentence task where continuous sentences are not provided.

One potential solution to address this issue, akin to the approach used by SPEAR-TTS which maintains consistent performance in cross-sentence tasks, could be to explicitly inform the discontinuity between the audio prompt and the decoder output by a prompt separator token. This simple yet auxiliary method might help alleviate the identified problem. We acknowledge an error in the CER score (2.21) for SPEAR-TTS in the cross-sentence task and have corrected it to 1.92 in our manuscript (Table 1).

<!-- To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model. In Appendix D.2, we describe an ablation study where we replaced the encoder in our base-size model with different pretrained encoders. The results, shown in Table 11, reveal that the highest performance is achieved when using the phoneme encoder. -->

<!-- R2 pointed out that Voicebox outperforms our method, and noted that the incorporation of phoneme and duration information may not be a significant factor in explaining this performance gap. Additionally, R3 highlighted a deficiency in our comparison with other models and demo cases. In light of these comments, we present a detailed analysis of our method's performance relative to other models and evaluate the underlying reasons. -->

<!-- Although we have validated and discussed several methods for performance enhancement in the subsequent paragraphs, we would like to reiterate why, unlike other approaches that employ these methods, we have chosen to rely solely on plain text input. Our model directly employs pretrained language models, enabling us to seamlessly integrate them, which have been trained on a diverse array of texts and tasks, in a plug-and-play fashion. -->

<!-- We proceed to discuss the performance gap between Voicebox and our method, particularly in the cross-sentence task, where our method exhibits greater degradation relative to other models. This is primarily due to our latent language model being trained to generate continuous and natural utterances from a sequence of naturally connected sentences using a pretrained text encoder and a transformer decoder. Consequently, it underperforms in cross-sentence tasks involving discontinuous texts and partially truncated audio prompts. -->
<!-- This discussion includes considering the effects of phonemes and duration modeling in our method as well as in previous text-to-speech (TTS) systems. -->

<!-- To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model.

In Appendix D.2, we describe an ablation study where we replaced the encoder in our base-size model with different pretrained encoders. The results, shown in Table 11, reveal that the highest performance is achieved when using the phoneme encoder. -->

<!-- Moreover, since the introduction of non-autoregressive TTS methods, models using explicit duration modeling have shown more robust generation [1, 2]. -->
<!-- To further explore the effectiveness of applying the duration of text input in our method, we incorporated the alignment search method from YourTTS to apply identified input durations and then evaluated its performance. The training involved adding the results of applying strided convolution to expanded character embeddings according to duration, to our decoder input embeddings. It is important to note that the only additional parameters in the resulting model are 164 character embeddings and a strided convolutional layer. The performance in terms of WER, CER, and speaker similarity is shown in the table below. When alignment from the alignment search was provided, we observed no quality gain in the continuation task, but a noticeable performance improvement in all metrics for the cross-sentence task. This suggests that even with only negligible additional parameters, significant performance enhancement occurs in our method when duration is provided, particularly in cross-sentence cases where the input text is concatenated with partially truncated sentences. This also explains the limitations of our model's performance in the cross-sentence task where continuous sentences are not provided. -->

<!-- One potential solution to address this issue, akin to the approach used by SPEAR-TTS which maintains consistent performance in cross-sentence tasks, could be to explicitly inform the discontinuity between the audio prompt and the decoder output by a prompt separator token. This simple yet auxiliary method might help alleviate the identified problem.

Firstly, we acknowledge an error in the CER score (2.21) for SPEAR-TTS in the cross-sentence task, presented in Table 1, and have corrected it to 1.92 in our manuscript. -->

<!-- | Model | WER | CER | SIM-o | SIM-r |
| ---------------- | ---- | ---- | ------ | ------ |
| CLaM-en | 2.36 | 0.79 | 0.4767 | 0.5128 |
| CLaM-en-duration | 2.53 | 0.86 | 0.4876 | 0.5203 | -->

<!-- | Model | WER | CER | SIM-o | SIM-r |
| ---------------- | ---- | ---- | ------ | ------ |
| CLaM-en | 5.11 | 2.87 | 0.4951 | 0.5382 |
| CLaM-en-duration | 3.59 | 1.55 | 0.5122 | 0.5484 | -->

<!-- Next, we discuss the shortcomings in our comparison with other competitive models, including VALL-E, NaturalSpeech 2, and Mega-TTS. While official codes or models for these three are not available, we have added samples from each model's demo pages to our demo page for comparison, in line with your comments. Although there are limitations in the statistical significance of performance analysis using this small number of additional samples, we sincerely request the reviewer to consider these additional samples in their evaluation. -->

<!-- [1] Ren, Yi, et al. "Fastspeech: Fast, robust and controllable text to speech." Advances in neural information processing systems 32 (2019).
[2] Peng, Kainan, et al. "Non-autoregressive neural text-to-speech." International conference on machine learning. PMLR, 2020. -->

## Author's Respond to Reviewer w2yt
We do appreciate your positive review and valuable feedback, and we hope our response fully addresses your concern.

> `CLaM-TTS with and without the two-stage system`

Thank you for your valuable comment. While our method seamlessly generates from coarse to fine-grained tokens without the necessity for a two-stage pipeline, we do not believe that modeling our approach as a two-stage system would inherently degrade its performances. We want to clarify that, in this work, we provide a method that efficiently addresses the complex modeling challenges inherent in text-to-speech synthesis systems at scale. Our proposed probabilistic residual vector quantization not only enhances the reconstruction of audio quality but also offers a principled way to generate multiple codes within a variational inference framework. This approach effectively allows for large-scale text-to-speech synthesis without adding modeling complexities or compromising inference speed.

> `scaling up dataset and model sizes`

Thank you for highlighting the importance of conducting additional experiments to substantiate our claims. We have carried out an ablation study. Following your recommendation, we will incorporate these results into the revised manuscript.

**Scaling up dataset size**: The results in the below table show that the performance of our model improves with an increase in the volume of training data. Specifically, we trained ByT5-base models on different sizes of training datasets: 1. (*small dataset*) 50% of the MLS English subset; 2. (*normal dataset*) the full MLS English subset; and 3. (*large dataset*) a comprehensive English dataset that includes MLS English subset, Gigaspeech, LibriTTS-R, VCTK, and LJSpeech. We observed a noticeable degradation in performances (WER, CER, and speaker similarity) when the small dataset was employed. Conversely, training on the large dataset resulted in a performance enhancement in speaker similarity while maintaining WER and CER compared to the model trained on the normal dataset. This evidence strongly indicates that scaling up training data can significantly boost our model's performance.

| Dataset | WER | CER | SIM-o |
| -------- | -------- | -------- | -------- |
| small (22K hr) | 3.06 | 1.15| 0.3851|
| normal (44K hr) | 2.79 | 1.00| 0.3879|
| large (55K hr) | 2.79 | 0.98| 0.4001|

**Scaling up model size**: Due to time constraints, we were unable to train ByT5-base model on the large dataset with as many steps as the trained ByT5-large model. Consequently, a precise comparison is challenging at this stage, but we share the current results. We will include the experimental results in the revised manuscript as soon as they become available.
| Model size | WER | CER | SIM-o |
| -------- | -------- | -------- | -------- |
| ByT5-base | 2.79 | 0.98| 0.4001|
| ByT5-large | 2.36 | 0.79| 0.4767|

## Author's Respond to Reviewer yc4a
We would like to express our sincere gratitude for the constructive feedback and detailed suggestions, which helped us improve the manuscript. We provide point-by-point replies below.

> `Regarding the performance of the proposed method compared to VoiceBox`

Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. We acknowledge the observed performance gap of our method compared to Voicebox.

In response to the concerns, we would like to clarify that our model can easily incorporate phoneme or duration to improve the performance. We have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. We provide the empirical results and detailed discussion in the second section of “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

<!-- Nonetheless, we would like to reiterate why, unlike other approaches that employ these methods, we have chosen to rely solely on plain text input. Our model directly employs pretrained language models, enabling us to seamlessly integrate them, which have been trained on a diverse array of texts and tasks, in a plug-and-play fashion. -->

<!-- To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. -->

<!-- In light of these comments, we would like to clarify that our model can easily incorporate duration to improve the performance. We have validated and discussed several methods for performance enhancement in the subsequent paragraphs. Nonetheless, we would like to reiterate why, unlike other approaches that employ these methods, we have chosen to rely solely on plain text input. Our model directly employs pretrained language models, enabling us to seamlessly integrate them, which have been trained on a diverse array of texts and tasks, in a plug-and-play fashion. -->

<!-- We acknowledge the observed performance gap of our method compared to Voicebox, particularly in cross-sentence tasks where models are required to generate outputs from discontinuous texts and partially truncated audio prompts. -->

<!-- Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. In response to the concerns you raised, we have provided detailed explanations in the second section of our "Common Response to Reviewers". We demonstrate that the incorporation of phonemes and duration as inputs to our model leads to performance enhancements. This finding is pivotal in explaining why our model might experience a performance drop in cross-sentence tasks compared to other models. Furthermore, we have updated our demo page that showcases sample-to-sample comparisons with other works. We sincerely request that you review these additions for a more comprehensive understanding of our approach. -->

<!-- We acknowledge the observed performance gap of our method compared to Voicebox, especially in cross-sentence tasks, where our method exhibits greater degradation relative to other models. This is primarily due to our latent language model being trained to generate continuous and natural utterances from a sequence of naturally connected sentences using a pretrained text encoder. Consequently, it underperforms in cross-sentence tasks involving discontinuous texts and partially truncated audio prompts. -->

<!-- To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) and durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. Additionally, we have updated our demo page to include sample-to-sample comparisons with other works. We kindly request that you review these additions for a more thorough comprehension of our approach. -->

> `Clarification of the descriptions`

Thank you for pointing out the parts of our paper where the explanation was insufficient. We are grateful for your detailed feedback. In our work, the latent language model produces three distinct outputs, representing 1) the mixture weights, 2) the means of the mixture of Gaussian distribution, and 3) the probability of ending the generation. Initially, we described these three components collectively as “parallel predictors”. However, in light of your feedback, we have replaced this term in the text with the aforementioned description. Furthermore, we elucidate the decoding step in which multiple tokens are generated from the output mixture of Gaussians (MoG) distribution, consisting of the mixture weights and the means. During each decoding step, a latent vector sampled from this distribution is transformed into latent codes through the residual vector quantizer of Mel-VAE. The autoregressive decoding step concludes when the probability of ending the generation reaches or exceeds 0.5.

> `Missing references & typos`

In response to your feedback, we have added the missing reference and corrected typos, as well as other areas requiring revision, in the updated version of the paper.

> `Ablation study regarding the frequency`

To address the critical concern you raised, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance. This finding illustrates that our main model operates at a balance point in this trade-off. We sincerely request you to review them for additional context.

> `Using mel-spectrogram for the desired compression`

Our choice to use mel-spectrograms instead of raw waveforms as the input for the VAE is driven by a practical consideration. It requires substantially more memory and trainable parameters to train VAE with raw waveforms due to the extended input length. In contrast, using mel-spectrograms allows us to train VAE with considerably fewer resources as well as to use an off-the-shelf vocoder to convert mel-spectrograms to raw waveforms. Nonetheless, as the reviewer pointed out, pursuing higher reconstructed audio quality with the desired compression in an end-to-end manner with raw waveforms can be a promising direction for future work.

## Author's Respond to Reviewer ZbF5
We would like to thank the reviewer for providing constructive feedback and spotting unclear points.

> `Contribution of this paper`

To address the reviewer's concerns and clarify our methodology, we emphasize the distinctions between our approach and similar existing generative methodologies, VQGAN and RQ-Transformer, among generative models such as VAE and latent diffusion.

1. **Introduction and utilization of probabilistic residual vector quantization**: This feature not only enhances codebook usage to improve reconstruction quality but also offers a principled approach to generate residual codes from latent language modeling within a variational framework.
2. **Employing a single autoregressive transformer**: Our method proposes a process where a single autoregressive transformer can generate $N$ tokens at each decoding step by utilizing the probabilistic modeling of the residual vector quantizer, without the need for depth-transformers or non-autoregressive (NAR) transformer decoders. Our method shares similarities with VQGAN in that both involve 1) autoencoding of input data using VQVAE and 2) generation of the quantized latent representation using an autoregressive transformer. However, the vector quantizer in VQGAN employs plain vector quantization, producing a sequence of tokens, thereby enabling the transformer to model the discrete sequence with a softmax output. In contrast, RQ-Transformer and neural audio codec language models, which use $N$-depth residual vector quantization, produce $N$ sequences of tokens. This results in the need for not only an autoregressive transformer to decode along the time axis but also an additional depth-transformer to model $N$ tokens along the depth axis or a NAR transformer decoder to generate a token sequence at each decoding step.

<!-- Firstly, the primary distinction of our proposed method compared to others is the introduction and utilization of probabilistic residual vector quantization. This feature not only enhances codebook usage to improve reconstruction quality but also offers a principled approach to generate residual codes from latent language modeling within a variational framework.

Our method shares similarities with VQGAN in that both involve 1) autoencoding of input data using VQVAE and 2) generation of the quantized latent representation using an autoregressive transformer. However, the vector quantizer in VQGAN employs plain vector quantization, producing a sequence of tokens, thereby enabling the transformer to model the discrete sequence with a softmax output. In contrast, RQ-Transformer and neural audio codec language models, which use N-depth residual vector quantization, produce N sequences of tokens. This results in the need for not only an autoregressive transformer to decode along the time axis but also an additional depth-transformer to model N tokens along the depth axis or a non-autoregressive (NAR) transformer decoder to generate a token sequence at each decoding step. Our method, however, proposes a process where a single autoregressive transformer can generate N tokens at each decoding step by utilizing the probabilistic modeling of the residual vector quantizer, without the need for depth-transformers or NAR transformer decoders. -->

<!-- To facilitate a clearer understanding of these differences from existing methodologies, we add a detailed explanation of this in the appendix. -->

> `Regarding the performance of the proposed method`

<!-- Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. In response to the concerns you raised, we have provided detailed explanations in the second section of our "Common Response to Reviewers". We demonstrate that the incorporation of phonemes and duration as inputs to our model leads to performance enhancements. This finding is pivotal in explaining why our model might experience a performance drop in cross-sentence tasks compared to other models. Furthermore, we have updated our demo page that showcases sample-to-sample comparisons with other works. We sincerely request that you review these additions for a more comprehensive understanding of our approach. -->
Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review.

We acknowledge the observed performance gap of our method compared to Voicebox, particularly in cross-sentence tasks where models are required to generate outputs from discontinuous texts and partially truncated audio prompts. To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) and durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. In response to the concerns you raised, we have provided detailed explanations in the second section of our “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

Moreover, as per your great suggestion, we have included samples from other competitive models (VALL-E, NaturalSpeech 2, and Mega-TTS) on our demo page for comparison. These samples were taken from each model's demo pages. We note that the small number of these additional samples is due to the lack of official codes or pre-trained models. We acknowledge that the performance analysis using this limited number of additional samples may not have statistical significance. Nevertheless, we kindly request the reviewer to consider these additional samples in their evaluation.

demo page: https://clam-tts-mos.s3.us-east-2.amazonaws.com/demo/index.html

<!-- Regarding the shortcomings in our comparison with other competitive models including VALL-E, NaturalSpeech 2, and Mega-TTS, while official codes or models for these three are not available, we have added samples from each model's demo pages to our demo page for comparison, in line with your comments. Although there are limitations in the statistical significance of performance analysis using this small number of additional samples, we sincerely request the reviewer to consider these additional samples in their evaluation. -->

## Author's Respond to Reviewer CLFG
We would like to thank the reviewer for providing a positive review and spotting unclear points.

> `Regarding the effect of codeword emitting frequency`

Thank you for pointing out the parts in our paper that were not explained clearly. The semantic tokens in AudioLM are a sequence of 50Hz, while our tokens consist of 32 sequences of 10Hz due to the use of 32-depth residual vector quantization. This means that when representing audio, we can achieve a higher "length" compression rate compared to AudioLM by compressing the audio into multiple sequences using RVQ. We have conducted experiments and evaluations regarding the codeword emitting frequency, which we addressed in the first section of “General Response to AC and Reviewer”. These results demonstrate that our main model operates in the sweet spot for the trade-off. We sincerely request you to review them for additional context.

> `Definitions of $y$ and $\hat{y}$`

For the definitions of the variables, please refer to the second paragraph in Sec. 3.2.1. $y$ and $\hat{y}$ denote a mel-spectrogram and its reconstruction from the decoder, respectively.

## Author's Respond to Reviewer D6DJ
We sincerely appreciate your positive review and constructive feedback.

> `Regarding the tradeoff between efficiency and effectiveness in varying codeword rate`

As per your great suggestion, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the first section of “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: *reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance*. We also reported on the inference speed of the entire system in relation to varying code frequencies. This finding illustrates that our main model operates at a balance point within this trade-off, efficiently managing both aspects.