## Reviewer #1:
**Q1.1**: The acronyms for the three metrics-MIG, SAP, and DCI-Disentanglement-should be expanded to their full names within the main text to clarify their meanings for all readers.

**A1.1**: As suggested, we explain three metrics as follows:

(1) DCI-Disentanglement measures the weighted average entropy of the probability of each latent code to predict each generative factor.

(2) Separated Attribute Predictability (SAP) attributes a prediction score to all code-factor pairs and measures the average difference between the two highest scores for all generative factors.

(3) Mutual information gap (MIG) is an information-theory-based score computing the average difference between the top two latent variables with the highest mutual information over all factors.

**Q1.2**: Regarding Table 1, is the SW-VAE [47] identical to the proposed Swap-VAE? If they are the same, this should be explicitly stated in Section 4, where the proposed methods are discussed. If they are not the same: a) A detailed comparison should be provided, especially since [47] also employs latent factor swapping.
**A1.2**: No, our Swap-VAE is different from SW-VAE in the following two aspects:

(1) More flexible and robust swapping strategy: SW-VAE contains two training stages: (i) the warm-up stage only swaps one latent factor; (ii) after warm-up, it increases the number of swapped latent factors in a stepwise manner from 1 to $k$. In contrast, our approach adopts two filters to ensure that (i) swapping is only implemented in disentangled latent factors, and (ii) swapped latent factors are bounded with KL values. By doing this, our method can mitigate frequent swapping to enhance the stability of model training.

(2) Weaker supervision: First, SW-VAE use $100\%$ pairwise labeled inputs and our methods only uses $10\%$. Secondly, SW-VAE generates pairwise data with up to $k$ distinct generative factors. Our method only requires input pairs to have one distinct generative factor. ==(Question: SW-VAE is fully supervised? ours is weakly supervised?)==

******

**Q1.3**: It is unclear to me why not including the Swap-VAE (without the second phase SPN) in the experimental section.

**A1.3**: We would like to clarify that Table 1-4 present the disentanglement results of Swap-VAE, which is the first stage of RID. We will change "RID" in these tables to "Swap-VAE" to avoid confusion.

**Q1.4**: The details about the comparison between various VAE variants and RID is not thoroughly discussed. To be specific, VAE provides the latent representation and RID further feeds the z to the prediction phase. The details about how to compare such two different things are not provided.
My understanding of 4 and 5 is as follows: For the disentanglement performance experiments (Tables 1-4), RID functions identically to Swap-VAE, with the second phase set aside. In the classification experiments, however, the second phase is utilized. Is this correct? Regardless, additional details about the experimental setup and procedures should be provided to clarify these points. Given these issues, my initial inclination is to lean towards a weak reject.
==(you need to answer his question, the first stage is Swap-VAE for disentanglement while the second stage is classification)==

**A1.4**: Your understanding is correct. RID consists of two parts: (1) Swap-VAE learns disentangled representations $z$ with more robust swapping strategies. Thus, Tables 1-4 present the results of Swap-VAE for disentanglement. We will change the name RID in tables 1-4 to Swap-VAE.
(2) In the second stage, we feed the learned latent factors $z$ into SPN for classification.

Details about our experiments are as follows:
(1) **Data preparation** is introduced in the last paragraph of Section 5.1. We generate $10\%$ image pairs sharing the overlapping factors except for only one distinct factor.
(2) **Model configuration and hyperparameters** are discussed in Appendix C due to the length of main article.
(3) **Input of SPN**: The disentangled latent factor ($z_1, z_6, z_9$) has much higher KL values in training, as illustrated in [Figure 1](#fig1). We feed these three latent values with meaningful concepts to SPN and observe the contribution of each concept to the classification.
(4) **Testing adversarial robustness**: The attacked/corrupted images are fed into the trained VAE to generate attacked/corrupted concepts. If the attack has a small impact on the concept that has a significant contribution to classifications, the adversarial robustness would be verified with high adversarial accuracy.

<span id="fig1">


</span>

## Reviewer #2:
**Q2.1**: The writing needs to be polished and the content needs to be more accurate and clear. Unexplained terms and unsupported claims in abstracts and introductions will naturally cause some confusion and doubt about motivations.
For example, in lines 40~43, the idea that existing defense methods use low-level features thus resulting in a defense limited to specific attack types is not supported by the literature. Humans can robustly recognize because of the use of high-level concepts such as color and shape, but should these count as low-level features?

**A2.1**: We would like to note that "low-level features" here refer to pixel-level features of images, which are unexplainable to humans. High-level concepts concerning colors and shapes are extracted from the pixel-level features. We will polish our manuscript based on your suggestions.

**Q2.2**: The significance of synthetic datasets needs to be clarified. If it is a contribution to this paper it needs to be clearly presented in the introduction section.

**A2.2**: Synthetic dataset is one of main contributions in our work. Per your suggestion, we will discuss the synthetic dataset in the introduction section.

**Q2.3**: References need to be updated. Research from four or five years ago can't be considered recent work, can it, unless there is no progress in the field.

**A2.3**: We have updated references in line 100 with recent weakly-supervised works in 2021 [4], 2022 [5] and 2023 [8]. Swap-VAE [4] seperates latent representations to two blocks and whole blocks are swapped in training. SW-VAE [5] contains two training stages and only increases the number of swapping from one to maximum number of variants in the second stage. WS-VAE [8] leverages binary weak labels in latent spaces to provide weak supervision. In contrast, our approach flexibly swaps meaningful latent representations with two filters and doesn't need to warm up. We will add these discussions to our revised manuscript.

The misleading "recent work" is also corrected in line 89, 140 and 290.

==(no works in 2023? add one related work in 2023. In addition, please discuss the limitations of related work according to my slides)==

**Q2.4**: Knowledge about countering offense and defense needs to be more accurate. Image reconstruction cannot be categorized as model training.

**A2.4**: Offending: (1) **Common corruptions** alter the image using computer vision methods to simulate corruptions caused by noise, blur, weather, and digital artifacts. The severity of these corruptions is denoted by $s$. (2) In **$\mathcal{L}_{\infty}$ attack**, perturbations added to the input data are constrained by the L-infinity norm. This constraint aims to make the adversarial examples less noticeable to humans while still causing the model to make incorrect predictions. (3) **Sticker attack** places patches on images to cause misclassification. All parameter settings have been shown in Section 5.4.

Defending is achieved through the robustness of numerical latent features under attacks or corruptions. The table below illustrates the effect of adding shot noise to a stop sign image on latent values. It can be observed that these meaningful latent features remain robust under this attack. ==(we may not need to put the table here)==

| | $z_1$(shape) | $z_6$ (color) | $z_9$ (orient) |
| ---------- | ------------ | ------------- | -------------- |
| Shot noise | 0.7841 | -1.9806 | 0.0591 |
| Original | 0.8244 | -2.0001 | 0.0327 |
| $\Delta z$ | 0.0403 | 0.0195 | 0.0264 |

The decoder $p_{\theta}(x|z)$ reconstructs images from latent features. In the training process of VAE, the model aims to minimize the reconstruction error when decoding, which contributes to the VAE objective.

**Q2.5**: How is the value of gamma determined? Why is $argmax_k Diff(z_{k}^{+})$ need to be excluded?

**A2.5**: We conduct experiments with one dataset without the second filter, and $\gamma$ is determined by the highest dimension-wise $D_{KL}\left(q_{\phi, a}(z_{i}^+)|q_{\phi, b}(z_{i}^+)\right)$ among all meaningful factors $z_{i}^+$ in the complete training process. We also evaluate the effect of $\gamma$ in Appendix E. Then, we employ the selected $\gamma$ for other datasets.

$argmax_k Diff(z_{k}^{+})$ represents the most distinct latent dimension in one image pair. It is excluded from swapping because we aim to make the swapped reconstruction $\hat{x}^{\prime}$ identical to the original input image $x$.
<!-- If the training is stable, the largest KL of $z_i^+$ would not exceed the threshold $\gamma$. In this case, we maintain the latent value of the dimension with the highest dim-wise KL (denoted by $argmax_k Diff(z_{k}^{+})$), and swap all other meaingful latent spaces in $z_i^+$. -->

**Q2.6**: Corruption is not an adversarial attack, and the fact that only two adversarial attack methods have been chosen makes it difficult to prove the generalization of the methods.

**A2.6**: Following these prior works [1, 6, 7], ==(need to cite two papers)== we use these attacks and corruptions in our experiments and draw the generalization conclusion.

## Reviewer #3:

==（please answer the questions in more detail)==

**Q3.1**: The idea of swapping VAE dimensions is not new and can be found - using the same name Swap-VAE,- for example, in Liu, Ran et al. “Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity.” bioRxiv (2021): n. pag. The swapping procedure is there different though, relating to content and style, not specific factors of variation. Hence, the name SwapVAE might introduce ambiguity.

==(you need to answer the big difference between our method and prior Swap-VAE. Please refer to A1.2 above)==
**Q3.1**: Swap-VAE [4] seperates latent representations to two blocks and whole blocks are swapped in training. Our method introduces two filters to get meaningful representations, and implements flexible and smooth swapping.

As suggested, we have changed the name of our first stage to "FS-VAE" representing "filters-based swapping VAE".

**Q3.2**: How to appropriately adjust these thresholds for datasets. In other words, how, given a new dataset not tested by the authors, should one choose these thresholds?

**A3.2**: We tuned all thresholds in the TrafficSign dataset and found that these values are robust in other datasets we tested. [Figure 1](#fig1) illustrates the dimension-wise KL-divergence of all latents during training. Compared with disentangled dimensions ($z_1, z_6, z_9$), the KL values of other dimensions are small, with the order of magnitude of $e^{-6}$. $\alpha$ in Filter 1 is selected based on this observation to separate $z_1, z_6, z_9$ from others. We set it relatively low within the range $[0,1]$ so that it will not exceed the KL of meaningful factors in other datasets. The selection of $\gamma$ has been discussed in **A2.5**.

**Q3.3**: How the meaningful factors of variation were adopted in SPN and which effect this had on classification.

**A3.3**: We train the SPN with three meaningful factors of variation together with traffic sign type label. [Figure 2](#fig2) illustrates one branch of the trained SPN's root. One observation is that splitting color ($z_2$) and shape ($z_8$) has more important contributes to the classification because it gives more branches compared with orientation ($z_9$). This is reasonable because the type of traffic sign is mostly decided by color and shape.
==(answer which factors are most important rather than which is less important.)==
<span id="fig2">

</span>

**Q3.4**: It would have been nice to see a discussion on what the chosen metric measures and why they have been chosen. How it might have influenced why the introduced method performed better or worse.

**A3.4**: Disentanglement metrics have been introduced in **A1.1**. They provides different standards to evaluate disentanglement.

**How do they influence the performance?**

Both SAP and MIG measure the gap between the highest and second-highest scores of learned latent representations. Close scores may be caused by two reasons:
(1) Low explicitness: The latent representation cannot fully describe the explanatory factors, resulting in low scores for both measures. This can be reflected in the incomplete orient change in Figure 4 of our paper.
(2) Overlap between two latent factors: There may be overlapping between two factors, which can be observed in the orientation of objects when traversing shape in Figure 4.

Similarly, a single latent code may be important for predicting several factors (e.g., shape and orientation for dSprites) in DCI-Disentanglement, leading to a low score.

**Q3.5**: How would one apply the given method to a dataset with unknown factors of variation? In other words, at least a short discussion about how one might extend the given experimentation to a real world scenario would be very helpful.

**A3.5**: Locatello et al. [2] pointed out that unsupervised disentangled representation learning (DRL) is impossible without induced bias. Therefore, most recent works assume that some factors of variation are available. Although real-world data lack this information, we can reduce the pre-known knowledge by conducting partial labeling following prior work [2]. ==(cite weakly-supervised disentanglement)==

<!-- Additionally, real-world data always exhibit underlying causal dependencies, which conflict with the assumption of independent latent features in DRL. For example, a traffic sign is more likely to be red if its shape is an octagon or an inverted triangle. Such causal relations can be describe as a Directed Acyclic Graph (DAG) by adding a Structured Causal Model (SCM) [3]. -->

References:
[1] Gurel et al."Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attacks". ICML (2021).
[2] Locatello et al. "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations". ICML (2019).
[3] Yang et al. "CausalVAE: Structured Causal Disentanglement in Variational Autoencoder". CVPR (2021).
[4] Liu et al. "Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity". NeurIPS (2021).
[5] Zhu et al. "SW-VAE: Weakly Supervised Learn Disentangled Representation via Latent Factor Swapping". ECCV (2022).
[6] Zhao et al. "Improving certified robustness via statistical learning with logical reasoning." NeurIPS (2022).
[7] Pavlitska et al. "Adversarial Attacks on Traffic Sign Recognition: A Survey". ICECCME (2023).
[8] Tonolini et al. "Robust Weak Supervision with Variational Auto-Encoders". ICML (2023).