# Neurips 2022 dual batch normalization rebuttal

# Reviewer hNGX

Strengths: (1) The designed ablations are interesting and convincing.
(2) With empirical analysis, it reveals that the two sets of affine parameters are the key to Dual AT.

Weaknesses:
(1) In the paper, only Madary AT and TRADES are studied. Recent advanced methods, e.g., AWP[1] and LBGAT[2] are not considered. Especially, similar to the proposed method, LBGAT improves model robustness and accuracy on natural data simultaneously.
(2) Lack of ablations for their designed new normalization operation for cross-training. Does cross training outperform the simple post-processing described in Sec. 4.1?
(3) In Table 3, model robustness under 16/255 is much stronger than 8/255.
(4) As described in lines 316-317, Trades-AT performs much worse than Madry-AT for a reasonably large perturbation like ϵ = 16/255.

In TRADES, the hyper-parameter lambda is very important for model robustness. Usually, TRADES achieves stronger robustness than Madary-AT but inferior performance on natural data on CIFAR-10 with 8/255. However, in Table 4, TRADES achieve much higher natural accuracy than Madary-AT. Considering that, I think TRADES can achieve much stronger robustness than the numbers reported in Table 4 by adjusting lambda hyper-parameters.
(5) We usually use the perturbation size 8/255 on CIFAR for evaluation. For a fair comparison with previous work, the authors are encouraged to evaluate their model with perturbation size 8/255 for Tables 4 and 5.

[1] 'Adversarial Weight Perturbation Helps Robust Generalization', NeurIPS 2020.
[2] 'Learnable Boundary Guided Adversarial Training', ICCV 2021.

## Q1 Sota method which improves model robustness and accuracy on natural data simultaneously.

Our paper mainly study and investigate the batch normalization in the context of adversarial training.

## Q2 Ablations for the designed new normalization operation better than the simple post-processing described in Sec. 4.1
In row 2 and row 4 of Table 4 we compare the post-processing and the teh designed new normalization operation (OCT).

## Q3 Model robustness under 16/255 is stronger than 8/255 in Table 3.
Sorry for misleading, we wrongly put the corresponding attack strength. Here the 16/255 and 8/255 should be swapped. We will correct this in the later version.

## Q4 Trades-AT performs worse than Madry-AT under 16/255
Thank you for your great suggestion. By adjust we get the better performance, as shown in Table xx.

## Q5 Comparison with previous work under perturbation size 8/255.
We will give the results.

# Reviewer bDmB

Pros:

This paper is very interesting and delves into the very popular issue of "mixture BNs" in the adversarial machine learning area. The author pointed out that existing studies have shown wrong understandings of mixture BNs, and proposed to better utilize mixture BNs from the perspective of affine parameters rather than running statistics.
Cons:

My biggest concern is the writing of this paper. Even though I've published papers in this specific area, I still need to go over some paragraphs several times in order to understand the meaning. In particular, there are so many similar abbreviations such as AP, NS, NP, Running NS, Static NS, etc, which are very hard for me to understand.

The organization of this paper looks like a technical report rather than a research paper for me. The author conducted many ablation studies and found out BN affine parameters are important.

Some minor problems. The author missed some related studies [R1, R2] which should be discussed in the paper. Specifically, I think [R2] is very close to the discussion of "online cross training" in Section 4.2.

References

[R1] Towards defending multiple adversarial perturbations via gated batch normalization

[R2] Improving robustness against common corruptions by covariate shift adaptation.

## Q1 many similar abbreviations. Hard for reader to understand.
Thank you for pointing out this. We change the following abbreviations for eaisier understanding AP to AffPara, NP to NormPara, and NS to NormStat. Also we change the running to training and static to testing to represent the BN weight used during training and testing.

## Q2 The organization of the paper looks like a techinical report rather than a research paper.

## Q3 Missed related studies

# Reviewer hfB9

Pros:-

The paper is based on an interesting observation and analysis that affine parameters for dual BN models are responsible for improving the adversarial robustness.This result is interesting and provides a deeper understanding for the role of BN in robustness.

Required experiments are provided to verify this claim (Till section 3).

Cons:-

Section 4 (Cross Robustness): The authors claimed that cross-robustness increases the robustness by a visible margin. However, in Table 2, I can see that the improvement in robust accuracy is only 0.36% (46.19 to 46.55 for auto-attack). I fdo not think that this is sufficient to make any claim.

Typo in Table 3: epsilon values should be swapped. Even here, I find the performance improvement is <1% for both Hybrid-AT and Madry-AT models.

Table 4: Please provide results also for epsilon = 8/255.

Line 314-320: “ First, even 315 though Trades-AT is widely reported to outperform 316 Madry-AT for ϵ = 8/255, we find that Trades-AT performs much worse than Madry-AT for a 317 reasonably large perturbation like ϵ = 16/255.” — I am not sure if this is an interesting fact. Typically, we train a robust model after defining an appropriate threat boundary for attack. If we require robustness for a larger threat boundary, we should train the models at larger boundaries for comparison.

All the experimental results are provided using a single run of the experiments.it is recommended that the authors should provide the mean and variance of 5-10 runs (especially when the performance gaps are not significantly large).

## Q1 Limited improvement in robust accuracy in Table 2.
We verify the improvement is stable and can be obeserved on multip runs and different datasets and when using different model.

## Q3: Swap the epsilon values in Table 3.
Thank you for your remind. We will correct this in the new version.

## Q4: The results for epsilon=8/255.

| Method | Clean | PGD-10 | PGD-20 | 20 |
|---|---|---|---|---|
|Madry-AT | 1 | 1 | 1 | 1 |
| OCT | 1 | 1 | 1 | 1 |
| OCT (clean) | 1 | 1 | 1 | 1 |
| Mixture | 1 | 1 | 1 | 1 |
|---|---|---|---|---|
|Trades-AT| 82.00 | 53.00 | 52.60 | 48.80 |
| Dual BN (adv) | 81.100 | 52.700 | 52.100 | 47.90 |
| Dual BN (clean) | 81.700 | 52.900 | 52.100 | 47.10 |
| OCT | 82.40 | 53.30 | 52.10 | 47.60 |
| OCT (clean) | 82.00 | 53.50 | 52.00 | 49.10 |

## Q5: Line 314-320 misunderstanding.
Sorry for the misleading writting, here we want to say under the same setting the results that Madry-AT achieves higher robustness with PGD attack and autoattack as shown in Table 4.

## Q2

# Reviewer pa4X

Weakness
The novelty and contribution are limited. Although this paper shows an interesting finding that the benefit of Dual BN is from the independent affine parameters, it does not give a further analysis of why the independent affine parameters help to improve adversarial robustness.
The evaluations are not sufficient. Whole experiments are based on a small dataset CIFAR-10 and a small model ResNet-18, which is not enough to make solid conclusions. E.g., this paper evaluates the effectiveness of OCT simply on CIFAR-10 with ResNet-18 under white-box attack. It is unclear whether it is still effective on other datasets, e.g., CIFAR-100/Tiny ImageNet, and large models, e.g. WideResNet models. Besides, it can not be excluded that the improved adversarial robustness is from the 'gradient obfuscation '[1] effect if the evaluations are only done on the white-box attacks.
typos: in line 132, "Normalziation statistics"--> "Normalization statistics"
in line 181,"Inspired by our above visualization result", which figure?
[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning. PMLR, 2018.

Questions:
1. More analysis of why the independent affine parameters help to improve adversarial robustness.
2. More evaluations, e.g. Experiments on CIFAR-10 and other architectures.

## Q1 Limited novelty. Does not give a further analysis of why the independent affine parameters help to improve adversarial robustness.

## Q2 other datasets, larger models, and verify the 'gradient obfuscation' (only tested on white-box attacks)

## Q3 Typos in line 132, "Normalziation statistics"--> "Normalization statistics"
Thank you for your reminder, we will correct this.

## Q4 "Inspired by our above visualization result", which figure? in Figure [1] and Figure [3]
Sorry for the misleading here. In line 161-164 we mentioned that there is a higher gap between the affine parameter of BN_adv and BN_clean as shown in Figuer 3, while from figure 3 we can observer that with the same affine parameter the discrepancy of the normalization statistics between clean and adversarial example is much lower. So we conjecture that two sets of affine parameter is the key point of dual BN. And what makes the clean BN in dual BN has rarely robustness is affine parameter instead of normalization statistics.