To all reviewers

# To all reviewers We would like to thank all the reviewers including the area chair for their contribution to this conference and for helping us to improve the quality of our paper. We would further like to briefly summarize the major contributions of this paper and highlight the most prominent changes made to the revised version of our paper which already address some of the concerns of the reviewers. In addition to these changes, we point out that we have substantially improved the clarity of the paper in terms of structure and language by elaborating on some key concepts (e.g. motivation for consistent/inconsistent perturbations) and removing the typos. **Main contribution and relation to other works on overparameterized linear regression that show benefits for regularization around the interpolation threshold.** Our paper demonstrates how the robust risk of linear models benefits from explicit ridge regularization in overparameterized settings, even if the data is *entirely noiseless*. When the goal is to achieve good robust generalization, our results contradict the modern storyline that highly overparameterized models (e.g. $d/n \gg 1$ for linear models) perform best when interpolating the training data without the need for explicit regularization. To the best of our knowledge, we are the first to provide theoretical evidence that the phenomenon of robust overfitting occurs even in entirely noiseless settings. As noted by the reviewers, a number of recent related works [1,2,3] show that regularization in linear regression mitigates the peak of the standard risk at the interpolation threshold by reducing the variance in the presence of noise. In fact, it is well known that ridge regularization leads to a bias-variance tradeoff and it is therefore not surprising that explicit ridge regularization helps to mitigate the peak in the double descent curve, that is caused by the large variance. In contrast, in our paper we show the benefits of ridge regularization for the robust risk in - highly overparameterized settings where the effect of explicit regularization is negligible for the standard risk as also shown in [1,2]. - noiseless settings where the variance is zero and thus the standard risk does not benefit from regularization, even at the interpolation threshold. **Motivation for studying standard training for linear regression and adversarial training for logistic regression.** In our paper we study standard trained estimators for regression and adversarially trained estimators for classification. We motivated our choice by adding a short discussion in the revised version of our paper that captures the following argument: Since the goal of this paper is to study the shortcomings of interpolating estimators compared to regularized estimators, we only analyze training algorithms that allow interpolation. For regression, inconsistent adversarial training prevents interpolation (as discussed in Appendix C). On the other hand, consistent adversarial training simply recovers the ground truth for linear regression when training with noiseless samples. In contrast, for linear classification, interpolation is easier to achieve -- it only requires the sign of $\langle x_i , \theta\rangle$ to be the same as the label $y_i$ for all $i$. In particular, when the data is sufficiently high-dimensional, it is possible to find a classifier that interpolates the adversarially perturbed training set. **Motivation for adversarial training with consistent perturbations for logistic regression.** As mentioned before, our main contribution is to show that robust overfitting occurs even in entirely noiseless settings, where we expect the phenomenon to happen the least. However, training with inconsistent perturbations induces noise during training, even if the clean data is noiseless. To challenge the hypothesis that noise is responsible for robust overfitting, we decided to focus on training with consistent perturbations instead. Even though the procedure is not practically feasible since it requires full knowledge of the ground truth, consistent adversarial training allows for an entirely noiseless setting. Thus, we can rigorously confirm that robust overfitting occurs even in the complete absence of noise in the training data. However, based on the reviewers’ feedback, we think that it is important to also discuss inconsistent adversarial training more in detail. We have revised our paper accordingly and now present empirical results for linear regression with both consistent and inconsistent adversarial training in the main text. Moreover, we have adapted Theorem 4.1 such that it holds for both training procedures. We motivate our choice of the training algorithm in the revised paper as follows: > Adversarial training with respect to inconsistent perturbations is a popular choice in the literature to improve the robust risk (e.g., [4,5]). The terminology is derived from the fact that adversarial training with such perturbations is inconsistent in the infinite data limit for fixed $d$. The failure to recover the ground truth even for noiseless data is caused by perturbed samples that may cross the true decision boundary. In other words, inconsistent perturbations effectively introduce noise during the training procedure. > As mentioned in the introduction, in this paper we are interested in verifying whether regularization can be beneficial in high dimensions even in the absence of noise. However, adversarial training with inconsistent perturbations can induce noise even if the data is clean. In particular, in the data model with noiseless observations that we introduce in Section 2.1 **(previously Section 4.1)**, the ground truth function misclassifies approximately $8\%$ of the labels when perturbing the training data with inconsistent perturbations of size $\epsilon = 0.1$. Therefore, in the sequel we study the impact of both inconsistent and consistent perturbations on robust overfitting. [1]: Optimal Regularization Can Mitigate Double Descent. Preetum Nakkiran, Prayaag Venkat, Sham Kakade, Tengyu Ma. ICLR 21. \ [2]: Surprises in High-Dimensional Ridgeless Least Squares Interpolation. Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J. Tibshirani. Arxiv 18. \ [3] Benign overfitting in ridge regression. A. Tsigler, P. L. Bartlett. Arxiv 20. \ [4]: Explaining and Harnessing Adversarial Examples. Ian Goodfellow and Jonathon Shlens and Christian Szegedy. ICLR 15. \ [5]: Precise Statistical Analysis of Classification Accuracies for Adversarial Training. Adel Javanmard and Mahdi Soltanolkotabi. Arxiv 20.