### Overall Response

We thank all the reviewers for their insightful and constructive comments. All reviewers recognize that the question of addressing adversarial robustness when using compressed data is an important one, and our contribution of using minimum coreset (MCS) and the theoretical analysis presented as strengths of our work.

In terms of the experimental evaluation, we reported experiments on three datasets ("ICML", MNIST, CIFAR10) and a convolutional neural network (CNN) using the standard projected gradient descent (PGD) attack in the initial submission. However, most reviewers have asked for adding more experiments along different axes to further strengthen our empirical evaluation.

To address these comments jointly, we have added another set of experiments with larger datasets (Reviewers 5sPF, M8af), a larger neural network (Reviewers 5sPF, M8af), more comprehensive adversarial attacks (Reviewers EPTK, 5sPF), and additional dataset condensation (DC) methods (Reviewers LAC8, M8af):

* **Datasets**: CIFAR100, SVHN, TinyImageNet (in addition to MNIST, CIFAR10);
* **Network**: ResNet-18;
* **Attacks**: AutoAttack including 2/4 attacks (only APGD-CE and APGD-DLR) for $\ell_{\infty}$-norm, we still use PGD-$\ell_{\infty}$ attack for adversarial training;
* **DC methods**: distribution matching (DM), gradient matching (GM), trajectory matching (TM);
* **Training methods**: standard training, adversarial training with PGD-$\ell_{\infty}$ attack.

The number of epochs in training and PGD attack is fixed to 20, learning rate is set to 1e-2 (based on selection from the best performance), adversarial perturbation is set to 0.1 for MNIST, 8/255 for CIFAR10, CIFAR100, SVHN, and 4/255 for TinyImageNet. Each experiment is repeated for 3 runs. The results below show the mean and standard deviation of test and robust score for each case.

<style scoped>table{font-size:12px;}</style>
| Dataset | Acc. | RAND-50 | MCS-50 | DM-50 | GM-50 | TM-50 |
| :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| MNIST | Test <br> Auto | 96.40 $\pm$ 0.16 <br> 92.50 $\pm$ 0.37 | 97.06 $\pm$ 0.11 <br> **92.83 $\pm$ 0.07** | **97.10 $\pm$ 0.09** <br> 91.20 $\pm$ 0.27 | 96.34 $\pm$ 0.31 <br> 83.68 $\pm$ 0.74 | -- |
| CIFAR10 | Test <br> Auto | 32.29 $\pm$ 0.76 <br> **8.79 $\pm$ 1.05** | 24.88 $\pm$ 3.12 <br> 5.26 $\pm$ 0.16 | **32.59 $\pm$ 2.49** <br> 2.68 $\pm$ 0.30 | 30.67 $\pm$ 2.10 <br> 0.73 $\pm$ 0.21 | 30.62 $\pm$ 3.03 <br> 0.87 $\pm$ 0.19 |
| CIFAR100 | Test <br> Auto | 20.24 $\pm$ 0.54 <br> **6.59 $\pm$ 0.23** | 13.71 $\pm$ 1.05 <br> 4.84 $\pm$ 0.58 | -- | -- | **24.45 $\pm$ 0.14** <br> 1.09 $\pm$ 0.10 |
| SVHN | Test <br> Auto | 65.41 $\pm$ 5.82 <br> 9.53 $\pm$ 1.12 | **66.60 $\pm$ 5.91** <br> **14.34 $\pm$ 1.38** | -- | 50.27 $\pm$ 14.17 <br> 0.19 $\pm$ 0.04 | -- |
| TinyImageNet | Test <br> Auto | **18.23 $\pm$ 0.41** <br> 0.17 $\pm$ 0.04 | 14.34 $\pm$ 0.20 <br> **4.48 $\pm$ 1.57** | -- | -- | 14.54 $\pm$ 1.61 <br> 3.22 $\pm$ 0.66 |

These new experiments show similar trends to the already reported results in the submission. DC methods like DM, GM, TM achieve slightly higher test performance in specific instances but have poor adversarial performance. MCS achieves better adversarial robustness compared to all DC methods. The performance of MCS in some settings is still lower than Random sampling, which is to be expected in low budget regimes (here with 50 samples). Note that the DC methods with no entries did not provide synthetic examples for these specific datasets, and we did not recompute them to save compute.

Results from these experiments further strengthen the key conclusion in our work: **The efficacy of adversarial training diminishes as compressed datasets increasingly overfit to optimize test performance.**

### Response to Reviewer M8af (Rating: 4 / Confidence: 4)
#### General Comments:
We thank the reviewer for their effort and for recognising some of its key strengths. The major concerns expressed by the reviewer are addressed below.

* **Proposed method for fairness and privacy**: We appreciate this perceptive comment from the reviewer, which also points at the general usefulness of the proposed method. As indicated by the reviewer, investigating the joint-improvements of effeciency with other aspects such as fairness (in terms of both test performance and robustness), and privacy (membership inference privacy) are direct extensions of the current work which only focuses on robustness. We will point to these extensibility in the future works.

* **Lack of analysis and discussion**: We agree with the reviewer in that additional discussions could have helped further. We limited these due to lack of space; we are considering moving some of the proofs/theorems to the Appendix based on reviewer feedback. We will use this additional space to elaborate on the relation between random sampling and MCS. Combined with the response for reviewers 5sPF and LAC8, MCS can be regarded as the deterministic version of random sampling, and can be relaxed by minimizing the relaxed Hausdorff distance. We will discuss this connection, the new results, and extensions to fairness/privacy in the updated version of the manuscript.

* **Technical contribution**: We respectfully disagree with the reviewer in considering the technical contribution to be little. This is the first principled approach to performing dataset compression with an emphasis on adversarial robustness, with an appropriate theoretical analysis of the same. The exact formulation of the minimum finite covering (MFC) problem can also be useful in real-life apllications for coreset selection. Also, by relaxing the definition of Hausdorff distance, one is able to obtain a soft-margin version of MFC (see the response to reviewer LAC8). We agree with the reviewer that some of the propositions/corollaries could be moved to the Appendix as they can be considered simple after the main theorems.

* **Clarity**:
* **Inconsistencies in Fig. 2 and 3:** Thanks for pointing out, we will correct them.
* **$\eta_1,\eta_2$:** This is a typo, it should be $\eta_l, \eta_u$.
* **Distance $d(\mathcal{T}_i, \mathcal{T}_j)$**: This distance is computed by definition $\inf \{ \|\mathbf{x} - \mathbf{y}\|_2: \mathbf{x} \in \mathcal{T}_i, \mathbf{y} \in \mathcal{T}_j\}$.

* **Additional experiments**: We extend our experiments to larger datasets (CIFAR100, SVHN, TinyImageNet), a larger network (ResNet-18), more DC methods (DM, GM, TM), and comprehensive attacks (AutoAttack). Please see the general comment for a discussion on these results. We hope these will alleviate the concerns expressed by the reviewer.

### Response to Reviewer 5sPF (Rating: 4 / Confidence: 5)
#### General Comments:
We thank the reviewer for their effort and for recognising its key strengths. The main concerns expressed by the reviewer are addressed below.

* **Less effective than random sampling**: This is expected since many of the coreset methods performs worse than random sampling in terms of test performance, according to literature such as [1]. However, we would like to emphasize that, downstream performance is just one aspect of our method. We have measured how much information we extract from the full dataset by minimizing the volume of the finite covering. Moreover, MCS can be regarded as the deterministic version of random sampling, since both of them converges to the full dataset as the budget increases.

* **Extension to other dataset compression methods**: Any coreset or DC method could be applied for adversarial training. The theoretical guarantees we presented in this work can be derived for any condensed dataset since _any dataset is a finite covering_ for some radius $\eta$. However, MFC is the optimal one since we minimize the radius $\eta$ to ensure the minimality of the covering.

* **Visualization of MCS samples**: We thank the reviewer for this suggestion. We will add them in the Appendix in the updated manuscript.

* **ResNet and AutoAttack**: We extend our experiments to larger datasets (CIFAR100, SVHN, TinyImageNet), a larger network (ResNet-18), more DC methods (DM, GM, TM), and comprehensive attacks (AutoAttack). Please see the general comment for a discussion on these results. We hope these will alleviate the concerns expressed by the reviewer.

[1] Zhao, Bo, et al. "Dataset Condensation with Gradient Matching." (ICLR 2021).

### Response to Reviewer EPTK (Rating: 7 / Confidence: 2)
#### General Comments:
We thank the reviewer for their effort and for being enthusiastic about our work. Some of the concerns expressed by the reviewer are addressed below.

* **More advanced attacks**: We extend our experiments to larger datasets (CIFAR100, SVHN, TinyImageNet), a larger network (ResNet-18), more DC methods (DM, GM, TM), and comprehensive attacks (AutoAttack). Please see the general comment for a discussion on these results. We hope these will alleviate the concerns expressed by the reviewer.

* **Applicability of Theorem 5.5**: The assumption made in Theorem 5.5 regarding the isolation property of the dataset is difficult and challenging to meet in practical scenarios. However, as highlighted in the response to reviewer LAC8, this assumption can be relaxed. Conducting generalized adversarial training on datasets that do not adhere strictly to the RIP condition only offers an approximation of robustness performance.

* **Computational costs of MCS**: Among the listed dataset compression methods, the computation of MCS is the least expensive, with the order of cost being MCS < DM < GM < TM.

### Response to Reviewer LAC8 (Rating: 4 / Confidence: 3)
#### General Comments:
We thank the reviewer for their effort and for recognising the key contributions. The major concerns expressed by the reviewer are addressed below.

* **Running Isolation Property**: Proposition 4.3 suggests that discovering an MFC is equivalent to minimizing the Hausdorff distance, ensuring a precise and deterministic covering of the dataset. We have the option to ease the constraints of the Hausdorff distance by substituting the maximum with the mean of all distance pairs, $$d_H^r (\mathcal{X}, \mathcal{Y}) = \frac{1}{2} \bigg(\frac{1}{|\mathcal{X}|} \sum_{\mathbf{x} \in \mathcal{X}} \min_{\mathbf{y} \in \mathcal{Y}} d(\mathbf{x}, \mathbf{y}) + \frac{1}{|\mathcal{Y}|} \sum_{\mathbf{y} \in \mathcal{Y}} \min_{\mathbf{x} \in \mathcal{X}} d(\mathbf{x}, \mathbf{y})\bigg).$$ Despite $d_H^r (\mathcal{X}, \mathcal{Y})$ losing its property as a distance metric, it remains feasible to derive a coreset that minimizes the relaxed Hausdorff distance. This approach yields a soft-margin adaptation of the proposed method.

* **More DC methods and results on CIFAR100**: We extend our experiments to larger datasets (CIFAR100, SVHN, TinyImageNet), a larger network (ResNet-18), more DC methods (DM, GM, TM), and comprehensive attacks (AutoAttack). Please see the general comment for a discussion on these results. We hope these will alleviate the concerns expressed by the reviewer.

* **Provably robust claim in Theorem 5.5**: Thanks for the comment, we will rephrase the sentence as "It is provably robust in theory while approximately robust in practice".

* **Explanation of good performance without RIP condition**: RIP condition is related to downstream property but not during the dataset compression phase. The MCS approximates the distribution of the entire dataset, which does not directly impact downstream performance. Consequently, it demonstrates generally modest performance across various downstream tasks.

* **Writing**: Thanks for the comment, we will revise the introduction accordingly.