# General Response

**[Scope]** We appreciate the constuctive reviews from all reviewers. The reviewers had a common comment that our contributions are catestrophic data leakage in vertical federated learning (VFL) with theoretical analysis and emprirical results, but the scope is not clearly reflected in our title and early sections. We will make the suggested title changes and revise the early sections for clarity.

**[VFL]** We would also like to emphasize that our considered FL setting is typical and there are many practical industrial examples of VFL such as finance, e-commerce, and health.
For example, many VFL protocols provide data indices, and VFL is an important FL paradigm which is suitable in cases where multi local datasets or data owners share the same data identity (ID) space but their data differ in feature space. Some industrial examples of VFL are:
+ [FedML at Webank AI](https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/classical_vertical_fl) [C.He et al., 2020]
+ [FATE at Webank AI](https://github.com/FederatedAI/FATE)
+ [TF-Encrypted by Alibaba Gemini Lab](https://alibaba-gemini-lab.github.io/docs/blog/tfe/)

Moreover, we can find similar VFL protocols that fit into our studied scenario in other papers such as

+ Cheng, Kewei and Fan, Tao and Jin, Yilun and Liu, Yang and Chen, Tianjian and Papadopoulos, Dimitrios and Yang, Qiang "SecureBoost: A Lossless Federated Learning Framework", *IEEE Intelligent Systems, 2021*
+ Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, Zhiyuan Zong "Privacy Leakage of Real-World Vertical Federated Learning", *arXiv, eprint:2011.09290*
+ Fanglan Zheng, Erihe, Kun Li, Jiang Tian, Xiaojia Xiang "A Vertical Federated Learning Method for Interpretable Scorecard and Its Application in Credit Scoring", *arXiv, eprint:2009.06218*
+ Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, Brian Thorne "Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption", *arXiv, eprint:1711.10677*
+ Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Xinghua Zhu, Jianzong Wang, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, Salman Avestimehr "FedML: A Research Library and Benchmark for Federated Machine Learning" *NeurIPS 2020*

Below we summarize the key points in our response to each reviewer. The detailed point-by-point response can be found underneath each reviewer's review.

**Reviewer kAc7**
* Added more references and real-world examples of VFL on Q1 (The learning protocol is not typical)
* Added experimental baseline comparison results to our CAFE on Q2 (Insufficient baseline comparison)
* Added experimental results comparing different number of workers on Q3 (Insufficient number of workers)
* Added more clear descriptions on Q4 (Unclear concepts)

**Reviewer a7yr**
* Revised some statement on Q1 (Regarding the title and the scope)
* Added both discussion and experimental results on batch size $K$ and number of iterations $T$ on Q2 (The term $T$ in our experiment)
* Revised some typos on Q3 (Some typos)
* Added discussion and experimental results on the attacking while training case on Q4 (Convergence plot with a small learning rate)

**Reviewer PDMg**
* Added experimental training and testing accuracy of fake gradient based defense on Q1 (training and testing accuracy of fake gradient based defense)
* Added theoretical analysis on input recovery to a convolutional layer on Q2 (Finer analysis on recovering the inputs to the convolutional layers)
* Added more references and real-world examples of VFL on Q3 (Not general enough FL setting)
* Added experimental baseline comparison results to our CAFE on Q4 (Issue with the baseline comparision)
* Added both experimental differential privacy (DP) results and discussions on Q5 (The difference of the method to differential privacy)
* Revised some typos on Q6 (Some typos)

We would like to make the most use of the interactive discussion function provided by Openreview to clarify any concern the reviewers may have. We look forward to the rolling dicussion and further engagement with the reviewers and area chair(s)!

-------

## We thank Reviewer kAc7 for detailed and constructive comments!

First of all, we thank the reviewer for the careful review and constructive feedback. Below we address your concerns.

**Q1. The considered federated learning protocol is not typical.**

We would like to clarify that our proposed CAFE framework actually applies to the described typical federated learning (FL) setting. Because CAFE is a server-end attack, it is agnostic to the assumption of inter-worker communication (or not), as long as data indices are aligned (see Line 121-124).

For example, many vertical federated learning (VFL) protocols provide data indices, and VFL is an important FL paradigm which is suitable in cases where multi local datasets or data owners share the same data identity (ID) space but their data differ in feature space. Some industrial examples of VFL are:
+ [FedML at Webank AI](https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/classical_vertical_fl) [C.He et al., 2020]
+ [FATE at Webank AI](https://github.com/FederatedAI/FATE)
+ [TF-Encrypted by Alibaba Gemini Lab](https://alibaba-gemini-lab.github.io/docs/blog/tfe/)

Moreover, we can find similar VFL protocols that fit into our studied scenario in other papers such as
+ Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, Zhiyuan Zong "Privacy Leakage of Real-World Vertical Federated Learning", *arXiv, eprint:2011.09290*
+ Cheng, Kewei and Fan, Tao and Jin, Yilun and Liu, Yang and Chen, Tianjian and Papadopoulos, Dimitrios and Yang, Qiang "SecureBoost: A Lossless Federated Learning Framework", *IEEE Intelligent Systems, 2021*
+ Fanglan Zheng, Erihe, Kun Li, Jiang Tian, Xiaojia Xiang "A Vertical Federated Learning Method for Interpretable Scorecard and Its Application in Credit Scoring", *arXiv, eprint:2009.06218*
+ Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, Brian Thorne "Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption", *arXiv, eprint:1711.10677*
+ Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Xinghua Zhu, Jianzong Wang, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, Salman Avestimehr "FedML: A Research Library and Benchmark for Federated Machine Learning" *NeurIPS 2020*

Therefore, our considered FL setting is typical and our contributions are not limited.

**Q2. Insufficient reference to more advanced and recent papers such as e.g., Inverting gradients [Geiping et al., 2020], GradInversion [Yin et al., 2021].**

In fact, we have already compared to the requested baseline [Geiping et al., 2020] and discussed these two mentioned works in Table $1$. In our paper we call the baseline [Geiping et al., 2020] "cosine similarity'' since the authors didn't give a name of their algorithm.

In GradInversion [Yin et al., 2021], the authors proposed several assumptions such as non-repeating labels in the batch, which is hard to be satiesfied in some datasets we used like CIFAR-10, MNIST and Linnaeus-5. In those datasets, we use batch size of more than $40$, which is larger than the number of classes ($10$ or $5$), and therefore the assumption in the paper does not hold. Nevertheless, during the limited rebuttal period, we still compared our CAFE to the methods by using the batch normalization regularizer (eq. (10) in [Yin et al., 2021]) and group consistency regularizer (eq. (11) in [Yin et al., 2021]) mentioned in their work in CAFE. We report the results below using the same experiment setup in Table 2. The results show that CAFE outperforms these alternatives. We will add these results in the revised version.

| Baselines\PSNR\Datasets|CIFAR-10| MNIST |Linnaeus-5|
| :------------- | :----------: | :-----------: | :-----------: |
| CAFE | 31.83 | 43.15 | 33.22 |
| BN regularizer [Yin et al., 2021] | 18.94 | 13.68 | 8.09 |
| Group consistency regularizer [Yin et al., 2021] | 13.63 | 9.24 | 12.32 |

[Ref [10]] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, "Inverting gradients – how easy is it to break privacy in federated learning?" *NeurIPS 2020*
[Ref [27]] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov "See through Gradients: Image Batch Recovery via GradInversion" *CVPR 2021*

**Q3. The number of local workers is not sufficient.**

We explain the effect of the number of local workers on CAFE in the VFL case from both theoretical and practical aspects.

In the paper, we define the input data $\textbf{x}_n$ in line $87$-$88$. Although the data is partitioned on feature space by local workers, the dimension of the entire data feature space is fixed and indepedent of the number of workers. Therefore, increasing number of workers theoretically does not change the dimension of variables assosicated with data recovery in equation (3) in our paper.

In practice, different from HFL, where there could be hundreds of workers, in VFL, the workers are typically financial organizations or companies. Therefore, the number of workers is usually small (See FedML at Webank AI as an example).<br />

In the table below we added a new experiment with 16 workers following the same experiment setup as in Table 2. The results show that the CAFE performances are comparable.

| Baselines\PSNR\Datasets|CIFAR-10| MNIST |Linnaeus-5|
| :------------- | :----------: | :-----------: | :-----------: |
| CAFE w/ $4$ workers | 31.83 | 43.15 | 33.22 |
| CAFE w/ $16$ workers | 28.39 | 39.28 | 39.85 |

**Q4. Definition of data index alignment.**

We are sorry for the confusion. We now realize we discussed this term in Line 121-124 but did not define it formally. The term "data index alignment" means that for each batch, the server (acting as the attacker) sends a data index or data id list to all the local workers to ensure that data with the same id sequence have been selected by each worker, which is a typical setting in VFL. Fig 2(b) in [Yang et al ACM 2019] gives an appropriate illustration of this step. We will clarify this concept in detail in the next version.

[Ref [26]] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications” *ACM Trans. Intell. Syst. Technol., vol. 10, no. 2, Jan. 2019*

## We thank Reviewer a7yr for detailed and constructive comments!
First of all, we thank the reviewer for the constructive comments and careful reading. We will address your concerns below.

**Q1. Regarding the title and the scope.**

Following your suggestion, we will change the title and descriptions in the article from FL to VFL, which is a more precise statement. In Section 3 we have clearly mentioned that we focus on VFL setting. But we agree we should make our scope clearer by specifying VFL upfront to avoid unnecessary confusion. We will also follow your suggestion to mention CAFE is less effective when the learning rate is large in earlier pages.

**Q2. The term $T$ in our experiment.**

Theoretically, given infinite number of iterations, we prove that we can recover $\nabla_{U}\mathcal{L}$ and $H$ because the respective objective function in equation (7) and (9) in our paper is strongly convex as long as $N < d_2$ and $\text{Rank}(\textbf{V}^{*})=N$ (see details in the supplementary material $D$ and $E$).

In our experiment, we did fix the number $T$ for each dataset and it shows that large batch size indeed helps the CAFE algorithm to approximate $H$, especially in MNIST. Following the reviewer's comment, we also conducted an experiment using the same number of epochs on Linnaeus-5 (same setup in Table 3) and reported the results in the table below. The results suggest that increasing batch size $K$ (Table 3 in the paper) and number of iterations $T$ (Table below) both contribute to the attack performnace. When we fix the number of epoches, the attacker with a smaller batch size has more iterations to recover data, leading to a better performance. We will add this result in the revised version.

| **Epoch**\PSNR\Batch size| $10$|$20$|$40$|$80$|$100$|
| :------------- | :----------: | :-----------: | :-----------: | :-----------: | :-----------: |
| 100 | 12.30 | 14.76 | 15.33 | 11.84 | 11.79 |
| 150 | 15.83 | 17.92 | 16.26 | 14.28 | 13.21 |
| 200 | 17.63 | 19.38 | 17.20 | 16.24 | 14.46 |
| 250 | 21.80 | 21.49 | 19.09 | 18.11 | 16.14 |
| 300 | 22.92 | 24.00 | 21.14 | 19.83 | 17.29 |
| 350 | 24.86 | 25.86 | 22.62 | 21.05 | 18.90 |

**Q3. Typos in line 240-243 and Algorithm 3, 4**

Thank you for correcting the typos. It is indeed a typo in line 240-243 and it should be "As shown in Table 5, CAFE single-loop requires fewer number of iterations. Meanwhile, it is difficult to set the loop stopping criteria in the CAFE nested-loop mode."

In addition, line 5 in Algorithm 4 should be “Run step 11-13” and line 6 should be “Run step 11-13”. We apologize for our typos again.

**Q4. It may be interesting to show the convergence plot for experiment 4.3 with small learning rate. Is CAFE better on untrained model?**

Following the reviewer's suggestion, we have conducted a new experiment on MNIST with $4$ workers. The model indeed converges with a relative small learning rate (we used Adam optimizer, learning rate = $10^{-6}$, trained on $800$ images, tested on $100$ images, batch size $K=40$) and we attach our experimental results below.

| $\#$ of iterations $T$ |PSNR value|Training loss|Testing accuracy|
| :------------- | :----------: | :-----------: | :-----------: |
| 0 | 5.07 | 2.36 | 0.11 |
| 2K | 11.68 | 2.31 | 0.27 |
| 4K | 14.39 | 2.14 | 0.34 |
| 6K | 18.07 | 1.99 | 0.54 |
| 8K | 18.16 | 1.87 | 0.63 |
| 10K | 18.12 | 1.82 | 0.64 |
| 15K | 16.86 | 1.63 | 0.65 |
| 20K | 20.72 | 1.68 | 0.68 |

The data indeed leaks to a certain level (PSNR above 20) while the model converges at a certain testing accuracy ($0.68$), which indicates that CAFE works in an attacking while training scenario. We note that due to the stealthy nature of CAFE, it did not impede the performance of the final model. The relatively low accuracy is merely because only $800$ samples are used for training.

CAFE is indeed better to recover data when the model is untrained comparing with the stage when the model fully converges. We use our theoretical analysis to explain the reason. When the model is trained or even fully converged, the real gradients of loss with repect to (w.r.t) model parameters will be very small. It is possible that the value of the recoverd $\nabla_{U}\mathcal{L}$ in equation (7) will also be close to 0. In that case, it can be difficult to recover $H$ in equation (9).

## We thank Reviewer PDMg for detailed and constructive comments!
We thank the reviewer’s support and the constructive comments. We address your remaining concerns below.

**Q1. The training and testing accuracy of fake gradient based defense**
We attached the training accuracy and testing accuracy when training with (Fake gradient) and without (True gradient) the proposed defense at $T=2500$ iterations based on the experiment setup of Figure 6 except for Linnaeus-5 with learning rate=0.01. From the table below, our defense strategy didn't affect model performance much. Note that one should compare the different between the results using true v.s. fake gradients to study the effect of the defense. The low absolute value of the accuracy is an artifact of insuffieint training iterations due to time constraints.

| Algorithm|Training loss| Training accuracy |Testing accuracy |
| :-------------: | :----------: | :-----------: | :-----------: |
| True gradient (MNIST) | 0.299 | 0.884 | 0.885 |
| Fake gradient (MNIST) | 0.404 | 0.857 | 0.858 |
| True gradient (CIFAR-10) | 2.198 | 0.263 | 0.263 |
| Fake gradient (CIFAR-10) | 2.236 | 0.247 | 0.248 |
| True gradient (Linnaeus-5) | 1.568 | 0.310 | 0.302 |
| Fake gradient (Linnaeus-5) | 1.652 | 0.292 | 0.279 |

**Q2. Finer analysis on recovering the inputs to the convolutional layers**

We apprecite the reviewer's insightful question. We have indeed tried to recover the inputs before a convolutional layer. Unfortunately, the attempt was not successful. Below we provide derivations to show why such recovery is impossible.

In the simplest case, we assume a single input as $X \in \mathbb{R}^{M\times N\times C}$, the convolutional layer as $\Theta \in \mathbb{R}^{H\times W\times C \times G}$, the bias as $b \in \mathbb{R}^{G}$, the output as $Y \in \mathbb{R}^{I\times J \times G}$, the loss as $\mathcal{L} \in \mathbb{R}$, where $M$ and $N$ are the width and height of input $X$, $I$ and $J$ are the width and height of output $Y$，$C$ and $G$ are the number of input and output channels.

We have
\begin{align}
Y_{i,j,g} = b_g + \sum_h\sum_w\sum_c \Theta_{h,w,c,g}X_{h+i, w+j,c}
\end{align}
and we can compute the gradient as

\begin{align}
\nabla_{\Theta_{h,w,c,g}}\mathcal{L} &= \frac{\partial\mathcal{L}}{\partial Y}\cdot\frac{\partial Y}{\partial \Theta_{h,w,c,g}} \notag \\
% &= \sum_i\sum_j \frac{\partial\mathcal{L}}{\partial Y_{i,j,g}}\cdot\frac{\partial Y_{i,j,g}}{\partial \Theta_{h,w,c,g}} \notag \\
&= \sum_i\sum_j \frac{\partial\mathcal{L}}{\partial Y_{i,j,g}}\cdot X_{h+i, w+j,c}.
\end{align}

Then we need to deal with the term $\frac{\partial\mathcal{L}}{\partial Y_{i,j,g}} \in \mathbb{R}^{I\times J\times R}$ for data recovery, and
the gradient of bias is
\begin{align}
\nabla_{b_{g}}\mathcal{L} &= \frac{\partial\mathcal{L}}{\partial Y}\cdot\frac{\partial Y}{\partial b_{g}} \notag \\
% &= \sum_i\sum_j \frac{\partial\mathcal{L}}{\partial Y_{i,j,g}}\cdot \frac{\partial Y_{i,j,g}}{\partial b_g} \notag \\
&= \sum_i\sum_j \frac{\partial\mathcal{L}}{\partial Y_{i,j,g}}.
\end{align}

In both equations of $\nabla_{\Theta_{h,w,c,g}}\mathcal{L}$ and $\nabla_{b_{g}}\mathcal{L}$, they contain the summation operations on both width and height dimension of output image $Y$. For each $i$ and $j$, $\frac{\partial\mathcal{L}}{\partial Y_{i,j,g}}$ is unknown to the server in standard VFL protocols. As the result, we cannot recover each element of $X$ from $\nabla_{\Theta_{h,w,c,g}}\mathcal{L}$ and $\nabla_{b_{g}}\mathcal{L}$ even when the batch size is $1$. Furthermore, there usually exist max or average pooling layers in convolutional neural network structures, making it more difficult to recover the input $X$. However, in fully connected (FC) layers, there is no summation operations when we compute the gradients of loss w.r.t bias of an FC layer (equation (8) and (9) in [Geiping et al., 2020]), where we can directly recover the input $H$ to an FC layer. We will add this discussion to the revised version.

[Ref [10]] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, "Inverting gradients – how easy is it to break privacy in federated learning?" *NeurIPS 2020*

**Q3. The method is not general enough and only applies to the special VFL setting.**

We understand the reviewer's concern; but we disagree our attack only applies to "special" VFL setting. There are many practical industrial examples of VFL such as finance, e-commerce, and health. For example, vertical federated learning (VFL) provides data indices for CAFE's implementation, and it is an important FL paradigm which is suitable in cases where multi local datasets or data owners share the same data identity (ID) space but there data differ in feature space. There are many practical industrial examples of VFL such as
+ [FedML at Webank AI](https://github.com/FedML-AI/FedML/tree/master/fedml_experiments/standalone/classical_vertical_fl) [C.He et al., 2020]
+ [FATE at Webank AI](https://github.com/FederatedAI/FATE)
+ [TF-Encrypted by Alibaba Gemini Lab](https://alibaba-gemini-lab.github.io/docs/blog/tfe/)

Moreover, we can find the similar VFL protocol in other papers such as
+ Haiqin Weng, Juntao Zhang, Feng Xue, Tao Wei, Shouling Ji, Zhiyuan Zong "Privacy Leakage of Real-World Vertical Federated Learning", *arXiv, eprint:2011.09290*
+ Cheng, Kewei and Fan, Tao and Jin, Yilun and Liu, Yang and Chen, Tianjian and Papadopoulos, Dimitrios and Yang, Qiang "SecureBoost: A Lossless Federated Learning Framework", *IEEE Intelligent Systems, 2021*
+ Fanglan Zheng, Erihe, Kun Li, Jiang Tian, Xiaojia Xiang "A Vertical Federated Learning Method for Interpretable Scorecard and Its Application in Credit Scoring", *arXiv, eprint:2009.06218*
+ Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, Brian Thorne "Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption", *arXiv, eprint:1711.10677*
+ Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Xinghua Zhu, Jianzong Wang, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, Salman Avestimehr "FedML: A Research Library and Benchmark for Federated Machine Learning" *NeurIPS 2020*

We will change the title and descriptions in the article from FL to VFL to avoid uncessary confusion.

**Q4. Issue with the baseline comparision**

Following your suggestion, we expand the reasons to why some methods summarized in Table 1 are not compared to our proposed CAFE method.

Theory-driven label inference methods are proposed in [B. Zhao et al., 2020] and in [W. Wei et al., 2020], and the authors proposed label-based regularizers. However, our attack mainly deals with training data leakage, so we didn't compare our work with training label inference methods. In [X. Pan et al., 2020], the authors proposed a sufficient requirement that "each data sample has at least two exclusively activated neurons at the last but one layer". However, in our training protocol, the batch size is too large and it is almost impossible to ensure that each selected data sample has at least two exclusively activated neurons. In [J. Zhu et al. 2020], the authors claim that in the batch setting, their method will only return a linear combination of the selected training data, which is a very restricted assumption.

In GradInversion [Yin et al., 2021], the authors proposed several assumptions such as non-repeating labels in the batch, which is hard to be satiesfied in some datasets we used like CIFAR-10, MNIST and Linnaeus-5. In those datasets, we use batch size of more than $40$, which is larger than the number of classes ($10$ or $5$), and therefore the assumption in the paper does not hold. Nevertheless, during the limited rebuttal period, we still compared our CAFE to the methods by using the batch normalization regularizer (eq. (10) in [Yin et al., 2021]) and group consistency regularizer (eq. (11) in [Yin et al., 2021]) mentioned in their work in CAFE. We report the results below using the same experiment setup in Table 2. The results in the table below show that CAFE outperforms these two methods by a large margin. We will add these results in the revised version.

| Baselines\PSNR\Datasets|CIFAR-10| MNIST |Linnaeus-5|
| :------------- | :----------: | :-----------: | :-----------: |
| CAFE | 31.83 | 43.15 | 33.22 |
| BN regularizer [Yin et al., 2021] | 18.94 | 13.68 | 8.09 |
| Group consistency regularizer [Yin et al., 2021] | 13.63 | 9.24 | 12.32 |

If the reviewer has any specific baseline in mind, please let us know and we are willing to compare and report the results.

[Ref [21]] X. Pan, M. Zhang, Y. Yan, J. Zhu, and M. Yang, “Theory-oriented deep leakage from gradients via linear equation solver”, *ArXiv eprint:2010.13356 2020*
[Ref [24]] W. Wei, L. Liu, M. Loper, K. H. Chow, M. E. Gursoy, S. Truex, and Y. Wu, “A framework for evaluating gradient leakage attacks in federated learning”, *ArXiv eprint:2004.10397 2020*
[Ref [27]] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, Pavlo Molchanov "See through Gradients: Image Batch Recovery via GradInversion" *CVPR 2021*
[Ref [28]] B. Zhao, K. R. Mopuri, and H. Bilen, “idlg: Improved deep leakage from gradients”, *arXiv,eprint:2001.02610, 2020*
[Ref [29]] J. Zhu and M. B. Blaschko, “R-GAP: Recursive gradient attack on privacy”, *in International Conference on Learning Representations, 2021*

**Q5. Discussion on the difference of the defense method to the differential privacy**

Following the reviewer's suggestion, we have also added the experiment to discuss the difference of our fake gradient method to differential privacy.

The results below show the training loss of no defense (true gradient), differential privacy (DP) defense, and our defense (fake gradient). For DP, we followed the gradient clipping approach [M. Abadi et al. *ACM CCS, 2016*] to apply DP to the gradients from workers. In particular, the gradient norm was clipped to 3, as suggested by [M. Abadi et al. *ACM CCS, 2016*]. As shown in the table below, the training loss cannot be effectively reduced using DP. This is also consistent with the result in [Zhu et al. *NeurIPS, 2019*], as one proposed to add noise to gradient as a candidate defense. [Zhu et al. *NeurIPS, 2019*] concluded that the noise with sufficient magnitude required to avoid information leakage from gradients degrades the accuracy significantly. As the noise magnitude required by DP is even stronger than the one needed for the ad hoc privacy in [Zhu et al. *NeurIPS, 2019*], it is inevitable to lead to a similar conclusion, which is the difficulty in reducing the training loss. In our fake gradient defense, all of the gradients will be projected to a set of predefined gradients before being sent to the server, with the purpose of restricting the attacker’s knowledge from gradient leakage. Our defense is still deterministic in its essence and therefore does not satisfy the DP. However, as DP may largely destabilize the training process, our defense strikes a balance between the empirical data privacy and model accuracy. In other words, our experiments demonstrate that the attacker is unable to recover the worker’s data and at the same time the training loss can be reduced effectively. We will add this discussion in the revised version.

| Steps\Training loss\DP|DP $\epsilon=10$|DP $\epsilon=5$|DP $\epsilon=1$|DP $\epsilon=0.1$|Fake gradients|True gradients|
| :------------- | :----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |
| 0 | 2.78 | 2.77 | 2.77 | 2.77 | 2.77 | 2.77 |
| 1K | 2.69 | 2.69 | 2.69 | 2.69 | 1.95 | 1.08 |
| 2K | 2.85 | 2.85 | 2.85 | 2.85 | 1.38 | 0.54 |
| 3K | 2.85 | 2.85 | 2.85 | 2.85 | 0.65 | 0.23 |
| 4K | 2.92 | 2.92 | 2.92 | 2.92 | 1.09 | 0.38 |
| 5K | 2.61 | 2.61 | 2.61 | 2.61 | 0.92 | 0.31 |
| 6K | 2.69 | 2.69 | 2.69 | 2.69 | 0.62 | 0.31 |
| 7K | 2.77 | 2.77 | 2.77 | 2.77 | 1.08 | 0.15 |
| 8K | 2.69 | 2.69 | 2.69 | 2.69 | 1.15 | 0.46 |

[Ref [30]] Zhu et al. "Deep Leakage from Gradients"* NeurIPS, 2019*
M. Abadi et al. "Deep Learning with Differential Privacy", *ACM CCS, 2016*

**Q6. Some typos**

Thank you for the careful reading. We will correct all the typos that you have pointed out in our next version.