# NeurIPS-2021 Rebuttal

## General Author Response

We thank all the reviewers for the careful reading and detailed comments. As stated by reviewers, the paper is insightful and interesting (Reviewer eC3H and 3Ucv), well-written, easy to follow, contains detailed analysis and convincing ablation study (Reviewer zcQX and 3Ucv), and shows strong empirical results (Reviewer pL1o). We will address the reviewers' concerns individually and will incorporate all the feedback in the final version.

## Response to Reviewer eC3H

Thank you for the insightful comments. Below we address the concerns.

***Q1: Comparison with other methods like BYOL, MoCo, etc.***

**A1**: We have performed additional 5-way 5-shot evaluation of *BYOL*, *MoCo*, *Transfer+BYOL*, and *Transfer+MoCo*, and report the results in the following table. *BYOL* and *MoCo* are trained on the unlabeled target images only, and *Transfer+(BYOL/MoCo)* is trained on both labeled base dataset (mini-ImageNet) and unlabeled target dataset. Similar to our comparison with *SimCLR* and *Transfer+SimCLR* in Table 1 in the main paper, our method outperforms all other models in all datasets except the ChestX dataset. We will include them in the camera-ready version.

| Model | EuroSAT | CropDisease | ISIC | ChestX |
| :-------------- | :------ | :---------- | :---- | :----- |
| BYOL | 82.95 | 91.52 | 41.22 | 26.44 |
| Transfer + BYOL | 85.59 | 89.83 | 45.57 | 29.10 |
| MoCo | 83.44 | 85.20 | 46.86 | 28.30 |
| Transfer + MoCo | 84.42 | 87.56 | 47.20 | **29.52** |
| Ours | **89.07**| **95.54** | **49.36** | 28.31 |

***Q2: Compariosn with other in-domain semi-supervised few-shot learning baselines.***

**A2:** Thank you for pointing this out. We show additional results for 5-way 5-shot evaluation below for semi-supervised soft k-Means Prototypical Network from [18], which uses both labeled base dataset and unlabeled target dataset to create class prototypes. Apart from the CropdDisease dataset, the results are worse than simple ProtoNet. It suggests that soft k-means ProtoNet is not optimal for cross-domain few-shot setting where the unlabeled samples are obtained from a different domain than the base dataset.

| Model | EuroSAT | CropDisease | ISIC | ChestX |
| :-------------------- | :------ | :---------- | :---- | :----- |
| ProtoNet | 76.92 | 81.84 | 42.49 | 24.72 |
| Soft k-Means ProtoNet | 72.10 | 82.43 | 41.44 | 24.26 |
| Ours | 89.07 | 95.54 | 49.36 | 28.31 |

Note that, [14] uses transductive inference that classifies the entire test set at once, hence it's not directly applicable with our evaluation protocol.

[14] Y. Liu, J. Lee, M. Park, S. Kim, E. Yang, S. J. Hwang, and Y. Yang. Learning to propagate labels: Transductive propagation network for few-shot learning.

[18] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification

***Q3: I also am not convinced by the conclusion of lines 226-227, as another possible explanation is that the target tasks are simply too easy for extra large data to show any benefit."***?

**A3:** Thanks for the excellent suggestion. We agree that this could be another explanation, and will update it in the camera ready version.

***Q4: I found it surprising that training with Ours-All in Table 6 did not yield the best results in all target tasks. Why do the authors think that is?***

**A4:** Excellent question! It has been shown in the literature that not every unlabeled sample is equally helpful for the downstream task in self-supervised or semi-supervised learning [1]. Our experiment suggests that unlabeled samples from a completely different domain than the downstream task might actually hurt the performance.

[1] Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing. Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning. NeurIPS'20.

## Response to Reviewer zcQX

Thank you for the insightful comments. Below we address the concerns.

***Q1: Limited novelty since the approach combines the mean teacher approach from [23] and self-supervised loss from [5] and [2], yet achieving good performance in the cross-domain few-shot learning context.***

**A1**: To clarify, there are crucial differences between our methods and other self-supervised methods like [2, 5]. The projection head we are using to calculate the final predictions of the two different views of an unlabeled image is the same classification head that is used to predict the classification logits of the labeled base samples. This is important as we show in Table 5 in the main paper that separate projection head performs much worse. We refer to the section "Comparison with self-supervised learning" (line 262-283) for detailed comparison with the other self-supervised learning methods. We also agree that our teacher-student approach is inspired from [23]. However, the other details are important (e.g., using the same classification head for distillation loss, augmentation, two-step training) to make it work on the cross-domain few-shot setting.

[2] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers.

[5] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning.

[23] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.

***Q2: The cross-entropy function in the distillation loss \\( l_u \\) cannot guarantee that the prediction of the student is equal to the prediction of the teacher. Please clarify this.***

**A2**: To clarify, the cross entropy function \\( l_u(p^w, p^s) = H(p^w, p^s) \\) is similar to KL divergence \\( D_{KL}(p^w, p^s) = H(p^w, p^s) - H(p^w) \\) from the perspective of optimization for \\( p^s \\), as \\( p^w \\) is held constant with *stop-grad* operation. Hence, \\( l_u(p^w, p^s) \\) will be minimized when the distributions \\( p^w \\) and \\( p^s \\) are the same.

***Q3: Why and how to choose the weak and strong data augmentations?***

**A3**: For weak augmentation, we use random-resize-crop, horizontal flip and normalization, and for strong augmentation we additionally use additionally use the color jitter, Gaussian blur, and random gray scale transformations (line 183-186 in the main paper).

To answer why we chose weak and strong augmentation, we provided ablations in the main paper (line 289-297). We infer that *weakly-augmented images provide better pseudo-labels* from the teacher network such that the student network can be optimized to make the outputs from strongly-augmented images to be consistent with the outputs from the weakly-augmented images.

## Response to Reviewer pL1o

We appreciate the review and thank the reviewer for the thoughtful feedback.

***Q1: My main concern to this paper is the motivation. Using student-teacher knowledge distillation is not a new idea in training deep neural networks. It remains unclear to me why such method is strong on cross-domain few-shot learning without seeing the experimental results. I expect the authors to provide more discussion on the advantages of the proposed methods, e.g., what specific natures of cross-domain few-shot learning is exploited? The current discussion shows only 'another way of using unlabeled data' in cross domain few-shot learning, yet this is something can be exploited in many tasks like standard few-shot learning, domain generalizations, etc..***

**A1**: We thank the reviewer for the detailed comments. We provide several points below:

1. Although student-teacher knowledge distillation has been exploited in several computer vision problems, to the best of our knowledge, we are the first to apply this in the cross-domain few-shot learning problem.
2. We hypothesize that using both labeled base data and unlabeled target data during training provides a common embedding for both base and target domain. Then the natural question could be - why not use the unlabeled target data only, it might provide more target-specific representation. One issue with this approach is that self-supervised learning generally requires a large amount of unlabeled data to work. Secondly, it has been shown that combining supervised and unsupervised learning during training provides more transferable representation (*Islam et al., A Broad Study on the Transferability of Visual Representations with Contrastive Learning*). We argue that similar conclusion holds for cross-domain few-shot learning, i.e., combining supervised and unsupervised loss provides better representation for the downstream task.
3. An important aspect of our method is sharing the same head for supervised loss and distillation loss. Our distillation loss is similar to non-contrastive self-supervised loss like BYOL or DINO. However, it requires much more data and several training tricks to make the non-contrastive method like DINO to work. We argue that using the same distillation head for supervised loss resolves these issues. Please refer to line 262-283 in the main paper.

5. As for why our proposed method works on the cross-domain setting, we also refer to the "Effect of dynamic distillation" section of the main paper (line 242-261), where we show that our method creates better grouping on the embeddings of the target datasets even though we do not use any labels from the target dataset during pretraining.
6. We agree with the reviewer that this approach can be exploited in standard few-shot setting too, which we verified in "Few-shot performance on similar domain" section (line 228-240).

***Q2: The paper is somehow poorly organized. It is surprising to see that the main discussion (methodology) is not even 1.5 page long with a Figure in it. I think this is also part of the reason that the motivation of this paper is not well presented.***

**A2**: Thanks for the suggestion. We will provide more details about the motivation of our proposed method in the final version.

***Q3. Some widely used cross-domain few-shot learning settings were set in [1], and are not included in the paper.***

**A3**: For cross-domain few-shot evaluation, [1] uses only mini-ImageNet->CUB, i.e., training on mini-ImageNet dataset and evaluation on CUB dataset. We argue that this setting is rather limited as CUB contains only natural images like ImageNet. We adopt BSCD-FSL benchmark [6] which has a better distribution of downstream datasets from natural to medical images.
[1] Chen, Wei-Yu, et al. "A closer look at few-shot classification." ICLR 2019.
[6] Guo et al. A broader study of cross-domain few-shot learning. ECCV 2020.

***Q4: The performance on standard few-shot classification datasets are actually not comparable to SOTA. E.g., according to the [leaderboard](https://few-shot.yyliu.net/miniimagenet.html), with the standard inductive setting, many methods can achieve over 54% with simple Conv-4 architecture on miniImageNet 5way 1shot. While in-domain few-shot classification is obviously less challenging, it is confusing to me why the proposed method performs poorly.***

**A4**: Thanks for the comment. In Table 3, we show the in-domain performance comparison with similar training and test set and similar evaluation protocol for the methods considered for cross-domain few-shot learning. First, we want to clarify that our goal is not meta-learning for in-domain few-shot evaluation. Our approach is about having a stronger pretraining if some unlabeled target-related data are available, which is not the evaluation protocol of the leaderboard. Moreover, as stated in lines 233-234, our method needs unlabeled data from novel classes, which results in the different test set for the evaluation than the leaderboard uses. Thus the results are not comparable.

***Q5: Can the authors provide more discussion on how the proposed method is 'dynamic'?***

**A5**: The term 'dynamic' refers to the momentum teacher, as the parameters of the teacher network are updated during training from the parameters of the student network. We provided ablation on the importance of the momentum update in Table 11 in the supplementary material, which shows that we get around 1.47\% average improvement over fixed teacher for 5-way 5-shot evaluation.

## Response to Reviewer 3Ucv

We thank the reviewer for the positive and detailed review as well as the suggestions for improvement. Our response to the reviewer’s question is below:

***Q1: In Tables 1 and 2, the performance of the proposed method is compared with the STARTUP and SimCLR-based baselines. However, BYOL is the most similar SSL method to the proposed method. How is the performance of the **BYOL** or **Transfer + BYOL** in cross-domain FSL?***

**A1**: We thank the reviewers for the suggestion. We performed additional experiments for *BYOL* and *Transfer+BYOL*, and report the 5-way 5-shot results below. Similar to our comparison with *SimCLR* and *Transfer+SimCLR* in Table 1 in the main paper, our method outperforms BYOL in all datasets except the ChestX dataset. We will include them in the camera-ready version.

| Model | EuroSAT | CropDisease | ISIC | ChestX |
| :-------------- | :------ | :---------- | :---- | :----- |
| BYOL | 82.95 | 91.52 | 41.22 | 26.44 |
| Transfer + BYOL | 85.59 | 89.83 | 45.57 | 29.10 |
| Ours | 89.07 | 95.54 | 49.36 | 28.31 |

***Q2: In ablation experiments, the authors presented the performance of Ours (w/o base), that pretrain the model on miniImageNet (base dataset) and then learn representation without labeled base dataset. What will happen if someone pretrains the model on base dataset, and then learn representation using the unlabeled images and images from base dataset (use the images only, and not use the label. Consider the images from base dataset as unlabeled data)? It can be an interesting experiment to figure out the effect of supervised loss in the proposed method.***

**A2**: Thank you for the suggestion, it is an interesting experiment that we didn't consider in the main paper. We show the 5-way 5-shot results of the suggested experiment (*Ours w/o base-labels*) below.

| | EuroSAT | CropDisease | ISIC | ChestX |
| :------ | :------ | :---------- | :---- | :----- |
| Ours w/o base-labels | 88.19 | 93.94 | 38.03 | 23.50 |
| Ours | 89.07 | 95.54 | 49.36 | 28.31 |

Although the method works well for EuroSAT and CropDisease dataset, it performs poorly for ISIC and ChestX. It further clarifies the importance of the supervised loss in our method.

***Q3: In line 212 to 214, the authors stated that the proposed method does not use any self-supervised training or distillation. However, the proposed method is dynamic distillation, and it utilizes the unlabeled samples like the self-supervised learning model, BYOL. Can you explain the exact meaning of the sentence in line 212 to 214?***

**A3**: Thanks for pointing this out. Line 212 is an unintentional typo from our side. We meant to say that "the proposed method does not use any supervision from the target domain during pretraining". We will change it in the final version.