---
title: 'Transfer learning for (?): a review'
disqus: hackmd
---

Transfer learning for (?): a review
===
**(?) = object detection? image classification? computer vision in general?**

## Table of Contents

[TOC]

## Papers to review

#### Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
[https://arxiv.org/abs/](https://arxiv.org/abs/2001.06268)
They modify ResNet-50 (it becomes something more similar to mxresnet, but even more complex), add mixup for training: the result is 3x faster to train than EfficientNet and slightly less accurate.

#### A Study on CNN Transfer Learning for Image Classification [^hussain2018study]
[researchgate.com](https://www.researchgate.net/publication/325803364_A_Study_on_CNN_Transfer_Learning_for_Image_Classification)
This work proposes the study and investigation of such a CNN architecture model (i.e. Inception-v3) to establish whether it works best in terms of accuracy and efficiency with new image datasets via Transfer Learning. The retrained model is evaluated, and the results are compared to some state-of-the-art approaches.

#### Multifaceted Analysis of Fine-Tuning in Deep Model for Visual Recognition [^li2019Multifaceted]
[https://arxiv.org/](https://arxiv.org/abs/1907.05099)
In this paper, we introduce and systematically investigate several factors that influence the performance of fine-tuning for visual recognition. These factors include parameters for the retraining procedure (e.g., the initial learning rate of fine-tuning), the distribution of the source and target data (e.g., the number of categories in the source dataset, the distance between the source and target datasets) and so on. We quantitatively and qualitatively analyze these factors, evaluate their influence, and present many empirical observations. The results reveal insights into what fine-tuning changes CNN parameters and provide useful and evidence-backed intuitions about how to implement fine-tuning for computer vision tasks.

#### Factors in Finetuning Deep Model for object detection [^Ouyang2016Factors]
[https://arxiv.org/](https://arxiv.org/abs/1601.05150)
Finetuning from a pretrained deep model is found to yield state-of-the-art performance for many vision tasks. This paper investigates many factors that influence the performance in finetuning for object detection. There is a long-tailed distribution of sample numbers for classes in object detection. Our analysis and empirical results show that classes with more samples have higher impact on the feature learning. And it is better to make the sample number more uniform across classes. Finetuned on the GoogLeNet model, experimental results show 4.7% absolute mAP improvement of our approach on the ImageNet object detection dataset without increasing much computational cost at the testing stage.

#### A Survey on Deep Transfer Learning [^Tan2018survey]
[https://arxiv.org/](https://arxiv.org/abs/1808.01974)
As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition and costly annotation, which limits its development. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed (i.i.d.) with the test data, which motivates us to use transfer learning to solve the problem of insufficient training data. This survey focuses on reviewing the current researches of transfer learning by using deep neural network and its applications. We defined deep transfer learning, category and review the recent research works based on the techniques used in deep transfer learning.

#### Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective [^Zhang2019Recent]
[https://arxiv.org/](https://arxiv.org/abs/1705.04396)
This paper takes a problem-oriented perspective and presents a comprehensive review of transfer learning methods, both shallow and deep, for cross-dataset visual recognition. Specifically, it categorises the cross-dataset recognition into seventeen problems based on a set of carefully chosen data and label attributes. Such a problem-oriented taxonomy has allowed us to examine how different transfer learning approaches tackle each problem and how well each problem has been researched to date. The comprehensive problem-oriented review of the advances in transfer learning with respect to the problem has not only revealed the challenges in transfer learning for visual recognition, but also the problems (e.g. eight of the seventeen problems) that have been scarcely studied. This survey not only presents an up-to-date technical review for researchers, but also a systematic approach and a reference for a machine learning practitioner to categorise a real problem and to look up for a possible solution accordingly

#### Domain Adaptation for Visual Applications: A Comprehensive Survey [^Csurka2017Domain]
[https://arxiv.org/](https://arxiv.org/abs/1702.05374)
The aim of this paper is to give an overview of domain adaptation and transfer learning with a specific view on visual applications. After a general motivation, we first position domain adaptation in the larger transfer learning problem. Second, we try to address and analyze briefly the state-of-the-art methods for different types of scenarios, first describing the historical shallow methods, addressing both the homogeneous and the heterogeneous domain adaptation methods. Third, we discuss the effect of the success of deep convolutional architectures which led to new type of domain adaptation methods that integrate the adaptation within the deep architecture. Fourth, we overview the methods that go beyond image categorization, such as object detection or image segmentation, video analyses or learning visual attributes. Finally, we conclude the paper with a section where we relate domain adaptation to other machine learning solutions.

## On transfer learning for cv

***There are too many of them***

check for other resources here https://arxiv.org/pdf/1807.05511.pdf

#### Rich feature hierarchies for accurate object detection and semantic segmentation [^girshick2014Rich]
One of the first papers applying succesfully transfer learning on a large dataset.

#### Revisiting Fine-tuning for few-shot learning [^Nakamura2019Revisiting]
[https://arxiv.org/](https://arxiv.org/abs/1910.00216)
Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used lowresolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.

#### A Framework of Transfer Learning in Object Detection for Embedded Systems [^Athanasiadis2018Framework]
[https://arxiv.org/](https://arxiv.org/abs/1811.04863)
Transfer learning is one of the subjects undergoing intense study in the area of machine learning. In object recognition and object detection there are known experiments for the transferability of parameters, but not for neural networks which are suitable for object detection in real time embedded applications, such as the SqueezeDet neural network. We use transfer learning to accelerate the training of SqueezeDet to a new group of classes. Also, experiments are conducted to study the transferability and co-adaptation phenomena introduced by the transfer learning process. To accelerate training, we propose a new implementation1 of the SqueezeDet training which provides a faster pipeline for data processing and achieves 1.8 times speedup compared to the initial implementation. Finally, we created a mechanism for automatic hyperparameter optimization using an empirical method.

#### Pay Attention to Features, Transfer Learn Faster CNNs
[openreview.net](https://openreview.net/pdf?id=ryxyCeHtPB)
Deep convolutional neural networks are now widely deployed in vision applications, but the size of training data can bottleneck their performance. Transfer learning offers the chance for CNNs to learn with limited data samples by transferring knowledge from weights pre-trained on large datasets. On the other hand, blindly transferring all learned features from the source dataset brings unnecessary computation to CNNs on the target task. In this paper, we propose attentive feature distillation and selection (AFDS) that not only adjusts the strength of regularization introduced by transfer learning but also dynamically determines which are the important features to transfer. When deploying AFDS on ResNet-101, we achieve state-of-theart computation reduction at the same accuracy budget, outperforming all existing transfer learning methods

## References
[^hussain2018study]: Hussain, Mahbub & Bird, Jordan & Faria, Diego. (2018). A Study on CNN Transfer Learning for Image Classification.

[^li2019Multifaceted]: Li, Xiangyang, Luis Herranz, and Shuqiang Jiang. "Multifaceted Analysis of Fine-Tuning in Deep Model for Visual Recognition." arXiv preprint arXiv:1907.05099 (2019).

[^girshick2014Rich]: Girshick, Ross, et al. "Rich feature hierarchies for accurate object detection and semantic segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.

[^Ouyang2016Factors]: Ouyang, Wanli, et al. "Factors in finetuning deep model for object detection with long-tail distribution." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[^Tan2018survey]: Tan, Chuanqi, et al. "A survey on deep transfer learning." International conference on artificial neural networks. Springer, Cham, 2018.

[^Zhang2019Recent]: Zhang, Jing, et al. "Recent advances in transfer learning for cross-dataset visual recognition: A problem-oriented perspective." ACM Computing Surveys (CSUR) 52.1 (2019): 1-38.

[^Csurka2017Domain]: Csurka, Gabriela. "Domain adaptation for visual applications: A comprehensive survey." arXiv preprint arXiv:1702.05374 (2017).

[^Nakamura2019Revisiting]: Nakamura, Akihiro, and Tatsuya Harada. "Revisiting Fine-tuning for Few-shot Learning." arXiv preprint arXiv:1910.00216 (2019).

[^Athanasiadis2018Framework]: Athanasiadis, Ioannis, Panagiotis Mousouliotis, and Loukas Petrou. "A Framework of Transfer Learning in Object Detection for Embedded Systems." arXiv preprint arXiv:1811.04863 (2018).