DL Surveyception: a survey of overviews, reviews and surveys in deep learning for computer vision, time-series and others

Summary of slides

Introduction

In order to catch up with the progress of the field, I did a meta-survey, a survey of surveys, in different areas of deep learning. It allows for understanding the main challenges and trends in particular areas.

Methodology

I found the following papers by using the following keywords on Google, Google Scholar, Microsoft Academic, SemanticScholar, and others were found through connections on Connected Papers or other websites. I divide them into papers before 2020, and since 2020. The former includes 14 papers, while the latter includes 15, for a total of 29 papers.

For all the papers I list the title, the date it was last updated (for most Arxiv papers), the number of citations (estimate of how impactful/useful it's been for others), the number of pages (how much content does it pack), and the first author (just to get familiar with some big names in the field). The classification in this section is relatively arbitrary, since the original purpose of this was to do a survey on GANs but then I expanded it to other topics. However, just for reference, the idea is that the blue DL/CV represented more general-purpose surveys, the green AS meant application-specific where it focused on a particular area or application, and the purple GAN represented general papers related to GANs.

Surveys before 2020.

Surveys since 2020.

Surveyception

I'll try to group up similar reviews and list them here, along with one or two reviews that I found particularly informative, or helpful, for any reason. I'll link the topic in the tables to the slide in the album.

Note: these slides are definitely not for everyone; they're extremely information dense.

Disclaimer: the content of these slides correspond to the respective authors.

Time-series

Topic	Year / Month	First Author
Physiological Signals	2018 / 05	Faust
EEG Analysis	2019 / 01	Roy
Classification	2019 / 05	Fawaz

Computer vision

Topic	Year / Month	First Author
CV using DL in General	2017 / 06	Voulodimos
CV using DL in General	2020 / 05	Khan A.

GANs

Topic	Year / Month	First Author
GANs in General	2017 / 09	Creswell
GANs in General	2018 / 01	Wang K.
GANs in General	2019 / 03	Pan
GANs in General	2020 / 06	Jabbar
GANs in General	2020 / 06	Gui
GANs in General: Pt. 1 / Pt. 2 / Pt. 3	2020 / 09	Saxena
GANs in General	2020 / 12	Wang Z.
GANs for Medical Imaging	2019 / 09	Yi
GANs Stabilizing	2020 / 03	Wistrak
GANs Loss Functions	2020 / 05	Pan
GANs for DeepFakes	2020 / 06	Tolosana
GANs for Video	2020 / 11	Aldausari
GANs for Image Synthesis: Pt. 1 / Pt. 2	2020 / 12	Shamsolmoali

Transformers

Topic	Year / Month	First Author
Efficient Transformers: Pt. 1 / Pt. 2 / Pt. 3	2020 / 09	Tay
Transformers for Vision: Pt. 1 / Pt. 2 / Pt. 3	2021 / 01	Khan S.
Transformers for Vision	2021 / 01	Han

Representation, transfer and not-fully supervised learning

Topic	Year / Month	First Author
Not Fully-Supervised Learning for Medical Image	2017 / 04	Cheplygina
Self-Supervised Learning	2019 / 02	Jing
Transfer Learning	2019 / 05	Zhang
Transfer Learning	2019 / 11	Zhuang
Self-Supervised Learning: Pt. 1 / Pt. 2	2020 / 07	Liu

Others:

Topic	Year / Month	First Author
DL for Healthcare in General	2017 / 05	Miotto
DL for Edge Computing	2019 / 08	Chen
DL for Audio-Visual	2020 / 06	Zhu

Open-source repo020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2001.04758.

[4]M. Wiatrak, S. V. Albrecht, and A. Nystrom, “Stabilizing Generative Adversarial Networks: A Survey,” arXiv:1910.00927 [cs], Mar. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1910.00927.
[5]Z. Wang, Q. She, and T. E. Ward, “Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy,” arXiv:1906.01529 [cs], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1906.01529.
[6]R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection,” arXiv:2001.00179 [cs], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2001.00179.
[7]Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient Transformers: A Survey,” arXiv:2009.06732 [cs], Sep. 2020, Accessed: Jan. 22, 2021. [Online]. Available: http://arxiv.org/abs/2009.06732.
[8]P. Shamsolmoali et al., “Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies,” arXiv:2012.13736 [cs, eess], Dec. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2012.13736.
[9]D. Saxena and J. Cao, “Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions,” arXiv:2005.00065 [cs, eess, stat], Oct. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2005.00065.
[10]Z. Pan et al., “Loss Functions of Generative Adversarial Networks (GANs): Opportunities and Challenges,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 4, no. 4, pp. 500–522, Aug. 2020, doi: 10.1109/TETCI.2020.2991774.
[11]X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” arXiv:2006.08218 [cs, stat], Jul. 2020, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/2006.08218.
[12]B. Lim and S. Zohren, “Time Series Forecasting With Deep Learning: A Survey,” arXiv:2004.13408 [cs, stat], Sep. 2020, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/2004.13408.
[13]A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artif Intell Rev, vol. 53, no. 8, pp. 5455–5516, Dec. 2020, doi: 10.1007/s10462-020-09825-6.
[14]A. Jabbar, X. Li, and B. Omar, “A Survey on Generative Adversarial Networks: Variants, Applications, and Training,” arXiv:2006.05132 [cs, eess], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2006.05132.
[15]S. Hooker, “The Hardware Lottery,” arXiv:2009.06489 [cs], Sep. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2009.06489.
[16]J. Gui, Z. Sun, Y. Wen, D. Tao, and J. Ye, “A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications,” arXiv:2001.06937 [cs, stat], Jan. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2001.06937.
[17]N. Aldausari, A. Sowmya, N. Marcus, and G. Mohammadi, “Video Generative Adversarial Networks: A Review,” arXiv:2011.02250 [cs, eess], Nov. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2011.02250.
[18]F. Zhuang et al., “A Comprehensive Survey on Transfer Learning,” Nov. 2019, Accessed: Jan. 13, 2021. [Online]. Available: https://arxiv.org/abs/1911.02685v3.
[19]J. Zhang, W. Li, P. Ogunbona, and D. Xu, “Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective,” arXiv:1705.04396 [cs], May 2019, Accessed: Jan. 24, 2021. [Online]. Available: http://arxiv.org/abs/1705.04396.
[20]X. Yi, E. Walia, and P. Babyn, “Generative Adversarial Network in Medical Imaging: A Review,” Medical Image Analysis, vol. 58, p. 101552, Dec. 2019, doi: 10.1016/j.media.2019.101552.
[21]Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep learning-based electroencephalography analysis: a systematic review,” arXiv:1901.05498 [cs, eess, stat], Jan. 2019, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1901.05498.
[22]Z. Pan, W. Yu, X. Yi, A. Khan, F. Yuan, and Y. Zheng, “Recent Progress on Generative Adversarial Networks (GANs): A Survey,” IEEE Access, vol. 7, pp. 36322–36333, 2019, doi: 10.1109/ACCESS.2019.2905015.
[23]L. Jing and Y. Tian, “Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey,” arXiv:1902.06162 [cs], Feb. 2019, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/1902.06162.
[24]H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” Data Min Knowl Disc, vol. 33, no. 4, pp. 917–963, Jul. 2019, doi: 10.1007/s10618-019-00619-1.
[25]J. Chen and X. Ran, “Deep Learning With Edge Computing: A Review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, Aug. 2019, doi: 10.1109/JPROC.2019.2921977.
[26]R. Miotto, F. Wang, S. Wang, X. Jiang, and J. T. Dudley, “Deep learning for healthcare: review, opportunities and challenges,” Brief Bioinform, vol. 19, no. 6, pp. 1236–1246, Nov. 2018, doi: 10.1093/bib/bbx044.
[27]O. Faust, Y. Hagiwara, T. J. Hong, O. S. Lin, and U. R. Acharya, “Deep learning for healthcare applications based on physiological signals: A review,” Computer Methods and Programs in Biomedicine, vol. 161, pp. 1–13, Jul. 2018, doi: 10.1016/j.cmpb.2018.04.005.
[28]A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative Adversarial Networks: An Overview,” IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 53–65, Jan. 2018, doi: 10.1109/MSP.2017.2765202.
[29]V. Cheplygina, M. de Bruijne, and J. P. W. Pluim, “Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis,” Apr. 2018, Accessed: Jan. 13, 2021. [Online]. Available: https://arxiv.org/abs/1804.06353v2.
[30]V. A, D. N, D. A, and P. E, “Deep Learning for Computer Vision: A Brief Review.,” Comput Intell Neurosci, vol. 2018, pp. 7068349–7068349, Feb. 2018, doi: 10.1155/2018/7068349.
[31]K. Wang, C. Gou, Y. Duan, Y. Lin, X. Zheng, and F. Wang, “Generative adversarial networks: introduction and outlook,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4, pp. 588–598, 2017, doi: 10.1109/JAS.2017.7510583.
[32]Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539.
[33]M. Längkvist, L. Karlsson, and A. Loutfi, “A Review of Unsupervised Feature Learning and Deep Learning for Time-Series Modeling,” Pattern Recognition Letters, vol. 42, Jun. 2014, doi: 10.1016/j.patrec.2014.01.008.
[34]S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, Oct. 2010, doi: 10.1109/TKDE.2009.191.