# DL Surveyception: a survey of overviews, reviews and surveys in deep learning for computer vision, time-series and others [Summary of slides](https://imgur.com/a/02pZTKh) ## Introduction In order to catch up with the progress of the field, I did a meta-survey, a survey of surveys, in different areas of deep learning. It allows for understanding the main challenges and trends in particular areas. ## Methodology I found the following papers by using the following keywords on Google, Google Scholar, Microsoft Academic, SemanticScholar, and others were found through connections on [Connected Papers](https://www.connectedpapers.com/) or other websites. I divide them into papers before 2020, and since 2020. The former includes 14 papers, while the latter includes 15, for a total of 29 papers. For all the papers I list the title, the date it was last updated (for most Arxiv papers), the number of citations (estimate of how impactful/useful it's been for others), the number of pages (how much content does it pack), and the first author (just to get familiar with some big names in the field). The classification in this section is relatively arbitrary, since the original purpose of this was to do a survey on GANs but then I expanded it to other topics. However, just for reference, the idea is that the blue DL/CV represented more general-purpose surveys, the green AS meant application-specific where it focused on a particular area or application, and the purple GAN represented general papers related to GANs. <figure> <img src="https://i.imgur.com/vrlMyaE.png" alt="Surveys Before 2020"> <figcaption style="align: left; text-align:center;">Surveys before 2020.</figcaption> </figure> <figure> <img src=" https://i.imgur.com/e84uHyg.png" alt="Surveys Since 2020"> <figcaption style="align: left; text-align:center;">Surveys since 2020.</figcaption> </figure> ## Surveyception I'll try to group up similar reviews and list them here, along with one or two reviews that I found particularly informative, or helpful, for any reason. I'll link the topic in the tables to the slide in the album. Note: these slides are definitely not for everyone; they're extremely information dense. Disclaimer: the content of these slides correspond to the respective authors. ### Time-series | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [Physiological Signals](https://i.imgur.com/0f4JcXA.png) | 2018 / 05 | Faust | | [EEG Analysis](https://i.imgur.com/Zq6TzcO.png) | 2019 / 01 | Roy | | [Classification](https://i.imgur.com/o9EFF9m.png) | 2019 / 05 | Fawaz| ![](https://i.imgur.com/Zq6TzcO.png) ### Computer vision | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [CV using DL in General](https://i.imgur.com/hUELhrd.png) | 2017 / 06 | Voulodimos| | [CV using DL in General](https://i.imgur.com/TWJt0sO.png) | 2020 / 05 | Khan A. | ![](https://i.imgur.com/TWJt0sO.png) ### GANs | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [GANs in General](https://i.imgur.com/xnQMPnz.png) | 2017 / 09 | Creswell| | [GANs in General](https://i.imgur.com/N5CyJkD.png) | 2018 / 01 | Wang K. | | [GANs in General](https://i.imgur.com/OwN5VSj.png) | 2019 / 03 | Pan | | [GANs in General](https://i.imgur.com/QzCdCUJ.png) | 2020 / 06 | Jabbar | | [GANs in General](https://i.imgur.com/40fMF6i.png) | 2020 / 06 | Gui | | [GANs in General: Pt. 1](https://i.imgur.com/TZrCqew.png) / [Pt. 2](https://i.imgur.com/k35zPak.png) / [Pt. 3](https://i.imgur.com/UpbG3aq.png) | 2020 / 09 | Saxena | | [GANs in General](https://i.imgur.com/mHXkA0p.png) | 2020 / 12 | Wang Z. | | [GANs for Medical Imaging](https://i.imgur.com/ZdggTTP.png) | 2019 / 09 | Yi | | [GANs Stabilizing](https://i.imgur.com/QI19oys.png) | 2020 / 03 | Wistrak | | [GANs Loss Functions](https://i.imgur.com/X3rhpWp.png) | 2020 / 05 | Pan | | [GANs for DeepFakes](https://i.imgur.com/EHQqoUg.png) | 2020 / 06 | Tolosana | | [GANs for Video](https://i.imgur.com/ntaO9j8.png) | 2020 / 11 | Aldausari | | [GANs for Image Synthesis: Pt. 1](https://i.imgur.com/OiDxN8l.png) / [Pt. 2](https://i.imgur.com/duLqFJ9.png) | 2020 / 12 | Shamsolmoali | ![](https://i.imgur.com/mHXkA0p.png) ![](https://i.imgur.com/QI19oys.png) ### Transformers | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [Efficient Transformers: Pt. 1](https://i.imgur.com/bTshOlz.png) / [Pt. 2](https://i.imgur.com/dlVPD6P.png) / [Pt. 3](https://i.imgur.com/CQVnmTE.png) | 2020 / 09 | Tay| | [Transformers for Vision: Pt. 1](https://i.imgur.com/r67KWwS.png) / [Pt. 2](https://i.imgur.com/YgYsa1Q.png) / [Pt. 3](https://i.imgur.com/3abuhoC.png) | 2021 / 01 | Khan S. | | [Transformers for Vision](https://i.imgur.com/BuZhSlz.png) | 2021 / 01 | Han | ![](https://i.imgur.com/r67KWwS.png) ### Representation, transfer and not-fully supervised learning | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [Not Fully-Supervised Learning for Medical Image](https://i.imgur.com/c00cwJK.png) | 2017 / 04 | Cheplygina | | [Self-Supervised Learning](https://i.imgur.com/cjSOLcZ.png) | 2019 / 02 | Jing | | [Transfer Learning](https://i.imgur.com/uIitJzJ.png) | 2019 / 05 | Zhang | | [Transfer Learning](https://i.imgur.com/M1HrFL6.png) | 2019 / 11 | Zhuang | | [Self-Supervised Learning: Pt. 1](https://i.imgur.com/ymzHSd9.png) / [Pt. 2](https://i.imgur.com/XI1YT5B.png) | 2020 / 07 | Liu | ![](https://i.imgur.com/cjSOLcZ.png) ![](https://i.imgur.com/ymzHSd9.png) ### Others: | Topic | Year / Month | First Author | | ------- | ---------- | ------ | | [DL for Healthcare in General](https://i.imgur.com/8lOL4nV.png) | 2017 / 05 | Miotto| | [DL for Edge Computing](https://i.imgur.com/t6PqFsF.png) | 2019 / 08 | Chen | | [DL for Audio-Visual](https://i.imgur.com/ClRRADj.png) | 2020 / 06 | Zhu | ![](https://i.imgur.com/t6PqFsF.png) ![](https://i.imgur.com/ClRRADj.png) ## Open-source repo020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2001.04758. [4]M. Wiatrak, S. V. Albrecht, and A. Nystrom, “Stabilizing Generative Adversarial Networks: A Survey,” arXiv:1910.00927 [cs], Mar. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1910.00927. [5]Z. Wang, Q. She, and T. E. Ward, “Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy,” arXiv:1906.01529 [cs], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1906.01529. [6]R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection,” arXiv:2001.00179 [cs], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2001.00179. [7]Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient Transformers: A Survey,” arXiv:2009.06732 [cs], Sep. 2020, Accessed: Jan. 22, 2021. [Online]. Available: http://arxiv.org/abs/2009.06732. [8]P. Shamsolmoali et al., “Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies,” arXiv:2012.13736 [cs, eess], Dec. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2012.13736. [9]D. Saxena and J. Cao, “Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions,” arXiv:2005.00065 [cs, eess, stat], Oct. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2005.00065. [10]Z. Pan et al., “Loss Functions of Generative Adversarial Networks (GANs): Opportunities and Challenges,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 4, no. 4, pp. 500–522, Aug. 2020, doi: 10.1109/TETCI.2020.2991774. [11]X. Liu et al., “Self-supervised Learning: Generative or Contrastive,” arXiv:2006.08218 [cs, stat], Jul. 2020, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/2006.08218. [12]B. Lim and S. Zohren, “Time Series Forecasting With Deep Learning: A Survey,” arXiv:2004.13408 [cs, stat], Sep. 2020, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/2004.13408. [13]A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” Artif Intell Rev, vol. 53, no. 8, pp. 5455–5516, Dec. 2020, doi: 10.1007/s10462-020-09825-6. [14]A. Jabbar, X. Li, and B. Omar, “A Survey on Generative Adversarial Networks: Variants, Applications, and Training,” arXiv:2006.05132 [cs, eess], Jun. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2006.05132. [15]S. Hooker, “The Hardware Lottery,” arXiv:2009.06489 [cs], Sep. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2009.06489. [16]J. Gui, Z. Sun, Y. Wen, D. Tao, and J. Ye, “A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications,” arXiv:2001.06937 [cs, stat], Jan. 2020, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/2001.06937. [17]N. Aldausari, A. Sowmya, N. Marcus, and G. Mohammadi, “Video Generative Adversarial Networks: A Review,” arXiv:2011.02250 [cs, eess], Nov. 2020, Accessed: Jan. 28, 2021. [Online]. Available: http://arxiv.org/abs/2011.02250. [18]F. Zhuang et al., “A Comprehensive Survey on Transfer Learning,” Nov. 2019, Accessed: Jan. 13, 2021. [Online]. Available: https://arxiv.org/abs/1911.02685v3. [19]J. Zhang, W. Li, P. Ogunbona, and D. Xu, “Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective,” arXiv:1705.04396 [cs], May 2019, Accessed: Jan. 24, 2021. [Online]. Available: http://arxiv.org/abs/1705.04396. [20]X. Yi, E. Walia, and P. Babyn, “Generative Adversarial Network in Medical Imaging: A Review,” Medical Image Analysis, vol. 58, p. 101552, Dec. 2019, doi: 10.1016/j.media.2019.101552. [21]Y. Roy, H. Banville, I. Albuquerque, A. Gramfort, T. H. Falk, and J. Faubert, “Deep learning-based electroencephalography analysis: a systematic review,” arXiv:1901.05498 [cs, eess, stat], Jan. 2019, Accessed: Nov. 13, 2020. [Online]. Available: http://arxiv.org/abs/1901.05498. [22]Z. Pan, W. Yu, X. Yi, A. Khan, F. Yuan, and Y. Zheng, “Recent Progress on Generative Adversarial Networks (GANs): A Survey,” IEEE Access, vol. 7, pp. 36322–36333, 2019, doi: 10.1109/ACCESS.2019.2905015. [23]L. Jing and Y. Tian, “Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey,” arXiv:1902.06162 [cs], Feb. 2019, Accessed: Dec. 16, 2020. [Online]. Available: http://arxiv.org/abs/1902.06162. [24]H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” Data Min Knowl Disc, vol. 33, no. 4, pp. 917–963, Jul. 2019, doi: 10.1007/s10618-019-00619-1. [25]J. Chen and X. Ran, “Deep Learning With Edge Computing: A Review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, Aug. 2019, doi: 10.1109/JPROC.2019.2921977. [26]R. Miotto, F. Wang, S. Wang, X. Jiang, and J. T. Dudley, “Deep learning for healthcare: review, opportunities and challenges,” Brief Bioinform, vol. 19, no. 6, pp. 1236–1246, Nov. 2018, doi: 10.1093/bib/bbx044. [27]O. Faust, Y. Hagiwara, T. J. Hong, O. S. Lin, and U. R. Acharya, “Deep learning for healthcare applications based on physiological signals: A review,” Computer Methods and Programs in Biomedicine, vol. 161, pp. 1–13, Jul. 2018, doi: 10.1016/j.cmpb.2018.04.005. [28]A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative Adversarial Networks: An Overview,” IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 53–65, Jan. 2018, doi: 10.1109/MSP.2017.2765202. [29]V. Cheplygina, M. de Bruijne, and J. P. W. Pluim, “Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis,” Apr. 2018, Accessed: Jan. 13, 2021. [Online]. Available: https://arxiv.org/abs/1804.06353v2. [30]V. A, D. N, D. A, and P. E, “Deep Learning for Computer Vision: A Brief Review.,” Comput Intell Neurosci, vol. 2018, pp. 7068349–7068349, Feb. 2018, doi: 10.1155/2018/7068349. [31]K. Wang, C. Gou, Y. Duan, Y. Lin, X. Zheng, and F. Wang, “Generative adversarial networks: introduction and outlook,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4, pp. 588–598, 2017, doi: 10.1109/JAS.2017.7510583. [32]Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, May 2015, doi: 10.1038/nature14539. [33]M. Längkvist, L. Karlsson, and A. Loutfi, “A Review of Unsupervised Feature Learning and Deep Learning for Time-Series Modeling,” Pattern Recognition Letters, vol. 42, Jun. 2014, doi: 10.1016/j.patrec.2014.01.008. [34]S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, Oct. 2010, doi: 10.1109/TKDE.2009.191. </details>
Last changed by  
 
Edwin Arkel Rios
About me: https://arkel23.github.io/
2113

Read more

Vision Transformer training with PyTorch Lightning, HuggingFace and lucidrains's vit-pytorch

What's PyTorch Lightning Fundamentally it's a wrapper for PyTorch training and testing code; it allows for encapsulation and organization of PyTorch models and corresponding associated components and functions, into Lightning Modules. PT Lightning Modules and PT Trainer: PT Lightning Modules help organize all the components of a machine learning system into one single class, including the training, validation and testing, along with optimizers, (and optionally data) into one single class. The PT Trainer takes the Lightning Module and helps abstract the loops and other mostly engineering aspects of the training and evaluation system, while keeping it easily accessible in case there needs to be any modification. Of particular noteworthiness it massively simplifies acceleration using CPU/GPU/TPUs and allows for easy parallel training, usage of mixed-precision, logging, and other (sometimes) cumbersome optimization tricks. Example: Pt Lightning Module For a simple image classification problem using lucidrains's implementation of the Vision Transformer (ViT) and an extended version of HuggingFace's configuration wrapper:

4/21/2021
Getting started with computer vision using deep learning in 2021: resources and comments for beginners and intermediate practitioners

There's many online resources for learning computer vision. The goal of this post is to put some of (what I consider) the best, along with a plan and suggestions, to become an adept practitioner in this field. Motivation The last decade has seen as exponential increase in works that use computer vision (CV), and its endless derivative applications are only starting to get explored. The number of academic publications in the field is booming, and so is the interest from industries and governments, which is shown in an ever growing market all over the world. Computer vision research publications Computer vision market projection

2/9/2021
How to read a paper on 10 minutes (or less): a case study for more efficient reading on 4 papers

Motivation If you're interested in starting research on anything related to artificial intelligence (AI), deep learning (DL) or its many applications, or like I was, a graduate student with not much experience reading papers, navigating through the relevant literature can feel like a daunting task. Every week there's probably dozens, if not hundreds, of new research articles posted to journals, conferences, and repositories like Arxiv. Keeping up with the literature is a challenge by itself, let alone getting started. This post aims to enable new researchers, like me, to familiarize themselves with the paper reading process. Reading a paper Someone once mentioned to me that some researchers intentionally over-complicate their papers to inflate the perception of how much work has been put into it, make it more difficult for others to (potentially) judge negatively, and make themselves appear smarter. I won't comment on this, but reading scientific papers, can definitely be challenging, stressing and time-consuming, specially at first. However, with time, and good technique, one gets used to the process.

2/5/2021
How to start programming with Python: a compilation of resources

Why learn how to program? Learning to write programs stretches your mind, and helps you think better, creates a way of thinking things that I think is helpful in all domains. -Bill Gates，Microsoft. I think everybody in this country should learn how to program a computer...because it teaches you how to think. -Steve Jobs, Apple Inc. In fifteen years we'll be teaching programming just like reading and writing...and wondering why we didn't do it sooner. -Mark Zuckerberg, Facebook.

2/5/2021
Read more from Edwin Arkel Rios
Published on HackMD