# Linear Algebra and Optimisation

Following your lectures and assignments on Probability and Statistics, we move on to the final segment of the mathematics background required for your ML Journey. These resources will broadly cover the areas of linear algebra and optimisation theory. As mentioned previously, some of the resources are not disjoint and cover essentially the same material. These redundant resources have been purposefully added so that you can figure out what works best for you. Please refer to individual sections (in the proper order) to read (or watch) the relevant resources. For any doubts, do not hesitate in contacting a DSG member (these resources were specifically prepared by Sukrit and Aakash). While the optional resources are, as the name suggests, optional, we still encourage you to go through them at least once. We also suggest you go over any material provided in your mathematics' courses regarding linear algebra if needed. Also, sometimes the optional material may be better for your understanding as opposed to the compulsory resources, so do go through them for sure.

### Fundamentals of Linear Algebra
This section covers linear algebra topics such as scalars, vectors, matrices and tensors. Further, it includes terminol as linear (in)dependence, span, basis, norm and vector spaces. Also included are topics like linear transformations and matrices
* [Introduction to Linear Algebra](https://math.mit.edu/~gs/linearalgebra/ila6/indexila6.html) - Chapters 1, 2, 3 (Compulsory) and 8 (Optional)
* [3Blue1Brown Linear Algebra](https://www.3blue1brown.com/topics/linear-algebra) - Video Chapter 1-5
* [Deep Learning Chapter 2](http://imlab.postech.ac.kr/dkim/class/csed514_2019s/DeepLearningBook.pdf) - Sections 2.1-2.6
* [Gilbert Strang Linear Algebra](https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8) - Optional, cover it alongside theory if needed
* [CS 229 (Anand Avati) Lecture 1](https://www.youtube.com/watch?v=KzH1ovd4Ots&list=PLoROMvodv4rNH7qL6-efu_q2_bPuy0adh&index=1) - Optional

### Eigenvectors, Eigenvalues, SVD and the Pseudoinverse
This includes topics such as eigenvalues, eigenvectors, eigendecomposition and singular value decomposition. SVD is commonly used in PCA, so that is also covered in this section itself. Finally, using this, we move on to the concept of the Moore-Penrose pseudoinverse
* [Introduction to Linear Algebra](https://math.mit.edu/~gs/linearalgebra/ila6/indexila6.html) - Chapters 4, 6 and 7
* [3Blue1Brown Linear Algebra](https://www.3blue1brown.com/topics/linear-algebra) - Video Chapter 14 (Compulsory) and 16 (Optional)
* [Intuitive understanding of SVD](https://gregorygundersen.com/blog/2018/12/10/svd/)
* [Deep Learning Chapter 2](http://imlab.postech.ac.kr/dkim/class/csed514_2019s/DeepLearningBook.pdf) - Sections 2.7-2.9, 2.12
* [SVD Applications in Pseudoinverses](https://www.cis.upenn.edu/~cis5150/cis515-12-sl13.pdf) - Sections 13.1 and 13.3 (Optional)

### Optimisation Theory
* [Optimisation Lecture Notes](https://www.seas.ucla.edu/~vandenbe/ee236b/lectures/) - 'intro', 'sets', 'functions', 'kkt-scribed', 'problems', 'duality'
* [Blog on KKT Conditions and Lagrange Multipliers](https://towardsdatascience.com/lagrange-multipliers-kkt-conditions-duality-intuitively-explained-de09f645b068)
* [Blog 1](https://q2liu.wordpress.com/2015/01/23/lagrangian-dual-problem-and-weak-duality/) - Lagrangian Dual and Weak Duality, Optional but encouraged
* [Blog 2](https://q2liu.wordpress.com/2015/01/27/strong-duality-and-slaters-theorem/) - Strong Duality and Slater's Theorem, Optional but Encouraged
* [Convex Optimisation Book](https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf) - Chapters 1-5, Optional

### Gradient Descent and Optimisers
* [Goodfellow Chapter 4](https://www.deeplearningbook.org/contents/numerical.html)
* [Common Optimisers](https://artemoppermann.com/optimization-in-deep-learning-adagrad-rmsprop-adam/)
* [Momentum Demonstration](https://distill.pub/2017/momentum/)
* [Michigan Online Lecture 4](https://www.youtube.com/playlist?list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r) - Optional

### Matrix Calculus and Kernel Mathematics
* [Matrix Calculus](https://explained.ai/matrix-calculus/)
* [CS229 Linear Algebra Notes](https://cs229.stanford.edu/summer2020/cs229-linalg.pdf) - Optional
* [CS 229 Lectures 1 and 2 (Anand Avati)](https://www.youtube.com/watch?v=KzH1ovd4Ots&list=PLoROMvodv4rNH7qL6-efu_q2_bPuy0adh) - Optional
* [Kernel Mathematics](https://people.eecs.berkeley.edu/~jordan/kernels/0521813972c03_p47-84.pdf) - Optional, only after all resources have been covered