# Optimisation

Optimization algorithms are the backbone of machine learning models as they enable the modeling process to learn from a given data set. This is the final segment of the mathematics background you require for understanding the modules coming ahead, but please do not let that limit you if you're interested in diving deeper into topics discuss throughout the prior lectures or this one. In case of any doubts specifically pertaining to the following resources, approach Simar or Advika; if unavailable, feel free to reach out to any DSG member.

#### !!Guide to these resources:
There are multiple exhuastive resources given below. Unlike previous lectures, this lecture will cover multiple topics that are lengthy and would ordinarily require a whole lecture for themselves. In light of that, go through all the exhaustive resources because no two touch the same topic. In the non-exhaustive and optional resources, you'll find recurring topics explained in a different manner.

### Optimisation Theory
This theory covers various types of optimization problems, including unconstrained, constrained, and convex optimization, which are essential for building efficient machine learning models.
> Exhaustive Resources
1. [Visually Explained Playlist - 3 videos](https://www.youtube.com/watch?v=AM6BY4btj-M&list=PLqwozWPBo-FuPu4d9pFOobsCF1vDGdY_I)
2. [Blog on Lagrangian Dual](https://q2liu.wordpress.com/2015/01/20/lagrangian-dual/)
3. [Blog on Lagrangian Dual Problem and Weak Duality](https://q2liu.wordpress.com/2015/01/23/lagrangian-dual-problem-and-weak-duality/)
4. [Blog on Strong Duality and Slater’s Theorem](https://q2liu.wordpress.com/2015/01/27/strong-duality-and-slaters-theorem/)

> Non-Exhaustive Resources
1. [Optimisation Lecture Notes](https://www.seas.ucla.edu/~vandenbe/ee236b/lectures/) - 'intro', 'sets', 'functions', 'kkt-scribed', 'problems', 'duality'
2. [Duality: Lagrange Dual Problem and Slater’s Constraint Qualification](https://youtu.be/uhm750aLoh8?si=Ne-IdcncHGHnGj5x) - YouTube video under EE563 Convex Optimization, good for mathematical understanding
3. [Karush-Kuhn-Tucker (KKT) Optimality Conditions](https://www.youtube.com/watch?v=j36nj8TfboY) - YouTube video under EE563 Convex Optimization, good for mathematical understanding

> Optional Resources
1. [Convex Optimisation Book](https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf) - Chapters 1-5

### Gradient Descent and Optimisers
Gradient descent is the most commonly used optimization algorithm for training machine learning models, particularly neural networks. There are also several variations of gradient descent, such as stochastic and mini-batch gradient descent, as well as advanced optimizers like Adam, which adapt the learning rate over time.
> Exhuastive Resources
1. [3Blue1Brown - How Neural Networks Learn](https://www.3blue1brown.com/lessons/gradient-descent#title)
2. [Goodfellow Ch4 - Mathematics behind Gradient Descent](https://www.deeplearningbook.org/contents/numerical.html)
3. [Momentum Demonstration](https://distill.pub/2017/momentum/)
4. [Common Optimisers](https://neptune.ai/blog/deep-learning-optimization-algorithms)

> Non-Exhaustive Resources
1. [Michigan Online Lecture 4](https://youtu.be/YnQJTfbwBM8?si=KzIGX8xcufRva2Qw)

### Optional, Comprehensive Resource
[Optimisation and Mathematical Theory](https://fmin.xyz/)

### Matrix Calculus and Kernel Mathematics
Matrix calculus is crucial for optimization in machine learning models, especially when dealing with high-dimensional data and kernel methods are primarily used in support vector machines and other algorithms for non-linear classification. Most of this portion would have been covered in your MAI/MAN course in the 2nd Semester but it is essential for you to go through these to check if you've missed any key concepts.

> Exhaustive Resource
1. [Matrix Calculus](https://explained.ai/matrix-calculus/)

>Non-Exhaustive Resources
1. [CS229 Linear Algebra Notes](https://cs229.stanford.edu/summer2020/cs229-linalg.pdf)
2. [CS 229 Lectures 1 and 2 (Anand Avati)](https://www.youtube.com/watch?v=KzH1ovd4Ots&list=PLoROMvodv4rNH7qL6-efu_q2_bPuy0adh)

> Optional Resources
1. [Kernel Mathematics](https://www.youtube.com/watch?v=KzH1ovd4Ots&list=PLoROMvodv4rNH7qL6-efu_q2_bPuy0adh)