![](https://i.imgur.com/sZdQY0g.png) # Mathematics of Machine Learning - Summer School ###### tags: `academic-training` `summer-school` `learning-at-the-turing` ## Useful info Welcome to the Summer School. This working and collaborative document comprises everything you need to know before the summer school start, and during the week. ::::info **Dates** 28 June - 2 July 2021 **Time** 9.30 - 17.30 UK time zone (BST) **Instructors**: Patrick Rebeschini, Tomas Vaskevicius **Sponsors**: The Alan Turing Institute, Google **Contact**: Mishka Nemes (mnemes@turing.ac.uk) :::: **Schedule** - same each day, all in UK timezone (UTC/BST) | Time | Agenda item | Length | Mode of delivery | |------|--------|--------|------------------| |9.30-10.30 |lecture 1 |1 hour | Everyone | |10.30-12.30 |research practical 1 | 2 hours | Groups of 3| |12.30-13.10 | lunch break |40 minutes | Groups of 3/ Away from desk| |13.10-13.50| research practical 1 follow-up and Q&A |40 minutes| Everyone | |13.50-14.50| lecture 2 |1 hour| Everyone| |14.50-16.50| research practical 2 |2 hours| Groups of 3 | |16.50-17.30| research practical 2 follow-up and Q&A | 40 minutes| Everyone| Every day: 2 hours lecture 4 hours research practicals 1.20 hours practicals follow-up and Q&A 40 lunch break **Joining instructions on Zoom** :::info [Download link for adding the iCalendar (.ics)](https://turing-uk.zoom.us/meeting/tJEvdu2rpjkuEtF30Hg8yxjTchji27U5hbQb/ics?icsToken=98tyKuCtrDouGdyXtBGCRowMAIj4LOnxiHpbj7d7z0_NBHhKegnCNepIO-AoJY3Y) files to your calendar system. [Zoom Meeting link](https://turing-uk.zoom.us/j/95214382485?pwd=Rld2NFN0eVpQVDM4ZGsrSlFGMUF3Zz09) Meeting ID: 952 1438 2485 Passcode: 422157 [Find your local number link](https://turing-uk.zoom.us/u/agp6jj02C) ::: **Zoom etiquette** - :microphone: make sure you have the mic off at all times, unless you are speaking - :raised_hand: to ask questions during the sessions with __Everyone__ please raise your hand by pressing *Reactions -> Raise hand* ![](https://i.imgur.com/O4rVjMv.png) - :question: to ask for support from the instructors during the breakout room sessions __Groups of 3__ please click the button *Ask for help* and one of the instructors will join your breakout room ![](https://i.imgur.com/roxfKEc.png) - :camera: try to keep the camera on as much as possbile, to allow for *enhanced* remote interaction with the instructors and the other peers - :memo: use the Zoom chat if you prefer to add any thoughts in an ongoing discussion - :black_nib: if you require/ would like to access typed captions during the summer school, please click *Live Transcript* - :wrench: make sure to install the latest version of Zoom before the summer school - :computer: we recommend for best experience to join the Zoom session from a laptop/ PC as the phone experience will be lacking certain elements - :exclamation: to note: we are not recording any session so that we can allow for more interaction between the students and the instructors **How to edit this document** Everyone should be able to edit this live document by clicking `edit` if in view mode, and the `double pane` or the `pencil` button (on the top left corner) if in edit mode already. ## :pushpin: Summer school materials and useful links :floppy_disk: [Application website for the summer school](https://www.turingevents.co.uk/turingevents/frontend/reg/thome.csp?pageID=19480&eventID=60&CSPCHD=000001000000iUS8lHBmR6AfxWuzWejmXsQb0pXDC4erYIE70E) :book: Useful materials to go through in preparation for the summer school - [Algorithmic foundation of learning](https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.stats.ox.ac.uk%2F~rebeschi%2Fteaching%2FAFoL%2F20%2Findex.html&data=04%7C01%7Cmnemes%40TURING.ac.uk%7C67729de4b5a4470fd1bd08d934987768%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C637598651714330400%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vF3xJyC5ia4AfmAr4Ra5GoeWU4bu5lAQQaWN8chn%2FjQ%3D&reserved=0) :bookmark: Lectures notes/ handouts and notebooks for the practical sessions can be found on the [GitHub repository](https://github.com/alan-turing-institute/mathematics-of-ml-course) :dart: [Feedback form throughout the summer school](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fforms%2Fd%2F1kf3QihO23f54JdXCLkc9CBgdhVWLp74KlL-d_AqSCzE&data=04%7C01%7Cmnemes%40TURING.ac.uk%7C516cc3e34bab4e6e8e6808d93a09e7c5%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C637604637074107853%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=412ZZKuOuRMxriuIcBUgAarmarhmHknnRzCu5ccRP70%3D&reserved=0) :performing_arts: [Feedback form following the summer school](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fforms.office.com%2FPages%2FResponsePage.aspx%3Fid%3Dp_SVQ1XklU-Knx-672OE-QsFMYnhUOZIq6vkLRtyNQFUMzgyOFZCR0tBVzhXMzhOTjQ4SVNRTDZQQS4u&data=04%7C01%7Ctraining%40turing.ac.uk%7C00c985834b9c4ca3dacc08d93d30aa37%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C637608101480186689%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=p%2BiL8OL50OOtHUOVT6OEh4oU45uYjH155Pm7ZulHuZc%3D&reserved=0) :left_speech_bubble: [Slack workspace link](https://join.slack.com/t/turing-vay8057/shared_invite/zt-safc75ya-tFfTxmMYDcmpK6cwIsx2UQ) :key: [Markdown cheatsheat](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) ## :dart: About the practicals You are allocated in groups of 3 people which will be the same throughout the practical sessions as following: - day 1-2 - day 3-4 - day 5 The group allocation below will only apply in day 5 of the summer school. :exclamation: **The instructors will open the breakout rooms and allow you to choose your own room. Please use the assignment below to pick the relevant breakout room. If you don't get a pop-up message or you had to rejoin the meeting, click Breakout Room and then click Join as you hover over the number of participants in the relevant room.** In the first session with each breakout room formation (Monday morning, Wednesday morning, Friday morning), please use the icebreaker questions below to get to know each other.. and feel free to go beyond these questions. > **Icebreaker questions** > > Introduce yourself - what organisation/ university you're from, what is your field of work > > If you could live in any country, which country would you pick? > > :notebook: Please note all practical sessions will be carried out through Google Colab notebooks and you don't need to install any specialist software. #### Q&A Q: Exercise 3, it is not very clear to me what the bolded emphasis is trying to make a point --- from the maths it seems that CLT kicks in with a $n^{-\frac{1}{2}}$ rather than saying "non-asymptotic bounds". So perhaps there is another perspective of this equation which I have not been able to appreciate. A: It kicks in with a $n^{-1/2}$ rate, but for what sizes of $n$? Suppose you observe 100 samples $Z_{1}, \dots, Z_{100} \sim P$ from some distribution $P$. How do you know if the CLT has already "kicked in" for this particular distribution $P$? [Q1] PY \& HZ: Do we expect the training to converge to the Neural Network as we designed Exercise 5.1? E.g. in some cases if we mamanged to initialise the weights that are close to optimal, it would be intuitive that within small steps, we get to the theoretical optimal. [A] Tomas: as we have seen in Exerise 2, the training does not converge to the configuration of weights shown in Exercise 5. The idea of Exercise 5 is to show that a configuration of weights realizing a given parity function **exists**, so that the observed failure in our experiments (of Exercise 2) for training neural networks is **not** due to the fact that our class of functions is not expressive enough (it is just that gradient descent cannot find the right configuration of weights!) Regarding the question on initialization: there is no way to initialize close to the true parity function without knowing the support of the true parity function. [Q2] PY: Could you please elaborate "procedures that do not inspect the individual data points but are instead based on some aggregate statistics of the observed data. Gradient descent can be shown to fall into this category"? [A] Ideally, you want to minimize the population risk $r$, thus in the perfect world you would use gradients of the population risk $\nabla r$. However, we do not know the underlying data distribution, so we approximate the population gradient by empirical gradients $\nabla R \approx \nabla r$. This approximation is computed by taking an average (or as written in the practical session text "aggregate statistic") of the full dataset by averaging the gradients at individual training points. The computed $\nabla R$ does not "reveal" too much information about any individual training point. Thus, $\nabla R$ can be seen as an output of a *statistical query* "output some vector $g$ such that $\|g - \nabla r\|$ is not too large". See, for example, the following paper for many useful references to works showing that "usual" machine learning algorithms are statistical query algorithms. V. Feldman, E. Grigorescu, L. Reyzin, S. S. Vempala, and Y. Xiao. Statistical algorithms and a lower bound for detecting planted cliques. Journal of the ACM (JACM), 64(2): 1–37, 2017. [Q3] PY \& HZ: At the end of Exercise 5, it is unclear between the link of the inequality and the conclusion stating `'directions independent of the true signal.'` [A] If the variance of the gradient is exponentially small, random realizations of the gradient are exponentially close to its expected value. The expected value is computed with a random draw of a target parity function; hence, this expected value is independent of the problem that you are trying to solve. [Q4, maybe to Patrick] PY: Any literature on penalised likelihood estimation (pMLE)? What we have covered today are penalised least squares, so just wonder if there are any past research on pMLE, which seems to be a relevant avenue on top. [...] ## Improvements and suggestions space ### Is there anything we could do to improve your learning experience? Let us know below! * [suggestion 1] * [suggestion 2] * [...] ## Code of conduct We are very keen to ensure that you both benefit from training and feel part of the community, and if you think you experienced any harassment, discomfort, or anything else caused by the instructors, your peers, the course materials or the training team, please get in touch by email at training@turing.ac.uk. There is no formal process of reporting incidents and we will assess each case individually. We will also ensure confidentiality where it's appropriate. As a general guideline, we try to abide to the Turing Way Code of Conduct principles, which you can read [here](https://the-turing-way.netlify.app/community-handbook/coc/coc-details.html). We can only improve and ensure you are having a good experience if you let us know what might have gone wrong or just felt exclusionary. We'd be pleased not receive any complaints, but as you well know, that doesn't always mean everything is perfect - so let's work together to make the Learning at the Turing space is as inclusive as possible.