Time Series and Machine Learning Reading Group

![](https://i.imgur.com/Zh2P8k6.jpg) <style>.markdown-body {max-width: 1000px; }</style> <h1>Time Series and Machine Learning Reading Group (2023 Summer) </h1> *February-June 2023, University of Southampton* :::warning In this semester we will read [a series of papers](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/reading_list.pdf) on deep nerual network (DNN) theories for nonparametric/quantile regression, and state-of-art time series forecasting using DNN. . This reading group is hybrid --- we meet weekly on **Friday 13:30-15:00** (UK time), both at **B54/5001** and via **MS Teams**. Feel free to choose your preferred method to join in. ::: - Coordinator/Contact: [Chao Zheng](https://www.personal.soton.ac.uk/cz1y20/) - [Web link to join our Microsoft Teams Group](https://teams.microsoft.com/l/team/19%3ajuMgMyrXZ2myNKmFU3bfv6_8HK_8YslTUQNJxHwOkx81%40thread.tacv2/conversations?groupId=343691ce-de1d-47cc-be79-f7ebfd6da135&tenantId=4a5378f9-29f4-4d3e-be89-669d03ada9d8) - [Web link to join the reading group sessions via MS Teams](https://teams.microsoft.com/l/meetup-join/19%3ajuMgMyrXZ2myNKmFU3bfv6_8HK_8YslTUQNJxHwOkx81%40thread.tacv2/1677063114466?context=%7b%22Tid%22%3a%224a5378f9-29f4-4d3e-be89-669d03ada9d8%22%2c%22Oid%22%3a%22758cc7d0-3e91-4fd7-aef9-8931e7112dfb%22%7dhttps://teams.microsoft.com/l/meetup-join/19%3ajuMgMyrXZ2myNKmFU3bfv6_8HK_8YslTUQNJxHwOkx81%40thread.tacv2/1677063114466?context=%7b%22Tid%22%3a%224a5378f9-29f4-4d3e-be89-669d03ada9d8%22%2c%22Oid%22%3a%22758cc7d0-3e91-4fd7-aef9-8931e7112dfb%22%7d) Timetable (provisional) --- ++Please check this website regularly for the most up-to-date arrangement.++ | | Date | Topic | Presenter| Discussant | |---| :----: | :--------------: | :-----: | --------- | | 1 | 24 Feb | [Deep Neural Networks for Estimation and Inference (part 1)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week1/Econometrica-2021.pdf) | Chao| Christis | | 2 | 03 Mar | [Deep Neural Networks for Estimation and Inference (part 2)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week1/Econometrica-2021.pdf) | Baiyu | Shubin | | 3 | 10 Mar | [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week3/TFT.pdf)| Huan | Chao | 4 | 17 Mar | [Error Bounds for Approximations with Deep ReLU Networks (part1)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week4/Yarotsky_2017.pdf) | Shubin | Baiyu | | 5 | 24 Mar | [Error Bounds for Approximations with Deep ReLU Networks (part2)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week5/Yarotsky_2017.pdf) | Chao | Zudi | | 6 | 31 Mar | [ Optimal Approximation of Continuous Functions by Very Deep ReLU Networks](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week6/yarotsky18a.pdf) | Shubin | Baiyu | | | | Easter Break | | | | 7 | 14 Apr | [Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition (part 1)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week10/2107.04907.pdf) | Baiyu | Chao | | 8 | 21 Apr | [N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week8/N-BEATS.pdf) | Huan | Chao | | 9 | 28 Apr | [Combining Counterfactual Outcomes and ARIMA Models for Policy Evaluation](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week9/utac024.pdf) | Yunlong | Zudi | | 10 | 05 Mar | [Causal Inference Using Potential Outcomes](https://www.dropbox.com/s/zlkkjbugylw1aho/Causal%20Inference%20Using%20Potential%20Outcomes_Rubin2005.pdf?dl=0) | Yan | Zudi | 11 | 12 May | [Robust inference on average treatment effects with possibly more covariates than observations](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week11/Farrell_2015.pdf) | Christis | Zudi | | 12 | 19 May | [A Penalized Synthetic Control Estimator for Disaggregated Data](https://www.dropbox.com/s/atlo5y5bw1wc2pl/A%20Penalized%20Synthetic%20Control%20Estimator%20for%20Disaggregated%20Data.pdf?dl=0) | Yan | Zudi | | 13 | 26 May | [On the rate of convergence of a neural network regression estimate learned by gradient descent](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week13/Brau_2019.pdf) | Christis | Chao | | 14 | 02 Jun | [Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition (part 2)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week10/2107.04907.pdf) | Baiyu | Chao | | 15 | 09 Jun | [Asymptotic Properties of Neural Network Sieve Estimators](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week15/xiaoxi_2023.pdf) | Shubin | Chao | | 16 | 16 Jun | [The Augmented Synthetic Control Method](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week16/Augmented_Synthetic_Control_Method.pdf) | Yan | Zudi  Materials --- - Week 1. Deep Neural Networks for Estimation and Inference (part 1) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Meeting%20in%20_General_-20230224_133044-Meeting%20Recording.mp4?web=1) + [Nearly-Tight VC-Dimension Bounds for Piecewise Linear Neural Networks](https://jmlr.org/papers/volume20/17-612/17-612.pdf) by Peter Bartlett, Nick Harvey, Christopher Liaw, Abbas Mehrabian (2019) + [Learning Without Concentration](http://proceedings.mlr.press/v35/mendelson14.pdf) by Shahar Mendelson (2014) + [Rademacher Processes and Bounding the Risk of Function Learning](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week1/rad1.pdf) by V. Koltchinskii. D. Panchenko (2000) - Week 2. Deep Neural Networks for Estimation and Inference (part 2) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230303_133009-Meeting%20Recording.mp4?web=1) + [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://www.mdpi.com/2227-7390/7/10/992) by Boris Hanin (2019) + [Deep vs. Shallow Networks : An Approximation Theory Perspective](https://www.worldscientific.com/doi/10.1142/S0219530516400042) by Hrushikesh Mhaskar and Tomaso Poggio (2016) + [Double/Debiased Machine Learning for Treatment and Structural Parameter](https://economics.mit.edu/sites/default/files/2022-08/2017.06%20Double%20Debiased%20Machine%20Learning%20for%20Treat.pdf) by Victor Chernozhukov et al. (2018) - Week 3. Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230310_133603-Meeting%20Recording.mp4?web=1) + [Language modeling with gated convolutional networks](http://proceedings.mlr.press/v70/dauphin17a/dauphin17a.pdf) by Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2017). + [ Attention is all you need](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) by Vaswani, A., et al. (2017). - Week 4. Error Bounds for Approximations with Deep ReLU Networks (part 1) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Meeting%20in%20_General_-20230317_133010-Meeting%20Recording.mp4?web=1) - Week 5. Error Bounds for Approximations with Deep ReLU Networks (part 2) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Meeting%20in%20_General_-20230324_133300-Meeting%20Recording.mp4?web=1) + [Neural network learning: Theoretical foundations](https://www.cambridge.org/us/academic/subjects/computer-science/pattern-recognition-and-machine-learning/neural-network-learning-theoretical-foundations?format=PB) by Martin Anthony and Peter Bartlett. + [Why Deep Neural Networks?](https://arxiv.org/pdf/1610.04161.pdf) by Liang, S. and Srikant, R. (2017). + [Learning Real and Boolean Functions: When is Deep Better Than Shallow.](https://core.ac.uk/download/pdf/78068834.pdf) by Mhaskar, H., Liao, Q., and Poggio, T. (2017). + [Deep vs. Shallow Networks: An Approximation Theory Perspective](https://www.worldscientific.com/doi/10.1142/S0219530516400042) by Mhaskar, H. and Poggio, T. (2016) - Week 6. Optimal Approximation of Continuous Functions by Very Deep ReLU Networks + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230331_133358-Meeting%20Recording.mp4?web=1) - Week 7. Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition (part 1) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Meeting%20in%20_General_-20230414_133702-%E4%BC%9A%E8%AE%AE%E8%AE%B0%E5%BD%95.mp4?web=1) + [Handout](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week7/handout7.pdf) by Baiyu + [Nonlinear Approximation via Compositions](https://www.sciencedirect.com/science/article/pii/S0893608019301996) by Zuowei Shen, Haizhao Yang and Shijun Zhang (2019). + [Deep Network Approximation Characterized by Number of Neurons](https://arxiv.org/abs/1906.05497) by Zuowei Shen, Haizhao Yang and Shijun Zhang (2021). + [Nearly-Tight VC-Dimension Bounds for Piecewise Linear Neural Networks](https://jmlr.org/papers/volume20/17-612/17-612.pdf) by Peter Bartlett, Nick Harvey, Christopher Liaw, Abbas Mehrabian (2019). - Week 8.N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230421_133256-Meeting%20Recording.mp4?web=1) + [A Hybrid Method of Exponential Smoothing and Recurrent Neural Networks for Time Series Forecasting](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week8/Smyl_2020.pdf) by Slawek Smyl (2020) - Week 9. Combining Counterfactual Outcomes and ARIMA Models for Policy Evaluation + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230428_132804-Meeting%20Recording.mp4?web=1) + [Estimating Causal Effects in the Presence of Partial Interference Using Multivariate Bayesian Structural Time Series Models](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week9/21-AOAS1498.pdf) by Fiammetta Menchetti and Iavor Bojinov (2022) - Week 10. Causal Inference Using Potential Outcomes + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230505_133018-Meeting%20Recording.mp4?web=1) + [Handout](https://www.dropbox.com/s/wfqq4jpb4gs9gbt/Causal%20Inference%20Using%20Potential%20Outcomes_Handout.pdf?dl=0) by Yan + [Bayesian Inference for Causal Effects](https://www.dropbox.com/s/gcv7vja4c20hzft/Bayesian%20Inference%20for%20Causal%20Effects.pdf?dl=0) by Rubin(1978) - Week 11. Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230512_132735-Meeting%20Recording.mp4?web=1) + [Doubly Debiased LASSO: High-Dimensional Inference Under Hidden Confounding](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week11/DOUBLY_DEBIASED_LASSO.pdf) by Zijian Guo, Domagoj Cevid and Peter Buhlmann (2022) + [Choosing Exogeneity Assumptions in Potential Outcome Models](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week11/Matthew_2022.pdf) by Matthew A. Masten and Alexandre Poirier (2022) - Week 12. A Penalized Synthetic Control Estimator for Disaggregated Data + [Handout](https://www.dropbox.com/s/wcluxv4noijuah7/Handout.pdf?dl=0) by Yan + [Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program ](https://www.dropbox.com/s/w520pc40ilmm6nl/Synthetic%20Control%20Methods%20for%20Comparative%20Case%20Studies%20Estimating%20the%20Effect%20of%20California%20s%20Tobacco%20Control%20Program.pdf?dl=0) by Alberto Abadie, Alexis Diamond & Jens Hainmueller (2012) - Week 13. On The Rate of Convergence of A Neural Network Regression Estimate Learned by Gradient Descent + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230526_132846-Meeting%20Recording.mp4?web=1) + [Asymptotic and Finite-sample Properties of Estimators Based on Stochastic Gradients](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week13/panos_2017.pdf) by Panos Toulis and Edoardo M. Airoldi (2017) + [A Sieve Stochastic Gradient Descent Estimator for Online Nonparametric Regression in Sobolev Ellipsoids](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week13/tianyu_2022.pdf) by Tianyu Zhang and Noah Simon (2022) + [Statistical Inference for Model Parameters in Stochastic Gradient Descent](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week13/xi_2020.pdf) by Xi Chen, Jason D. Lee, Xin T. Tong and Yichen Zhang (2020) - Week 14. Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition (part 2) + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230602_133102-Meeting%20Recording.mp4?web=1) + [Handout](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week14/handout14.pdf) by Baiyu - Week 15. Asymptotic Properties of Neural Network Sieve Estimators + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230609_133030-Meeting%20Recording.mp4?web=1) + [Handout](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week15/shubin.pdf) by Shubin - Week 16. The Augmented Synthetic Control Method + [Video recording](https://sotonac.sharepoint.com/teams/EmpiricalProcessesReadingGroup2022/Shared%20Documents/General/Recordings/Machine%20Learning%20and%20Time%20Series%20Reading%20Group%20S2%202023%20(Weekly%20Meetings)-20230616_132926-Meeting%20Recording.mp4?web=1) + [Handout](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-2023/week16/Handout.pdf) by Yan  Group Members --- *Please contact Chao if want to join in the group* - Prof. Zudi Lu (<z.lu@soton.ac.uk>) - Dr. Chao Zheng (<chao.zheng@soton.ac.uk>) - Dr. Christis Katsouris (<C.Katsouris@exeter.ac.uk>) - Dr. Maurizio Daniele (<daniele@kof.ethz.ch>) - Dr. Huan Yu (<Huan.Yu@soton.ac.uk>) - Dr. Libo Li (<libo.li@soton.ac.uk>) - Dr. Fangsheng Ge (<f.ge@soton.ac.uk>) - Dr. Jiangtao Wang(<wjtao1983@163.com>) - Dr. Hongyi Peng(<penghyi@163.com>) - Lulu Wang (<lulu.wang@soton.ac.uk>) - Shubin Wu (<sw6y19@soton.ac.uk>) - Baiyu Wang (<bw1g21@soton.ac.uk>) - Yan Zhang (<yz4u18@soton.ac.uk>) - Yunlong Wang (<yw33u22@soton.ac.uk>) - Dong Qiu (<dong.qiu@warwick.ac.uk>)  Supplementary References --- 1. **Quantile Regression.** Roger Koenker, *Cambridge Uni Press*, 2005. 3. **Introduction to Nonparametric Estimation.** Alexandre Tsybakov, *Springer*, 2009. 4. **A Distribution-Free Theory of Nonparametric Regression.** László Györfi, Michael Kohler, Adam Krzyżak, and Harro Walk, *Springer*, 2002. Roles of Presenter and Discussant --- If it is your first time attend a reading group, you might find the [reading group tips by Lester Mackey and Percy Liang](https://docs.google.com/document/d/1KqtfhKbePLfSsJ-_hR6kBditC0uMSo0BXAZP5Mm_hPw/edit) helpful. Every time we will have one people (<font color="#3ABF78">presenter</font>) present the main contents and another people (<font color="#7733FF">discussant</font>) raise questions and lead the discussion. - **<font color="#3ABF78">As a presenter</font>**: you should have an in-depth reading and develop a solid understanding of all the details in the assigned topic. You should prepare well, and make sure you deliver a logically clear and technically accessible presentation. In short words, it is your job to have everyone in the meeting understand the main ideas of the reading. - **<font color="#7733FF">As a discussant</font>**: you should be more familiar with the content than if you were simply in the group. You don’t need to know everything. You can pause the presentation, ask questions (to the presenter or to the audience), and facilitate discussions. It is your job to help the presenter to have everyone (yourself included!) in the meeting understand the main ideas of the reading and having learned something. Before each session, although not compulsory I would recommend following amount of time spent on reading: - <font color="#3ABF78">Presenter</font>: > 10 hours; - <font color="#7733FF">Discussant</font>: 5 hours; - <font color="#FFBD33">General audience</font>: 2 hours. > If you encounter any problem during your reading, feel free to discuss with me or other staff members. Past Reading Groups: --- - [Empirical Process Theory and Applications Reading Group (March-July 2022)](http://www.personal.soton.ac.uk/cz1y20/Reading_Group/ep-group.html) - [Time Series and Machine Learning Reading Group (Oct 2022-Feb 2023)](https://www.personal.soton.ac.uk/cz1y20/Reading_Group/mlts-group-2022.html)  --- <h6 style="text-align: center;"><font size="-2">Webpage maintained by Chao Zheng. Last updated on 18/11/2022</font></h6>