---
# System prepended metadata

title: Untitled

---

## DD Intro

The classical wisdom in statistical learning theory predicts the existence of a trade-off between the generalization ability of a machine learning model and its complexity [Hastie OR Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of SL]. If complexity is measured by the number of parameters or training time, deep learning models appear to go against this prediction [See for example [Zhang et al]]. Deep neural networks trained by stochastic gradient descent exhibit non-trivial generalization behaviors as the complexity increases. One such behavior is Double Descent [].

Double Descent was first introduced in [belkin 2018], which study learning and generalization of over-parameterized neural networks. [belkin 2018] show that consistent with lessons from statistical learning theory, the generalization error decreases and then increases as the number of parameters grows; however, generalization error decreases once again beyond a transition threshold. The "cusp" appears at intermediate levels of complexity, specifically when the model can perfectly fit the training data.

In a follow-up paper, [Nakiran et al] showed that double descent is not limited to model size but is also observed as the training time proceeds, specifically in the presence of label noise. The so-called epoch-wise double descent is in apparent contradiction with the classical understanding of overfitting [] where predicts that longer training of a sufficiently large model beyond a certain threshold should result in overfitting. Epoch-wise DD also suggests against early-stopping, perhaps the most widely used regularization method in deep learning [Cite].

Although the term Double Descent was introduced recently, a similar phenomenon had already been studied in several decades-old work, including []. Recent work has provided explanations for this phenomenon in the context of modern machine learning from at least three perspectives: bias-variance trade-off [], matrix theory [], and statistical physics []. Despite significant progress, the current state of the theory does not provide an accurate picture of the generalization dynamics and particularly why epoch-wise double descent occurs.

In this work, we build upon early work of statistical physics and neural networks [] to provide a unifying theory explaining both model-wise and epoch-wise double descent. We study a linear teacher-student framework where the teacher generates a dataset on which the student is trained. We propose simple models for the teacher and student that allow for analytical treatment while exhibiting double descents similar to that of neural networks. In the limit of high-dimensions, we derive closed-form scalar expressions describing the generalization error in terms of the student size and training time.