# Notes on Deep Equilibrium Models #### Author: [Sharath Chandra](https://sharathraparthy.github.io/) ## [Paper Link](https://arxiv.org/pdf/1909.01377.pdf) The present day deep learning architectures have this notion of explicit layers which model the relationship between inputs and outputs with the help of a well defined computational graph. For example, if we consider a one hidden layer feed forward network with sigmoid non-linearity, we can define the explicit hidden layer as a layer which performs the specified computation to get calculate the output $$ z_1 = \sigma(W_1x + b_1) $$ For deep networks these explicit layers are stacked together and are trained using backpropagation. Yet one needs to store the intermediate values to carry the backpropagation which puts a heavy burden on the memory considering the huge capacity modern deep learning architectures. This problem has driven a new research direction which aimed at being memory efficient without compromising on the powerful representational capacity that present deep architectures offer. Instead of specifying the underlying computational graph which a layer should perform, one can think of other class of layers where we can just specify the joint conditions that layer should satisfy to get the output. **Implicit layers**, unlike explicit layers, define some conditions that the inputs and outputs satisfy. To understand this consider a weight-tied neural network applied to an input $x$. A weight-tied network is that network which has shared parameter space and these are shown to work well in practice. So, the output at any hidden layer $L_i$ can be written as $z_i = \sigma(W z_{i - 1} + x)$ where $W$ is the common weight matrix for all the layers, $z_{i-1}$ is the output of layer $L_{i-1}$ and $\sigma$ is the nonlinear activation function. One can view this as a discrete time dynamical system with the hidden state $z_i$ evolving (discretely) through time. Assuming that there is a fixed point to such dynamical system, one can defined an implicit layer is the solution to the following nonlinear system $$ z^\star = \sigma(Wz^\star + x) $$ In fact this can be extended to any nonlinear function $f$. Given this structure of implicit layer ($z^\star = f(z^\star, x, \theta)$ where $\theta$ is the parameters of the implicit layer), our aim is to calculate the fixed point $z^\star$ by using any fixed point iteration solver. Since we are interested in memory efficiency, we need to have a mechanism to backpropagate through the layers without storing any intermediate values. This can be achieved by using Implicit function theorem \cite{implicit} which directly gives us the total derivative of the loss function $l(\theta)$ which is given by \begin{equation} \partial l(\theta) = \partial_{z^\star}(z^\star, y)(I - \partial_{z^\star}f(z^\star, x, \theta))\partial_x f(z^\star, x, \theta) \end{equation} Deep equilibrium models leverages this idea of implicit layers to more general settings where $f$ is a ResNet block, transformer block etc., and uses implicit function theorem to carry the backpropagation. In the forward pass, DEQ computes the equilibrium point $z^\star$ using a fixed point iteration solver like Anderson acceleration and then compute the loss function which depends on this fixed point. As the gradients we compute are invariant to the fixed point solvers, one can use any solver to calculate the fixed points. During the backpropagation, DEQ calculates the gradients using the equation above. One can view DEQ models as a single layer neural network which can represent any feed forward neural network. And furthermore since we are using a weight-tied neural network there is no exponential blow up of parameters in single layer universal function approximation theorems. One more view of these networks is to consider each the fixed point approximation step as an instantiation of a layer resulting in arbitrary depth (sometimes infinite depth) networks which are memory efficient. <!-- The paper considers one instantiation of the model $f$ which considers a layer as a "cell" rather than the single true layer. And the experiments are shown to have good performance as compared with baselines which considers explicit layers -->
Last changed by  
 
SharathRaparthy
I am a Masters student at Mila working on continual reinforcement learning.
396

Read more

When do curricula work?

Author: Sharath Paper Link Overview Curriculum learning is inspired by the way human learns, where the examples are shown in the increasing order of the difficulty. More sepcifically te network is exposed to the easier examples in the early stages of training and then gradually to the tougher ones. This paper studies the benifits of showing this sequential ordered eamples to the network and comments about when it works and when it doesn't. Contributions: This paper introduces a phenomenon called implicit curricula. One of the claims they make is that the the ordered learning (curriculum, anti-curriculum and random) almost performs same in the standard settings. Curricula is benificial when there is a limited time budget and in noisy regime

7/4/2021
Radial Bayesian Neural Networks

Author: Sharath Overview This paper discusses how the BNNs behave as we scale to the large models. This paper discusses the "soap-bubble" issue in case of high dimensional probability spaces and how MFVI suffers from this. As a way to tackle this issue, the authors propose a new variational posterior approximation in hyperspherical coordinate system and show that this overcomes the soap-bubble issue when we sample from this posterior. The geometry of high dimensional spaces One of the properties of high dimensional spaces is that there is much more volume outside any given neighbourhood than inside of it. Betacount et al explained this behaviour visually with two intuitive examples. For first example let us consider partitioning our parameter space in equal rectangular intervals as shown below. We can see that as we increase the dimensions the distribution of volume around the center decreases. This becomes almost negligible as compared to the its neighbourhood in high dimensional cases where $D$ is very large. We can observe a similar behaviour if we consider spherical view of parameter space, where the exterior volume grows even larger than the interior in high dimensional spaces as shown in the figure. . How this intuition of volumes in high dimensions explain soap bubble phenomenon?

7/4/2021
Notes on convexity and gradient descent

Author: Sharath What is a convex function? Let's try to define a convex function formally and geometrically. Formally, a function $f$ is said to be a convex function if the domain of $f$ is a convex set and if it satisfies the following $\forall x \ \text{and} \ y \in \text{dom} f$; \begin{equation} f(\theta x + (1 - \theta)y) \leq \theta f(x) + (1 - \theta)f(y) \end{equation} Geometrically it means that the value of a function at the convex combination of two points of the function always lies below the convex combination of the values at the corresponding points. It means that if we draw a line at any two points $(x, y) \in \text{dom} f$, then this line/chord always lies above the function $f$.

7/4/2021
Notes on equilibrium and stability of the dynamical systems.

Author: Sharath What is a dynamical system? It is any system that evolves and changes through time governed by a set of rules. Using dynamical systems we can study the long term behavior of an evolving system. Formally, it is a triplet $(X, T, \phi)$ where $X$ denotes the state space, $T$ denotes the time space and $\phi: X \times T \rightarrow X$ is the flow (this is the rule that governs the evolution). There are few properties of flow: $\phi(X, 0) = X$ Principle of compositionality: $\phi(\phi(x, t), s) = \phi(x, t+s)$

7/4/2021
Read more from SharathRaparthy
Published on HackMD