# Problem 1
We make the induction assumption for that for $m \to \infty$:
$$ f^{(h)} \sim GP(0, \Sigma^{(h-1)})$$

First, we for the mean have :

$$
\mathbb{E}\left[[f^{(h+1)}(x)]_i
\mid f^{(h)}(x) \right]
= \mathbb{E}\left[\sum_{j=1}^mW^{(h+1)}_{i,j}[g^{(h)}(x)]_i
\mid f^{(h)}(x) \right]
= \sum_{j=1}^m\mathbb{E}\left[W^{(h+1)}_{i,j}\right][g^{(h)}(x)]_i
= 0
$$

Now, for the covariance matrix, we have :
$$
\Sigma^{(h)}=\lim_{m\to \infty} \frac{2}{m}\sum_{i=1}^m\sigma(u_i)\sigma(v_i)\qquad [u;v]\sim\mathcal{N}(0,\Lambda^{(h)})
$$

$$
\begin{align}
\mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i[f^{(h+1)}(y)]_i
\mid f^{(h)}(x) \right]
& =\mathbb{E}\left[\sum_{j=1}^m\sum_{k=1}^m W_{i,j}^{(h+1)}W_{i,k}^{(h+1)}[g^{(h)}(x)]_j[g^{(h)}(y)]_k
\mid f^{(h)}(x) \right] \\
& = \mathbb{E}\left[\sum_{j=1}^m \left(W_{i,j}^{(h+1)}\right)^2 [g^{(h)}(x)]_j[g^{(h)}(y)]_j
\mid f^{(h)}(x) \right] \\
& = \sum_{j=1}^m [g^{(h)}(x)]_j[g^{(h)}(y)]_j\\
& = \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j)\sigma([f^{(h)}(y)]_j)
\end{align}
$$
Hence with $m\to\infty$ we get:
$$
\begin{aligned}
Cov([f^{(h+1)}(x)]_i,[f^{(h+1)}(y)]_i\mid f^{(h)}(x) ) &= \mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i[f^{(h+1)}(y)]_i
\mid f^{(h)}(x) \right]
\\& - {\mathbb{E}_{x,y}\left[[f^{(h+1)}(y)]_i
\mid f^{(h)}(x) \right]}\\&- \mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i
\mid f^{(h)}(x) \right]\\
&=\lim_{m\to \infty} \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j)\sigma([f^{(h)}(y)]_j)\\
&=2\mathbb{E}[\sigma(u)\sigma(v)] = \Sigma^{(h)}(x,y)\\
\end{aligned}
$$
Where $u,v$ are drawn from:
$$
\begin{bmatrix}
[f^{(h)}(x)]_j\\
[f^{(h)}(y)]_j
\end{bmatrix}\sim \mathcal{N}\left(
\begin{bmatrix}
0\\
0
\end{bmatrix},
\begin{bmatrix}
\Sigma^{(h-1)}(x,x) & \Sigma^{(h-1)}(x,y)\\
\Sigma^{(h-1)}(y,x) & \Sigma^{(h-1)}(y,y)
\end{bmatrix}
\right) = \mathcal{N}(0,\Lambda^{(h)})
$$
Thanks to the indjuction assumption. Since this is all gaussian then also $[f^{(h+1)}(x)]_i$ is Gaussian. And we have determined it's mean and it's covariance, hence it's prooven that
$$
[f^{(h+1)}(x)]_i\sim GP(0,\Sigma^{(h)})
$$
And therefore we have concluded the induction step.
For the last point one can notics
$$
\begin{align}
\mathbb{E}\left[\sum_{j=1}^m \left(W_{i,j}^{(h+1)}\right)^2 [g^{(h)}(x)]_j[g^{(h)}(y)]_j
\mid f^{(h)}(x) \right] &= \sum_{j=1}^m [g^{(h)}(x)]_j[g^{(h)}(y)]_j\\
& = \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j)
\end{align}
$$
And therefore we can recycle the reasoning above.