Problem 1

We make the induction assumption for that for \(m \to \infty\):
\[ f^{(h)} \sim GP(0, \Sigma^{(h-1)})\]

First, we for the mean have :

\[ \mathbb{E}\left[[f^{(h+1)}(x)]_i \mid f^{(h)}(x) \right] = \mathbb{E}\left[\sum_{j=1}^mW^{(h+1)}_{i,j}[g^{(h)}(x)]_i \mid f^{(h)}(x) \right] = \sum_{j=1}^m\mathbb{E}\left[W^{(h+1)}_{i,j}\right][g^{(h)}(x)]_i = 0 \]

Now, for the covariance matrix, we have :
\[ \Sigma^{(h)}=\lim_{m\to \infty} \frac{2}{m}\sum_{i=1}^m\sigma(u_i)\sigma(v_i)\qquad [u;v]\sim\mathcal{N}(0,\Lambda^{(h)}) \]

\[ \begin{align} \mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i[f^{(h+1)}(y)]_i \mid f^{(h)}(x) \right] & =\mathbb{E}\left[\sum_{j=1}^m\sum_{k=1}^m W_{i,j}^{(h+1)}W_{i,k}^{(h+1)}[g^{(h)}(x)]_j[g^{(h)}(y)]_k \mid f^{(h)}(x) \right] \\ & = \mathbb{E}\left[\sum_{j=1}^m \left(W_{i,j}^{(h+1)}\right)^2 [g^{(h)}(x)]_j[g^{(h)}(y)]_j \mid f^{(h)}(x) \right] \\ & = \sum_{j=1}^m [g^{(h)}(x)]_j[g^{(h)}(y)]_j\\ & = \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j)\sigma([f^{(h)}(y)]_j) \end{align} \]
Hence with \(m\to\infty\) we get:
\[ \begin{aligned} Cov([f^{(h+1)}(x)]_i,[f^{(h+1)}(y)]_i\mid f^{(h)}(x) ) &= \mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i[f^{(h+1)}(y)]_i \mid f^{(h)}(x) \right] \\& - {\mathbb{E}_{x,y}\left[[f^{(h+1)}(y)]_i \mid f^{(h)}(x) \right]}\\&- \mathbb{E}_{x,y}\left[[f^{(h+1)}(x)]_i \mid f^{(h)}(x) \right]\\ &=\lim_{m\to \infty} \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j)\sigma([f^{(h)}(y)]_j)\\ &=2\mathbb{E}[\sigma(u)\sigma(v)] = \Sigma^{(h)}(x,y)\\ \end{aligned} \]
Where \(u,v\) are drawn from:
\[ \begin{bmatrix} [f^{(h)}(x)]_j\\ [f^{(h)}(y)]_j \end{bmatrix}\sim \mathcal{N}\left( \begin{bmatrix} 0\\ 0 \end{bmatrix}, \begin{bmatrix} \Sigma^{(h-1)}(x,x) & \Sigma^{(h-1)}(x,y)\\ \Sigma^{(h-1)}(y,x) & \Sigma^{(h-1)}(y,y) \end{bmatrix} \right) = \mathcal{N}(0,\Lambda^{(h)}) \]
Thanks to the indjuction assumption. Since this is all gaussian then also \([f^{(h+1)}(x)]_i\) is Gaussian. And we have determined it's mean and it's covariance, hence it's prooven that
\[ [f^{(h+1)}(x)]_i\sim GP(0,\Sigma^{(h)}) \]
And therefore we have concluded the induction step.
For the last point one can notics
\[ \begin{align} \mathbb{E}\left[\sum_{j=1}^m \left(W_{i,j}^{(h+1)}\right)^2 [g^{(h)}(x)]_j[g^{(h)}(y)]_j \mid f^{(h)}(x) \right] &= \sum_{j=1}^m [g^{(h)}(x)]_j[g^{(h)}(y)]_j\\ & = \frac{2}{m}\sum_{j=1}^m \sigma([f^{(h)}(x)]_j) \end{align} \]
And therefore we can recycle the reasoning above.