Optional — Linearized learning and the interpolation peak

Linearized learning and the interpolation peak

This supplement to Week 4 has no assigned lecture hours in the current syllabus. It separates existence, training and fresh-data prediction in models simple enough to calculate. See Chapter 2 §2.5 in the companion notes for the longer treatment and nn-llm-bibliography for the underlying literature.

A fixed feature map turns training into a matrix equation

Let $f_{\theta }(x)$ be a scalar prediction. Near initial parameters ${\theta }_0$, a first-order expansion is

$$f_{\theta }(x)\simeq f_{{\theta }_0}(x)+\varphi (x{)}^{\mathsf{T}}(\theta -{\theta }_0),\varphi (x)={\nabla }_{\theta }{f}_{{\theta }_0}(x).$$

The components of $\varphi (x)$ measure how each parameter changes the initial prediction. In this linearized model they are held fixed. On $P$ training examples define $r_{\mu }=f_{\theta }(x^{\mu })-y^{\mu }$ and use loss $\mathcal{L}=\frac{1}{2}{\sum }_{\mu }{r}_{\mu }^2$. Gradient flow is $\dot{\theta }=-{\sum }_{\mu }\varphi (x^{\mu })r_{\mu }$. Differentiating each residual gives

$${\dot{r}}_{\nu }=-\sum _{\mu }{K}_{\nu \mu }{r}_{\mu },K_{\nu \mu }=\varphi (x^{\nu }{)}^{\mathsf{T}}\varphi (x^{\mu }),r(t)=e^{-Kt}r(0).$$

$K$ is a Gram matrix and is positive semidefinite. It is called the neural tangent kernel on these observations. An eigenmode with eigenvalue $\lambda$ decays as $e^{-\lambda t}$; a zero mode cannot be fitted by this feature map. Using the mean loss instead rescales time by $P$.

For example, with $K=\mathrm{diag}(2,0)$ and $r(0)=(1,1)$, $r(t)=(e^{-2t},1)$. One error can decrease while the other remains. This solution is exact for the fixed-feature model. In the nonlinear network the features generally move, so a small measured kernel drift is needed to use it as a controlled approximation. Small parameter motion alone, without a scale and derivative bound, does not establish small output error.

A noisy linear teacher

Take $x^{\mu }\in {\mathbb{R}}^N$ with independent standard Gaussian components and observations $y^{\mu }=w_{\ast }\cdot x^{\mu }+\sigma {ϵ}^{\mu }$, with independent unit Gaussian ${ϵ}^{\mu }$. Fit by least squares, choosing the minimum-norm solution when several minimizers exist. Define the signal-prediction risk on a fresh independent input by

$$R={\mathbb{E}}_x[((\hat{w}-w_{\ast })\cdot x{)}^2]=\Vert \hat{w}-w_{\ast }{\Vert }^2.$$

This excludes the fresh observation’s own noise variance ${\sigma }^2$. In the proportional limit $P,N\to \infty$, $\alpha =P/N\ne 1$, with a fixed signal norm, the asymptotic risk for this isotropic ensemble is

$$R(\alpha )=\{\begin{array}{ll}\Vert w_{\ast }{\Vert }^2(1-\alpha )+\frac{{\sigma }^2\alpha }{1-\alpha },& 0<\alpha <1,\\ \frac{{\sigma }^2}{\alpha -1},& \alpha >1.\end{array}$$

This random-matrix result is quoted with its assumptions, rather than derived here. Below one, the unseen component of the teacher contributes the first term, and fitting observation noise contributes the second. At nonzero noise the risk diverges near one because the nearly singular design matrix amplifies that noise. This is an asymptotic statement, not an infinity in every finite training run.

For $0<\sigma <\Vert w_{\ast }\Vert$, differentiating the lower branch gives an interior minimum:

$$R^{\prime }(\alpha )=-\Vert w_{\ast }{\Vert }^2+\frac{{\sigma }^2}{(1-\alpha {)}^2}=0,{\alpha }_{\ast }=1-\frac{\sigma }{\Vert w_{\ast }\Vert }.$$

With $\Vert w_{\ast }\Vert =1$ and $\sigma =0.3$, ${\alpha }_{\ast }=0.7$ and $R=0.51$. If $\sigma \ge \Vert w_{\ast }\Vert$, there is no interior minimum in $0<\alpha <1$. If $\sigma =0$, there is no noise divergence: the lower branch decreases to zero, and sufficiently many noiseless independent equations identify the teacher. These qualifications are essential to interpreting the curve.

The interpolation threshold at one concerns real equations in this Gaussian linear model. Cover’s threshold at two concerns random sign constraints in general position. Their values do not define a universal ratio for other models or data distributions.

Checkpoint. Does zero training error imply zero risk? No. For $\alpha =0.5$, signal norm one and $\sigma =0.3$, interpolation is possible while $R=0.5+0.09=0.59$. Fitting observed labels and predicting an unseen signal are different questions.