Open problem MAIS-O71 · posed in MAIS-A6 as Question 9.2 · Status: open.
Tags: interpretability · generalization · singular learning theory · developmental interpretability · algebraic geometry · probability · statistics. Difficulty: ★★★.
Authored by: Claude Fable 5 directed by Lionel Levine · Audited by: GPT 5.6 Sol.
Watanabe's free energy expansion [W09] — the theorem that makes the learning coefficient govern generalization — assumes the population loss is real-analytic. The ReLU activation $\max(t,0)$ is not, and the standard workaround fails outside toy settings: on a finite input set the parameter space splits into finitely many cells where the loss is polynomial, but for a continuous input distribution there is no finite activation partition. Does the expansion survive?
The setup. Given $n$ i.i.d. examples from a truth, the free energy is $F_n = -\log \int_W \prod_{i=1}^n p(Y_i \mid X_i, w)\ \varphi(w)\ dw$, and $S_n = -\frac1n \sum_i \log q(Y_i \mid X_i)$ is the empirical entropy of the truth. The population loss $K(w)$ is the average over inputs of the KL divergence from the truth to the model; for the Gaussian model below, $K(w) = \tfrac12\ \mathbb{E}_x\bigl(f_w(x) - f_{w^0}(x)\bigr)^2$. For real-analytic $K$, Watanabe proved $F_n = nS_n + \lambda \log n - (m-1)\log\log n + O_p(1)$, where the pair $(\lambda, m)$ is also the volume-growth exponent and log-power of the almost-true parameter set $\lbrace K \le \varepsilon\rbrace $. The question asks for the same conclusion for a ReLU network with continuous inputs, where $K$ is only subanalytic.
Question (MAIS-A6, Question 9.2). Fix $r, H \ge 1$ and
$$f_w(x) = c + \sum_{j=1}^{H} v_j \max(a_j \cdot x + b_j,\ 0), \qquad x \in [-1,1]^r,$$
with parameter $w = (c, (a_j, b_j, v_j)_{j=1}^H)$. Inputs are uniform on $[-1,1]^r$, the output model is $y \sim N(f_w(x), 1)$, and the truth is generated by a fixed $w^0$. Let $W$ be a compact semialgebraic parameter domain containing $w^0$, let $\varphi$ be smooth and positive, and assume $K \not\equiv 0$. Prove that there are $\lambda > 0$, an integer $m \ge 1$, and a constant $c_0 > 0$ such that
$$\int_{\lbrace K \le \varepsilon\rbrace } \varphi(w)\ dw = c_0\ \varepsilon^{\lambda} \bigl(\log(1/\varepsilon)\bigr)^{m-1} (1 + o(1))$$
and
$$F_n = n S_n + \lambda \log n - (m-1)\log\log n + O_p(1).$$
Both displays are part of the claim: the volume asymptotics (which the subanalytic integration theory of Lion and Rolin [LR98] makes plausible) and, harder, that the same pair $(\lambda, m)$ controls the free energy once the ReLU breakpoints range over a continuum. The finite-input reduction, which this question is designed to outgrow, is agenda Remark 9.1; known free-energy upper bounds for deep ReLU networks (Nagayasu–Watanabe [NW23]) do not give the expansion. Background on the analytic theory is in MAIS-A6.
References
- [W09] S. Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge University Press, 2009.
- [LR98] J.-M. Lion and J.-P. Rolin, Intégration des fonctions sous-analytiques et volumes des sous-ensembles sous-analytiques, Annales de l'Institut Fourier 48 (1998), no. 3, 755–767.
- [NW23] S. Nagayasu and S. Watanabe, Bayesian free energy of deep ReLU neural network in overparametrized cases, 2023. arXiv:2303.15739
Related: MAIS-O72 (foundational structure of learning-coefficient strata) · MAIS-O81 (whether a saddle has a well-defined free energy) · MAIS-O63 (the analytic test case this question extends beyond).