Open problem MAIS-O70 · posed in MAIS-A6 as Problem 6.3 · Status: open; full solution pending review (issue #3).
Tags: interpretability · generalization · singular learning theory · developmental interpretability · algebraic geometry · statistics. Difficulty: ★★.
Authored by: Claude Fable 5 directed by Lionel Levine · Audited by: GPT 5.6 Sol.
The one singular model family solved completely — reduced-rank regression — was solved globally: Aoyagi and Watanabe [AW05] computed the minimum of the local learning coefficient over the whole set of exact fits, as a closed form in four integers. But the estimators now used on neural networks, introduced by Lau, Furman, Wang, Murfet, and Wei [LFWMW23], are local, evaluated at a point, and the pointwise values on this benchmark family have never been tabulated. Those estimators matter for safety because they measure which internal mechanism training has selected: two networks fitting the same data by different mechanisms can behave differently on inputs no test anticipated. They now run on billion-parameter models, with exact ground truth known only for a short list of classical families. The benchmark is missing its own answer key.
Reduced-rank regression is the model $y = BAx + \text{noise}$: standard Gaussian output density with mean $BAx$, input $x \sim N(0, I_N)$ in $\mathbb{R}^N$, output in $\mathbb{R}^M$, and parameter $w = (A, B)$ with $A \in \mathbb{R}^{H \times N}$, $B \in \mathbb{R}^{M \times H}$ — a two-layer linear network whose exact-fit set for a rank-$r$ truth $B_0 A_0$ is the fiber $W_0 = \lbrace (A,B) : BA = B_0 A_0\rbrace $. The population loss is $K(A,B) = \tfrac12\ \mathbb{E}\lVert (BA - B_0 A_0)\ x \rVert^2$. The local pair $(\lambda(w^\ast ), m(w^\ast ))$ at a point $w^\ast \in W_0$ is the local learning coefficient and multiplicity: the volume of $\lbrace K \le \varepsilon\rbrace $ in a small ball around $w^\ast $ scales as $\varepsilon^{\lambda(w^\ast )}(\log(1/\varepsilon))^{m(w^\ast )-1}$. Different points of the fiber can be differently singular; the Aoyagi–Watanabe formula, a closed-form case analysis in the four integers $(N, M, H, r)$ stated as Theorem 3.1 in MAIS-A6, is the minimum over the fiber.
Problem (MAIS-A6, Problem 6.3). For reduced-rank regression with parameters $(N, M, H, r)$, prove that the local pair $(\lambda(w^\ast ), m(w^\ast ))$ at a factorization $w^\ast = (A, B) \in W_0$ depends only on $(\mathrm{rank} A, \mathrm{rank} B)$, and compute the resulting table. Hence characterize the strata on which $\lambda(w^\ast )$ equals the minimum in the Aoyagi–Watanabe theorem. For example, when $N = M = H = 2$ and $r = 0$, a neighborhood of $(I_2, 0)$ has local coefficient $2$, whereas the table gives $3/2$.
In the statement's last sentence, "the table" is the Aoyagi–Watanabe case list, which gives $3/2$ for these parameters. The example shows the stratification is genuine: at $(I_2, 0)$ the equation $BA = 0$ reduces to $B = 0$, a smooth point of the fiber, and the local coefficient $2$ exceeds the global minimum $3/2$. The convergence theory for the estimator at a point is already available via Watanabe's widely applicable Bayesian information criterion (WBIC) [W13]: restricted to a small ball around $w^\ast $, the WBIC estimate converges in probability to $\lambda(w^\ast )$ (agenda Remark 6.4). What is missing is the table itself. A starter sized for one paper: the case $N = M = H = 2$, listing the local pair on each rank stratum of $\lbrace (A,B) : BA = C\rbrace $. Background and the global theorem are in MAIS-A6.
References
- [AW05] M. Aoyagi and S. Watanabe, Stochastic complexities of reduced rank regression in Bayesian estimation, Neural Networks 18(7) (2005), 924–933.
- [LFWMW23] E. Lau, Z. Furman, G. Wang, D. Murfet, and S. Wei, The local learning coefficient: a singularity-aware complexity measure. arXiv:2308.12108
- [W13] S. Watanabe, A widely applicable Bayesian information criterion, Journal of Machine Learning Research 14 (2013), 867–897.
Related: MAIS-O69 (the localization question this table would calibrate) · MAIS-O77 (learning coefficients on the matrix factorization fiber and its saddles) · MAIS-O67 (the attention analog of the global formula).