Open problem MAIS-O59 · posed in MAIS-A5 as Problem 6.1 · Status: open.
Tags: interpretability · mechanistic interpretability · training dynamics · universality of circuits · monosemanticity · dynamical systems · probability · harmonic analysis. Difficulty: ★★.
Authored by: Claude Fable 5 directed by Lionel Levine · Audited by: GPT 5.6 Sol.
Addition mod 5 has two Fourier frequencies; give the network exactly two quadratic neurons, and the competition between representations for a finite supply of neurons appears in its smallest form. The whole system is an explicit polynomial ODE on $\mathbb{R}^{30}$ with Gaussian initial data — small enough to attack directly, and the smallest open case of the headline selection problem MAIS-O5.
The network has parameters $\theta = (u_i, v_i, w_i)_{i=1,2}$ with $u_i, v_i, w_i \in \mathbb{R}^{C_5}$, and logits $f_\theta(a,b)(c) = \sum_{i=1}^2 (u_i(a)+v_i(b))^2\ w_i(c)$. Training is gradient flow on the cross-entropy loss over the full 25-entry addition table, with no weight decay, from independent standard Gaussian coordinates. The irreducible characters of $C_5$ are $\rho_\zeta(a) = e^{2\pi i \zeta a/5}$; the two frequency classes, which make up $\mathcal{R}(C_5)$, are $[\rho_1] = \lbrace \rho_1, \rho_{-1}\rbrace $ and $[\rho_2] = \lbrace \rho_2, \rho_{-2}\rbrace $. Neuron $i$ is $(\delta,[\rho_\zeta])$-pure if, after centering (subtracting from each of $u_i, v_i, w_i$ its mean, which removes the trivial character), at least a $(1-\delta)$ fraction of the combined squared norm of the three vectors lies in the isotypic component of $[\rho_\zeta]$. In the statement below, $\sigma$, $m$, $\lambda$, and $\tau$ are the agenda's ensemble parameters — activation, number of neurons, weight-decay coefficient, initialization scale — specialized to the setup just described.
Problem (MAIS-A5, Problem 6.1). Let $G = C_5$, $\sigma(x) = x^2$, $m = 2$, $\lambda = 0$, and $\tau = 1$. The parameter space is $\mathbb{R}^{30}$ and $\mathcal{R}(C_5)$ has two frequencies. Write the gradient field in a real Fourier basis. As a first finite calculation, classify its stationary directions for which each neuron is supported on one frequency, determine their transverse stability, and decide which of the four ordered frequency assignments can attract an open set. Then prove or refute that almost surely each neuron aligns: the limit of $\theta(t)/\Vert \theta(t)\Vert $ exists and its neuron $i$ is $(\delta, [\rho_{\zeta_i}])$-pure for every $\delta > 0$, for some frequency $\zeta_i$. Compute the probability that the two neurons choose the same frequency. (The proved symmetry of the selection law makes the two frequencies exchangeable, so this one probability determines the law of the pair on the alignment event.)
The first step is a symmetry-reduced algebra problem; the second asks whether those local calculations control a random trajectory. A direction, here, is a unit vector in parameter space; it is stationary when the gradient field there is parallel to it — a fixed point of the flow induced on the unit sphere — and transverse stability means stability of that fixed point within the sphere. Normalized directions are the right object because at zero decay the cross-entropy may have no finite minimizer. The symmetry invoked in the closing parenthesis is the agenda's Proposition 4.4: the automorphism $a \mapsto 2a$ of $C_5$ makes the two frequency classes exchangeable. Existing theories miss this case: the small-initialization analyses of He, Wang, Chen, and Yang [HWCY26] start from controlled states rather than unit-scale Gaussians, and the asymptotic alignment theorem of He, Wang, Zhang, Chen, and Yang [HWZCY26] concerns a Taylor-surrogate projected flow, not the exact loss. Full conventions are in MAIS-A5.
References
- [HWZCY26] J. He, L. Wang, F. Zhang, S. Chen, and Z. Yang, Neural networks provably learn spectral representations for group composition, preprint, 2026. arXiv:2606.02993
- [HWCY26] J. He, L. Wang, S. Chen, and Z. Yang, On the mechanism and dynamics of modular addition: Fourier features, lottery ticket, and grokking, preprint, 2026. arXiv:2602.16849
The rectifier cousin of this problem, MAIS-O60, was resolved in the negative by explicit dead-neuron and memorization constructions; neither mechanism transfers to the quadratic activation here, which has no dead half-line, so the alignment question above stands.
Related: MAIS-O5 (the headline problem this is the foothold of) · MAIS-O60 (one rectifier neuron, the analogous reduction — resolved in the negative) · MAIS-O92 (the outcome law that resolution leaves open) · MAIS-O61 (a pilot experiment estimating the same-frequency probability).