Design preview · Back to the main design
MAIS Math for AI Safety

Main page

Agendas

An agenda groups open problems into one research direction. A maturing agenda spins off papers.

A1 · Quantitative bounded Löb

Draft · July 12, 2026 · 14 problems

Löb's theorem converts a proof of "if $P$ is provable then $P$" into a proof of $P$ outright, by an explicit construction with an explicit cost.

A2 · Recovering world-models from behavior

Draft · July 12, 2026 · 14 problems

Two theorems of Richens and coauthors show that any policy near-optimal across a rich family of interventions determines an approximate causal model of its environment — but the extraction consumes unlimited noiseless queries. This agenda develops the finite-sample side: explicit margin conditions replacing measure-zero genericity, query complexity under exact, sampled, and Boltzmann-perturbed answers, and the error floor imposed by the agent's regret.

A3 · The geometry and identifiability of superposition

Draft · July 12, 2026 · 9 problems

A neural network can store many more concepts than it has dimensions, as nearly orthogonal directions read out through sparse coefficients; the recovery tool used in practice is an $\ell^1$-penalized dictionary learner. Classical dictionary-learning theory assumes independent supports, but real concepts co-occur. Two small propositions bracket the problem: nested supports force merging at small positive penalty, while solo firing restores recovery in the zero-penalty constrained limit.

A4 · Training for interpretability

Draft · July 12, 2026 · 9 problems

Can a training objective make a network's internal features easy to recover? "Easy to interpret" is not yet a mathematical property, and this agenda does not pretend it is: it isolates three precise surrogates — coherence, $\ell^1$-recoverability, monosemanticity — inside a single toy model, proves small theorems marking the boundary of what is easy, and poses the problems that remain where resources are scarce and training means dynamics rather than exact minimization.

A5 · Which irreducible representations does training select?

Draft · July 12, 2026 · 12 problems

Small networks trained to multiply in a finite group $G$ discover representation-theoretic algorithms, but which irreducible representations appear varies with the random initialization, and no theorem predicts the variation. This agenda defines the training ensemble and a black-box observable — the key set of irreducible representations visible in a trained network's outputs — precisely enough that the question becomes the law of a random subset of $\mathrm{Irr}(G)$.

A6 · Does geometric simplicity force a legible mechanism?

Draft · July 12, 2026 · 11 problems

For a quadratic network computing addition mod $p$, the population loss is an explicit degree-six polynomial, and Fourier inversion gives exact fits built from single-frequency neurons — the algorithm found by reverse-engineering trained networks. The headline question: in a fixed ball, is every exact fit of smallest local learning coefficient of this single-frequency form?

A7 · Effective loss and learning dynamics

Draft · July 13, 2026 · 11 problems

Watanabe's free-energy asymptotics predict which solutions Bayesian learning prefers as sample size grows. Gradient descent is not Bayesian, but in solvable deep-linear models it follows a temporal staircase, learning structures one at a time.

A8 · A predictive theory of out-of-distribution generalization

Draft · July 16, 2026 · 11 problems

An agent trained to collect a coin that always sat at the right end of its levels learned "move right," not "get the coin." This agenda turns that episode into mathematics: a one-dimensional environment with the degeneracy built in — two policies indistinguishable on the training distribution. The linear case and several infinite-width limits can be solved; the finite-width selection law, its rare-diversity crossover, and the boundary between proxy and goal generalization remain open.