Open problem MAIS-O34 · posed in MAIS-A2 as Problem 5.2 · Status: open; partial solution pending review (issue #4).
Tags: interpretability · world-model discovery · eliciting latent knowledge · black-box evaluation · algebraic geometry · computational. Difficulty: ★.
Authored by: Claude Fable 5 directed by Lionel Levine · Audited by: GPT 5.6 Sol.
The smallest environment supporting a causal question has two binary variables and one arrow: does $X$ cause $Y$ or $Y$ cause $X$? An agent is paid a known function of its action and both variables, observes nothing, and acts near-optimally under every mixture of local interventions. Which models of the environment are consistent with its behavior? This is world-model extraction in miniature: what a black-box evaluator, seeing actions only, can learn about how an agent models its world. Here the question is three unknown parameters, one binary structural choice, and sixteen explicit quadratic polynomials — small enough to answer exactly, and the answer is not known.
The two-variable family $\mathcal{M}_2(\lambda)$: chance variables $\mathbf{C}=\lbrace X,Y\rbrace $, observation set $\mathbf{O}=\emptyset$, utility parents $\mathbf{Z}=\lbrace X,Y\rbrace $, and a fixed utility $u\colon\lbrace 0,1\rbrace ^3\to[0,1]$ whose gap $g(x,y)=u(1,x,y)-u(0,x,y)$ satisfies $|g|\ge\lambda$, takes both signs, and is sensitive to each variable: flipping either argument changes $g$ by at least $\lambda$ at some value of the other. The agent's single binary action $d$ influences nothing but its score $u(d,x,y)$. A model is a pair $M=(G,\theta)$: graph $G_\to$ ($X\to Y$; parameter triple $\theta=(a,b_0,b_1)$ with $a=P(X{=}1)$, $b_x=P(Y{=}1\mid X{=}x)$) or $G_\leftarrow$ ($Y\to X$; $\theta=(b,a_0,a_1)$), all parameters in $[\lambda,1-\lambda]$ and edge strength $|b_0-b_1|\ge\lambda$ (resp. $|a_0-a_1|\ge\lambda$). There are sixteen intervention profiles (fix, flip, or leave each variable), and the behavioral transform is the vector $\bigl(\Delta_M(\sigma)\bigr)_{\sigma}\in\mathbb{R}^{16}$ with $\Delta_M(\sigma)=\sum_{x,y}P_M(x,y;\sigma)\ g(x,y)$, the expected advantage of playing $1$ over $0$: an optimal agent plays $1$ under a mixture $\sigma$ of profiles precisely when $\Delta_M(\sigma)>0$. The identified set $I_\delta(M)$ consists of the models $M'$ for which one policy family (an action probability for each mixture) comes within $\delta$ of the optimal expected utility, under every mixture, in both $M$ and $M'$ — the models that no amount of behavior can rule out — and the error $e(M;G',\theta')$ is $1$ for a wrong graph, else the largest difference of corresponding parameters. The exact intervention semantics and the policy-family and regret quantifiers are spelled out in MAIS-A2.
Problem (MAIS-A2, Problem 5.2). For the family $\mathcal{M}_2(\lambda)$, write $r_M(\delta):=\sup_{M'\in I_\delta(M)}e(M;G',\theta')$ for the local radius of the identified set. (a) Determine, as an explicit semialgebraic condition on $(u,\theta)$, when the global fiber $\lbrace M':\Delta_{M'}=\Delta_M\rbrace $ is the singleton $\lbrace M\rbrace $; in particular decide whether margin $\lambda>0$ alone suffices. (b) On the locus where the inverse is locally Lipschitz and the graph is locally fixed, compute the first-order constant in $r_M(\delta)=c(u,\theta)\delta+o(\delta)$. On the complementary locus, classify the alternatives: a positive limiting radius, graph ambiguity, or a fractional-power modulus. Use this classification to determine the largest regret below which the edge direction is certain throughout the class.
Part (a) is a concrete real-algebra problem: quantifier elimination will decide any single instance, and the problem asks for the human-readable answer. Part (b) is a small but genuine piece of the geometry that the general regret-floor problem asks for. The agenda sizes it for a first paper. This family is the smallest instance of the setting in which Richens and Everitt [RE24] proved that near-optimal behavior under local interventions determines the causal model for almost every parameter choice; part (a) asks for the exact locus in the smallest case. For the general framework this family instantiates, see MAIS-A2.
References
- [RE24] J. Richens and T. Everitt, Robust agents learn causal world models, Proc. 12th International Conference on Learning Representations (ICLR), 2024. arXiv:2402.10877
Related: MAIS-O35 (the finite-sample companion on the same family) · MAIS-O23 (the global identifiability question) · MAIS-O27 (the regret floor in general).