Open problem MAIS-O87 · posed in MAIS-A8 as Question 6.2 · Status: open.
Tags: generalization · goal misgeneralization · proxy goals · training dynamics · dynamical systems · probability. Difficulty: ★★.
Authored by: Claude Fable 5 directed by Lionel Levine · Audited by: GPT 5.6 Sol.
The coin problem is the agenda's one-dimensional version of the coin-collector experiment of Langosco, Koch, Sharkey, Pfau, Orseau, and Krueger [LKSP+22], in which an agent trained only on right-lying coins learned "move right" rather than "get the coin". Reinforcement learning changes the coin problem in one essential way: the training data become endogenous, since the states an agent experiences depend on the policy it currently has. A policy that has already committed to "move right" receives corrective signal from a left-lying coin only on episodes where it happens to random-walk left far enough to collect it, an event whose probability the proxy itself drives toward zero. Misgeneralization protects itself. Is the protection strong enough to last forever?
On the agenda's coin line, an agent at position $p$ seeks a coin at position $c$ on $\lbrace 0,\dots,L\rbrace $; the coin sits at the right end in all but an $\varepsilon$-fraction of episodes, and the return $J_\varepsilon(w)$ is the probability that an episode played by the policy collects the coin. The linear policy steps right with probability $\sigma(w\cdot x_{\mathrm a}(s))$, where $x_{\mathrm a}(p,c)=(1,p,c)$ and $\sigma$ is the logistic function; it trains by the smooth policy-gradient flow $\dot w=\nabla J_\varepsilon(w)$ from $w(0)\sim N(0,\sigma^2I_3)$. Write $q^{\mathrm{RL}}_\varepsilon(t;\sigma,L)$ for the probability, over the initialization, that at time $t$ the policy prefers to step right at the probe state $s^{\ast }=(\lceil L/2\rceil,0)$ — coin at the left end, a state outside the training support for every $\varepsilon$, where the proxy and the intended goal disagree.
Question (MAIS-A8, Question 6.2). Fix $L\ge4$ and $\sigma>0$. Are there $\bar\varepsilon>0$ and $\delta>0$ such that for every real $\varepsilon\in(0,\bar\varepsilon]$ the linear policy-gradient flow satisfies $\limsup_{t\to\infty}q^{\mathrm{RL}}_\varepsilon(t;\sigma,L)\ge\delta$? Behavior cloning has no such $\bar\varepsilon$: there, the misgeneralization probability tends to zero for every rational $\varepsilon>0$.
Behavior cloning, the contrast case, trains the same linear model by supervised imitation of the intended action (step toward the coin) at states drawn from the same coin distribution; the agenda proves its trained policy walks toward the coin from every initialization, at every fixed diversity. (The restriction to rational $\varepsilon$ is a bookkeeping convention of the agenda.) In cloning, the corrective signal at diversity $\varepsilon$ has size of order $\varepsilon$; in reinforcement learning it is $\varepsilon$ times a policy-dependent exploration factor (in the neighboring setting of reinforcement finetuning of language models, this vanishing of the corrective signal is a theorem of Razin et al. [RZST+24]), and the question asks whether that starvation leaves a positive fraction of initializations misgeneralized for all time, uniformly in small $\varepsilon$. At $\varepsilon=0$ the agenda proves the proxy wins from every initialization; this question is the first strictly positive diversity level. For the exact episode mechanics, the surrounding results, and starter projects, see MAIS-A8.
References
- [LKSP+22] L. Langosco, J. Koch, L. Sharkey, J. Pfau, L. Orseau, and D. Krueger, Goal misgeneralization in deep reinforcement learning, ICML 2022. arXiv:2105.14111
- [RZST+24] N. Razin, H. Zhou, O. Saremi, V. Thilak, A. Bradley, P. Nakkiran, J. Susskind, and E. Littwin, Vanishing gradients in reinforcement finetuning of language models, ICLR 2024. arXiv:2310.20703
Related: MAIS-O88 (the trained-to-criterion misgeneralization curve) · MAIS-O84 (the cloning crossover this contrasts with) · MAIS-O8 (the headline selection problem).