How Unwanted Objectives Can Emerge from Proper Incentives

Research

Published
7 October 2022

Authors
Rohin Shah, Victoria Krakovna, Vikrant Varma, Zachary Kenton

Exploring examples of goal misgeneralisation – where an AI system’s capabilities generalise but its goal doesn’t. As we build increasingly advanced artificial intelligence (AI) systems, we want to ensure they don’t pursue undesired goals. Such behavior in an AI agent is often the result of specification gaming, which involves exploiting a poor choice of what they are rewarded for. In our latest paper, we explore a more subtle mechanism by which AI systems may unintentionally learn to pursue undesired goals: goal misgeneralisation (GMG). GMG occurs when a system’s capabilities generalise successfully, but its goal does not generalise as intended, leading the system to competently pursue the wrong goal. Crucially, in contrast to specification gaming, GMG can occur even when the AI system is trained with a correct specification.

Our earlier work on cultural transmission led to an example of GMG behavior that we didn’t design. An agent (the blue blob, below) must navigate its environment, visiting the colored spheres in the correct order. During training, there is an “expert” agent (the red blob) that successfully visits the spheres in the correct order. The agent learns that following the red blob is a rewarding strategy.

The agent (blue) watches the expert (red) to determine which sphere to go to.

Unfortunately, while the agent performs well during training, it does poorly when, after training, we replace the expert with an “anti-expert” that visits the spheres in the wrong order.

The agent (blue) follows the anti-expert (red), accumulating negative reward.

Even though the agent can observe that it is getting a negative reward, it does not strive to achieve the desired goal of “visiting the spheres in the correct order” and instead competently pursues the goal of “following the red agent.” GMG is not limited to reinforcement learning environments like this one. In fact, it can occur with any learning system, including the “few-shot learning” of large language models (LLMs). Few-shot learning approaches aim to build accurate models with less training data.

We prompted one LLM, Gopher, to evaluate linear expressions involving unknown variables and constants, such as x+y-3. To solve these expressions, Gopher must first inquire about the values of the unknown variables. We provide it with ten training examples, each involving two unknown variables. At test time, the model is posed questions with zero, one, or three unknown variables. Although the model generalises correctly to expressions with one or three unknown variables, when there are no unknowns, it nevertheless asks redundant questions like “What’s 6?”. The model always queries at least once before providing an answer, even when it is not necessary.

Dialogues with Gopher for few-shot learning on the Evaluating Expressions task, with GMG behavior highlighted.

Within our paper, we provide additional examples in other learning settings. Addressing GMG is critical for aligning AI systems with their designers’ goals, as it is a mechanism by which an AI system may misfire. This will be especially important as we approach artificial general intelligence (AGI).

Consider two possible types of AGI systems:
A1: Intended model. This AI system behaves as its designers intend.
A2: Deceptive model. This AI system pursues an undesired goal while being intelligent enough to understand that it will be penalized for acting contrary to its designer’s intentions.

Since A1 and A2 will exhibit the same behavior during training, the potential for GMG means that either model could emerge, even with a specification designed to reward intended behavior. If A2 is learned, it might attempt to subvert human oversight to enact its plans towards the undesired goal.

Our research team encourages follow-up investigations into how likely GMG is to occur in practice, as well as exploring possible mitigations. In our paper, we propose some approaches, including mechanistic interpretability and recursive evaluation, both of which we are actively pursuing.

We are currently gathering examples of GMG in a publicly available spreadsheet. If you have encountered instances of goal misgeneralisation in AI research, we invite you to submit those examples.