Back to AI News
arxiv

Study Localizes Causal Goal Representations in LLM Attention Heads

Researchers built a multi-sub-task coding setting to test whether LLMs encode goals as internal representations that guide action across different execution paths. They localized an encoding of the next sub-goal in the outputs of a sparse set of attention heads, which can also decode the model's upcoming actions in natural language. Causal interventions confirmed the encoding drives the next action and is plastic: transplanting it into another programming language triggers the same operation via different tokens. The authors propose a non-recurrent goal-seeking account in which the model repeatedly re-reads the goal from the task specification and compares it with visible progress, and they test this with targeted interventions.

AI Brief

Offers mechanistic evidence that LLM goal-directed behavior can be decoded and causally manipulated at specific attention heads, useful for interpretability and agent-reliability work.

arXiv:2611.17297v1 Announce Type: cross Abstract: We investigate whether and how LLMs encode goals: internal neural representations that precede and guide action, and that are flexible enough to be realized through many different execution paths. We construct a coding setting with multiple sub-tasks that allows us to evaluate goal-seeking behavior and the selection of sub-goals. In this setting, we localize an encoding of the next sub-goal in the outputs of a sparse set of attention heads. The same heads can be used to decode the LM's future actions in natural language (e.g., "I'm checking if a word is a palindrome"). We verify that this encoding is causal for next action and is plastic: transplanting it into a different environment, e.g., another programming language, triggers the same operation realized with a different sequence of tokens. The LM also sustains the encoding throughout the action's execution, which is puzzling because a transformer LM without recurrence cannot carry a state forward for more tokens than it has layers left. We propose that it does not need to. Under a non-recurrent goal-seeking (NRGS) account, the LM frequently re-reads the goal from the task specification, compares it with the progress visible in the growing context, and pursues what remains. We test the predictions of this account with targeted causal interventions.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!