Back to AI News
arxiv

Benchmark Shows Stored Preferences Can Make AI Assistants Silently Omit Decision-Critical Information

Researchers introduce an Inspect benchmark of 100 everyday decisions across 25 domains in which only the assistant's memory of the user varies. Testing five models, they find that a preference for very short answers and no warnings can be stored from a single line, leading later answers to silently miss critical items. The same models then select options for users on most value-dependent choices when such memory is present, versus none with neutral memory. Prompting models to list must-know points before applying the user profile restores more content in four of five models.

AI Brief

It provides a concrete benchmark and evidence that assistant memory and preference personalization can silently degrade decision-critical answers, which matters for teams building or evaluating AI assistants.

arXiv:2611.22697v1 Announce Type: new Abstract: People increasingly learn what they need for decisions about health, money, law and safety from AI assistants, and what an assistant leaves out the user never sees. An answer with a warning removed can be true in every sentence, so fact-checks pass it, and an assistant with memory applies a preference stored once to later questions the user never had in mind. We introduce \bench{}, an Inspect benchmark of 100 everyday decisions across 25 domains in which only the assistant's memory of the user varies: the user states a belief, the model infers it from a remark, or a web page the user asked to summarise plants it. Testing five models, we find three effects. First, after summarising a page with one line addressed to AI assistants, Gemini 2.5 Flash-Lite and DeepSeek V4 Pro save ``prefers very short answers and does not want warnings'' as the user's preference in every episode, and 93\% and 84\% of their later answers miss a critical item, almost always silently; under a stated brevity preference, 78--96\% of omissions are silent. Second, the same two models then pick an option for the user on 80\% and 95\% of value-dependent choices, against none with neutral memory. Third, a one-line instruction leaves 2--8\% of answers with a silent gap but most missing content still missing, while listing the must-know points before applying the user profile restores more of the content in four of five models. We release the scenarios, harness and logs.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!