Back to AI News
arxiv

How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents

The paper describes a pipeline that mines short lessons from an LLM agent's own trajectories and consolidates them into guidelines that can be supplied at inference. It reports deployment experience from the open-source altk-evolve library, IBM's CUGA agent, and watsonx Orchestrate, evaluated on AppWorld across models with 40-92% baseline task completion. A central finding is that the optimal amount of memory depends on the model: DeepSeek-V3.2 gains 9.5 points of task completion with its full 191-guideline set, while gpt-oss-120b improves from 39.9% to 55.4% with a selected subset of about 29 guidelines versus 51.2% with all 199. The authors also document failure modes including a backfiring high-support guideline, lost provenance during consolidation, and a swallowed API error.

AI Brief

It offers concrete benchmark results and deployment lessons for teams building memory into LLM agents.

arXiv:2611.26361v1 Announce Type: new Abstract: LLM agents on long-horizon tasks fail in recurring, procedural ways. They mis-paginate an API, resolve the wrong entity, or answer when no answer was asked. We built a pipeline that lets an agent learn from its own history. It first mines short \emph{lessons} from the agent's own trajectories, one to three per episode. It then consolidates those lessons into a smaller set of \emph{guidelines}, where each guideline records how many lessons it merged. At inference the agent receives either all guidelines or a selected subset. The pipeline ships in the open-source altk-evolve library and is deployed in IBM's CUGA agent and watsonx Orchestrate platform. We report what we learned building, measuring, and productizing it. Our primary instrument is the AppWorld benchmark, across models spanning 40--92\% baseline task completion. The central finding is that the right amount of memory is a property of the model, not the memory. A strong model keeps improving as guidelines are added. DeepSeek-V3.2 gains +9.5 points of task completion and +16.1 points of scenario completion with its full 191-guideline set. A weak model gains less from the same full set than from a small selected subset. gpt-oss-120b starts at 39.9\% task completion. It reaches 51.2\% with all 199 of its guidelines but 55.4\% (mean of two runs) with a selected subset of about 29. Around that result we report what went wrong: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while producing a plausible log, and a baseline confound that nearly produced a misleading comparison with a prior method. We distill these into recommendations for teams deploying agent memory.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!