Back to AI News
arxiv

AstroMemBench: A Benchmark for Memory Systems in Embodied Spacecraft Agents

Researchers introduce AstroMemBench, a benchmark for evaluating long-term memory systems in embodied spacecraft agents operating under communication delays, resource limits and safety constraints. The benchmark uses a Scenario-Challenge-Capability design method and covers 16 memory subchallenges through 140 templates and 2240 instances across nine tasks in planetary surface operations, on-orbit servicing and deep-space exploration. A deterministic environment scores actions for task completion, state changes, safety and resource use, and ablations under nine conditions test sensitivity to memory recovery and design differences. Experiments comparing six configurations find that Mem0, Chain-of-Memory, LightMem and RAG improve same-session mean scores over a native Pi Agent, with no clear advantage of the three memory systems over RAG, while Mem0 outperforms RAG by 12.25 points after a session restart. The benchmark and evaluation code are publicly released.

AI Brief

It provides a concrete, openly available benchmark and comparative evidence for memory-system design in embodied spacecraft agents, useful to researchers working on long-horizon embodied AI.

arXiv:2611.16590v1 Announce Type: new Abstract: Long-term spacecraft autonomy requires embodied agents to use operational history under communication delays, time-critical operations, environmental uncertainty, complex physical interactions, power and computational constraints, and safety and reliability requirements. We introduce AstroMemBench and a Scenario-Challenge-Capability design method that maps these requirements to executable memory tests. The benchmark covers 16 memory subchallenges through 140 templates and 2240 instances across nine tasks in planetary surface operations, on-orbit servicing, and deep-space exploration. A deterministic environment scores actions for task completion, state changes, safety, and resource use. Ablations under nine conditions show that the benchmark is sensitive to how accurately agents recover and use historical information and to differences in memory-system design. Main experiments compare six configurations. Mem0, Chain-of-Memory, LightMem, and retrieval-augmented generation (RAG) improve same-session mean scores over native Pi Agent, with no clear advantage of the three memory systems over RAG. After a session restart, Mem0 outperforms RAG by 12.25 points. Benefits favor instruction retention and result reuse over ongoing state tracking and energy reservation. AstroMemBench supports the design, development, and selection of memory systems for embodied spacecraft agents. AstroMemBench 1.0 and its evaluation code are publicly available: https://github.com/flyingView/AstroMemBench-v1.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!