StateBench: Testing how language models track state changes over long documents

StateBench is a new benchmark from AI2 that evaluates language models on tracking state changes across long documents, a task that requires sustained attention and memory. The benchmark includes 1,000+ tasks with synthetic and real-world documents, and initial results show that even top models struggle, with accuracy dropping significantly on longer contexts. This highlights a key limitation in current LLMs for tasks like code refactoring or document editing.