Live intelligence

Latest AI updates

2 updates
Research

LiveGeo benchmark evaluates geographic LLM agents in evolving worlds

Researchers introduced GeoWorld, a controllable geographic environment platform for LLM agents, and LiveGeo, a benchmark for evaluating reliable decision-making as real-world geographic states evolve. LiveGeo requires agents to actively acquire task-relevant information, verify whether prior evidence remains valid, and revise plans when necessary. The results show that frontier LLMs struggle even with static tasks, no model consistently excels, and strong static performance does not guarantee reliability in dynamic settings. Analysis points to a key bottleneck in acquiring and translating relevant evidence into valid decisions as the world changes.

LLM agentsgeographic reasoningbenchmarking
Research Editor selected

EnvOS Audits Computer-Use Agents on Mid-Task Changes

EnvOS is a research framework that checks whether mid-task environment changes execute correctly, tasks remain solvable, and graders distinguish correct from incorrect behavior, then independently rechecks verdicts against archived evidence. In the authors’ clinic-booking tests, Claude Opus 5 succeeded on 0 of 16 tasks without extra information, 2 of 16 with a reminder to recheck, and 15 of 16 when shown the updated value. Results varied by model and interface; on a separate externally generated set of corrupted episodes, the auditing pipeline caught 25 of 39 cases versus 4 for simple log checks.

Why it matters: The work offers a concrete method for separating agent failures from environment and grading failures, while showing how interface design can affect handling of stale information.

EnvOSComputer-use agentsAgent evaluation