Back to AI News
arxiv

Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents

Researchers introduce Soteria, a framework for LLM agents that performs multi-step tool calls while satisfying both task completion and domain policy conformance. Before execution, the agent generates a structured plan that is formally verified against specifications prior to any tool invocation; during execution, the verified plan constrains trajectories and guides recovery when unsafe actions are blocked. The authors report 1.7 times the average task success rate of the best baselines across multiple tool-use benchmarks, with no policy specification violations.

AI Brief

Offers AI practitioners a concrete, evaluated approach to resolving the safety-versus-utility tradeoff in LLM agents that execute multi-step tool calls.

arXiv:2611.17880v1 Announce Type: new Abstract: LLM agents that execute multi-step tool calls must satisfy two objectives simultaneously: completing the user's task (utility) and conforming to domain policies and correctness constraints (safety). These objectives are in tension -- blocking an unsafe action prevents a violation but can leave the agent stranded with no principled way to recover. Existing approaches sacrifice one objective for the other. We introduce Soteria, a framework that reconciles safety and utility through verified hierarchical planning and runtime enforcement of specifications. Before execution, the agent generates a structured plan that is formally verified against the specifications prior to any tool invocation. During execution, the verified plan serves dual roles: it ensures that the agent takes specification-conformant trajectories and provides guidance when unsafe actions are blocked. Across multiple benchmarks covering a diverse range of tool-use tasks, Soteria achieves 1.7 times the average task success rate of the best baselines, with no violations of policy specifications, demonstrating that safety and utility need not be traded off.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!