Back to AI News
arxiv

Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent Evaluation

Researchers propose Verification Transport, a reliability test that measures whether LLM-based verifier judgments remain stable when agent execution trajectories are exposed through abstractions such as composed tools, delegation boundaries, or compacted histories. Across four agent-evaluation benchmarks, seven verifier backbones, four verification protocols, and three abstraction boundaries, they find substantial transport failure, reaching about 31% under macro abstraction and delegation. Evidence-grounded and deliberative protocols tend to abstain under limited observability, while restoring execution provenance recovers 85.5% of transport failures. The results indicate that native-view accuracy alone is insufficient to characterize verifier reliability in deployed agent systems.

AI Brief

This helps readers understand hidden reliability risks when LLM verifiers judge agent executions across abstraction layers rather than raw trajectories.

arXiv:2611.18471v1 Announce Type: cross Abstract: LLM-based verifiers are increasingly used to evaluate agent executions, but their reliability is typically measured under a single representation of the trajectory. In deployed agent systems, the same execution may be exposed through composed tools, delegation boundaries, or compacted histories, changing what execution evidence is directly observable to the verifier. It therefore remains unclear whether correct verifier judgments persist across such abstraction boundaries. We introduce Verification Transport, a reliability test that measures judgment stability under abstraction-induced changes in execution observability. We construct matched native, abstracted, and restored views of the same frozen executions while holding the task, realized trajectory, outcome, ground truth, evaluation criterion, and verifier fixed. Across four agent-evaluation benchmarks, seven verifier backbones, four verification protocols, and three abstraction boundaries, we find substantial transport failure, reaching about 31% under macro abstraction and delegation. Verification protocols exhibit distinct failure modes: direct verification more often produces incorrect verdicts, whereas evidence-grounded and deliberative protocols tend to abstain under limited observability. This suggests that lower transport failure rates do not necessarily imply greater reliability. Restoring the corresponding execution provenance recovers 85.5% of transport failures, while recovery is lower when provenance is lossy or unavailable. These results show that native-view accuracy alone is insufficient to characterize verifier reliability when execution evidence crosses abstraction boundaries.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!