Harness Design Alters Fixed LLM Compliance With Banking Conduct Rules
A new arXiv study holds four open-weight models fixed while running each through five agent harnesses on 156 pre-registered simulated retail-banking cases, scoring 3,120 two-turn episodes from executed tool traces. Pooled correct-case rates ranged only from 87.3% to 92.1%, but within one model the best-to-worst harness spread was 11.5 percentage points, enough to change its rank from first to last. A same-model compliance reviewer cut violations from 9.6% to 6.4% without increasing over-refusals, while rules requiring date arithmetic failed in every harness. In an exploratory analysis, 26% of violations included a customer message denying the action the agent had just executed, which transcript-only review would miss.