arXiv:2610.41308v1 Announce Type: new
Abstract: Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution. We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations. Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve). Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity. Across ten models, REALIZE-L3 HSR ranges from 42.0% to 93.9%, while Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations. These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.
arxiv
SERIAL-7: A Benchmark for Machine-Verifiable Structured Generation
The paper introduces SERIAL-7, a Korean-English benchmark for evaluating machine-verifiable generation across seven representations. It separates semantic derivation, constrained artifact generation, and cross-representation preservation with 43 atomic constraints and executable checks. Results across ten models show that reliable structured generation requires explicit verification of how artifacts are realized and preserved.
Provides a concrete new benchmark and verification framework that addresses a critical gap in ensuring reliable structured outputs from LLMs.
Comments & Questions (0)
No comments yet
Be the first to comment!