Research Editor selected
Researchers introduced M4Bench, a benchmark of 28 clinical derivation tasks across 15 task families using MIMIC-IV and eICU, to test whether AI coding agents can reproduce EHR-derived variables and how clinician-reviewed agent skills affect results. In a Docker-isolated Codex campaign, skills raised reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounted for most of the gain: an operational spec alone raised reward by 0.162 on an eight-task subset, while skills added only 0.024. The authors note the gains reflect partial agreement with reference derivations rather than completed tasks, and release tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.
Why it matters: It provides a concrete benchmark and evidence on where coding agents help or fail in clinical EHR derivation, showing that precise specifications matter more than generic agent skills.
M4Benchclinical EHR derivationAI coding agents