Back to AI News
arxiv

Hx2Dx ARENA benchmarks LLMs on interactive clinical diagnosis

Researchers introduce Hx2Dx ARENA, an interactive benchmark of 1,000 clinical cases from the New England Journal of Medicine that simulates encounters from initial presentation through history taking, physical examination and diagnostic workup. Open- and closed-source LLMs are evaluated under multiple configurations. Every evaluated model loses diagnostic accuracy in the interactive setting compared with a static setting where the full case is presented upfront, and the ability to ask targeted questions accounts for much of the variation between models and the accuracy loss. The work highlights a gap between reasoning over available clinical data and hypothesis-driven information gathering.

AI Brief

It provides a concrete interactive benchmark showing that LLMs struggle with the questioning process needed for real-world diagnosis, an important limitation for clinical deployment.

arXiv:2611.14122v1 Announce Type: new Abstract: Large language models (LLMs) achieve expert-level diagnostic accuracy in static evaluations, where the model receives a curated vignette or case report that already condenses the diagnostically relevant findings. Such evaluations measure inference over already selected evidence; the evaluations do not account for the inquiry used to obtain it. In clinical practice, however, diagnosis is an iterative process of inquiry. Clinicians take the patient's history, or anamnesis (Hx), through targeted questioning, examine the patient, and request and interpret tests, progressively narrowing the differential until they reach a final diagnosis (Dx). We introduce Hx2Dx ARENA, an interactive benchmark of 1'000 clinical cases from the New England Journal of Medicine (NEJM). Each case is a simulated encounter that proceeds from initial presentation through history taking and physical examination to diagnostic workup. We evaluate open- and closed-source LLMs under multiple configurations. Performance varies considerably across models in the interactive setting, and every model loses diagnostic accuracy relative to the static setting, where the full case is presented upfront. The ability to ask the right questions accounts for much of both the gap between models and the loss of accuracy: diagnosis improves as questions become more targeted and build a more complete clinical picture. LLMs thus reason well once the relevant data are available but remain limited in hypothesis-driven information gathering, in knowing what to ask in order to know. Closing this gap is a prerequisite for their safe integration into real-world diagnostic workflows.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!