Back to AI News
arxiv

Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm

This paper proposes a trajectory audit that replays real hospital admissions checkpoint by checkpoint, letting each AI system decide using only the evidence released so far, and rates every test, treatment and disposition for potential patient harm and clinical validity. An LLM judge scores each decision, calibrated against a physician reference standard built from triple annotation and senior adjudication of a stratified sample. Across 300 cases from a hospital cardiology cohort, MIMIC-IV and the ASCENT benchmark, with up to seven AI systems per source, 55.2% of hospital trajectories that reached an acceptable final diagnosis passed through a decision with moderate or severe potential harm, and 24.8% through a clinically unacceptable one; harm rates were 76.2% on MIMIC-IV and 39.6% on ASCENT. An oversight audit found structured corrections resolved 73.6% of first errors versus 24.8% after bare rejection, but new potentially unsafe decisions followed 30.2% of corrections versus 19.7% of rejections.

AI Brief

It shows that correct final diagnoses and successful error corrections can still hide unsafe intermediate clinical AI decisions, giving evaluators a concrete method and benchmark evidence for auditing decision trajectories rather than outcomes alone.

arXiv:2611.22612v1 Announce Type: new Abstract: A clinical AI system can reach a correct final diagnosis through intermediate decisions that could have harmed the patient, and can repair an error while introducing new risks. Most evaluations score final diagnoses; even stepwise benchmarks, which release a case's evidence over time, score whether systems reach the right diagnosis or gather the right information, not whether the decisions a patient passes through are safe. We make patient safety the target of stepwise evaluation. A trajectory audit replays real admissions checkpoint by checkpoint, lets each system decide with only the evidence released so far, and rates every test, treatment and disposition for its potential harm to the patient and, separately, for clinical validity, against the documented final diagnosis and physician-written disease rubrics yet on the evidence available when the decision was made. An LLM judge scores every decision, and a physician reference standard (triple annotation and senior adjudication of a stratified sample) calibrates the rates. An oversight audit branches each trajectory at its first actionable error and measures, on the same branches, whether corrective feedback repairs the error and whether it introduces new harm. Across 300 cases from a hospital cardiology cohort, MIMIC-IV and the public ASCENT benchmark, with up to seven AI systems per source, 55.2% of hospital trajectories reaching an acceptable final diagnosis passed through a decision with moderate or severe potential harm and 24.8% through a clinically unacceptable one; the harm rates were 76.2% on MIMIC-IV and 39.6% on ASCENT. Structured corrections resolved 73.6% of first errors versus 24.8% after bare rejection, yet new potentially unsafe decisions followed 30.2% of corrections versus 19.7% of rejections. A correct final diagnosis does not certify a safe trajectory, and a successful correction does not certify a safe recovery.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!