Back to AI News
arxiv

One Scale, Four Texts: Gaps, Overlaps and Silent Repairs in Published Grading Schemes

This paper analyzes four published texts of the diabetic retinopathy grading scale, formalizing them into a rule language and exhaustively enumerating 1.4 billion finding profiles. It reveals that every text leaves some eyes uncategorized, the 2017 tabulation has an empty proliferative class and a macular gap absent in 2003, and English and Italian texts order some eyes oppositely. On 927 annotated photographs, the English text and 2017 tabulation disagree on referral for 26.8% of DDR eyes, and classifiers trained on each text's labels side with their own text. A second pre-registered study shows language models often silently repair gaps and overlaps, with Claude models reporting defects more often.

AI Brief

Provides concrete, quantified evidence of inconsistencies in real-world grading schemes, with implications for AI training and deployment in medical imaging.

arXiv:2611.06569v1 Announce Type: new Abstract: Category systems often reach their users as text, and readers apply them with tacit repairs. We ask what such a text determines once the repairs are removed, using diabetic retinopathy grading, where one international scale circulates in several documents. We transcribe four texts (the English screening definitions of 2025, the international scale as tabulated in 2017 and as published in 2003, and an Italian guideline of 2015) into one rule language over 24 finding variables, and make every point a text leaves open a named precisification. Exhaustive enumeration of 1,425,807,360 finding profiles, re-derived by an SMT solver and spot-checked by an OWL reasoner, shows that every text that assigns actions leaves some eyes without a category under every reading; that the 2017 tabulation has a proliferative class that is empty when read literally and a macular gap that the 2003 text lacks, so that no function maps the 2003 grades onto the 2017 grades; and that on image findings alone the English and Italian texts order some pairs of eyes in opposite ways. On 927 annotated photographs from three public datasets, with retinal thickening assumed absent, the English text and the 2017 tabulation disagree on referral for 26.8% of the eyes of one dataset (DDR) on which both decide, and in a pre-registered experiment classifiers trained on each text's labels side with their own text. Thickening cannot be seen on a colour photograph; left free, every disagreement involving England becomes undecidable, so it rests on what the camera cannot show. In a second pre-registered study, language models that answered without deliberating repaired most gap and overlap items even when told to read literally, while Claude Sonnet 5.5 and Claude Opus 5.5 reported the defects far more often; within two models, deliberation reduced silent repair without removing it.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!