Research
AltRA-Test: A Statistical Test for Replacing Human Annotators with LLM Judges
Researchers propose AltRA-Test, an alternative annotator test for settings where items are annotated by different subsets of annotators rather than a shared panel. The method uses a bootstrap that resamples items, draws k human annotations per item, and substitutes one with an LLM annotation to form paired all-human and mixed human-LLM panels. A replacement is supported when the mixed panel's inter-annotator agreement is statistically non-inferior to the all-human panel's. The authors establish asymptotic level-alpha consistency and evaluate the test on 18 LLMs and 5 prompting techniques across 20 datasets spanning text, image, audio, and video, finding about 100 items often suffice.
LLM evaluationhuman annotationstatistical testing