Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
Researchers introduce a counterfactual audit that reassigns category definitions among output labels to test whether LLM annotations follow supplied definitions or preserve a dataset's original label mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably improving definition adherence. In retrieval-augmented annotation, retrieving nearest training examples generally beats within-category retrieval despite using fewer demonstrations; on two tasks, direct k-nearest-neighbor classification recovers 74–93% of the improvement, and aligning demonstrations with reassigned definitions increases adherence to the new mapping.
It provides a concrete audit method and evidence that high annotation accuracy may reflect label or demonstration cues rather than adherence to researcher definitions.
Comments & Questions (0)
No comments yet
Be the first to comment!