Research
Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation
Researchers introduce a counterfactual audit that reassigns category definitions among output labels to test whether LLM annotations follow supplied definitions or preserve a dataset's original label mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably improving definition adherence. In retrieval-augmented annotation, retrieving nearest training examples generally beats within-category retrieval despite using fewer demonstrations; on two tasks, direct k-nearest-neighbor classification recovers 74–93% of the improvement, and aligning demonstrations with reassigned definitions increases adherence to the new mapping.
LLM annotationsocial science textretrieval augmentation