Latest AI updates

LLM Token Probabilities as Confidence Estimates in Human Behavior Simulation: A Luxembourgish Next Like Prediction Case Study

This paper investigates whether LLM token probabilities can serve as confidence estimates in human behavior simulation, using a Next Like Prediction task on Luxembourgish news platform data. The authors find that open-weight models (Ministral 3B/8B) are overconfident with in-context learning alone, and a simple statistical baseline outperforms them on both accuracy and calibration. Scaling model size improves accuracy but degrades calibration, while task-specific fine-tuning significantly improves both, reducing Expected Calibration Error from 0.2 to 0.006 and boosting accuracy from 0.536 to 0.758.