Live intelligence

Latest AI updates

2 updates

Research

LLM Token Probabilities as Confidence Estimates in Human Behavior Simulation: A Luxembourgish Next Like Prediction Case Study

This paper investigates whether LLM token probabilities can serve as confidence estimates in human behavior simulation, using a Next Like Prediction task on Luxembourgish news platform data. The authors find that open-weight models (Ministral 3B/8B) are overconfident with in-context learning alone, and a simple statistical baseline outperforms them on both accuracy and calibration. Scaling model size improves accuracy but degrades calibration, while task-specific fine-tuning significantly improves both, reducing Expected Calibration Error from 0.2 to 0.006 and boosting accuracy from 0.536 to 0.758.

arxiv1d agoScore 70

LLM calibrationhuman behavior simulationtoken probabilities

Research

Study Finds Prompted Linguistic Judgments Can Outperform Token-Probability Tests

A study compares explicit prompting with token-probability methods for evaluating language models’ understanding of how emphasis changes sentence meaning. Token probabilities perform above chance across all tested models, but prompting outperforms them by a wide margin in some larger models. Replacing conventional emphasis markers with randomly generated ones reduces token-probability performance more than prompting performance, suggesting that prompting can be more robust for stylistically unusual test data.

arxivOct 30, 2026Score 77

Language model evaluationLinguistic competencePrompting