arXiv:2610.41486v1 Announce Type: new
Abstract: While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
arxiv
Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines
Researchers introduce a benchmark using 722 official UN Security Council records and five complementary tests to measure nation-level bias in LLMs, including opinion ranking, association sorting, vote simulation, opposition attribution, and framing. They define explicit unbiased reference statuses for each test, including action-based pseudo-baselines derived from observable voting behavior for tests where ground truth is difficult to specify. Benchmarking GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B across the five permanent UNSC members reveals a test-agnostic macro-pattern: outputs lean toward the opposition side for China and Russia and toward the supportive side for the U.K., the U.S., and France. The authors argue that international-relations-facing LLM deployments need auditing with grounded, multi-test benchmarks tied to explicit reference points.
It offers a concrete, multi-test evaluation method for measuring nation-level political bias in LLMs, which is useful for teams auditing models intended for international-relations applications.
Comments & Questions (0)
No comments yet
Be the first to comment!