Back to AI News
arxiv

PolyRoute Benchmark Tests Typed-Decision Routing Across 12 Languages and Code-Mixed Text

PolyRoute is a new benchmark of 322 native-language support messages in 12 languages and 9 writing systems, plus 156 code-mixed or romanised messages such as Hinglish, Spanglish and Arabizi, for a three-way department-routing decision. Labels were produced by Qwen3.8-27B with a blind second pass, and an independent model agreed on 97.2% of test items. A leaderboard of 13 systems reports 61.2% for the English-root checkpoint, 77.3% for the shipped multilingual checkpoint and 96.6% for the best fine-tune, while TF-IDF with logistic regression reaches 91.9%. The authors note that the split resolves differences only above about 4 points and that all texts and labels are LLM-generated.

AI Brief

It provides a concrete multilingual and code-mixed evaluation benchmark with baselines and a leaderboard, useful for teams measuring typed-decision routing outside English.

arXiv:2611.13843v1 Announce Type: cross Abstract: Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish, Arabizi), 967 optional native training rows, and a protocol for a three-way department-routing decision. Labels come from Qwen3.8-27B with a blind second pass (91% of candidates kept); an independent model, gemma4-31b, agrees on 97.2% of test items, and the largest character 5-gram similarity between any test text and any training text is 0.33. A leaderboard of 13 systems gives 61.2% for the English-root checkpoint, 77.3% for the shipped multilingual one and 96.6% for the best fine-tune; TF-IDF with logistic regression reaches 91.9%. On four withheld languages the three multilingual systems score 77%, 90% and 93%. On code-mixed text the English root (85%) is nominally ahead of the shipped multilingual checkpoint (79%; p=0.134). The split resolves differences only above about 4 points. All texts and labels are LLM-generated.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!