Lara Mini Base: A Multilingual MoE Base Model Trained on 6T Tokens
Lara Mini Base is a 7.2B-parameter mixture-of-experts model with 0.6B active parameters, trained from scratch on 6T tokens spanning English and 44 other languages, covering all 24 official EU languages. It supports 128k-token contexts and is released under the CC BY-NC-SA 4.0 license. The authors report it achieves the highest average score among open base models with up to 2B active parameters on English, multilingual, math, and code benchmarks, with the largest margins on multilingual and math tasks. They also plan to release T-HellaSwag, a machine-translated commonsense reasoning benchmark in 110 language variants.
This is a concrete new model release with a novel multilingual MoE design and benchmark claims relevant to efficient multilingual AI.
Comments & Questions (0)
No comments yet
Be the first to comment!