Back to AI News
arxiv

Beam Technical Report Details 501B-Parameter Sparse MoE Built for Coding and Agentic Workloads

A new arXiv technical report describes Beam, a sparse Mixture-of-Experts model with 501.3B total parameters and 23.4B active parameters, targeted at coding, reasoning and agentic workloads. The model was pretrained on 23.8T tokens and reaches a 1M-token context window via a two-stage midtraining phase built on verifiable, correctness-anchored data. Post-training used a fully asynchronous reinforcement learning campaign across 10.5K GB300 GPUs, generating over 100M rollouts in four weeks, with reported continued gains as RL compute scaled. Serving efficiency comes from hardware-model co-design, FP8/NVFP4 quantization and a 1.93B-parameter speculative decoding drafter (DFlash 2).

AI Brief

It gives concrete architecture, training-scale and deployment details for a large open-weight-class MoE model aimed at coding and agent workloads, which is useful for readers tracking frontier model engineering.

arXiv:2611.26358v1 Announce Type: cross Abstract: Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!