Beam Technical Report Details 501B-Parameter Sparse MoE Built for Coding and Agentic Workloads
A new arXiv technical report describes Beam, a sparse Mixture-of-Experts model with 501.3B total parameters and 23.4B active parameters, targeted at coding, reasoning and agentic workloads. The model was pretrained on 23.8T tokens and reaches a 1M-token context window via a two-stage midtraining phase built on verifiable, correctness-anchored data. Post-training used a fully asynchronous reinforcement learning campaign across 10.5K GB300 GPUs, generating over 100M rollouts in four weeks, with reported continued gains as RL compute scaled. Serving efficiency comes from hardware-model co-design, FP8/NVFP4 quantization and a 1.93B-parameter speculative decoding drafter (DFlash 2).
It gives concrete architecture, training-scale and deployment details for a large open-weight-class MoE model aimed at coding and agent workloads, which is useful for readers tracking frontier model engineering.
Comments & Questions (0)
No comments yet
Be the first to comment!