Live intelligence

Latest AI updates

5 updates
Products Editor selected

Deploy frontier AI models on AWS Trainium with SGLang-Neuron

Amazon announced public availability of SGLang-Neuron, delivered through an AWS Deep Learning Container and released by RadixArk in collaboration with Annapurna Labs. The out-of-tree plugin lets users serve models and generate RL rollouts on AWS Trainium through the existing SGLang server interface and generation APIs. It runs on native PyTorch, uses NKI kernels compiled with torch.compile, and supports PD disaggregation with KV-cache transfer via NIXL over EFA. The post includes a walkthrough for launching a container and serving openai/gpt-oss-120b on a Trainium3 EC2 instance.

Why it matters: It gives AI engineers a concrete, documented path to run the familiar SGLang serving workflow on AWS Trainium, including deployment steps and supported architecture details.

AWS TrainiumSGLangAI inference
Research

MoE Routing Compared: DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss

This article compares routing mechanisms across 10 open Mixture-of-Experts (MoE) models, detailing expert counts, top-k values, shared experts, gating functions, and load-balancing methods. It highlights trends like growing expert counts (from 8 to 384) and variations in top-k (from 1 to 8). The comparison is based on verified config.json files and technical reports, offering a practical reference for model deployment.

Mixture of ExpertsModel ArchitectureOpen Source Models
Products Editor selected

Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI introduced Together Compiler (TKCC), an agent-driven system that continuously re-optimizes inference as production traffic changes. It uses coding agents to rewrite a model's forward pass)Skip and a persistent engine with hot reload to measure candidates, achieving up to 2.6x higher throughput than best published configurations within 24 hours. The system runs on multiple hardware platforms and has been tested on several models, with gains ranging from 4% to 2.6x depending on the setting.

Why it matters: This is a concrete, recent AI development with measurable performance gains and a novel approach to inference optimization, offering practical value for developers and researchers.

AI inferencecompilationagent-based optimization