AWS details fused Mixture-of-Experts kernel optimizations for SageMaker AI and vLLM
AWS described three kernel-level optimizations for fused Mixture-of-Experts inference on NVIDIA Hopper: a Swap-AB transposition to improve WGMMA utilization, routing FP8 tensor-quantized MoE to the Triton backend, and eliminating Triton JIT recompilations with do_not_specialize annotations. The post reports up to 10% higher end-to-end throughput and up to 9% faster Time to First Token on H200. The optimizations were upstreamed to vLLM and are used in the AWS LMI v28 container on Amazon SageMaker
Why it matters: Provides practical, open-source GPU kernel techniques and measured latency and throughput gains for teams deploying MoE models on Hopper and SageMaker.