AWS details fused Mixture-of-Experts kernel optimizations for SageMaker AI and vLLM
AWS described three kernel-level optimizations for fused Mixture-of-Experts inference on NVIDIA Hopper: a Swap-AB transposition to improve WGMMA utilization, routing FP8 tensor-quantized MoE to the Triton backend, and el...