Live intelligence

Latest AI updates

2 updates

Research Editor selected

AWS details fused Mixture-of-Experts kernel optimizations for SageMaker AI and vLLM

AWS described three kernel-level optimizations for fused Mixture-of-Experts inference on NVIDIA Hopper: a Swap-AB transposition to improve WGMMA utilization, routing FP8 tensor-quantized MoE to the Triton backend, and eliminating Triton JIT recompilations with do_not_specialize annotations. The post reports up to 10% higher end-to-end throughput and up to 9% faster Time to First Token on H200. The optimizations were upstreamed to vLLM and are used in the AWS LMI v28 container on Amazon SageMaker

Why it matters: Provides practical, open-source GPU kernel techniques and measured latency and throughput gains for teams deploying MoE models on Hopper and SageMaker.

AWS Machine Learning Blog5d agoScore 82

Mixture-of-ExpertsAmazon SageMaker AIvLLM

Tutorial

Deploy decision models with llama.cpp on Amazon SageMaker AI

The AWS Machine Learning Blog details serving quantized GGUF models with the AWS Deep Learning Container for llama.cpp on Amazon SageMaker AI, exposing OpenAI-compatible chat completions through InvokeEndpoint. It then extends the runtime with a SemIf child image that returns probabilities over caller-declared options using last-position logits, without generating answer tokens. A provided code sample covers CPU and GPU deployment paths, including CodeBuild/ECR image builds and SageMaker instanc

AWS Machine Learning Blog6d agoScore 76

Amazon SageMaker AIllama.cppSemIf