Live intelligence

Latest AI updates

4 updates
Products

Salesforce advances global model serving with SageMaker AI cross-Region inference routing

AWS details how Salesforce uses the SageMaker AI inference router to serve models across 30+ Regions and multiple AWS accounts. The router sends requests to local endpoints by default and fails over based on recent request outcomes, while residency-tagged traffic stays pinned in-Region. In a failover test, traffic shifted to a remote Region after local errors, with about 78% served remotely, under 100 ms added latency at p50, and a gradual return after recovery.

Amazon SageMaker AISalesforcecross-Region inference routing
Research

RESYNC: Residual Latent-State Synchronization for Deadline-Bound Streaming Inference

RESYNC addresses high-rate streaming inference where a large model may miss per-observation deadlines. The large model runs asynchronously while a fast path updates its stale latent state using newer observations and reuses the unchanged prediction head, learning from frozen-model representations without task labels. Across one real and two synthetic streams and five architecture families, RESYNC outperformed compact or distilled alternatives in all twelve dataset-architecture pairs at anchor ages of 4–12 observations, and in eleven pairs at older ages up to 32. In 1 ms arrival replay, it served at least 99.4% of arrivals within the interval with 0.35–0.62 ms median latency on one CPU thread.

RESYNCstreaming inferencelatency optimization
Research Editor selected

RecoGEM: A Serving System for Production-Scale Generative Recommendation

RecoGEM is a production serving system for generative recommendation, where item identifiers are generated autoregressively instead of scoring candidate items. It uses a runtime-engine decoupling architecture to handle GR-specific responsibilities such as candidate-state transitions and live artifact updates, while the execution engine optimizes self-attention, candidate selection, and execution planning for short decoding sequences and large candidate populations. The system has been deployed for over nine months across production lines serving hundreds of millions of users and more than ten GR model families. On OneModel workloads, it reports over 1,000 QPS per serving instance, P99 latency below 35 ms, and a 2.88x throughput improvement over representative baselines.

Why it matters: It offers concrete architecture and production performance details for serving generative recommendation models at industrial scale.

Generative recommendationModel servingRecommender systems
Products

AWS and NVIDIA detail Nemotron 3.5 Super VL deployment on EKS and SageMaker HyperPod

AWS announced that NVIDIA Nemotron 3.5 Super VL, a 120B-parameter MoE multimodal model, is now available on AWS and detailed its deployment on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. The model activates 12B parameters per token and supports text, image, document, and video understanding with a 1M-token context. The post provides architecture guidance, validation using NCCL benchmarks, and performance evaluation with NVIDIA aiperf, including concurrency sweeps to plan capacity. It also highlights integration with agent frameworks and NVIDIA Dynamo for serving.

NVIDIA Nemotron 3.5 Super VLAWS deploymentmodel serving