Live intelligence

Latest AI updates

5 updates
Research

LACCO Overlaps Communication and Computation for Distributed Language Model Training

Researchers proposed LACCO, a two-stage algorithm for training language models across low-bandwidth networks. It overlaps global communication with local computation while using globally aggregated updates shared by every replica, avoiding staleness in the outer update. The paper reports convergence guarantees for smooth non-convex objectives and experiments on FineWeb showing gains over Eager and Delayed DiLoCo at scales up to 0.5B parameters and 16 workers.

LACCODiLoCoFineWeb
Tutorial

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

AWS describes how to detect and recover from silent data corruption (SDC) on SageMaker HyperPod by adding three checks to a PyTorch DDP training loop: non-finite gradient checks, ring replay audits, and weight fingerprint comparisons. The post explains how to localize suspect ranks to nodes, contain the corruption, and replace hardware using HyperPod features. The solution was tested with software-injected faults on a 4-node cluster with 32 NVIDIA H100 GPUs, and full code is available in a GitHub repository.

Silent data corruptionSageMaker HyperPodDistributed training
Research

LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

LM-SimBench is a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. It decomposes simulation into functional stages from model compilation to schedule-level aggregation and connects execution representations and cost estimators to measurable evaluation targets. The dataset profiles 23 model settings from 10 dense and Mixture-of-Experts LLM families on an Ascend NPU cluster, yielding 348 feasible configurations across five parallelism dimensions and activation recomputation. Evaluations show the benchmark can support simulator accuracy comparisons, configuration-selection assessment, and prediction-error diagnosis, with matched GPU-NPU experiments validating cross-platform applicability.

LLM training simulationdistributed trainingbenchmarking
Research

DataHelix: CPU-GPU Co-Scheduling for Multimodal Model Training

DataHelix is a metadata-first CPU-GPU co-scheduling data plane designed to reduce pipeline bubbles in multimodal model training. It coordinates preprocessing across data-parallel ranks by treating GPU compute windows as readiness budgets and deciding sample placement while data remains lightweight descriptors. The system combines descriptor-level token-latent-aware batching with cost-gated collective rebalancing to correct residual skew only when transfer cost is justified. Across 11 video and multimodal workloads on 8 Ascend 910B accelerators, it reports a 67.8% lower mean deadline-miss rate, 87.0% lower mean p95 input-wait latency, and 2.72x geometric-mean throughput over rank-local DataLoaders.

multimodal trainingdata schedulingdistributed training
Tutorial

Distributed training with Ray Train DLCs and EFA on EKS

AWS published a walkthrough for distributed PyTorch training on Amazon EKS using the Ray Train Deep Learning Container (DLC), KubeRay, and Elastic Fabric Adapter (EFA). The post demonstrates fine-tuning Qwen2.5-1.5B with FSDP across eight NVIDIA L4 GPUs on two g6.12xlarge instances without building any custom container image. It details the cluster architecture, configuration via environment variables, and deployment scripts that handle capacity discovery and infrastructure setup.

distributed trainingAWS EKSPyTorch