arXiv:2611.17991v1 Announce Type: new
Abstract: Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation-communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift. We propose LACCO (Local Accumulation with Computation-Communication Overlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update. We establish convergence guarantees for smooth non-convex objectives. Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 0.5B parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible-or fully overlappable-computational or memory overhead, which demonstrates the strength of LACCO.
arxiv
LACCO Overlaps Communication and Computation for Distributed Language Model Training
Researchers proposed LACCO, a two-stage algorithm for training language models across low-bandwidth networks. It overlaps global communication with local computation while using globally aggregated updates shared by every replica, avoiding staleness in the outer update. The paper reports convergence guarantees for smooth non-convex objectives and experiments on FineWeb showing gains over Eager and Delayed DiLoCo at scales up to 0.5B parameters and 16 workers.
It gives readers a concrete method for reducing synchronization overhead in distributed LLM training without degrading convergence.
Comments & Questions (0)
No comments yet
Be the first to comment!