LACCO Overlaps Communication and Computation for Distributed Language Model Training
Researchers proposed LACCO, a two-stage algorithm for training language models across low-bandwidth networks. It overlaps global communication with local computation while using globally aggregated updates shared by every replica, avoiding staleness in the outer update. The paper reports convergence guarantees for smooth non-convex objectives and experiments on FineWeb showing gains over Eager and Delayed DiLoCo at scales up to 0.5B parameters and 16 workers.