LACCO Overlaps Communication and Computation for Distributed Language Model Training
Researchers proposed LACCO, a two-stage algorithm for training language models across low-bandwidth networks. It overlaps global communication with local computation while using globally aggregated updates shared by ever...