Study Finds Synthetic Pre-Pretraining Offers No Consistent Gain Over Natural-Language Training
A compute-matched study reports that synthetic pre-pretraining (PPT) never outperformed pure natural-language training across four curricula (k-Dyck, neural cellular automata, declarative reference corpora and code) and...