Temporal variability score shows high-variability knowledge improves faster late in pretraining
Researchers define a temporal variability score (TVS) as the coefficient of variation of an item’s negative log-likelihood across point-in-time language models trained on web snapshots from different years. Using six ben...