Live intelligence

Latest AI updates

2 updates

Research

Scaling Laws for SAE Training Data

This paper investigates how the number of unique activations affects sparse autoencoder (SAE) training when the total training budget is fixed. It finds that quality improves rapidly with buffer size but then plateaus, where replaying the buffer is nearly as effective as using fresh activations. The authors propose a scaling law for limited data and validate it across various models, layers, and SAE variants. They provide a practical rule: choose the smallest buffer that reaches the quality plateau and replay it, reducing activation storage by 8-64x with negligible quality loss.

arxiv1d agoScore 77

sparse autoencodersscaling lawsmodel interpretability

Research Editor selected

When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

A new arXiv paper decomposes the effects of quantization and pruning on language models, separating direct damage from network compensation. The authors find that for comparable damage, smaller models restore most of the lost performance, while larger models retain more loss because compensation offsets less. This implies that compression that is nearly harmless in small models can become costly in large ones, challenging the assumption that scaling alone mitigates compression side effects.

Why it matters: Provides a novel, empirically grounded decomposition of compression damage versus compensation across model scales, with direct implications for efficient deployment of large language models.

arxiv1d agoScore 88

model compressionscaling lawsattention mechanisms