Live intelligence

Latest AI updates

3 updates

Research

Scaling Laws for SAE Training Data

This paper investigates how the number of unique activations affects sparse autoencoder (SAE) training when the total training budget is fixed. It finds that quality improves rapidly with buffer size but then plateaus, where replaying the buffer is nearly as effective as using fresh activations. The authors propose a scaling law for limited data and validate it across various models, layers, and SAE variants. They provide a practical rule: choose the smallest buffer that reaches the quality plateau and replay it, reducing activation storage by 8-64x with negligible quality loss.

arxiv1d agoScore 77

sparse autoencodersscaling lawsmodel interpretability

Research

OT-CAL uses optimal transport to adaptively select training samples and cut backprop cost

Researchers introduce OT-CAL, a data selection method that adjusts which samples receive backward passes as training progresses. Each sample is described by five informativeness features such as difficulty, learnability and uncertainty, and optimal-transport matching assigns selection weights against a curriculum vector that is updated from the selected samples, removing the need for a manual schedule. On three standard visual learning benchmarks, selecting 50% of each mini-batch improved results by 1.3-2.1 points over AdaSelection while training 1.18-1.24 times faster than full-data training.

arxiv5d agoScore 68

optimal transportdata selectiontraining efficiency

Research Editor selected

Turing MoE Study Reports Strong Specialist Integration and 2.86× Faster Consolidation

A new paper combines calibrated multi-teacher on-policy distillation with execution optimizations for dynamically routed mixture-of-experts models. On Turing-20B-A2B, the authors report retaining at least 90.16% of measured specialist gains in every evaluated domain and outperforming shared supervised fine-tuning and Qwen3-8B on four reasoning-enabled mathematics and code benchmarks at a matched response cap. A PPU-based profile shows consolidation step time falling from 398.7 to 139.3 seconds, with the results attributed to the complete training and execution framework rather than a single component.

Why it matters: The study offers concrete evaluation and execution results for researchers seeking to combine specialist models while reducing MoE post-training costs.

arxivOct 30, 2026Score 86

Turing-20B-A2BMixture-of-expertsKnowledge distillation