Towards Reliable Scaling Laws for Tabular Foundation Models
The paper investigates whether scaling laws for TabPFNv2-style tabular foundation models reliably extrapolate to larger compute budgets and generalize across evaluation settings. It finds differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. Selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates, and compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. The insights are used to guide training of more efficient TabPFNv2-like models with the open-source nanoTabPFN pipeline.
Provides practical guidance for researchers training tabular foundation models, highlighting pitfalls in scaling-law extrapolation and compute allocation.
Comments & Questions (0)
No comments yet
Be the first to comment!