Back to AI News
arxiv

Towards Reliable Scaling Laws for Tabular Foundation Models

The paper investigates whether scaling laws for TabPFNv2-style tabular foundation models reliably extrapolate to larger compute budgets and generalize across evaluation settings. It finds differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. Selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates, and compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. The insights are used to guide training of more efficient TabPFNv2-like models with the open-source nanoTabPFN pipeline.

AI Brief

Provides practical guidance for researchers training tabular foundation models, highlighting pitfalls in scaling-law extrapolation and compute allocation.

arXiv:2611.24214v1 Announce Type: cross Abstract: Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!