This week at a glance4 items
01
Academia
Making TFMs cheaper: distillation into small students, a looped single-block TFM with ~90% fewer parameters, and linear-time attention for long contexts.
02
Academia
Understanding TFMs: sparse-prior architecture study (TabPFN v2 vs TabDPT) and synthetic-data attribution.
03
Academia
TabFM (zero-shot, 400M) and TabFM-Auto lead TabArena; RelICL pushes TFMs onto relational data without training.
04
Industry
NVIDIA released Kumo Tabular, open in-context tabular models with commercial-use weights.
01Academia
arXiv
Distillation of Tabular Foundation Models into Efficient Predictors
Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
Problem
TFMs predict well via in-context learning, but every prediction re-conditions on the labeled data, which is expensive at inference.
Method
Distil two TFMs into small neural or tree students, using the full training set as teacher context and training on teacher predictions for real plus synthetic queries.
Results
On TabArena students beat tuned-and-ensembled supervised models by 57-98 Elo; on TALENT they improve matched default students on 236-258 of 300 datasets. Median speedups 3-22x over the teacher.
Limitations
Students still lose 1.7-2.8% relative error to the teacher; gains shrink below ~1K context rows; synthetic query generators are simple and the 2:1 synthetic ratio is not tuned.
arXiv
Architecture Alignment With Sparse Priors in Tabular Foundation Models
Tianqi Zhao, Tianyi Zhuang, Shuo Duan, Guanyang Wang, Yan Shuo Tan, Qiong Zhang
Problem
Released TFMs differ in priors, architecture and objectives at once, so which inductive biases come from architecture is unclear.
Method
Compare irrelevant-feature suppression in row-token (TabDPT) vs cell-token alternating-axis (TabPFN v2) models, then train both under identical sparse-to-dense linear priors with exact Bayes analysis and interventions.
Results
With many null features TabDPT's R² drops ~46x more than TabPFN v2's on Friedman tasks. In controlled training the alternating-axis model has clearly lower excess error under sparse priors; the advantage vanishes under dense priors.
Limitations
Theory assumes Gaussian queries and a linear response; the advantage is not shown necessary or sufficient; the released-model comparison is observational; interventions show dependence but do not locate a circuit.
arXiv
Adapting Linear-Time Architectures for Tabular In-Context Learning
David Schnurr, Felix Sarnthein, Thomas Hofmann, Imanol Schlag
Problem
Softmax attention limits tabular ICL on large datasets; linear-time alternatives are mostly causal and little studied for tabular ICL.
Method
Compare linear sequence mixers in a controlled tabular ICL setup; DeltaNet wins, and a Final-State DeltaNet with write-rate decay is proposed to address recurrent-state drift beyond the pretraining context length.
Results
DeltaNet beats non-causal linear attention but degrades past 2-4x the pretraining context. The proposed variant nearly closes the gap to softmax on TabArena (ROC-AUC 0.814 vs 0.817). A state oracle shows capacity is not the issue.
Limitations
Small 12.5M-parameter models trained on ~6.4B rows versus 40B+ for TabPFN v2.5; row-wise architecture only (two-axis attention is expected, not shown, to transfer); the length-generalisation problem is mitigated, not solved.
arXiv
LoopICL: Looping a single transformer block to solve tabular tasks
Amir Rezaei Balef, Katharina Eggensperger
Problem
Recent mechanistic work suggests TFM parameters are largely redundant, so TFMs may be larger than needed.
Method
LoopICL is a single transformer block with cell and row streams, looped a variable number of times with a learned exit gate, decoupling parameters from depth.
Results
Matches TabICLv2 on TabArena and TALENT at equal FLOPs with nearly 90% fewer parameters. Early exit is faster on 33 of 38 TabArena datasets.
Limitations
Classification only (regression left for future work); early exit is erratic, with up to ~20% relative error increase on some TALENT datasets; sensitive to residual-scaling and exit-threshold tuning.
arXiv
Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shift
Seonghwi Kim, Sung Ho Jo, Minwoo Chae
Problem
TFMs can perform poorly on underrepresented groups under subpopulation shift, even with good mean accuracy.
Method
DR-TFM fine-tunes (or adds) only the query-scaling network that weights attention over context examples, with distributionally robust objectives using estimated groups, so no group labels are needed.
Results
Updates 0.016% of TabPFN-3 parameters. Average worst-group accuracy is 72.6% vs ~48% for the pretrained model across five benchmarks, with similar mean accuracy; 9x faster and 8.6x less memory than full fine-tuning.
Limitations
Likely: depends on clustering quality and cluster count to estimate groups; no limitations section; covers subpopulation shift only, not other shifts.
arXiv
TabFM: A Zero-Shot Foundation Model for Tabular Data
Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, Abhimanyu Das
Problem
Tabular ML usually means fitting trees or running AutoML per dataset; a single zero-shot model would avoid that.
Method
TabFM is a 400M-parameter in-context learner pretrained only on synthetic structural-causal-model tables, with optional ensembling/calibration (TabFM+) and LLM-guided feature engineering (TabFM-Auto).
Results
Zero-shot, it ranks first among default TFMs on TabArena (51 datasets) and beats tuned AutoML; TabFM+ adds ~69 (clf) / 134 (reg) Elo and TabFM-Auto ~172 / 337 Elo.
Limitations
Pretrained to at most 16,384 rows and 100 columns; numeric and categorical features only (no free text, timestamps, high-cardinality IDs); single flat tables, no relational input.
arXiv
From Behavior to Provenance: Attributing Tabular Foundation Models to Synthetic Pretraining Data
Mohamed Bouadi, Nassim Bouarour, Shivam Dubey, Aditya Tanna, Vinay Kumar Sankarapu
Problem
Attributing model behaviour to training data is hard to validate because causal influence is rarely observable.
Method
Use O'PRIOR, a provenance-tagged synthetic task generator, so each pretraining task carries known lineage; apply influence/TracIn attribution plus counterfactual retraining of a TFM.
Results
Removing the top-attributed 5% of tasks lowers held-out ROC-AUC by 0.013 vs 0.002 for random removal. Provenance discrimination is modest (~0.60 ROC-AUC).
Limitations
Only 5,000 tasks and a NanoTabPFN-scale model; results may change at scale; mechanisms co-occur, so enrichment is association rather than isolating a single mechanism; surrogate loss for non-differentiable metrics.
arXiv
RelICL: Training-free Relational Learning with Tabular Foundation Models
Simon Forbat, Rainer Gemulla
Problem
Applying TFMs to relational data via deep feature synthesis (DFS) suffers feature explosion and blindness to feature interactions.
Method
RelICL propagates and fuses information step by step along the schema graph, reusing the same frozen TFM at each step, without training.
Results
Beats the strongest DFS baseline (TabPFN-Rel) on 11 of 21 tasks and Relational Transformers on 11 of 18. Near-perfect AUROC on synthetic tasks where DFS sits near chance.
Limitations
Regression is mixed (e.g. R² of -22% on site-success); baseline numbers copied from papers, not re-run; wins may not be significant; DFS still does well on RelBench.
arXiv
Conformal Prediction and Conditional Coverage for Tabular Foundation Models
Sungwoo Park, Sunghee Park, Won Chang
Problem
TFM predictive distributions for regression can under- or over-cover even when point predictions are accurate.
Method
C-USIM applies highest-predictive-density split conformal prediction to TFM outputs, handling multimodal predictions without extra training or inference.
Results
With TabPFN and TabICL, marginal coverage is more accurate and conditional/group coverage error lower than plug-in HPD. Giving more of a fixed budget to calibration can help despite weaker point predictions.
Limitations
Guarantees rely on assumptions that TFMs only approximately meet (test-set invariance, continuous scores vs piecewise-constant densities); univariate regression with two TFMs only.
arXiv
TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models
Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong
Problem
TFMs ignore column names, task descriptions and metadata, while LLM agents that search pipelines from scratch overfit and are noisy.
Method
TabFM-Auto uses an LLM agent to evolve cleaning, feature engineering, context selection and post-processing around a frozen TFM, guided by metadata and validation feedback.
Results
Top five TabArena positions; best config raises TabFM from 1785 to 2013 Elo. Pipelines transfer to other frozen TFMs (+69 to +143 Elo).
Limitations
Gains are smaller on anonymized tables; about 6 GPU-hours of search per dataset; TabArena-validation-driven search may overfit; the TabArena comparison involves a model from the same paper.
02Industry
NVIDIA · 2026-09-29
Three pretrained sizes (28M-215M parameters), code under Apache 2.0, weights on Hugging Face under a commercial-use licence. NVIDIA reports 1,950 TabArena Elo from its own run (not independently verified). Pretrained up to 60K rows and 100 columns, numeric and categorical only.