arXiv
TabPFN-3.5: Technical Report
Benjamin Jäger, Nick Erickson, Léo Grinsztajn, Felix Birkel, Klemens Flöge, Oscar Key, Kürşat Kaya, Jonas Kübler, Adèle Frankel, Tobias Schröder, Anurag Garg, Jan Hendrik Metzen, David Salinas, Simon Bing, Kristina Collins, Tuana Çelik, Vahid Balazadeh, Lydia Sidhoum, Tomás Pereda, Brendan Roof, Andrej Tschalzev, Siyuan Guo, Philipp Singer, Lennart Purucker, Jake Robertson, Marie Salmon, Philipp Jund, Jerry Chen, Diana Kriuchkova, Arthur Cahu, Eliott Kalfon, Adrian Hayler, Georg Grab, Vitor Monteiro, Lilly Wehrhahn, Dominik Safaric, Clara Cornu, Alan Arazi, Rylee Grace, Simone Alessi, Mihir Manium, Bernhard Schölkopf, Yann LeCun, Madelon Hulsebos, Sauraj Gambhir, Noah Hollmann, Frank Hutter
Problem
Tabular foundation models need to keep improving to handle messy real-world tables (temporal splits, text, high-cardinality categories) beyond clean benchmark data.
Method
TabPFN-3.5 is Prior Labs' new flagship TFM, with Fast/Plus/Thinking variants and one multitask checkpoint spanning classification, regression, relational, and time-series tasks.
Results
Ranks #1 on TabArena (Elo 1910 Thinking, 1866 base) and BeyondArena, beating TabPFN-3 (p<0.0001) and AutoGluon 1.6; Fast is up to 3x quicker, Thinking up to 12x faster than its predecessor.
Limitations
Base model runs up to 2x slower than TabPFN-3 on large training sets; tuned MLPs still lead on very large (>100K row) grouped/temporal data; no dedicated time-series finetuning (6th on fev-bench).
arXiv
LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence
Xingxuan Zhang, Gang Ren, Hao Yuan, Hao Zou, Hongze Tan, Hui Wang, Jianhao Song, Jiansheng Li, Jiayao Zhang, Jinghan Zhang, Kaifang Li, Lang Mo, Li Mao, Mingchao Hao, Nuo Xu, Rui Ding, Ruiji Zhang, Shuyang Li, Siyu Mei, Tianyang Zhang, Weiyang Mu, Yancheng Dong, Yongxian Wei, Yuan Xue, Yuanrui Wang, Yue He, Zijia Yang, Ziyun Li, Dongzhe Li, Fuqiang Wang, Jiandong Liu, Jiawei Chen, Jiaxin Du, Kaijie Cheng, Kehan Li, Lei Sun, Linjun Zhou, Ningbo Dai, Qi Wang, Renzhe Xu, Shaoxing Du, Shumeng Yang, Wang Lu, Wenjing Chu, Xiannan Huang, Xiaoyu Lin, Xing Ai, Xinyan Han, Xuanyue Li, Xuanyue Su, Xukun Zhang, Yan Lu, Yaxin Zhang, Yi Qin, Yifei Huang, Yihan Xu, Yongle Lv, Yuanyuan Jiang, Yushan Han, Peng Cui
Problem
Conventional tabular PFNs model p(y|x, context) directly, which may miss the underlying causal/generative structure that could make predictions more robust and interpretable.
Method
LimiX-2 reframes in-context learning as joint modeling p(x,y|context) via "Contextual Mechanism Networks," pretrained on synthetic data generated from structural causal models.
Results
Outperforms TabPFN-3 on TabArena (Elo 1935 vs 1632), TALENT (1506 vs 1363), and BCCO (1432 vs 1295), and its feature attention recovers causal skeletons from the data.
Limitations
Likely: relies entirely on synthetic SCM-generated pretraining, so behavior on real causal structures outside that prior is less tested; causal-recovery claims read more as forecasts than fully measured results.
arXiv
LLMs as Feature Engineers for Text-and-Tabular Prediction
Merwan Barlier, Blaz Skrlj
Problem
Free-text fields in tabular data (reviews, descriptions) hold predictive signal that standard tabular models can't easily turn into usable features.
Method
A generator LLM proposes candidate text-derived features, an extractor LLM materializes them, and prediction errors (e.g. AUC ranking inversions) become natural-language feedback that steers further search.
Results
Across Kickstarter, Amazon Books, and Stack Overflow datasets, LLM features combined with TF-IDF/dense embeddings substantially lift AUC (e.g. 0.854→0.970 on Stack Overflow) while cutting search iterations up to 5x.
Limitations
Relies on a proprietary LLM (GPT-5.4), limiting reproducibility with open models; evaluated only on English text; deployment adds real inference latency (30-50%) and CPU overhead.
arXiv
Distilling Foundation Models for Agentic What-If Reasoning:Cost, Latency, and Governance in a Hybrid LLM+SLM Architecture
Sourish Dey, Aditya Kumar
Problem
TabPFN's in-context learning gives strong zero-training accuracy, but its inference latency is too high to serve as a hot-path decision backend inside interactive agent loops.
Method
Distills a TabPFN teacher into a tiny feed-forward student network via soft-label knowledge distillation, tested on tabular classification and regression tasks.
Results
Compresses a 53.2M-parameter head to 8,546 parameters (6,220x) while retaining 95.4-100.5% accuracy and 96.8-100% AUC across UCI Adult and OpenML benchmarks; soft targets add 2.1-7.0 AUC points over hard labels.
Limitations
Tested on only 11 small OpenML tasks with a single random seed; its own 50-query what-if benchmark is underpowered, and regression labels are rule-simulated rather than real ground truth.
arXiv
Agentic Search Spaces for Tabular Machine Learning
Renat Sergazinov, Artem Chistyakov, Sergey Pankevich, Artem Babenko
Problem
Standard hyperparameter-optimization search spaces for tabular models are hand-designed by their authors and may not be well-suited to a given task.
Method
LLM agents propose new candidate implementations for each pipeline module (preprocessing, embeddings, architecture, training), jointly tuned with classical HPO alongside default hyperparameters.
Results
Expanded search spaces improve nearly every model family across 45 datasets (avg. +0.6%, up to +2.0% on small/medium regression) at no extra tuning budget, lifting 4-of-5 families' TabArena Elo past AutoGluon's ensemble.
Limitations
Gains are modest and problem-dependent; larger spaces risk overfitting to validation data, fit-time can rise up to 2.4x, and agent-proposed code needs manual vetting for bugs.
arXiv
Benchmarking Tabular Foundation Models as Surrogates in Expensive Evolutionary Optimization
Lu Han, Jin Wang, Yuchen Li, Haoran Gu, Shulei Liu, Ziyang Shi, Wenao Lu, Handing Wang
Problem
Surrogate-assisted evolutionary algorithms need cheap, accurate fitness predictors, and it's unclear whether tabular foundation models like TabPFN are a good universal choice.
Method
Systematically benchmarks TabPFN as a surrogate across offline/online settings and single-, multi-objective, constrained, combinatorial, and mixed-variable optimization problems, versus GPs and RBFNs.
Results
TabPFN dominates on many low/medium-dimensional problems (e.g. 82 vs 5 wins over GPs in one setting), but its conservative, mean-anchored extrapolation fails on 1000D and some constrained problems (36% vs 100% success rate).
Limitations
Effectiveness is highly problem-dependent; the paper explicitly states TabPFN "cannot replace conventional surrogates universally" and needs role-specific algorithm design to avoid its pitfalls.
arXiv
MetaRTL: Meta-path Attention Enhanced Relational Table Learning
Ken Zhong, Weichen Li, Zheng Wang
Problem
Deep GNN/HGNN stacks for learning over relational databases are computationally expensive and scale poorly to large real-world multi-table data.
Method
A two-stage framework that pretrains lightweight table embeddings, then replaces deep message passing with non-parametric meta-path feature extraction aggregated by an attention module (MetaAttn).
Results
Outperforms RDL, RelGNN, BRIDGE, and LightRDL on SJTUTables and RelBench (best on 18 of 21 tasks, e.g. 0.8415 vs 0.7723 ROC-AUC on rel-f1), with training time scaling roughly linearly in graph size.
Limitations
Gains shrink or even reverse on simpler datasets (e.g. rel-hm); performance is sensitive to pretraining duration, and the pretraining stage's own cost still grows with graph size.
arXiv
Tables Decoded: DELTA for Structure, TARQA for Understanding
Jahanvi Rajput, Dhruv Kudale, Saikiran Kasturi, Utkarsh Verma, Ganesh Ramakrishnan
Problem
Table structure recognition and table QA usually rely on vision-language models processing table images, which is costly and hard to extend to multilingual documents.
Method
DELTA separates layout/OCR/logical structure recognition and outputs a compact textual format (OTSL); TARQA is an LLM fine-tuned on OTSL sequences for downstream table QA.
Results
DELTA matches state-of-the-art TEDS-Structure scores (97.5-98.2) on FinTabNet/PubTabNet/PubTables-1M and 85.4 on a new Hindi benchmark; TARQA gains +9.3pp on WTQ and +9.2pp on FinTabNetQA.
Limitations
OCR is the main error bottleneck (ground-truth text pushes TEDS above 90%); the method struggles with abstractive/comparative QA and degrades on noisy scans, slanted text, and low-resolution input.