arXiv
Tydra: An Efficient Hybrid Model for Tabular Data
Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting
Problem
TabPFN-style tabular in-context learners get strong accuracy but pay quadratic attention cost in context length, hurting scaling to larger datasets and local, resource-constrained deployment.
Method
Tydra interleaves TabPFN's transformer blocks with bidirectional Hydra SSM layers (quasiseparable matrix mixing, linear time) inside one tabular ICL model, mixing quadratic attention and subquadratic SSM layers rather than replacing attention.
Results
On 30 OpenML datasets, an 8-layer Tydra (4 attention+SSM pairs) matches TabPFN's AUROC (88.61 vs 88.90, mean |ΔAUROC|=0.006) while cutting inference time ~25-30% (19.99ms vs 27.12ms), and beats a ~10x larger pure-Hydra model.
Limitations
Paper states Tydra's speed edge shrinks with context length, falling to roughly a third of Hydra-16M's speed at 2^15 samples; evaluated only on small OpenML tasks (≤2000 rows, ≤100 features, ≤10 classes).
arXiv
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun
Problem
TFMs (TabPFN, TabICL, TabDPT) need the full labeled context table at every inference call, making latency and memory scale badly and blocking CPU/large-scale deployment.
Method
Two-stage distillation: Stage 1 queries a frozen TFM teacher on synthetic covariates (from TabPFGen/CTGAN/TabDiff/Copula), discarding synthetic labels, training the student only on soft targets to expand query coverage; Stage 2 re-anchors on real data via out-of-fold teacher predictions mixed with true labels, avoiding self-labeling leakage.
Results
On TALENT/TabArena, GEAR MLPs beat supervised MLPs by 1.81-2.00 AUC pts (binary) and 1.19-1.35 (multiclass), beat real-only distillation by up to 2.4 pts, and (with LightGBM/XGBoost too) beat CatBoost, while cutting inference time 57-2866x and peak memory 1.9-3.3x.
Limitations
Paper notes: gains are generator- and student-dependent (weak generators like Copula can underperform real-only distillation; LightGBM stays below supervised on multiclass); only classification and single-teacher distillation studied, with regression/multi-teacher left to future work.
arXiv
Understanding the Surprising Generalization Properties of Tabular Foundation Models
Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini
Problem
Explains what actually drives cross-dataset transfer in tabular foundation models (TabPFN-style ICL) — strong generalization is observed empirically, but which pretraining-corpus properties and mechanisms cause it is poorly understood.
Method
Pretrain TabPFN/TabDPT-style transformers on single real tables (not large multi-dataset corpora) and probe generalization via feature/row ablations, dataset- vs column-level corpus preprocessing, and retrieval/attention-similarity analyses across 88 single-table models and a 1,732-dataset corpus.
Results
A model pretrained on just one table (e.g. Colleges) reaches 0.918 AUC on CC-18; feature count, not row count, predicts transfer (XGBoost R²=0.67); retrieval-AUC correlates 0.89 with generalization AUC; column-level preprocessing beats dataset-level filtering (0.9763 vs 0.9757 AUC).
Limitations
Authors couldn't rule out generalization mechanisms beyond learned retrieval; results may not fully extend to purely synthetic-data pretraining, and apply only to row-based in-context attention architectures, not cell-based TFM variants used by some competitors.
arXiv
Localized TabICLv2: Scaling Tabular In-Context Learning through k-NN
Beimnet Bekele Guta
Problem
TabICLv2's Stage-3 in-context attention scales with the full training set, making inference costly on large tabular datasets and limiting practical deployment as training size grows.
Method
Cache Stage-2 row embeddings, retrieve each test point's k nearest training neighbors by cosine similarity in that learned space, run Stage-3 ICL only on the neighbors, then jointly fine-tune Stages 2-3 on synthetic data to recover accuracy.
Results
On 38 TabArena classification datasets, the fine-tuned localized model keeps 98.64% of full TabICLv2 accuracy (0.827 vs 0.855 ROC AUC), gives 2.18x median batch speedup and ~249x single-query speedup, and beats kNN-only/XGBoost-on-neighbors baselines on 37/38 datasets.
Limitations
Paper states localization underperforms when relevant evidence falls outside the retrieved k neighbors, retrieval overhead can erode speedups on small datasets or large query batches, and speedup is non-monotonic across datasets; regression tasks and LoCalPFN comparison left for future work.
arXiv
Advancing Open and Reproducible Relational Learning: RelArena-$α$, TabPFN-Rel and RPI
Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, Jonas Kübler, Siyuan Guo, Oscar Key, Jan Hendrik Metzen, Rylee Grace, David Salinas, Arthur Cahu, Simon Bing, Benjamin Jäger, Tuana Çelik, Mihir Manium, Vitor Monteiro, Jake Robertson, Jerry Chen, Eliott Kalfon, Tomás Pereda, Lilly Wehrhahn, Dominik Safaric, Tobias Schroeder, Georg Grab, Diana Kriuchkova, Clara Cornu, Philipp Singer, Nick Erickson, Vahid Balazadeh, Marie Salmon, Simone Alessi, Kürşat Kaya, Philipp Jund, Léo Grinsztajn, Yann LeCun, Bernhard Schölkopf, Madelon Hulsebos, Lennart Purucker, Sauraj Gambhir, Frank Hutter, Noah Hollmann
Problem
Relational learning (predicting over multi-table databases) lacks a standardized, reproducible benchmark with consistent data loading, tuning, and evaluation, making method comparisons across GNN and tabular approaches unreliable.
Method
TabPFN-Rel flattens a relational DB into one table via deep feature synthesis over foreign-key paths, then runs TabPFN-3 in-context; RelArena-α standardizes eval on 21 RelBench v1 tasks; RPI adds a YAML-based, code-free interface for custom databases.
Results
On RelArena-α's Elo leaderboard, TabPFN-Rel (API) ranks first among model submissions (~1821 Elo), beating GraphSAGE, RelGT and RelGNN; dropping text features cuts it to ~1706. System submission RT-PluRel scores highest overall but runs ~5x slower.
Limitations
Explicitly an alpha release: single-seed results with no confidence intervals, uneven tuning budgets across methods, a temporal-leakage bug found in one task, entity-level-forecasting only, missing baselines, and a stated conflict of interest since the authors release both the benchmark and its top method.
arXiv
ARASH: Adaptive Retrieval And Shot Selection for Tabular Prediction
Samirasadat Jamalidinan, Yue Xu, Kazem Cheshmi
Problem
TFMs like TabPFN/TabDPT do ICL over the full training set, giving quadratic memory scaling with shot count; naive locality-based retrieval degrades 2-3% when neighborhoods have mixed-label (low-purity) regions.
Method
ARASH clusters training data, scores each cluster's difficulty from entropy and purity to set an adaptive per-cluster shot budget, then routes each query to nearest-neighbor, diversity-aware, hybrid, or global retrieval.
Results
On OpenML-CC18/Combo, ARASH+TabDPT matches full-context accuracy (0.9017 vs 0.9009) using ~29 demos vs 30,805; ARASH+TabPFN 0.8955 vs 0.8948 with ~24 demos; prompt length cut 1261.5x, memory 2.56x, latency 1.37x.
Limitations
No explicit limitations section; the paper's own regression diagnostic shows performance depends on dataset locality and cluster purity (both significant, p<0.01), implying degradation on datasets lacking clear clustering structure.
arXiv
Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference
Blazej Banaszewski, Andrew W. Fitzgibbon
Problem
Molecular property prediction in drug discovery suffers from small, noisy labeled datasets, so foundation models need to generalize well from few examples per assay rather than relying on large task-specific fine-tuning data.
Method
Pretrain a GRIT graph encoder on 81M PM6 molecules plus 1,089 PCBA assays (1,152 tasks) with conformer denoising and embedding decorrelation; for downstream tasks, frozen embeddings and labels are fed directly to TabPFN for in-context posterior-predictive inference, replacing fine-tuning entirely.
Results
On 28 Polaris ADMET tasks Monroe wins 100% of pairwise comparisons (mean rank 3.71); on 30 MoleculeACE activity-cliff tasks it gets 86.7% win rate (RMSE 0.629 vs MolFormer's 0.748). Ablation: TabPFN beats an MLP head (100% vs 28.6% win rate) and LoRA fine-tuning (89.3%).
Limitations
Paper states Monroe is not a good conformation predictor (denoising doesn't improve geometry) and activity-cliff error remains well above overall error for every method, so cliffs are unsolved. Also flags curation flaws (duplicates, bad structures, stereo ambiguity) in benchmarks like TDC.
arXiv
Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths
Andrej Tschalzev, Stefan Lüdtke, Heiner Stuckenschmidt, Christian Bartelt
Problem
Benchmarks like TabArena aggregate scores (avg rank, Elo) into single leaderboards, which reward consistent all-rounders and hide which models are the only ones reaching peak performance on specific datasets.
Method
Define a per-dataset 'peak performance frontier' via paired Wilcoxon tests on CV-fold differences, classifying each model per dataset as irreplaceable, sufficient, redundant, or fallible relative to the best statistically-supported result.
Results
On TabArena (27 models, 51 datasets), RealMLP ranks 1st on Elo/avg-rank but is never irreplaceable (13th on sufficiency); only 9/27 models are irreplaceable anywhere, e.g. TabICLv2, TabPFN-2.5, CatBoost, TabDPT, ModernNCA each uniquely best on distinct datasets, and TabPFNv2 would have been irreplaceable on 17.
Limitations
Paper states: only predictive accuracy is considered (no runtime/memory/robustness); the frontier depends on which models are included in the zoo; needs repeated CV for significance; the four categories oversimplify a continuum; foundation-model overrepresentation may bias conclusions.
arXiv
TabNSM: Neural Sparse Mixer for Tabular Regression
Ali Eslamian, Qiang Cheng
Problem
Deep tabular regression needs cheap feature-interaction modeling that ignores noisy/redundant features, since full pairwise attention is costly and GBDTs don't learn end-to-end representations.
Method
TabNSM's Adaptive Sparse Interaction Module gives each feature token instance-adaptive sparse attention (sliding windows + compressed blocks) plus dual-axis token mixing, paired with a multi-stage regression head, ordinal GridLoss, and error-based RISE resampling.
Results
On 9 regression benchmarks vs 38 baselines (XGBoost/LightGBM/CatBoost, FT-Transformer, TabR, Mamba, TabPFN/TabPFNv2), TabNSM gets lowest RMSE on 7/9, e.g. +22% on Traffic Optimization, +21% on EV Population; Wilcoxon p≤0.02.
Limitations
GBDTs still win on low-dimensional data (TabNSM ranks 2nd on Crime Data); needs GPU training with more overhead than tree methods; requires a feature-ordering choice for sparse attention; GridLoss adds hyperparameters (K, τ).
arXiv
From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics
Heyang Gong
Problem
Attention masks control which tokens may interact, but give no inert, zero representation state — a zero-vector token still perturbs outputs, breaking compositional insertion/deletion of tokens.
Method
OAttention adds a per-token presence coefficient (a saturating function of hidden-state norm) that gates its contribution to attention; paired FFN/norm/injection components (O-closure) let zero-vector tokens pass through a full transformer block as exact no-ops.
Results
TabPFN v3 retrofit on six OpenML regression datasets shows only tiny RMSE shifts (+0.088% to +0.177% mean, near-identity). The stronger result is an ablation: only full O-closure gives exact zero-token insertion (L∞=0) vs ~7.98 for standard attention.
Limitations
Authors state presence-via-norm isn't guaranteed to track true epistemic absence, weak nonzero signals can be attenuated, cross-token centering can accidentally zero out active tokens, and results don't establish universal missing-value semantics or non-regression across hosts/tasks.