»Forward
Pass.

Problem

Build a small, open tabular foundation model that matches much larger models on real-world classification/regression, while generalizing to many-class problems and larger in-context support than seen in pretraining.

Method

77M-param 2D transformer (row- and column-wise attention) trained purely on synthetic data, mixing a new Hybrid SCM prior (NN/tree/GP/autoregressive mechanisms in one causal DAG) with base SCM and tree priors, Muon+AdamW optimization, a distributional regression head, plus wrapper mechanisms (feature selection/SVD, hierarchical many-class decomposition) for out-of-distribution table sizes.

Results

On TabArena (51 datasets), Mitra-v2 reaches Elo 1774.6, a statistical tie with the 1.6B-parameter TabFM (1773.5) and ~137 Elo above TabPFN-3, winning 31/38 classification datasets vs TabPFN-3 (p=0.0001); it ranks first on many-class classification on TALENT (2204.2 Elo) despite pretraining on ≤10 classes.

Limitations

Authors caveat that leading-group gaps (vs TabFM, EXAONE) fall inside overlapping bootstrap intervals, so results are a statistical tie, not a strict ordering; the headline system needs fine-tuning plus 8-fold bagging, so it is "not zero-shot or Pareto-dominant"; individual design changes are not ablated separately.

Problem

Tests whether tabular foundation models (TabPFN-3, TabICLv2, TabDPT, Real-TabPFN-2.5), which never see physical laws in pretraining, actually internalize physics-like structure (determinism, dimensional units) rather than just interpolating well, and whether any edge is pretraining contamination.

Method

Regenerate 316 physics equations (Feynman, LSR-Transform, LSR-Synth) fresh rather than loading published tables, sweeping sample size/noise/extrapolation; audit pretraining corpora for leakage; use Buckingham's π-theorem to test sensitivity to dimensional structure; check predictive-interval shrinkage at zero noise (deterministic limit).

Results

TabPFN-3 achieves the best mean rank (8.47 vs 9.85 for the best tuned+ensembled GP baseline) across 314 tasks with no detectable contamination advantage, but at noise σ=0 its predictive intervals stay nonzero (0.010-0.020 normalized) instead of collapsing to a delta, and it gets significantly worse (not better) as tasks have more dimensionless π-groups, unlike CatBoost/RealMLP.

Limitations

Authors note contamination effects below roughly one rank position would be undetectable at their pool size; the comparison is imbalanced because trained baselines sacrifice 20% of samples for internal model selection while TFMs condition on all of it; under extrapolation, all models collapse to a common failure mode rather than degrading gracefully.

Problem

Tabular foundation models (TabPFN, TabDPT, TabICL) have no native way to handle censored outcomes, so it's unclear which adaptation interface best transfers a frozen pretrained backbone to time-to-event/survival prediction.

Method

Systematically compares three frozen-backbone interfaces—zero-shot horizon-wise classification, classification fine-tuning, and survival heads (Cox, DeepHit, MTLR)—with adaptive event-time discretization and resampled in-context conditioning.

Results

Across 74 single-risk SurvSet datasets, zero-shot wins on small N; TabPFN-Cox gives the most consistent IBS gains as data scales (e.g. C_td≈0.67-0.70 on DIALYSIS, N=6,805, vs RSF≈0.65); cause-specific MTLR ranks best on 4 competing-risk sets.

Limitations

Competing-risk benchmark is only 4 datasets, called 'preliminary' by the authors; mean ranks obscure per-dataset variation; interface comparisons are confounded (different target constructions/objectives), so no causal claim about which head is truly best.

Problem

TabPFN's permutation-invariant feature assumption and synthetic pretraining ignore the temporal and inter-channel structure of physiological signals, limiting it for low/medium-data clinical time-series classification.

Method

Redesigns TabPFN with channel-wise rotary temporal embeddings (RoPE) plus learnable channel-identity embeddings, and replaces synthetic pretraining with ~140k real EEG/ECG/ICU series (TUEV, TUAB, PTB-XL, HiRID).

Results

On 5 physiological benchmarks (100-shot support, 5-fold CV), TSPFN averages 57.3% AUPRC / 82.5% AUC vs plain TabPFN's 49.2%/77.5% and XGBoost's 51.0%/77.1%, and beats MiniRocket (53.6% AUPRC) with DeLong p=3.7×10⁻⁵.

Limitations

Architecture caps input at T×C≤500, limiting very long recordings; specialized baseline MiniRocket still beats TSPFN on seizure detection (89.8% vs 82.7% AUC); only N=100 support tested, so scaling to larger labeled sets and failure-mode analysis are largely unexplored.

Problem

Building an in-context-learning tabular foundation model that matches or beats TabPFN/TabICL-style rivals on classification and regression while cutting training and inference cost.

Method

Transformer ICL model with dual-stream Set-Transformer column embedding, AttnRes cross-layer fusion, sparse MoE feed-forward blocks, query-aware scalable softmax, pretrained on SCM-generated synthetic data via a three-stage curriculum.

Results

Ranks first on OpenML-CTR23 (avg rank 3.03) and second on TabArena regression (Elo 1900 vs TabFM's 2019) and BCCO (mean rank 2.94), while using 82% less training time and 68% less inference time than TabFM (6.99s vs 38.85s per 1K samples train; 3.12s vs 9.67s predict).

Limitations

No dedicated limitations section; classification is weaker/secondary to regression, overall TabArena rank drops to fourth (Elo 1659); classification capped at 10 classes via mixed-radix ensembling; benchmarks skew toward datasets under 10K rows, so large-scale generalization is untested.

Problem

Relational foundation models (RT, Griffin, RelGT) cap per-entity neighborhood/context size, so they may silently truncate history for high-cardinality entities with many related records, undermining their use as general-purpose relational learners.

Method

Introduces Animus, a synthetic financial benchmark (100K customers, skewed transaction volumes, 5% of customers with 1,000-10,000 transactions) predicting future income from raw per-transaction data versus a temporally pre-aggregated (Agg-Simple, monthly-by-category) representation.

Results

On raw data RT/Griffin/RelGT scored R^2 = 0.184, 0.110, and OOM (RelGT ran out of memory materializing the graph) respectively; pre-aggregation lifted them to R^2 = 0.654, 0.384, and 0.230, while GraphSAGE (larger sampling budget) already reached 0.693 raw and 0.752 aggregated.

Limitations

Authors themselves note Agg-Simple is just routine SQL aggregation any analyst would do, so recovering only 0.2-0.5 R^2 via it is a "brittle" fix; a true foundation model should learn what to aggregate itself. Benchmark is synthetic and single-domain (finance), and exact budget sizes used (seq_len=1024 for RT, fanout=20 for Griffin, num_neighbors=300 for RelGT) are somewhat arbitrary hyperparameter choices.

Problem

Tests whether frozen Relational Transformer (RT) embeddings, injected as soft tokens into an LLM, let the LLM do relational prediction over multi-table databases without lossy text serialization of schema and rows.

Method

Project RT ego-graph embeddings (256 tokens, dim 256) through a 2-layer MLP into Qwen3.5-4B, adapt with LoRA (r=16, alpha=32), and train via SFT on 7,869 CoT traces followed by GSPO RL with binary correctness reward.

Results

On RelBench (10 binary-classification tasks, 6 DBs), standalone RT beats every hybrid regime on average — e.g. 71.9 vs 51.1 AUROC single-task, and holds a lead cross-dataset (69.7); SFT-only collapses to 48.3 AUROC.

Limitations

Authors report the hybrid is often below random, unstable under RL, and highly serialization-sensitive (JSON vs flat format shifts AUROC by up to +0.63), suggesting the LLM leans on text tokens, not embeddings; binary tasks only, one 4B model, single seed.

Problem

Tests whether foundation models (time-series and tabular) can replace market-specific electricity price forecasting models for day-ahead prices and downstream battery-arbitrage decisions, evaluated zero-shot across three European markets.

Method

Zero-shot evaluation of 9 FM variants from 5 families (Chronos-2, Moirai-2, TimesFM-2.5, TabPFN-2/3, TabPFN-TS-3, Mitra+CP) against LEAR-conformal and DDNN-JSU benchmarks on Germany/Poland/Spain day-ahead prices (2021-2025), scored by MAE/RMSE/CRPS and BESS arbitrage profit.

Results

TabPFN-3 gets the lowest CRPS in all three markets (Germany 9.15 vs DDNN-JSU 10.70; Poland 9.61 vs 10.96; Spain 8.15 vs 9.30, p<0.01) — 'Only the TabPFN models consistently and significantly outperform the benchmarks across all three markets and all statistical measures.'

Limitations

Statistical dominance doesn't transfer to economics: DDNN-JSU beats TabPFN on profit under risk-averse bidding (alpha=50-60%), with only TabPFN-3/TabPFN-TS-3 winning under unlimited-bid arbitrage; other time-series FMs (Moirai-2, TimesFM-2.5) were consistently weakest; limited to 3 markets, 1 battery spec.

Problem

LLM-based automated feature engineering for tabular data lacks structured domain knowledge, rigorous verification, and traceable provenance, limiting trust in high-stakes settings like AML.

Method

KnowFeat injects five typed domain-knowledge sources (schema, regulatory indicators, detection rules, expert opinions, court evidence) into an LLM agent, then filters candidate features through a three-stage pipeline: code-execution sandbox, statistical quality checks (IV, KS, correlation), and XGBoost effectiveness gain across 5 seeds.

Results

Under a leakage-free held-out protocol across 12 OpenML benchmarks and 7 methods, KnowFeat ranks first on average (rank 2.3, Wilcoxon p=0.017), with peak gain +11.6 pp AUC on churn (0.737→0.853); on SimECNY AML it holds F1=0.971 while FeatLLM's 1,124 unverified features drop F1 to 0.968.

Limitations

Gains shrink to <1 pp on saturated benchmarks (breast_w, adult AUC>0.93); knowledge curation took ~2 person-days for the AML taxonomy (309 court docs); feature-name overlap across reruns is near-zero (Jaccard 0.000-0.036) despite stable AUC, and the semantic-dimension taxonomy is manually specified.

Problem

Tests whether neural-scaling-law regularities (loss improving as a power law in data/parameters/compute) hold for actuarial ratemaking on tabular, noisy, zero-inflated insurance data, where GLMs remain the incumbent baseline.

Method

Fit power-law loss curves L(P,N,C) for GLM, FFN, TabM/TabM-mini, vanilla Transformers, and TabM-adapted/TokenMoE Transformer variants, training on nested data fractions (5%-100%) with 5 seeds each, measuring out-of-sample Poisson deviance on a ~4.5M-record motor insurance portfolio.

Results

TabM/TabM-mini shows the strongest data-scaling exponent (fit L(N)-0.28749≈1.03·N^-0.409, so each data doubling cuts ~25% of remaining gap to the floor / multiplies reducible loss by ~0.75); vanilla Transformers show near-zero parameter-scaling exponent unless given TabM-style inductive bias or self-supervision.

Limitations

IID split only (no timestamp for temporal validation), pre-COVID single-portfolio data limiting external validity; continuous features quantile-binned rather than learned embeddings (authors call this 'debatable'); hyperparameters set heuristically, not tuned per model; results are explicitly conditional on this dataset/protocol, not universal.