»Forward
Pass.

Problem

Table foundation models need strings encoded as numbers, and the usual cell-by-cell LLM embedding wastes the LLM's knowledge; larger LLMs don't help.

Method

TILE: serialize the whole row for the LLM, take intermediate-layer states, pool per column, and bypass numbers and categories around the LLM, then feed per-column PCA features to a TFM.

Results

On 100 tables with strings it beats the strongest of 30 baselines on 89% of splits, improves with LLM size (7 LLMs, 10 learners), and ranks first on the 108-dataset STRABLE benchmark.

Limitations

Needs a GPU for embeddings; row serialization is bounded by LLM context; causal-mask LLMs make features depend on column order; no gain on purely numerical tables.

Problem

Survival analysis on small censored clinical cohorts is a weak spot: no existing tabular foundation model beats the Cox model.

Method

SurvICL builds on TabICLv2, outputs a continuous quantile function with a plain quantile loss, and is pretrained on synthetic data from a proportional-odds prior that provides uncensored latent event times.

Results

On 63 held-out datasets (of a 103-dataset benchmark) it beats 20 baselines without tuning: 82.9% of pairwise IBS wins versus 71.3% for Cox, the first model to rank above Cox on small datasets.

Limitations

Gradient boosting stays slightly stronger on the largest datasets; only right-censored single-event data (no competing risks or time-varying covariates).

Problem

TFMs run every input through the same deep stack, but it is unclear which inputs actually need the later layers, which matters for efficiency and early exit.

Method

Layerwise study of all 24 ICL layers of TabPFN-3 on 272 datasets (plus four other TFMs): representation geometry, prediction trajectories, truncation, and structural interventions such as repeating a block or shuffling layer order.

Results

Class information is readable early, yet geometry keeps evolving (expansion then compression); commitment depth varies per sample, ambiguous queries gain most from late layers, and repeating a block or reordering layers lowers accuracy.

Limitations

Focuses on TabPFN-3 and classification; no practical early-exit method yet since stabilization depth is measured retrospectively; regression not covered.

Problem

TFMs are pretrained for IID data, but real datasets often contain repeated measurements at the same inputs. How do TFMs behave then?

Method

Controlled synthetic, semi-synthetic and real experiments on six TFMs, plus a statistical argument (PFN's IID rule treats repeats as fresh evidence) and attention-scaling interventions on TabPFN-v3 row count.

Results

Extra repeats can improve predictions at the measured inputs but degrade them elsewhere, most often at low noise and with irrelevant features. Averaging repeats usually beats feeding them separately; resetting the row count removes some harm.

Limitations

Likely: mitigations are partial and diagnostics are mostly synthetic; real-data evidence is limited to a few repeated-measurement datasets.

Problem

Most TFMs apply the same dense computation to every feature and dataset, which is wasteful for heterogeneous tables.

Method

OrionMoE-Small: a dual-axis (feature and sample) Transformer with sparse mixture-of-experts along the feature axis, routing informed by feature and support-derived dataset context; 37.7M parameters, pretrained on synthetic causal-prior classification tasks.

Results

Competitive with several medium-sized TFMs and ahead of larger ones on some benchmarks (TabArena, BeyondArena, OpenML-CC18, TALENT, BCCO), strongest on small-context and high-dimensional data.

Limitations

Top large models still lead overall, especially under distribution shift and at scale; no parameter-matched dense ablation, so the MoE gain is not isolated.

Problem

Relational foundation models typically need tens of thousands of examples to adapt to a new database.

Method

RT-J: an 86M relational transformer pretrained with up to 8,192-cell contexts on the Join (639 web-crawled databases, 6,023 LLM-generated tasks), with a random-walk retriever for in-context examples and context tuning/ensembling at test time.

Results

On RelBench in-context regression it needs about 10x fewer examples to match LLM-agent or RDBLearn feature-engineering pipelines with TabICLv2 or LightGBM predictors.

Limitations

No link prediction or recommendation; context length capped; evaluated only on RelBench.

Problem

PFN-style synthetic pretraining works for transformer relational models, but it was untested for message-passing graph models over multi-table databases.

Method

Pretrain the graph-based relational FM Griffin on synthetic relational databases with a curriculum (schema-guided then schema-agnostic), then pass its frozen latents to an in-context predictor, with no target-task training.

Results

On 16 RelBench-style classification tasks over five seeds, the synthetic curriculum beats real-data-only pretraining and unordered mixtures, and is best overall when appended after real-data pretraining.

Limitations

Likely: gains are task-dependent (some tasks dip slightly); limited to 16 classification tasks and one encoder family.

Problem

PFN-based TFMs predict in one forward pass but cannot say which inputs drive their predictions.

Method

SelectPFN: a PFN pretrained on a spike-and-slab sparse-network meta-prior that outputs posterior inclusion probabilities and predicts only from sampled subsets; knockoff columns give finite-sample FDR control.

Results

On out-of-prior nonlinear designs with up to 500 variables it improves PR-AUC by 0.23-0.28 over lasso, BART, DART, LassoNet and a sparse NN, at about 0.07 false discovery proportion and 0.04 s per dataset.

Limitations

Regression only and small scale; factorized selection head ignores posterior dependence; dense-signal data (e.g. spectra) not in the prior; the larger model's prediction regresses on one design.

Problem

Causal foundation models (CausalPFN and similar) are used for group-averaged effects, which only helps if their credible intervals are calibrated.

Method

Show formally that combining per-unit variances as if independent ignores correlation induced by posterior uncertainty over the data-generating process; test two neural-process models with joint predictions as heads on a frozen CausalPFN.

Results

Coverage falls with group size (on ACIC 2016, 95% intervals drop below nominal beyond groups of ~30 units; full-sample ATE covered in only 20% of datasets); joint-prediction neural processes substantially improve coverage across six benchmarks.

Limitations

Joint heads do not always reach nominal coverage; causal-effect setting rather than standard tabular prediction.

Problem

Tabular data generation is needed when data are scarce, but TFM-style in-context methods usually need large context sets.

Method

TabGift: a frozen SwiftPFN encoder maps rows to a 192-d latent, an 18-layer in-context diffusion Transformer denoises conditioned on a few clean examples, and a dataset-specific MLP decoder maps back; pretrained on synthetic data, then 72 real datasets.

Results

On 7 classification and 11 regression datasets it gives best or near-best synthetic-data quality with 16-64 context rows, with much faster adaptation and generation than most baselines.

Limitations

Likely: small benchmark (18 datasets) and per-dataset decoder needed; advantage is mainly in the small-context regime.