»Forward
Pass.

Problem

Tabular ICL models alternate row and column attention over 2D embedding grids, a regime unlike the 1D sequences LLM attention kernels were designed and benchmarked for.

Method

Systematically benchmark FlashAttention-2/3/4, PyTorch SDPA (efficient and cuDNN), vLLM, and SageAttention on realistic row/column attention shapes across A100, H100, and B200 GPUs, with roofline analysis.

Results

No backend wins everywhere: cuDNN SDPA is best for column attention, while FlashAttention-3 is up to 3.5x faster than SDPA and 1.5x faster than cuDNN for row attention on H100; best-vs-worst gaps exceed 3x.

Limitations

Benchmarks isolated attention kernels on synthetic tensors, not end-to-end training/inference; only bfloat16 and non-causal attention with batch size 1 are tested; SageAttention's accuracy impact is not evaluated.

Problem

Multimodal Alzheimer's diagnosis models combining MRI and tabular biomarkers are typically dataset- and task-specific, and need retraining when applied to cohorts with different features or imaging protocols.

Method

Differentiable prompt-tuning through a frozen TabPFN: 3D ResNet-18 and FT-Transformer encoders produce disentangled shared/modality-specific embeddings fed into TabPFN's context, trained end-to-end via a class-prototype alignment loss and a gated tabular-only shortcut.

Results

On ADNI 3-way CN/MCI/AD classification, M2PFN reaches 67.18% accuracy and 82.21% macro-AUC vs. 64.50%/81.14% for the best baseline (MMPFN); it also transfers zero-retraining to OASIS-3 and SCAN, beating MMPFN there too.

Limitations

MCI remains the hardest class to classify across all methods; support-set caching during training causes representation staleness; zero-shot transfer without support-set replacement is notably weaker than with it.

Problem

Tabular foundation models adapt to a new table purely via a forward pass over labeled support examples, but it is unclear what internal computation actually implements that in-context adaptation.

Method

Derives an update rule from a leave-one-out support objective, splitting it into a similarity-weighted attention readout over support states and a query-dependent gate, implemented per-block as Norm(A)⊙SiLU(W_g h) with no separate FFN.

Results

L24 model reaches 1644.8 Elo on TabArena (38 datasets), 31.4 Elo above TabPFN-v3, and 0.87173 accuracy/0.93836 AUC on AMLB29 (+0.30/+0.22 over TabPFN-v3); disabling support updates raises query cross-entropy by 0.0510 on average across 72 episodes.

Limitations

Authors note mechanism claims rely mainly on synthetic tasks with known rules; TabArena result is one fold-0 snapshot with heterogeneous reference budgets; regression is out of scope; trained weights and benchmark data are not released.

Problem

Feeding time series into a frozen tabular foundation model needs a fixed-width table, but naive pooling discards temporal order and per-sample coordinate systems make local-dynamics features incomparable across sequences.

Method

Fits local affine dynamics operators over sliding windows in a shared PCA coordinate system, soft-assigns them to a shared k-means regime codebook (K=12), and feeds resulting transition/residual/signature features into a frozen TabPFN classifier.

Results

On 8 UEA datasets, SwitchPFN gets 82.63±0.45% mean accuracy vs. 79.09±0.40% for the best baseline (MultiRocket+TabPFN), a 3.54pp gain, winning on 7/8 datasets; swapping TabPFN for ridge regression costs 5.67pp.

Limitations

Authors state evaluation covers only eight UEA datasets; they flag future work to refine feature selection via independent validation and reduce computational cost (full pipeline takes 560.63s per dataset on an RTX 4090).

Problem

Wide tables give tabular foundation models more evidence, but full pairwise feature interactions scale quadratically in memory; naive feature selection avoids this only by discarding data.

Method

Training-free inference: rank columns via support-set ANOVA into Core/Tail, split both into bounded ≤128-column leaves encoded independently by the frozen backbone, average Core leaves, and optionally add a validated Tail residual (binary-only) before one contextual prediction.

Results

Across 18 wide-table datasets and 6 backbones, matched-width SCFF cut error by up to 26.1% (TabICLv1) versus native full-width processing, with median GPU memory savings of 2.09-2.36x and peak reductions up to 34.3x.

Limitations

Tail residual recovery works only for binary classification (multiclass keeps exact Core); TabPFNv1 lost accuracy on held-out folds; TabICL backbones ran slower (0.56-0.62x) from repeated encoder calls; evaluated on 18 curated OpenML datasets, not an official leaderboard.

Problem

Neural Processes are strong meta-learners for tabular data but standard architectures struggle to represent shared structure with discontinuities or irregular patterns across tasks, since prior NP extensions mostly tweak the network rather than the underlying decomposition.

Method

NPBoost decomposes task responses into gradient-boosted trees as fixed effects shared across tasks plus a Neural Process modeling task-specific random effects, trained jointly via boosting with a cross-fitting scheme giving unbiased gradients for the trees.

Results

On 8 real-world tabular meta-learning datasets, NPBoost beat a standard NP on within-task/few-shot RMSE in 7 of 8 dataset-scenario combinations (e.g., Cars: 0.257 vs 0.266 RMSE); CRPS improvements were less consistent.

Limitations

RMSE gains did not consistently extend to CRPS, partly attributed to model selection via validation RMSE; cross-attention (ANPBoost) gave no consistent benefit on real data; evaluated only on tabular tasks, not other domains NPs typically target.

Problem

Causal tabular foundation models trained on mechanism-level effect labels reward posterior shrinkage, attenuating estimates toward the prior center rather than giving correct repeated-sample inference at a fixed population.

Method

Fluctuation-supervised pretraining (FSP) labels synthetic tables with ATE plus a weighted efficient-influence-function fluctuation term, T = theta(P) + lambda*(1/n)*sum(psi_P), deployed frozen with no test-time nuisance refitting.

Results

Continuous-row FSP cut macro RMSE 7.0% versus an S-learner across 24 nonlinear cells (winning all 12 weak-overlap cells); on NSW RCT data it lowered post-treatment RMSE from .0596 to .0522 with .952 coverage; Summary FSP ran 11.6x faster per table than S-learner.

Limitations

Deployment ATE risk stays order n^-1 even with perfect pretraining, an irreducible finite-sample floor; coverage needs accurate variance-head learning, which degraded badly under weak overlap (mean V-hat/V of only .328).

Problem

Tests whether a JEPA-style latent-prediction objective (predict masked representations, not values) can be trained alongside standard value-prediction pretraining on a tabular foundation-model prior, since naive JEPA collapsed the encoder to a constant map.

Method

Tabular JEPA recipe: the value head reads the shared encoder's own field rather than the predictor output, and the latent target is an EMA target-encoder's full-table minus masked-table embedding difference; both arms trained to a validation plateau rather than a fixed step budget.

Results

The fix stops collapse (target effective rank rose from 1.5 to 98.1) but at convergence JEPA still trails the value-only baseline on 147 real datasets: 32:70 wins:losses on classification (.809 vs .815 mean accuracy) and 8:24 on regression (.635 vs .665 mean R2), while needing 1.42x the steps and 1.66x the wall-clock to converge.

Limitations

Authors note only one seed per arm and no ablation with the latent weight set to zero, so the accuracy deficit cannot be attributed to the latent term alone; only one model size and context budget was tested, and the latent term bought no measured robustness or accuracy gain.

Problem

RelGT, the leading graph transformer for relational deep learning, samples loosely-connected local subgraphs and relies on a single seed-based global memory, missing broader topological, temporal, and collaborative signals across a relational database.

Method

QUARTET pairs a causal random-walk sampler (recency-truncated personalized PageRank with hub-degree capping) for denser local subgraphs with a quad-branch global module: two codebook branches (feature, topological) plus two Perceiver-style cross-attention branches (temporal, collaborative).

Results

On 12 RelBench classification tasks QUARTET beats RelGT on 8/12, up to +6.4% ROC-AUC (rel-trial study-outcome) and +5.6% (rel-event user-repeat, rel-f1 driver-top3), but trails by up to -2.7% on some churn/badge tasks; the sampler alone adds +1.6% average.

Limitations

No dedicated limitations section; authors note RelGT still wins on large, stationary tasks where dense local topology dominates. Restricted to binary classification (regression, link prediction left to future work); large tasks require four A100 GPUs.

Problem

Repeatedly training gradient-boosted trees on massive, highly imbalanced tabular datasets (fraud, ads) is expensive, and standard coresets lose accuracy once most majority-class rows are pruned, especially above 90% reduction.

Method

CRISP scores negative examples with a cheap proxy GBDT, stratifies them into equal-count score quantiles, allocates the negative sampling budget proportionally to each stratum's mean importance, and reweights kept negatives with clipped inverse-propensity weights.

Results

On a 25M-row production dataset at 95% negative reduction, CRISP retains 99.7% of full-data average precision vs. 94.2% (random) and 94.1% (CCS); on CriteoPrivateAds (85M rows) at 99.4% reduction it retains 94.2% AP vs ~90% (random/CCS) and 76.7% (CoreTab).

Limitations

Only viable when the positive class is small enough to leave a usable negative budget; reported cost savings exclude proxy-fitting and scoring overhead; evaluation uses only 3-5 seeds without paired significance tests; coreset selection can shift subgroup recall/calibration even when aggregate AP holds.