»Forward
Pass.

Problem

Tabular foundation models (TabPFN, TabICL, etc.) are being deployed in high-stakes settings, but it's unclear whether their in-context-learning gains survive real-world distribution shift, which is common and safety-critical for tabular data.

Method

Benchmarked nine TFMs spanning real-data, mixed, and synthetic-prior pretraining (TabPFN v2/v2.5/v2.6/v3, TabFM, Mitra, TabICL, TabICLv2, LimiX) on three TableShift datasets (HELOC, Childhood Lead, Voting), comparing in-distribution vs out-of-domain ROC-AUC to quantify each model's 'shift gap.'

Results

All nine models degraded under shift (gaps 0.003-0.060 AUC). Real-data-pretrained models led OOD (median AUC ~0.78 vs ~0.77 synthetic) but had similar shift gaps — the edge is better in-distribution fitting, not true OOD robustness; Mitra was most label-shift-sensitive.

Limitations

Only three shift datasets/types tested; TabFM and LimiX had prohibitive memory growth (Voting results omitted for both), Mitra capped at 10,000 training rows, and no fine-tuning was applied, so default-config results may underestimate true adaptable OOD capability.

Problem

Relational in-context learners like RDB-PFN need large synthetic multi-table pretraining corpora, but existing generators are tightly coupled to the model, making it unclear if an external, decoupled synthetic-DB generator can substitute.

Method

Pipe PluRel-generated synthetic relational databases through a conversion pipeline into RDB-PFN's DFS-linearized training format, comparing curricula that order real vs. synthetic schemas differently (Schema-Guided First/Last, Fully Synthetic).

Results

With only ~33K tasks (55x less data), Schema-Guided-First reaches 0.6346 ROC-AUC at 1024-shot (87.6% of published RDB-PFN's 0.7245) and 93.8% at 64-shot; beats Fully-Synthetic by 2-3 points, Schema-Guided-Last by 5-6.

Limitations

Unmatched training scale vs. original RDB-PFN, features restricted to INTEGER/FLOAT (no text/categorical fidelity), heuristic task construction, evaluation limited to binary classification, and gap widens at longer contexts, suggesting synthetic data lacks richer long-range dependencies.

Problem

Relational PFNs (e.g., RDB-PFN) need millions of synthetic multi-table pretraining examples, making pretraining expensive; the paper asks whether curriculum ordering, rather than sheer synthetic data volume, can make training far more data-efficient.

Method

Using PluRel synthetic data with a small (0.7M-param) bidirectional transformer, the authors train progressive curricula—single-table then relational, gradually widening schema complexity—versus feeding all data at once, and test transfer from single-table to relational tasks.

Results

Progressive single-table curriculum hits 0.703 ROC-AUC with 45x fewer tables than RDB-PFN's recipe (vs 0.541 all-at-once, 0.800 RDB-PFN); relational curriculum reaches 0.638 (88% of RDB-PFN's 0.725) using 220x less data; single-table-only transfer scores 0.631.

Limitations

Gaps to full RDB-PFN baseline remain largest on big/data-rich tasks (Higgs, Polish, MiniBooNE), suggesting curriculum can't fully substitute for scale; per-task variance is high, and PluRel's synthetic temporal patterns may not match real event distributions.

Problem

Tree ensembles give strong point predictions on tabular data but don't naturally output full predictive distributions; prior tree-based diffusion (Treeffuser) borrowed neural-diffusion defaults that may not suit histogram-based trees.

Method

DiffGBM trains LightGBM to predict scores or velocity fields instead of using neural nets, exposing tunable design axes (residualization, EDM preconditioning, noise-level features) and adding a flow-matching variant for fast deterministic sampling.

Results

Across 11 UCI regression datasets, tuned DiffGBM beat published Treeffuser on all 11 (Wilcoxon p=4.9e-4), best CRPS-skill 0.725 vs 0.699 baseline; flow-matching variant was best-calibrated and ~5x faster to sample (35s vs 183s).

Limitations

Baselines got fewer tuning trials than DiffGBM variants (25 vs 40); tested only scalar UCI-style regression, not categorical-heavy or multi-output data; stochastic samplers don't dominate deterministic ones on aggregate metrics; CT-slice evaluation leaks within-patient data.

Problem

Tabular foundation models like TabPFN outperform classical baselines but their deployment memory footprint (model weights, storage/distribution) has received little attention, limiting practical industry adoption under hardware and bandwidth constraints.

Method

Apply post-training INT4 weight quantization—K-means clustering, GPTQ, and AWQ (adapted from LLM compression, calibrated on the in-context train/test tuple)—to TabPFN v2.5 and v2.6.

Results

Across 30 OpenML datasets, INT4 quantization gives 6.0-7.65x compression (K-means best, 41MB to 5.4MB) with accuracy loss <=0.005 (0.872 to 0.871), still beating tuned LightGBM (0.864). Inference memory (~4.7GB) and speed barely change (<2%/10%).

Limitations

Only post-training quantization tested, not quantization-aware training, pruning, distillation, or LoRA; requires dequantization before inference so runtime memory is unchanged; no hardware/kernel optimization; doesn't address attention-driven context-size memory limits.

Problem

TabPFN classifies via in-context learning over labeled rows, but practitioners lack guidance on how much context size and row-selection choices affect accuracy and stability on small tabular datasets, where labeled data is scarce and every design choice matters.

Method

Systematic experiments on 15 OpenML datasets varying context size (k=16 to 256) and constructing contexts with controlled geometric properties (feature-mean shift, diversity) to causally isolate what makes a good context, versus just measuring correlations.

Results

Larger context cuts AUC coefficient-of-variation from ~6-18% (k=16) to ~1-4% (max k). Diversity predicts accuracy (β=+0.23, p=3e-12) but distribution-match doesn't (β=-0.01, p=0.71); forcing low feature-shift dropped diabetes AUC to 0.49 vs 0.86 random. Random sampling matched costlier methods at ~1000x lower cost.

Limitations

Evidence is associational/constructed rather than fully causal/randomized; per-dataset effects are mixed (diversity helped on only 9/15 datasets); scope limited to small tabular data (200-5000 rows), TabPFN v3 only, three selection methods, and coarse distribution-shift metrics.

Problem

When a tabular feature bundles a true causal signal with a spurious systematic artifact (e.g., hospital equipment quirks), in-context learners can't tell them apart, so predictions silently rely on the artifact and fail on new sites/environments.

Method

The authors prove ridge-regression ICL (and empirically TabPFN) inevitably routes through spurious signal when confounded in-context, quantified via a closed-form Causal Sensitivity Ratio (CSR); they fix it with environment-stratified context sampling and 'S-swap' augmentation using only weak environment labels.

Results

CSR predictions matched theory (r=0.997 linear ICL, r=0.979 TabPFN); larger context worsened spurious routing up to 1.74x. S-swap cut CSR 74% (linear)/98.8% (TabPFN) and raised TabPFN's causal sensitivity 8.4x; on real scIB pancreas data, S-swap cut CSR 88.9%.

Limitations

Theory assumes orthogonal causal/spurious subspaces and ridge-ICL estimators; robustness was only checked up to 40-60 degree rotation errors. S-swap needs multiple labeled training environments, which may not always be available in practice.

Problem

Predictive process monitoring (forecasting next activity, remaining time, next-event time from event logs) has long used LSTMs; it's unclear whether these still beat new tabular ICL and LLM foundation models.

Method

Controlled benchmark on 5 real event logs (BPI12/17/20) comparing LSTM/Transformer, LoRA-tuned small LLMs (Llama-3.2-1B, Gemma-2-2b), and tabular ICL models (TabPFN-3, ConTextTab) on next-activity, remaining-time, next-event-time tasks.

Results

LSTM/Transformer win next-activity prediction (e.g. BPI17: 85-86% vs ConTextTab 65%; BPI12: LSTM 78% vs TabPFN-3/Gemma 64%). Tabular models are competitive on temporal tasks; LLMs underperform despite higher cost; TabPFN-3 failed on BPI17 (context-size limits).

Limitations

Only small LLMs tested (1-2B params); no exploration of alternative tabular feature representations; TabPFN-3 hits context-length limits on larger logs; benchmark restricted to a handful of logs/tasks.

Problem

Table embeddings underpin retrieval and data discovery, but existing benchmarks judge them only via retrieval performance, conflating representation quality with one downstream task and hiding what embeddings actually capture.

Method

Extends TEmBed with three disentangled table-level tasks: cross-domain retrieval, a shuffle-vs-permute triplet test for structural fidelity, and header-stripped Schema.org type classification, evaluated across MiniLM, Granite-R2, GritLM, HyTrel, and TF-hashing.

Results

No encoder wins all three axes: GritLM leads retrieval (MRR@10 ~0.69), HyTrel dominates structural fidelity (0.87 triplet accuracy vs <0.20 for others), yet hashing/GritLM top type detection (F1 ~0.93/0.91) while HyTrel scores worst (0.64).

Limitations

Assumes row/column permutations are semantically valid positives, which may not hold for order-sensitive tables (e.g., time series); small workshop-scale study over 5 models and existing corpora rather than new data.

Problem

Building one anomaly detector that works across many tabular datasets is hard because features differ across domains and labeled or synthetic anomalies aren't reliably available for unseen data.

Method

CORE aligns datasets via variance/correlation-based feature selection (not opaque transforms), then detects anomalies by reconstructing each sample from attention-retrieved normal in-context examples, using reconstruction error as the anomaly score.

Results

Across 34 target datasets, CORE gets 0.8488 average AUROC, beating the prior unified method OFA-TAD by 1.43 points and winning on 26/34 datasets; out-of-domain AUROC is 0.8178 (+2.59 points over OFA-TAD).

Limitations

Ablations show in-context reconstruction is the main driver (-6.33% AUROC if removed); feature alignment adds computational cost scaling with feature dimension. Likely: relies on having enough confidently normal context samples per dataset.