»Forward
Pass.

Problem

Attention-heavy tabular ICL foundation models (TabPFN, TabICL) are accurate but slow and memory-hungry, especially as in-context example count grows. Can attention be dropped from parts of the pipeline?

Method

SOMTab uses cross-attention to map unordered table tokens into ordered latent slots, applies Mamba-2 state-space mixing there for row/column representations, then keeps attention only for the final query-conditioned prediction step. Pretrained on a new DCH-TailMix synthetic prior (degree-corrected graph structure + heavy-tailed Student-t/Pareto mixtures).

Results

On TALENT, SOMTab is ~24x faster per 1K samples than TabICLv2 (0.019s vs 0.463s) but trails in normalized log-loss (0.054 vs 0.029) and accuracy (0.849 vs 0.854); efficiency advantage grows with context length up to 28,672 rows tested.

Limitations

Clear accuracy/log-loss gap vs full-attention TabICLv2 (log-loss ~1.86x higher); ablations show replacing the attention reader with Mamba hurts accuracy, confirming attention is still needed there. Untested beyond ~28K context rows; pretraining took 23 GPU-days on 4 A100s.

Problem

Existing tabular ICL foundation models compress each row into a fixed embedding before a separate row-level learner, potentially losing feature-level detail; the paper seeks a more efficient architecture without sacrificing accuracy.

Method

EXAONE Tabular (Cross-Axis Summary Transformer) interleaves feature-axis attention (within each row, using per-item summary tokens) and support-conditioned item-axis attention (within each feature, using per-feature summary tokens) at every layer, keeping cell-level representations throughout. Pretrained only on an SCM synthetic prior; 20.81M params (classification), 21.11M (regression).

Results

On TabArena (51 datasets) the 20.81M-param classifier ranks first overall, beating TabPFN-3 by ~125 Elo and outperforming tuned ensembles/4hr AutoML; regression matches the 1.64B-param TabFM's accuracy at ~1/11 the inference cost (0.605s vs 7.0s per 1K samples). Second on TALENT/BCCO classification (0.006-0.007 behind TabFM), first on regression R2 and on ScoringBench's R2/RMSE/CRPS ranks.

Limitations

Native classification head caps at 10 classes; larger label spaces need Error-Correcting Output Code decomposition at inference, adding compute cost. Large support sets beyond configured limits require subsampling; authors flag context compression and representative-context selection as unexplored future work.

Problem

Deep tabular transformers are hard to interpret and their attention heads' individual roles are poorly understood, unlike in vision/NLP; the paper studies which heads actually matter for tabular prediction.

Method

Applies a gradient-based head importance score (expected absolute inner product of a head's activation with the loss gradient w.r.t. that activation) to TransTab (6 layers x 8 heads = 48 heads), plus a soft dynamic gating mechanism that learns per-head scaling during training, evaluated across 40 OpenML datasets (500-96,320 rows, 4-256 features).

Results

In 72.5% of experiments, pruning low-importance heads first causes the smallest AUC drop, while removing the highest-importance head first causes the largest drop (random removal falls in between); important heads are scattered across layers with no consistent layer-wise pattern, unlike vision/NLP transformers.

Limitations

All experiments use a single architecture, TransTab; the authors explicitly state generalization to other tabular transformer architectures (e.g., FT-Transformer, TabPFN-style ICL models) is untested.

Problem

Dense bidirectional attention makes large-context inference (e.g. TabPFN-style ICL over many rows) slow, and existing sparse-attention speedups need input structure or slow amortized clustering, so unstructured single-pass settings lacked a fast option.

Method

Fast recursive PCA-based clustering groups keys/queries into fixed power-of-two clusters, giving block-sparse attention dense-level per-interaction GPU latency, plus centroid-based compensation for excluded clusters to control error.

Results

On TabPFN-3 over TALENT datasets (93k-634k rows), speeds up inference 2-6x (scaling with data size) while keeping ≥99% of dense accuracy; also beats video-gen method SVOO (1.8x vs 1.4x speedup).

Limitations

Clustering/routing overhead dominates below ~25-30k tokens, making it slower than plain SpargeAttn there; compensation cost dominates past ~250k tokens; fixed (non-adaptive) cluster sizes are acknowledged as suboptimal.

Problem

It's unclear whether self-supervised pretraining actually helps tabular deep models when labels are scarce or data is missing at test time, despite common assumptions that it does.

Method

A mask-and-recover SSL objective (dual-branch MLP encoder, missingness masks, reconstruction plus consistency loss) is pretrained then fine-tuned, benchmarked against scratch training, Random Forest/LightGBM, and VIME/SCARF/SubTab across 14 OpenML tasks.

Results

SSL barely beats scratch (~0.895 vs ~0.890 AUC at 10% labels) but not significantly (p=0.626), still trails Random Forest (0.9015); helps under MCAR/MNAR test-time missingness (+0.0245/+0.0418 AUC) but not after multiple-comparison correction.

Limitations

All reported gains are statistically non-significant after Holm-Bonferroni correction; SSL degrades performance on datasets with high native missingness; robustness results use a single fine-tuning seed and small task counts limit statistical power.

Problem

Tabular transfer learning normally assumes shared features between source and target tables, which rarely holds in practice, blocking knowledge transfer across genuinely disjoint domains.

Method

CATTLE freezes a source transformer's top-layer key/value projection weights and inserts them into a target model's bottom two layers, letting the target learn only query weights, transferring generic attention 'rules' instead of domain-specific activations.

Results

Across 10 disjoint source-target OpenML pairs, CATTLE gets the best average rank (2.9) among 9 baselines (XGBoost, TransTab, XTab, CM2, FT-Transformer, etc.), a statistically significant 3.7% average AUROC gain.

Limitations

Which layers to transfer is chosen from prior heuristics, not theoretical grounding; authors admit they cannot fully explain why data-agnostic weight transfer works, since knowledge distribution across attention layers is not understood.

Problem

Global feature-attribution methods aggregate local SHAP scores but don't directly decompose predictive performance (R^2), leaving no exact accounting of each feature's contribution to a gradient-boosted model's overall fit.

Method

qshap computes an exact Shapley decomposition of R^2 by expanding squared loss into linear and quadratic SHAP terms, aggregating over leaf pairs instead of enumerating feature coalitions; a specialized fast path exploits oblivious trees' leaf symmetry.

Results

Feature contributions sum exactly to model R^2 (efficiency axiom holds, e.g. 0.7646=R^2 on California housing, p=9). On oblivious CatBoost (100 trees, p=100) runtime stays near-flat, ~0.09s to 0.14s as n grows from 1k to 100k.

Limitations

Restricted to squared-error loss and to XGBoost/LightGBM/CatBoost implementations; general (non-oblivious) trees keep O(L^2D^2) per-tree cost. The paper reports no head-to-head runtime comparison against TreeSHAP or SAGE.

Problem

Tabular foundation models and GBDTs are accurate but opaque; high-stakes decisions need models that are interpretable by design, not just accurate as a post-hoc explanation.

Method

Local distillation fits a regularized lasso linear "student" at each query point, using a black-box "teacher" (e.g. TabPFN) to define locality via prediction-similarity weights and anchor the fit via a weighted pseudo-observation; Gaussian-randomized refits assess feature-selection stability.

Results

On 17 UCI/OpenML datasets it nearly matches its TabPFN teacher's accuracy; on BRCA1 gene expression (536 patients) it edges out TabPFN (R^2 0.670 vs 0.666) and beats global lasso (0.579), with sparser local fits (94 vs 123 genes).

Limitations

When the teacher offers no global improvement (mu_hat<=1) the method just reverts to a plain global linear fit rather than trying to help locally; stability guarantees are proven only for the lasso (alpha=1), not elastic net.

Problem

Kinyarwanda, a morphologically rich Bantu language with 12M+ speakers, has no tabular representation-learning resource, blocking table QA and structured-data NLP for the language.

Method

Extends KinyaBERT-large (two-tier morphological transformer) with additive row/column/cell-type embeddings plus a learned table-structure attention bias (row/column/header bonuses); pre-trains with two new objectives, Masked Cell Recovery and Column Type Prediction.

Results

62.0% EM on new TabQA-kin benchmark, beating KinyaBERT-large by 5.7pts and mBERT/XLM-R by 11.7-12.7pts; however zero-shot GPT-4o/4o-mini match or edge it out overall (64.0%), dominating on lookup but far behind on aggregation.

Limitations

Pre-training corpus is only 172 tables (~35K cells), far smaller than English tabular corpora; evaluated on a single low-resource language with a small table corpus, limiting generality; learned attention-bias weights converged to ~5e-6 (effectively inactive); relies on a closed-source morphological analyzer.

Problem

In ERP-derived data lakes, column names are coded/abbreviated and metadata is weak, making it hard to automatically discover joinable or semantically related columns across tables for data integration.

Method

ColRel: Stage 1 embeds each column from schema cues + sampled values via a sentence-transformer (MiniLM/MPNet/BGE); Stage 2 optionally has GPT-4o-mini generate a short business-dictionary-grounded description appended before re-embedding, then ranks by cosine similarity.

Results

On the hard 'semantically related' industrial Public Housing benchmark, Stage 2 (BGE-m3) hits Hit@5/MRR 0.82/0.68 vs best classical matcher (Similarity Flooding) 0.53/0.53 and Stage 1 alone 0.65/0.43; gains are smaller on easier/clean ChEMBL data.

Limitations

On easy 'joinable' cases with strong value overlap, classical matchers (Cupid, COMA) still match or beat ColRel; one-to-one precision/F1 stay modest (~0.16-0.6) even with best settings; needs a curated business dictionary and per-column LLM calls, raising cost/privacy questions the authors leave to future work.