»Forward
Pass.

Problem

TFM inference is expensive: they stack many transformer blocks while GBDTs stay competitive on CPU. Is that depth actually needed?

Method

Replace a contiguous window of up to 96% of the blocks with a closed-form, label-free affine map fitted on intermediate activations, keeping the pretrained head; analyse per-block criticality across model families.

Results

On 8 TFMs and 171 classification datasets, dropping half the blocks costs ≤0.002 median AUROC; half-depth TabFM still beats all full-depth TFMs and tuned GBDTs. Up to 6× GPU speedup at ~0.03 AUROC cost.

Limitations

Single-block compression fails for regression rank quality and for LimiX-16M on very wide tables. Compression vs. shallow pretraining only compared at nanoTabPFN scale (1.4M params); classification only.

Problem

Which forms of inference-time compute actually help strong pretrained TFMs, and at what cost?

Method

Systematic study over adaptation (new DiagScale: a diagonal query-key similarity update training 0.003–0.03% of parameters), prediction aggregation, and context construction on TabArena and large OpenML tables.

Results

DiagScale matches full fine-tuning across three backbones. Greedy selection over 96 configurations cuts error 2.4% vs. default, while uniform averaging hurts. Context expansion gives no consistent gain; attention-guided retrieval helps only on some large tables.

Limitations

Adaptation plus aggregation stack but need much more compute than default inference. Context-construction benefits are task- and regime-dependent; Elo gains are not additive.

Problem

PFNs approximate a Bayesian posterior predictive from a synthetic prior, but how well they do so is rarely measured.

Method

Decompose deployment error into prior, context and amortization error, and measure amortization error against a reference PPD (MCMC) under a TabICLv2-style graph-SCM prior; then test Bayesian model averages of independently pretrained PFNs.

Results

PFNs generally recover the target PPD well, but residual amortization error persists in rare settings of the prior even when scaling model size, so independently trained PFNs disagree. Model averaging reduces this uncertainty and improves predictions, up to full-size TabICLv2.

Limitations

Likely: reference-PPD comparison is only feasible on small synthetic prior settings, and conclusions are tied to the chosen prior. Averaging multiplies inference cost.

Problem

Is ICL in TFMs explained by a fixed-weight account (features alone determine each context label's weight), as for linear regression or kernel smoothing?

Method

Show that derivative-based tests are confounded by the label standardization TFM packages apply ('standardization trap'); propose two certificates using only predictions at standardized labels that reject fixed-weight and separable-nonlinear explanations.

Results

On five public TFMs, changing one context label changes how other labels influence the prediction ('joint processing'). It emerges during training, and attention scores carry most of the measured interaction.

Limitations

Diagnostic only: it rejects simple explanations without giving a replacement. Main experiments use few datasets (four in the core grid, mostly regression-style).

Problem

Predictive refinement across TFM depth is uneven and concentrated in late layers. Can depth be used better?

Method

Retro: a TFM with retrospective inference, combining Attention Residuals (adaptively reweight earlier-depth outputs) and query-conditioned element-wise Gated Attention to revise contextual updates.

Results

Retro shifts refinement earlier and more evenly across depth and ranks in the top three by Elo on TabArena, TALENT and RelArena, on the Pareto frontier.

Limitations

Likely: top-three rather than first, within the evaluated candidate sets; extra historical-read parameters and compute are added, and the gains rest on pretraining a new model.

Problem

TFMs reprocess the whole training set at every forward pass; truncating the context cuts cost but hurts accuracy.

Method

Activation alignment: fit a lightweight linear map on synthetic unlabeled data so a partial-context 'student' activations match a full-context 'teacher'; no GPU needed, trains in seconds to minutes.

Results

On 38 TabArena classification datasets with TabPFN-3 and TabFM, aligned students beat unaligned ones at all context budgets, recovering nearly half the teacher's advantage in low-data regimes.

Limitations

Classification only and two models tested. Five of 38 datasets show a negative effect for both models, with no characterisation of why. Regression is untested.

Problem

Compressing in-context examples for tabular ICL trades accuracy against throughput: fixed subsets lose accuracy, per-query retrieval kills cache reuse.

Method

QCOC compiles the full KV cache once into compact memory: cluster states into joint-KV prototypes with multiplicities, then calibrate values against attention queries via an anchored closed-form solution.

Results

On 64 OpenML-CC18 datasets it has the best mean accuracy among compression/retrieval methods; on long tables it is 0.23 pp below full context. 8,192→512 slots (10.5× compression); up to 508× faster than dynamic retrieval and 1.98× faster than full context on CPU.

Limitations

Speedups exclude one-time compilation cost. Likely: classification-focused and a small accuracy gap remains versus full context.

Problem

It is unclear how individual in-context demonstrations shape a TFM's prediction, and mislabeled or redundant context examples can hurt.

Method

TICDA: train linear surrogates on TFM latent embeddings to score each demonstration's influence in a single forward pass, avoiding Shapley-style resampling and gradients that ICL lacks.

Results

Gives the best compromise against competitors on four tasks: label-error detection, context curation that keeps accuracy at lower cost, scores transferable across TFMs, and active-learning acquisition.

Limitations

Likely: attributions come from a linear surrogate, so they approximate nonlinear ICL behaviour; 'best compromise' rather than best on every task.

Problem

Tabular anomaly detection lacks strong deep features, and no anomalies exist for tuning; the reference set may itself contain anomalies.

Method

ZEN: kNN distance on frozen TabPFN features from selected layers with a task-suited extraction procedure; FOCUS fine-tunes TabPFN on the reference set to separate normal from anomalous samples.

Results

On ADBench, ZEN reaches higher mean AUROC than every baseline without fine-tuning, FOCUS improves further, and the approach transfers across PFN models.

Limitations

Needs a GPU (~3,500 samples/min on an A100), slower than classical detectors. Scale is limited: ADBench subsamples large sets, and cost grows with reference-set size.

Problem

TFM quality depends on inferring task-specific relationships from context, which standard synthetic pretraining may not encourage.

Method

TAFFY: an In-Context Diversity Prior (contexts from related environments via interventions/shifts on a shared causal process) plus a Task-Conditioned Looped Transformer with a task-level gate for iterative refinement.

Results

Lowest average rank across six classification and five regression benchmark datasets.

Limitations

Gating is a single task-level scalar on the final refinement, so it cannot vary per example, feature or layer. Likely: evaluation on only 11 datasets is small versus TabArena-scale.