--- abstract: | Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in `\ntfms{}`{=latex} state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only $20\%$ of the original model's parameters while achieving comparable performance. The code is available at . bibliography: - strings.bib - references.bib - lib.bib - proc.bib - myproc.bib --- ```{=latex} \newcommand{\theHalgorithm}{\arabic{algorithm}} ``` ```{=latex} \newcommand{\takeaway}[1]{% \refstepcounter{takeaway}% \noindent\fcolorbox{blue!40!black}{blue!5}{% \parbox{0.97\columnwidth}{% \textbf{Takeaway~\thetakeaway.} #1% }% }\par\medskip } ``` ```{=latex} \newcommand*\numcircledmod[1]{\raisebox{.5pt}{\textcircled{\raisebox{-.9pt} {#1}}}} ``` ```{=latex} \newcommand{\numcircledlabel}[1]{% \refstepcounter{numcircled}% makes it ref-able \label{#1}% \tikz[baseline=(char.base)]{ \node[shape=circle,draw,inner sep=1.2pt] (char) {\thenumcircled}; }% } ``` ```{=latex} \newcommand{\TabICL}{\textit{TabICL}} ``` ```{=latex} \newcommand{\TabPFNone}{\textit{TabPFN(v1)}} ``` ```{=latex} \newcommand{\RetrainedTabPFNone}{\textit{TabPFN(v1)-Retrained}} ``` ```{=latex} \newcommand{\TabForest}{\textit{TabForest}} ``` ```{=latex} \newcommand{\TabForestPFN}{\textit{TabForestPFN}} ``` ```{=latex} \newcommand{\TabPFNMix}{\textit{TabPFN-Mix}} ``` ```{=latex} \newcommand{\TabPFNtwo}{\textit{TabPFN(v2)}} ``` ```{=latex} \newcommand{\TabPFN}{\textit{TabPFN}} ``` ```{=latex} \newcommand{\TabPFNtwohalf}{\textit{TabPFN(2.5)}} ``` ```{=latex} \newcommand{\LimixS}{\textit{LimiX-2M}} ``` ```{=latex} \newcommand{\LimixL}{\textit{LimiX-16M}} ``` ```{=latex} \newcommand{\realTabPFNtwo}{\textit{RealTabPFN(v2)}} ``` ```{=latex} \newcommand{\realTabPFNtwohalf}{\textit{RealTabPFN(2.5)}} ``` ```{=latex} \newcommand{\nanoTabPFN}{\textit{nanoTabPFN}} ``` ```{=latex} \newcommand{\loopedNanoTabPFN}{\textit{nanoTabPFN$_{looped}$}} ``` ```{=latex} \newcommand{\ntfms}{6} ``` ```{=latex} \newcommand{\nexps}{6} ``` ```{=latex} \newcommand{\TabArena}{\textit{TabArena}} ``` ```{=latex} \newcommand{\PMLB}{\textit{PMLBmini}} ``` ```{=latex} \newcommand{\orangesolid}{{\color{orange}\textbf{--}}} ``` ```{=latex} \newcommand{\orangedashed}{{\color{orange}\textbf{-~-}}} ``` ```{=latex} \newcommand{\bluesolid}{{\color{blue}\textbf{--}}} ``` ```{=latex} \newcommand{\bluedashed}{{\color{blue}\textbf{-~-}}} ``` ```{=latex} \newcommand{\greensolid}{{\color{green}\textbf{--}}} ``` ```{=latex} \newcommand{\darkbluesolid}{{\color{darkblue}\textbf{--}}} ``` ```{=latex} \newcommand{\darkorangesolid}{{\color{darkorange}\textbf{--}}} ``` ```{=latex} \newcommand{\redsolid}{{\color{red}\textbf{--}}} ``` ```{=latex} \newcommand{\brownorangesolid}{{\color{brownorange}\textbf{--}}} ``` ```{=latex} \newcommand{\rqone}{RQ1: How does inference unfold across depth in TFMs during prediction?} ``` ```{=latex} \newcommand{\rqtwo}{RQ2: How do the inference dynamics of TFMs compare to those observed in LLMs?} ``` ```{=latex} \twocolumn[ \icmltitle{Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models} % It is OKAY to include author information, even for blind submissions: the % style file will automatically remove it for you unless you've provided % the [accepted] option to the icml2026 package. % List of affiliations: The first argument should be a (short) identifier you % will use later to specify author affiliations Academic affiliations % should list Department, University, City, Region, Country Industry % affiliations should list Company, City, Region, Country % You can specify symbols, otherwise they are numbered in order. Ideally, you % should not use this facility. Affiliations will be numbered in order of % appearance and this is the preferred way. \icmlsetsymbol{equal}{*} \begin{icmlauthorlist} \icmlauthor{Amir Rezaei Balef}{yyy,comp,sch} \icmlauthor{Mykhailo Koshil}{yyy,comp} \icmlauthor{Katharina Eggensperger}{yyy,comp} % \icmlauthor{Firstname4 Lastname4}{sch} % \icmlauthor{Firstname5 Lastname5}{yyy} % \icmlauthor{Firstname6 Lastname6}{sch,yyy,comp} % \icmlauthor{Firstname7 Lastname7}{comp} % %\icmlauthor{}{sch} % \icmlauthor{Firstname8 Lastname8}{sch} % \icmlauthor{Firstname8 Lastname8}{yyy,comp} %\icmlauthor{}{sch} %\icmlauthor{}{sch} \end{icmlauthorlist} \icmlaffiliation{yyy}{TU Dortmund University, Dortmund, Germany} \icmlaffiliation{comp}{Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany} \icmlaffiliation{sch}{University of Tübingen, Tübingen, Germany} \icmlcorrespondingauthor{Amir Rezaei Balef}{amir.balef@tu-dortmund.de} % You may provide any keywords that you find helpful for describing your % paper; these are used to populate the "keywords" metadata in the PDF but % will not be shown in the document \icmlkeywords{Machine Learning, ICML} \vskip 0.3in ] ``` `\printAffiliationsAndNotice{}`{=latex} # Introduction {#sec:intro} Tabular foundation models (TFMs) have demonstrated that transformer-based architectures using in-context learning (ICL) can achieve state-of-the-art performance on small to medium-sized predictive tabular tasks [@erickson2025tabarena]. Extending their flexibility and increasing performance, as well as exploring their potential for real-world industrial applications, form active areas of research. However, despite the ubiquity of tabular data, particularly in high-stakes settings such as healthcare and finance, the internal mechanisms of TFMs remain poorly understood. Studying mechanism of inference by means of mechanistic explainability is necessary for two reasons. Firstly, it enables the identification of limitations in existing architectures and the detection of failures during model training and generalization, which are critical for advancing model development. Secondly, understanding how models use data for inference improves the predictability of model behavior in unseen settings, which is critical for reliable deployment [@sharkey2025open]. It is unclear whether insights from architecturally similar large language models, i.e., LLMs [@gromov2025the; @sun2025layesaspainters], transfer directly due to differences in inference strategies and training. Unlike most LLMs, prominent TFMs (e.g., `\TabICL{}`{=latex}, `\TabPFN{}`{=latex}) are encoder-only, smaller in size, do not perform auto-regressive inference, use attention between features, and are row-invariant, operating over sets of examples rather than sequences. Furthermore, LLMs are trained for next-token prediction on real-world data, making memorization of surface-level facts viable. This enables LLMs to derive the associations between "Paris" and "France" [@petroni2019language] without such information being contained in the context. In contrast, TFMs are typically pre-trained to solve synthetic tabular tasks using ICL; therefore, it remains unclear whether and which inductive biases they learn or memorize.

Individually trained decoders (, ) exhibit good performance early, showing that representations are descriptive but not aligned with the original decoder. For the original decoder, the sudden increase in ROC-AUC () and balanced accuracy () at different layers suggests inference stages.
These key differences motivate our study of the internal dynamics of tabular ICL models. More importantly, the smaller size and lower inference costs of TFMs enable large-scale studies that are often infeasible for LLMs. We present an illustrative experiment that studies layer-wise performance, in which inference is halted after a given layer and the resulting hidden state is passed to a decoder to make a prediction. For LLMs, this evaluation typically applies the original decoder (i.e., the detokenizer) and is referred to as the \"logit lens\" [@nostalgebraist2020interpreting]. @belrose2023eliciting showed that this can be brittle and proposed a \"tuned lens\", a learned affine transformation in each layer. For TFMs, we can efficiently continue pre-training the original decoder and adapt it to each layer. We show in Figure `\ref{fig:early_exit_intro}`{=latex} that our \"tabular tuned lens\" leads to substantially different behaviour compared to the \"logit lens\" and that good performance is already possible in early layers. This raises the question of how many layers are actually needed to perform effective inference in TFMs, i.e., *Is one layer enough?* To study this in detail, we provide the first large-scale mechanistic study of layer-wise dynamics over six state-of-the-art TFMs. Specifically, we carefully designed and adapted `\nexps{}`{=latex} experiments to study TFMs, inspired by mechanistic studies of LLMs, including structural interventions, ablations, and probing. Each experiment provides distinct and insightful observations; by combining these insights, we address the following two research questions. **`\rqone{}`{=latex}** We identify where and how predictions are formed and assess the robustness of inference to layer ablations. We find that iterative inference and overlapping computations emerge across several state-of-the-art TFMs, which informs the design of more efficient architectures. **`\rqtwo{}`{=latex}** We relate our findings to observations in LLM behavior, focusing on the emergence of inference stages and specialized layers [@lad2025remarkable]. While it is evident that TFMs form block-layer structures like LLMs, they are more sensitive to layer swapping. Furthermore, although less pronounced, we observe similar stages of inference. Finally, based on these findings, we design a proof-of-concept experiment showing that a single-layer TFM can achieve similar performance as a six-layer TFM, if we train it by *looping* layers. We conclude our study with three suggestions to explore for architectural improvements. # Related Work and Background This section reviews prior work on understanding inference mechanisms, with a focus on representation analysis and layer-wise ablations in large language models. We also provide current views on how inference evolves in transformer models and related work on TFMs. `\label{sec:related_work}`{=latex} ## Methods for Studying Inference Mechanisms Studying neural networks by examining how their internal parts, such as neurons, layers, or attention heads, interact to perform computations is commonly referred to as mechanistic interpretability (MI) [@olah2018building; @olah2020zoom; @zou2023representation; @sharkey2025open]. MI methods broadly fall into two categories. **Reverse engineering** approaches decompose a trained network into its constituent components and seek to infer the functional role each component plays in the overall computation [@wang2022interpretability]. In contrast, **concept-based interpretability** starts from hypothesized concepts or variables and identifies network components that are critical to implement a concept [@elhage2021mathematical]. Overall, MI aims to understand model behavior in order to detect failures or unsafe behavior, predict generalization and emergent abilities, modify internal mechanisms to align behavior with human objectives, improve training and inference, and extract latent knowledge to better model the world. While models for language and vision are already studied at the level of neurons [@sharkey2025open], methods and insights for tabular models lag behind. Given their architectural similarity to TFMs, we continue by briefly describing related methods for studying LLMs. First, studying **learned representations and data embeddings** can provide insights into the properties of hidden states. Several studies focus on the interpretability of LLMs using representation analysis [@dar2023analyzing; @song2025llm; @bronzini2025hyperdimensional], for example, by using **probing classifiers** [@rogers2020primer] or projecting representations into the model's vocabulary space [@nostalgebraist2020interpreting; @din2024jump; @belrose2023eliciting]. Predictions of probes are then used to judge how much of the final prediction has already been formed at different stages of inference. Combining this with interventions, we can directly identify whether a representation is critical for certain predictions [@wang2022interpretability; @ghandeharioun2024patchscopes]. Another line of work focuses on **layer-wise dynamics** of transformer models in LLMs. @skean2025layer showed that intermediate layers often outperform final layers on downstream tasks. Similarly, @sun2025layesaspainters analyzed representational alignment via cosine similarity and found that middle layers occupy a shared representation space, while early and late layers are more specialized. Through structural interventions such as skipping, reordering, or repeating layers, they further show that some middle layers can be removed without catastrophic failure, whereas repeating a single layer severely degrades performance. Complementing these findings, @lad2025remarkable investigated the robustness of LLMs to structural interventions by deleting or swapping adjacent layers during inference. They found that early and final layers are the most sensitive, while middle layers are remarkably robust. This robustness is attributed to the transformer's residual architecture. They further identified four depth-dependent stages of LLM inference: early layers perform detokenization, middle layers refine task- and entity-specific features, mid-to-late layers ensemble predictions, and final layers sharpen outputs by suppressing irrelevant features. These analyses are straightforward to conduct for TFMs; hence, they provide our starting point. ## Two Views on Inference Dynamics There are two non-competing hypotheses offering explanations for how inference evolves in artificial neural models, specifically transformers. First, the **circuit hypothesis** presents a mechanistic view, proposing that individual model components perform specialized, modular roles along distinct computational pathways [@conmy2023towards]. Evidence includes knowledge neurons and circuits [@dai2022knowledge; @yao2024knowledge], MLP units that suppress token repetition in copy-suppression mechanisms [@mcdougall2023copy], and task-general or 'universal' units [@gurnee2024universal; @voita2024neurons]. Second, the **iterative inference hypothesis** proposes that the residual stream refines representations iteratively rather than learning entirely new ones [@greff2016highway; @jastrzebski2017residual], for example, through skip connections in ResNet architectures. Similarly, in transformer models, the residual stream accumulates each layer's contribution via linear projections, progressively refining representations in a flexible, high-dimensional space [@elhage2021mathematical]. This suggests that each layer incrementally updates the residual stream to improve the prediction [@geva2022transformer; @belrose2023eliciting]. Iterative inference is additionally supported by self-repair [@wang2022interpretability; @mcgrath2023hydra; @rushing2024explorations], showing that later layers correct or mitigate errors of earlier layers. This behaviour arises if multiple layers perform similar or overlapping computations [@rushing2024explorations]. As subsequent layers can compensate for removing earlier layers, we must consider this for ablation-based interpretability. Importantly, recognizing that models iteratively refine their internal representations allows us to make more effective use of a given model size, enabling smaller models without sacrificing performance. A popular example is a **looped transformer** where repeating (looping) transformer blocks (instead of training deeper models) improves performance [@dehghani2018universal; @gong2025makes; @zhu2025scaling]. We build on this idea and study the role of overlapping computations and iterative inference in TFMs to explore the potential of recurrent model components to improve efficiency and performance. ## Tabular Foundation Models TFMs are an emerging type of transformer-based models, pre-trained to solve supervised learning tasks via in-context learning (ICL). Specifically, provided with a support set of (labeled) training samples, the model has learned to approximate Bayesian inference for the (unlabeled) query samples. This is referred to as ICL, since the model performs inference without updating weights. The current generation of TFMs is commonly pre-trained on synthetic tasks generated by a prior distribution, i.e., they are based on a prior-data-fitted network [@muller-iclr22a]. `\TabPFNone{}`{=latex} [@hollmann-iclr23a] was the first widely used model operating in this fashion on tabular data, with a vanilla transformer backbone. `\TabPFNtwo{}`{=latex} [@hollmann-nature25a] improves by adding an attention mechanism within tokens, in addition to cross-token attention. `\TabICL{}`{=latex} [@qu2025tabicl] shares a similar backbone with `\TabPFNtwo{}`{=latex}, but additionally introduces a transformer-based compression that efficiently transforms rows into semantically rich embeddings. Specifically, `\TabICL{}`{=latex} employs a two-stage architecture: first, it compresses the data, and then uses these embeddings to make predictions. @LimiX further extends this line of work by introducing two TFMs, `\LimixS{}`{=latex} and `\LimixL{}`{=latex}, which offer enhanced handling of missing values and support retrieval-based ensemble methods. More recently, `\TabPFNtwohalf{}`{=latex} [@grinsztajn2025tabpfn] scales to datasets with up to $50\,000$ rows and $2\,000$ features, achieving state-of-the-art performance. Notably, `\TabICL{}`{=latex} and `\TabPFNone{}`{=latex} are the only open-source TFMs with access to training and prior-data generation. Prior work on understanding how these TFMs operate is largely limited to the `\TabPFN{}`{=latex} model family. @nagler-statistical23a analyzed how PFNs approximate predictive posteriors, providing a statistically grounded framework to understand how their ICL mechanism works with a focus on the bias-variance tradeoff and the need for localization. @mccarter2024exactly studied the behavior of `\TabPFNone{}`{=latex} and `\TabPFNtwo{}`{=latex} on out-of-distribution tasks to assess how well the model generalizes on data that is not described by its prior. @zheng2025tablessignalsrevealingspectral used concepts from signal reconstruction and frequency response analysis to investigate the inductive biases of `\TabPFNtwo{}`{=latex}, demonstrating that `\TabPFNtwo{}`{=latex} can dynamically adjust its frequency capacity to the number of support samples provided. @ye2025closer showed that `\TabPFNtwo{}`{=latex} learns highly predictive features and can be used to embed tabular data for downstream tasks. We aim to study general patterns in TFMs, going beyond focusing on individual models. # Empirical Experiments {#sec:experiments} Here, we present our main empirical results and start by describing the general setup and models considered. **Models.** In the experiments we study two state-of-the-art open-source tabular ICL models, `\TabPFNone{}`{=latex} [@hollmann-iclr23a] and `\TabICL{}`{=latex} [@qu2025tabicl] as well as four open-weight models, `\TabPFNtwo{}`{=latex} [@hollmann-nature25a], `\TabPFNtwohalf{}`{=latex} [@grinsztajn2025tabpfn], `\LimixS{}`{=latex} and `\LimixL{}`{=latex} [@LimiX]. We run all models in their default configurations with their standard data pre-processing; however, if applicable, we set the number of ensembles to $1$. Additional details on the model architectures and their key differences are provided in Appendix `\ref{app:architectures}`{=latex}. **Datasets.** For all experiments, we use binary classification tasks.[^1] Specifically, we use a subset of $15$ tasks from `\TabArena{}`{=latex}, selected to match the model constraints with $\leq10{,}000$ samples and $\leq100$ features [@erickson2025tabarena], as well as $34$ tasks from on `\PMLB{}`{=latex} [@knauer2024pmlbmini], containing small datasets with $\leq500$ samples. We report ROC-AUC, averaged across folds and repetitions (see Appendix `\ref{app:benchmarks}`{=latex} for more details). In the following, we describe our `\nexps{}`{=latex} experiments, each accompanied by its description, results, observations, and main takeaways highlighted in blue boxes. Each experiment studies one aspect of the inference process, and we order the experiments from low-level, comparing embeddings, to higher-level, ablating layers, as illustrated in Figure `\ref{fig:layer_experiments}`{=latex}.
Representation Analysis
Repeating layers
Skipping layers
Swapping layers
Experiments analyzing the inference process of tabular ICL models.
**`\numcircledlabel{exp:emb_sim}`{=latex} Embedding similarity.** First, we study the similarity of the representation space across layers. Following prior work [@sun2025layesaspainters; @lad2025remarkable], we examine both the averaged absolute cosine similarity and linear centered kernel alignment (CKA) [@kornblith2019similarity] between the output embeddings of each layer (see Appendix `\ref{app:metrics}`{=latex} for more details). High similarity between layers may suggest a shared representation space, whereas low similarity indicates that a layer significantly transforms the embedding space. Results in Figure `\ref{fig:layers_similarity}`{=latex} show that adjacent layers generally exhibit a high similarity. In addition, all models, except `\TabPFNone{}`{=latex} and `\TabICL{}`{=latex}, form clearly visible blocks of sequences of layers in which representation remains highly similar, suggesting small incremental updates to the embedding space. Moreover, while cosine similarity only reflects individual vector alignment, @davari2022reliability argued that CKA can be sensitive to certain affine transformations, e.g., non-isotropic scaling. This is evident in cases where high cosine similarity coexists with low CKA. One possible explanation is that specific attention heads stretch or scale certain embedding dimensions, producing high cosine similarity while reducing CKA. Per-benchmark results are reported in Appendix `\ref{app:embedding_similarity}`{=latex}.

Embedding similarity over different layers of the respective models (average over all datasets), upper triangular – linear CKA, lower triangular – cosine similarities.
```{=latex} \takeaway{TFMs often form blocks in which the embeddings remain similar.} ``` **`\numcircledlabel{exp:sep_gap}`{=latex} Separation gap.** Unlike LLMs, TFMs have a fixed task, for example, classification. This allows us to track progress towards the goal of separating classes. To do so, we study the distance between representations of samples within the same class and between different classes across layers. We refer to the *separation gap* as the difference between the intra-class and inter-class distances. We randomly sample $100$ pairs of data-points (within and across classes) and compute the pairwise cosine similarities between representations.[^2] We compute the gap for support and query samples. Additionally, for models that use attention across features, we also compute the gap value for the (grouped) features and label. See Appendix `\ref{app:separation_gap}`{=latex} for a formal definition and more details. A large separation gap indicates highly discriminative features, making samples from different classes easier to distinguish. We study how this gap develops across layers to determine whether it gradually increases or undergoes sudden jumps. In Figure `\ref{fig:separation_gap}`{=latex}, we show that the separation gap for all models generally increases with depth, with some fluctuations.[^3] Notably, the gap for the label embedding in the support set (`\darkbluesolid{}`{=latex}) stands out, as it already starts at a high value due to the presence of label information. Comparing the development of the gap of the label embedding (`\redsolid{}`{=latex}) and the feature embedding (`\brownorangesolid{}`{=latex}), we observe a slight delay, suggesting that the models first focus on forming separable representations for the features and then for the label. Furthermore, we observe significant differences between models: `\LimixS{}`{=latex} already shows a large gap in the early layers, whereas `\TabPFNtwo{}`{=latex} shows a noticeable jump in layer $5$. ```{=latex} \takeaway{TFMs incrementally increase the distance between samples from different classes.} ```

Separation gap (mean difference between inner-class and intra-class distances) across layers of the embedding network. Bold lines indicate the average across tasks, and thin lines represent results for individual datasets.
**`\numcircledlabel{exp:prob_cls}`{=latex} Probing classifiers.** We use probing classifiers to measure how much information relevant to a downstream task is contained in embeddings from different model layers [@belinkov2022probing]. We first extract query embeddings (as support embeddings inherently contain label information from the training data) from the hidden states of each layer during the forward pass. For this experiment, we extend the query set by including half of the original training set (excluded from the support set) to serve as training data for the probing classifier, in our case logistic regression. We perform linear probing across layers by evaluating classifiers on representations from the same and different layers. Additional details and alternative probing methods are provided in Appendix `\ref{app:probing_classifiers}`{=latex}. We use the performance of this probe as an approximation to the mutual information between the embedding and the quantity of interest (our label). The results in Figure `\ref{fig:linear_probing}`{=latex} show a consistent, though model-dependent, pattern: a probe trained on layer $(i)$ generalizes better to embeddings from later layers $(j > i)$ than the reverse, indicating that later layers retain information from earlier layers while also encoding new features that are not present in lower layers. ```{=latex} \takeaway{Each layer cumulatively enriches the representation by adding new features while preserving previous ones.} ```

Probing classifiers (logistic regression) normalized AUC trained on embeddings at different layers of the models.
**`\numcircledlabel{exp:tab_lens}`{=latex} \`\`Tabular" Logit Lens.** For the next experiment, we use our adapted version of the popular \"logit lens\" method [@nostalgebraist2020interpreting]. As shown in the introduction, we can not rely on the original final decoder and use individual decoders per layer (see also irregular entropy patterns across layers in Figure `\ref{app:fig:early_exit_predition_entropy}`{=latex}). Following @Kuken2025EarlyStopping, we continue pre-training our final decoder individually for each layer on synthetic datasets generated by `\TabICL{}`{=latex} priors (details are provided in Appendix `\ref{app:logit_lens}`{=latex}). Then, after each layer, we pass the embeddings to its individual decoders. We study the performance of individual decoders as a proxy to measure whether the features crucial for a prediction (for a given task) were already formed at a certain layer. Results in Figure `\ref{fig:early_exit}`{=latex} show that high performance can already be achieved in early layers for all models. Furthermore, performance increases abruptly for all models, and some models show further incremental improvements (e.g., `\TabPFNtwohalf{}`{=latex}). Notably, using individual decoders yields faster and more reliable predictions than applying the original decoder as a logit lens.

“Tabular” Logit Lens on the respective tabular foundation model.
```{=latex} \takeaway{Representations are already formed for a reliable prediction in the early layers, but not necessarily aligned with the original decoder.} ``` **`\numcircledlabel{exp:layer_abl}`{=latex} Layer ablation.** We perform layer ablation and reconfiguration experiments to study the role and contribution of individual layers. We manipulate the execution order of layers by skipping, repeating, and swapping them, and measure the resulting performance of the forward pass (see Appendix `\ref{app:layer_interventions}`{=latex} for more details and results). Skipping a layer indicates the layer contribution. By repeating a layer, we test whether the model has learned to perform the same refinement iteratively across layers. In such a case, repeating the layer would be expected to improve performance. Finally, swapping layers tests sequential representational alignment. If layers are not aligned sequentially, the order of execution can be changed without loss of performance. Figure `\ref{fig:layer_interventions}`{=latex} shows that skipping an early layer leads to the largest performance degradation, whereas skipping middle and later layers results in little to no performance drop. This indicates that layers contribute differently to forming the final predictions. Repeating certain layers results in slight performance improvements for models such as `\LimixL{}`{=latex} and `\TabPFNone{}`{=latex}, supporting the idea of iterative refinement. Models are generally sensitive to layer swapping. One reason could be that some layers learn specialized, co-adapted representations that swapping layers disrupts this feature hierarchy, leading to degraded performance. Another explanation is that swapping layers simultaneously applies interventions to two layers, thereby making recovery more difficult. ```{=latex} \takeaway{Early layers contribute the most, while later layers perform iterative refinement of the representation.} ``` **`\numcircledlabel{exp:self_rep}`{=latex} Self-repair.** Here, we study whether TFMs exhibit self-repair mechanisms and, thus, whether layers perform similar or overlapping computations. As observed, TFMs are generally robust to ablating layers. However, it is unclear whether this robustness arises from self-repair or from layer redundancy, as we have measured performance only at the final layer. We use our \"tabular logit lens\" to measure the performance of all subsequent layers after skipping a layer (see Appendix `\ref{app:self_repair}`{=latex} for more results and details). If the TFM can recover from dropping a layer, it has learned to self-repair, and layers overlap in functionality. Results in Figure `\ref{fig:self_repair}`{=latex} show that interventions in early layers cannot be recovered and, thus, implement unique functionality; however, self-repair is clearly visible in middle and later layers, especially for `\TabPFNtwo{}`{=latex}. ```{=latex} \takeaway{Self-repair generally occurs after layer ablations, except for the first layer.} ```

Layer ablation effect on the performance of model.
![**`\numcircledmod{\ref{exp:self_rep}}`{=latex} Self-repair** analysis under layer skipping. The Tabular Logit Lens measures model performance at each layer (intermediate performance). The solid black line shows intermediate performance without intervention. Colored lines, from blue (early) to orange (late), show intermediate performance after layer ablations, with cross markers indicating skipped layers. Dashed lines connect the first layer after a skipped layer to its original performance; a drop followed by recovery indicates self-repair. ](./figures/c2_skipping_layer_all.png){#fig:self_repair width="0.95\\linewidth"} # Discussion {#sec:discussion} We will now discuss the results in the context of our research questions and draw connections to prior work. **`\rqone{}`{=latex}** Our experiments show that the separation gap increases and each successive layer increases the distance between feature embeddings of different classes. Additionally, probing experiments show that each layer introduces increasingly stronger and more informative features. These results strongly support the iterative inference hypothesis, as they show that each layer's contribution is incremental. In the embedding similarity experiment, we observe that a block structure emerges in the largest models (e.g., `\TabPFNtwohalf{}`{=latex} and `\LimixL{}`{=latex}). Within each block, the embedding changes gradually, whereas it changes more drastically between blocks. Furthermore, we observed that while most layers are robust to intervention (see layer ablation and self-repair experiments), a small number, especially the first layer in most models, are highly sensitive to being skipped, suggesting specialization. Interestingly, two models showed a distinct pattern: `\TabICL{}`{=latex} and `\LimixS{}`{=latex} exhibit more robustness to skipping early layers compared to other models, as evidenced in Figure `\ref{fig:layer_interventions}`{=latex}. `\TabICL{}`{=latex}'s prediction module operates directly on already processed embeddings from the column embedder and row-wise interaction module, and `\LimixS{}`{=latex} uses an RBF-kernel preprocessing step. This results in strong features being produced upfront; consequently, the models rely less on the first transformer layer. Since `\LimixL{}`{=latex} does not employ the same preprocessing and is more dependent on the first few layers, we conclude that the primary functional role of the early layers is to map the representations from the input encoder to representations that are suitable for the residual stream operations. **`\rqtwo{}`{=latex}** Beginning with differences, most notably and in contrast to LLMs, redundancy plays a larger role in the middle layers, as suggested by the relatively stable, high performance observed in the early-exit experiment shown in Figure `\ref{fig:early_exit}`{=latex}. This statement is further supported by the stark decreasing entropy in early layers (as shown in Figure `\ref{app:fig:early_exit_predition_entropy}`{=latex} in the Appendix), which means that a confident prediction is formed early on. Additionally, TFMs are considerably more sensitive to layer swapping than LLMs [@sun2025layesaspainters], with this sensitivity being especially pronounced in the `\TabPFNtwo{}`{=latex} model. While the final layer in both model types increases the separation gap, its functional role differs: in LLMs, removing or disrupting the last layer typically leads to a noticeable performance drop, likely because it performs residual sharpening (i.e., amplifying and consolidating already-formed representations to ensure confident token-level predictions). In TFMs, by contrast, the last layer seems less important; although it improves class separation, its removal or perturbation does not degrade performance to the same extent, suggesting that TFMs rely less on late-stage residual refinement and more on earlier layers for forming decisive representations. However, we also find clear commonality in the inference dynamics of TFMs and LLMs. Firstly, early layers are crucial, and their ablation drastically reduces performance. Secondly, we observe the emergence of blocks of layers operating on similar embeddings. The presence of blocks of layers with high similarity suggests the existence of stages of inference (see related work in Section `\ref{sec:related_work}`{=latex}). However, we argue that these stages are distributed differently across the layers in TFMs. To reflect this, we adjust the names of the stages accordingly, see Appendix `\ref{app:further_discussion}`{=latex} for a detailed comparison. Based on Figure `\ref{fig:early_exit}`{=latex}, the first stage [latent mapping]{.smallcaps} is an extension of the input encoder, where representations are transformed into enriched feature representations for subsequent layers. For models with a more advanced encoding, such as `\LimixS{}`{=latex} and `\TabICL{}`{=latex}, this stage is less pronounced. The second stage is [feature engineering and labeling]{.smallcaps}, during which all models show rapid improvements in individual decoder performance. During this stage, features of the same class move closer together, while features of different classes gradually separate, and labels are formed iteratively. The third stage, [prediction ensembling]{.smallcaps}, is less pronounced and only clearly observed in `\TabPFNtwo{}`{=latex} and `\TabPFNtwohalf{}`{=latex}. It occurs when the early-exit performance of the individual decoder (`\orangesolid{}`{=latex}) has already converged, but the original decoder (`\bluesolid{}`{=latex}) continues to improve. In this stage, representations are transformed to better align with the original decoder. The final stage, [prediction calibration]{.smallcaps}, happens when both decoders have similarly high performance, but the balanced accuracy exhibits a noticeable jump, and the output entropy continues to change (see Figures `\ref{app:fig:early_exit_auc_acc}`{=latex} and `\ref{app:fig:early_exit_predition_entropy}`{=latex}). Notably, these stages can overlap due to computations that overlap and redundancy across layers, as well as the self-repair mechanism. # Is One Layer Enough? Our experiments show that most non-early layers appear redundant in tabular logit lens and layer-ablation experiments; however, we find that these layers still influence prediction quality, i.e., they are important for performance but might overlap in computation. This suggests that fewer, but repeated layers, could achieve similar performance, which connects to our initial question: Is one layer enough? In this concluding experiment, we illustrate the impact of this finding. Specifically, we leverage the open-source `\nanoTabPFN{}`{=latex} [@pfefferle2025nanotabpfn] codebase providing an architecture similar to `\TabPFNtwo{}`{=latex} and pre-train three models using the `\TabICL{}`{=latex} prior codebase: (1) the original implementation containing six layers without any modification (`\nanoTabPFN{}`{=latex}$_{6l}$), (2) a single-layer transformer (`\nanoTabPFN{}`{=latex}$_{1l}$) and (3) a single-layer transformer where we repeat the single layer up to six times during trained and inference (`\nanoTabPFN{}`{=latex}$_{looped}$) (see Appendix `\ref{app:nanotabpfn}`{=latex} for details). We highlight that `\nanoTabPFN{}`{=latex}$_{looped}$ matches the computational complexity of `\nanoTabPFN{}`{=latex}$_{6l}$ while retaining only the number of parameters of `\nanoTabPFN{}`{=latex}$_{1l}$. This controlled comparison suggests that the observed performance gains are not merely due to parameter count but rather to the iterative (looping) mechanism. We compare these models, including original `\TabPFNone{}`{=latex} and `\TabPFNtwohalf{}`{=latex} on the `\PMLB{}`{=latex} and `\TabArena{}`{=latex} benchmarks and show results in Figure `\ref{fig:looped_transformer}`{=latex}. We first observe that the single-layer model (`\redsolid{}`{=latex}) clearly compares worse than all other models, indicating that a single layer alone is insufficient. However, more importantly, `\loopedNanoTabPFN{}`{=latex} (`\bluesolid{}`{=latex}) performs almost identically to the six-layer `\nanoTabPFN{}`{=latex} (`\orangesolid{}`{=latex}); the model learns to iteratively refine its predictions, reusing a single layer. We also observe that performance is similar across the number of layers and loops. This supports the view that depth in TFMs primarily facilitates iterative computation rather than learning fundamentally distinct transformations at each layer, and that comparable predictive performance can be achieved through recurrent reuse of a single transformer block. Notably, one advantage of `\loopedNanoTabPFN{}`{=latex} is that it can produce immediate predictions without requiring pretraining of individual decoders. However, we note that our experiments were conducted at a small scale, and there remains a performance gap compared to state-of-the-art models such as `\TabPFNtwohalf{}`{=latex}.

Performance comparison between 6l, 1l, and looped. Repeating a single transformer block recovers performance comparable to the full-depth model.
# Conclusion In this work, we investigate open questions regarding the mechanisms inside TFMs and how they solve predictive tasks. We found that although the layer dynamics in the TFMs we studied differ from those in LLMs, iterative refinement plays a major role in how inference unfolds, layers form blocks with overlapping functionality, and different inference stages emerge. Specifically, we can derive insights that open new directions for a principled improvement of TFMs: - Using a strong encoder leads to more robust inference dynamics and forming highly descriptive features in early layers. Our results suggest that the compression stage of `\TabICL{}`{=latex} and the encoding used in `\LimixS{}`{=latex} yield a richer embedding of raw data. - The near-complete recovery of original performance after layer ablation indicates depth-wise redundancy. Our results show that TFMs can reliably recover from the removal of middle and late layers, suggesting that less capacity (i.e., shallower models) is required to achieve this performance. - The emergence of layer-wise blocks of highly similar representation reveals further opportunities for post-hoc model compression, such as merging layers or replacing multiple blocks with a recurrently applied shared layer to improve parameter efficiency [@mcleish2025teaching]. **Limitations.** Our work has several limitations. First, we did not study in detail when effective depth becomes necessary, and leave a more systematic investigation of this question, e.g., using controlled benchmarks with varying notions of tabular task complexity, for future work. For Tabular Logit Lens, we use the open-source priors from `\TabICL{}`{=latex}, which may be suboptimal for models that employ more expressive prior designs. For the looped transformer experiments, we use the smaller `\nanoTabPFN{}`{=latex} architecture. Although the results do not directly extrapolate to larger architectures without further experiments, they are encouraging and suggest that this approach can effectively scale to larger models such as `\TabPFNtwohalf{}`{=latex}. Finally, we evaluated the models without ensembling and on only two benchmark suites, studying mostly average performance; the results could be impacted by different benchmarks and ensembling. **Future Work.** We see several promising steps to build on our work: (1) extending our experimental setup to study TFMs at the neuronal and circuit level to gain deeper mechanistic insights, (2) using our methods to study the impact of design choices like the prior design and pre-training setup and (3) study whether to LLM-based models for predictive tabular tasks @hegselmann-icml23a [@gardner2024tabula8] exhibit similar behaviour. Such studies could reveal whether observed patterns are consistent and help identify potential weaknesses in the model architecture, learned biases, or the training data. Finally, we aim to further investigate recurrent models, such as looped transformers at larger scales, which offer benefits such as anytime predictions, parameter efficiency, and adaptive computation, which represent a particularly promising direction for future research. # Acknowledgments {#acknowledgments .unnumbered} This research has been funded by the Federal Ministry of Research, Technology and Space of Germany and the state of North Rhine-Westphalia as part of the Lamarr Institute for Machine Learning and Artificial Intelligence. Additionally, part of this research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. A. Balef and M. Koshil also thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS). # Impact Statement {#sec:impact_tatement .unnumbered} This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. ```{=latex} \bibliographystyle{icml2026} ``` ```{=latex} \newpage ``` ```{=latex} \appendix ``` ```{=latex} \counterwithin{figure}{section} ``` ```{=latex} \counterwithin{table}{section} ``` ```{=latex} \counterwithin{algorithm}{section} ``` ```{=latex} \onecolumn ``` # Experimental setup details ## Metrics {#app:metrics} **Linear centered kernel alignment (CKA)** is a metric [@kornblith2019similarity] used to evaluate the similarity between representations. Given two column-centered feature matrices $X \in \mathbb{R}^{n \times d_1}$ and $Y \in \mathbb{R}^{n \times d_2}$, linear CKA is defined as $$\text{CKA}(X, Y) = \frac{\| X^\top Y \|_F^2}{\| X^\top X \|_F \cdot \| Y^\top Y \|_F + \varepsilon}, \label{eq:cka}$$ where $\|\cdot\|_F$ denotes the Frobenius norm and $\varepsilon$ is a small constant for numerical stability. This metric quantifies the similarity between two representations while being invariant to isotropic scaling. **Cosine similarity** is a metric used to measure the similarity between two vectors. Given two vectors $x, y \in \mathbb{R}^{d}$, cosine similarity is defined as $$\text{CosSim}(x, y) = \frac{x \cdot y}{\|x\|_2 \, \|y\|_2}, \label{eq:cosine_similarity}$$ where $\|\cdot\|_2$ denotes the $\ell_2$ norm. This metric quantifies the angular similarity between two vectors, independent of their magnitudes. In our experiments, we report the average cosine similarities. ## Architectures {#app:architectures} In this section, we provide additional details on the tabular in-context learning (ICL) models used in our experiments. We focus on architectural differences, representation strategies, and key design choices. **`\TabPFNone{}`{=latex}** is the first version of the Tabular Prior-Data Fitted Network (TabPFN) designed for in-context learning on tabular data [@hollmann-iclr23a]. Unlike later TabPFN variants, `\TabPFNone{}`{=latex} employs a single embedding per feature and row, without additional cross-feature attention mechanisms as seen in Table `\ref{app:arch:tab:tabpfn_v1}`{=latex}. **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ----------------- -------------------- `\TabPFNone`{=latex} \- 25,821,706 Input encoder Linear + Linear 51,712 + 1,024 Transformer blocks 12 layers 25,233,408 `\quad `{=latex}Each block \- 2,102,784 `\quad `{=latex}`\quad `{=latex}MultiheadAttention \- 1,050,624 `\quad `{=latex}`\quad `{=latex}LayerNorm1 Post-LN 1,024 `\quad `{=latex}`\quad `{=latex}MLP Linear + Linear 525,312 + 524,800 `\quad `{=latex}`\quad `{=latex}LayerNorm2 Post-LN 1,024 Decoder Linear + Linear 525,312 + 10,250 : Layers and Parameters in `\TabPFNone{}`{=latex} `\label{app:arch:tab:tabpfn_v1}`{=latex} **`\TabPFNtwo{}`{=latex}** is an improved version of the original TabPFN [@hollmann-nature25a], designed to increase model capacity and generalization for in-context learning on tabular data. Compared to `\TabPFNone{}`{=latex}, it includes additional transformer layers and attention heads as seen in Table `\ref{app:arch:tab:tabpfn_v2}`{=latex}, enabling richer representations and more complex interactions between input features. ```{=latex} \small ``` **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ---------------------------- -------------------- `\TabPFNtwo`{=latex} PerFeatureTransformer 7,244,554 **Encoder** SequentialEncoder \- Feature Linear 768 Label Linear 576 Positional embedding Linear 9,408 **Transformer Blocks** 12 layers 7,077,888 `\quad `{=latex}Each block \- 589,824 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Items 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear + Linear 147,456 + 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 Decoder Linear + Linear 148,224 + 7,690 : Layers and Parameters in `\TabPFNtwo{}`{=latex} `\label{app:arch:tab:tabpfn_v2}`{=latex} ```{=latex} \FloatBarrier ``` **`\TabPFNtwohalf{}`{=latex}** [@grinsztajn2025tabpfn] is the latest TabPFN variant, roughly doubling the number of layers compared to `\TabPFNtwo{}`{=latex} (Table `\ref{app:arch:tab:tabpfn_v2}`{=latex}). This increase in depth allows the model to capture more complex feature interactions and improves generalization while maintaining in-context learning capabilities. ```{=latex} \small ``` **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ---------------------------- -------------------- `\TabPFNtwohalf`{=latex} PerFeatureTransformer 10,718,218 **Encoder** \- \- Feature Linear 1,152 Label Linear 576 Positional embedding Linear 9,408 **Transformer Blocks** 24 layers 10,616,832 `\quad `{=latex}Each block \- 442,368 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Items 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear + Linear 73,728 + 73,728 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 Decoder Linear + Linear 74,112 + 3,850 : Layers and Parameters in `\TabPFNtwohalf{}`{=latex} `\label{app:arch:tab:tabpfn_v2_5}`{=latex} ```{=latex} \FloatBarrier ``` **`\TabICL{}`{=latex}** [@qu2025tabicl] is a transformer-based in-context learner for tabular data that uses feature-wise embeddings, cross-feature attention, and learned positional encodings; from an in-context prediction perspective, it operates similarly to `\TabPFNone{}`{=latex}, without additional cross-feature mechanisms, as summarized in Table `\ref{app:arch:tab:tabicl}`{=latex}. ```{=latex} \small ``` **Layer Name** **Type** **Num Parameters** ---------------------------------------------------- --------------- -------------------- `\TabICL`{=latex} 27,051,666 **Feature Encoder (Column Embedder)** \- 877,824 `\quad `{=latex}Input Linear Linear 256 `\quad `{=latex}Transformer Blocks 3 layers 844,032 `\quad `{=latex}`\quad `{=latex}Each block \- 281,344 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention 1 66,048 + 16,512 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 33,024 + 32,896 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention 2 132,480 + 66,048 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 33,024 + 32,896 `\quad `{=latex}Output linear Linears 16,512 + 256 **Feature Encoder (Row Interaction)** \- 398,216 `\quad `{=latex}Rotary Embedding \- 8 `\quad `{=latex}Transformer Blocks 3 layers 397,440 `\quad `{=latex}`\quad `{=latex}Each block \- 132,480 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention 66,048 + 16,512 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 256 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 33,024 + 32,896 `\quad `{=latex}Output LayerNorm 256 **ICL predictor** \- 25,775,626 `\quad `{=latex}Transformer Blocks 12 layers 25,233,408 `\quad `{=latex}`\quad `{=latex}Each block \- 2,102,784 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 1,024 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention 1,050,624 +262,656 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 1,024 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 525,312 + 524,800 Decoder Linear layers 535,562 : Layers and Parameters in `\TabICL{}`{=latex} `\label{app:arch:tab:tabicl}`{=latex} ```{=latex} \FloatBarrier ``` **`\LimixS{}`{=latex} and `\LimixL{}`{=latex}** [@LimiX] are transformer-based tabular in-context learners that maintain separate embeddings per feature. `\LimixS{}`{=latex} is the smaller variant (Table `\ref{app:arch:tab:limix_2m}`{=latex}), emphasizing feature encoding with an RBF kernel, while `\LimixL{}`{=latex} is larger (Table `\ref{app:arch:tab:limix_16m}`{=latex}), enabling richer feature interactions. Despite these differences, both support in-context predictions similarly to `\TabPFNtwo{}`{=latex}, with separate feature embeddings facilitating detailed analysis of feature-wise interactions. **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ---------------------------- -------------------- `\LimixS`{=latex} PerFeatureTransformer 2,377,837 **Encoder** \- 47,712 `\quad `{=latex}Feature \- 18,048 `\quad `{=latex}Fusion Network \- 28,224 `\quad `{=latex}Label(cls) \- 1,056 `\quad `{=latex}Label(reg) \- 288 **Transformer Blocks** 12 layers 2,211,840 `\quad `{=latex}Each block \- 184,320 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Items 73,728 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 73,728 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 36,864 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN () 0 Decoder(cls) Linear layers 41,098 Decoder(reg) Linear layers 38,401 Decoder(feature) Linear layers 38,786 : Layers and Parameters in `\LimixS{}`{=latex} `\label{app:arch:tab:limix_2m}`{=latex} **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ---------------------------- -------------------- `\LimixL`{=latex} PerFeatureTransformer 16,526,413 **Encoder** \- 134,016 `\quad `{=latex}Feature \- 19,392 `\quad `{=latex}Fusion Network \- 111,744 `\quad `{=latex}Label(cls) \- 2,112 `\quad `{=latex}Label(reg) \- 576 `\quad `{=latex}Positional embedding Linear 9,408 **Transformer Blocks** 12 layers 15,925,248 `\quad `{=latex}Each block \- 1,327,104 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 294,912 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 294,912 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Items 147,456 `\quad `{=latex}`\quad `{=latex}LayerNorm Pre-LN 0 `\quad `{=latex}`\quad `{=latex}MLP Linear layers 294,912 Decoder(cls) Linear layers 155,914 Decoder(reg) Linear layers 150,529 Decoder(feature) Linear layers 151,298 : Layers and Parameters in `\LimixL{}`{=latex} `\label{app:arch:tab:limix_16m}`{=latex} ```{=latex} \FloatBarrier ``` ## `\nanoTabPFN{}`{=latex} {#app:nanotabpfn} **`\nanoTabPFN{}`{=latex}**[^4] is a simplified and lightweight implementation of the `\TabPFNtwo{}`{=latex} architecture [@pfefferle2025nanotabpfn]. The architecture, shown in Table `\ref{app:nanotabpfn:tab:nanotabpfn}`{=latex}, consists of six transformer blocks. For `\loopedNanoTabPFN{}`{=latex}, we use the same architecture but with a single transformer block, which is looped six times during the forward pass. This modification reduces the total number of parameters by 20%. ```{=latex} \small ``` **Layer Name** **Description** **Num Parameters** ---------------------------------------------------- ---------------------------- -------------------- `\nanoTabPFN`{=latex} PerFeatureTransformer 3,717,514 **Encoder** SequentialEncoder \- Feature Linear 384 Label Linear 384 **Transformer Blocks** 6 layers 3,560,832 `\quad `{=latex}Each block \- 593,472 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Features 185,280 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 384 `\quad `{=latex}`\quad `{=latex}MultiheadAttention Attention Between Items 185,280 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 384 `\quad `{=latex}`\quad `{=latex}MLP Linear + Linear 148,224 + 147,648 `\quad `{=latex}`\quad `{=latex}LayerNorm Post-LN 384 Decoder Linear + Linear 148,224 + 7,690 : Layers and Parameters in `\nanoTabPFN{}`{=latex} `\label{app:nanotabpfn:tab:lnanotabpfn}`{=latex} ```{=latex} \small ``` **Layer Name** **Description** **Num Parameters** ------------------------------------ ---------------------------- -------------------- `\loopedNanoTabPFN`{=latex} PerFeatureTransformer 750,154 **Encoder** SequentialEncoder \- Feature Linear 384 Label Linear 384 **Single Transformer Block** \- 593,472 `\quad `{=latex}MultiheadAttention Attention Between Features 185,280 `\quad `{=latex}LayerNorm Post-LN 384 `\quad `{=latex}MultiheadAttention Attention Between Items 185,280 `\quad `{=latex}LayerNorm Post-LN 384 `\quad `{=latex}MLP Linear + Linear 148,224 + 147,648 `\quad `{=latex}LayerNorm Post-LN 384 Decoder Linear + Linear 148,224 + 7,690 : Layers and Parameters in `\loopedNanoTabPFN{}`{=latex} `\label{app:nanotabpfn:tab:nanotabpfn}`{=latex} ```{=latex} \FloatBarrier ``` **Training.** We train all `\nanoTabPFN{}`{=latex} models using the hyperparameters listed in Table `\ref{app:tab:training_config}`{=latex}. We report the pretraining cost on an NVIDIA A100 GPU in Table `\ref{app:tab:pretraining_cost}`{=latex}. ```{=latex} \small ``` ::: {#app:tab:pretraining_cost} **Hyperparameter** **Value** -------------------------------- ------------------ Training steps $10\,000$ Batch size $512$ Micro-batch size $4$ Learning rate ($\eta$) $1\times10^{-4}$ Scheduler Cosine warmup Warmup steps $2\,000$ Gradient clipping $1.0$ Weight decay $0.0$ Cosine cycles $1$ Cosine amplitude decay $1.0$ Cosine final learning rate $0.0$ Polynomial final learning rate $1\times10^{-7}$ Polynomial decay power $1.0$ : Pretraining Runtime (hours) ::: ```{=latex} \hfill ``` ::: {#app:tab:pretraining_cost} **Model** **Training time (hours)** ------------------------------ --------------------------- `\nanoTabPFN`{=latex}$_{1l}$ 11.9 `\nanoTabPFN`{=latex}$_{6l}$ 62.3 `\loopedNanoTabPFN`{=latex} 68.8 : Pretraining Runtime (hours) ::: ```{=latex} \FloatBarrier ``` ## Reproducibility {#app:reproducibility} **Code.** The code will be made publicly available upon acceptance. **Compute cost.** All experiments were run on a compute cluster with $16$ CPU cores per node and NVIDIA A100 GPU nodes. The total CPU and GPU hours for each benchmark and model are reported in Table `\ref{app:tab:compute_budget}`{=latex}. GPU hours account only for active GPU utilization; the corresponding wall-clock GPU runtime is approximately given by the reported CPU hours divided by $16$. ::: {#app:tab:compute_budget} benchmark model CPU hours GPU hours --------------------- -------------------------- ----------- ----------- `\PMLB`{=latex} `\TabPFNone`{=latex} 5942.06 26.75 `\PMLB`{=latex} `\TabPFNtwo`{=latex} 6579.17 2.78 `\PMLB`{=latex} `\TabPFNtwohalf`{=latex} 25329.90 5.64 `\PMLB`{=latex} `\TabICL`{=latex} 15664.14 4.27 `\PMLB`{=latex} `\LimixS`{=latex} 5685.21 4.06 `\PMLB`{=latex} `\LimixL`{=latex} 9848.21 4.71 `\TabArena`{=latex} `\TabPFNone`{=latex} 18236.49 200.11 `\TabArena`{=latex} `\TabPFNtwo`{=latex} 25129.97 47.05 `\TabArena`{=latex} `\TabPFNtwohalf`{=latex} 74645.58 65.21 `\TabArena`{=latex} `\TabICL`{=latex} 38908.78 44.72 `\TabArena`{=latex} `\LimixS`{=latex} 26737.44 88.37 `\TabArena`{=latex} `\LimixL`{=latex} 39022.71 121.88 : Total CPU and GPU hours per benchmark and model ::: ## Benchmarks {#app:benchmarks} We use the `\TabArena{}`{=latex} and `\PMLB{}`{=latex} benchmarks to evaluate our experiments. For `\TabArena{}`{=latex}, we consider only binary classification tasks, in line with the limitations of the models (at most 10,000 samples and 100 features), resulting in 15 distinct tasks. Tables `\ref{app:tab:datasets_tabarena}`{=latex} and `\ref{app:tab:datasets_plmb}`{=latex} list all datasets used in our experiments. For each dataset, we report the OpenML task ID [@bischl2025], dataset name, number of samples, number of features, and number of categorical features. ```{=latex} \small ``` ::: {#app:tab:datasets_tabarena} index task id dataset name #samples #features #categorical features #folds #repetitions ------- --------- --------------------------------------- ---------- ----------- ----------------------- -------- -------------- 1 363619 Bank-Customer-Churn 10000 11 5 3 3 2 363624 coil2000-insurance-policies 9822 86 4 3 3 3 363623 churn 5000 20 5 3 3 4 363706 taiwanese-bankruptcy-prediction 6819 95 1 3 3 5 363689 NATICUSdroid 7491 87 87 3 3 6 363694 polish-companies-bankruptcy 5910 65 1 3 3 7 363700 seismic-bumps 2584 16 4 3 3 8 363674 hazelnut-spread-contaminant-detection 2400 31 1 3 10 9 363682 Is-this-a-good-customer 1723 14 9 3 10 10 363629 diabetes 768 9 1 3 10 11 363671 Fitness-Club 1500 7 4 3 10 12 363626 credit-g 1000 21 14 3 10 13 363684 Marketing-Campaign 2240 26 9 3 10 14 363621 blood-transfusion-service-center 748 5 1 3 10 15 363696 qsar-biodeg 1054 42 6 3 10 : Datasets in `\TabArena{}`{=latex}. ::: ```{=latex} \small ``` ::: {#app:tab:datasets_plmb} index task id dataset name #samples #features #categorical features #folds #repetitions ------- --------- ----------------------------- ---------- ----------- ----------------------- -------- -------------- 1 13 breast-cancer 286 10 10 10 1 2 27 colic 368 23 16 10 1 3 39 sonar 208 61 1 10 1 4 42 haberman 306 4 2 10 1 5 52 heart-statlog 270 14 1 10 1 6 54 hepatitis 155 20 14 10 1 7 55 vote 435 17 17 10 1 8 57 ionosphere 351 35 1 10 1 9 3495 SPECT 267 23 23 10 1 10 3496 SPECTF 349 45 1 10 1 11 3503 aids 50 5 3 10 1 12 3538 analcatdata-boxing2 132 4 4 10 1 13 3539 prnn-crabs 200 8 2 10 1 14 3540 analcatdata-boxing1 120 4 4 10 1 15 3542 analcatdata-lawsuit 264 5 2 10 1 16 3543 irish 500 6 4 10 1 17 3550 analcatdata-asbestos 83 4 3 10 1 18 3552 analcatdata-creditscore 100 7 4 10 1 19 3554 backache 180 32 27 10 1 20 3555 prnn-synth 250 3 1 10 1 21 3556 analcatdata-cyyoung8092 97 11 4 10 1 22 3558 analcatdata-japansolvent 52 9 2 10 1 23 3562 lupus 87 4 1 10 1 24 3565 analcatdata-bankruptcy 50 6 2 10 1 25 3568 analcatdata-cyyoung9302 92 10 5 10 1 26 3570 biomed 209 9 2 10 1 27 3819 molecular-biology-promoters 106 58 59 10 1 28 146188 analcatdata-fraud 42 12 12 10 1 29 146196 corral 160 7 7 10 1 30 146208 mux6 128 7 7 10 1 31 146210 postoperative-patient-data 88 9 9 10 1 32 146236 cleve 303 14 9 10 1 33 146240 parity5 32 6 6 10 1 34 3722 hungarian 294 14 8 10 1 : Datasets in `\PMLB{}`{=latex}. ::: ```{=latex} \FloatBarrier ``` ## Tabular Logit Lens {#app:logit_lens} Similar to [@Kuken2025EarlyStopping], we pretrain individual decoders for each tabular foundation model using `\TabICL{}`{=latex} priors (see Appendix `\ref{app:prior}`{=latex}). Notably, the individual decoders share the same architecture as the original decoder. During pretraining, all model parameters are frozen, and only the decoders are updated, using the settings shown in Table `\ref{app:tab:training_params}`{=latex}. Table `\ref{app:tab:training_runtime_hours}`{=latex} reports runtime of pretraining all decoders for each model on a single NVIDIA A100 GPU. Additional results are reported in Appendix `\ref{app:early_exit}`{=latex}. ::: {#app:tab:training_runtime_hours} **Parameter** **Value** --------------- ----------- Epochs 200 Batch size 8 #Steps/Epoch 512 Learning rate 3e-5 : Pretraining Runtime (hours) ::: ```{=latex} \hfill ``` ::: {#app:tab:training_runtime_hours} **Name** **Training time (hours)** -------------------------- --------------------------- `\TabPFN`{=latex} 7.06 `\TabPFNtwo`{=latex} 13.59 `\TabPFNtwohalf`{=latex} 18.14 `\TabICL`{=latex} 7.24 `\LimixS`{=latex} 6.58 `\LimixL`{=latex} 10.16 `\nanoTabPFN`{=latex} 12.89 : Pretraining Runtime (hours) ::: ```{=latex} \FloatBarrier ``` ## Synthetic Data Generation (Prior) {#app:prior} For pretraining the models (e.g., `\nanoTabPFN{}`{=latex} and `\loopedNanoTabPFN{}`{=latex}), as well as for pretraining the individual decoders used in the Tabular Logit Lens, we rely on the `\TabICL{}`{=latex} implementation to generate training priors. The configuration used for prior generation is detailed below: ::: {#app:tab:tabicl_prior_config} **Parameter** **Value** ---------------------------- ----------- Batch size per GP 4 Number of batches 10,000 Minimum number of features 2 Maximum number of features 30 Maximum number of classes 10 Maximum sequence length 1024 Log-scaled sequence length False Sequence length per GP False Minimum train size ratio 0.1 Maximum train size ratio 0.9 : Configuration used for synthetic data generation (prior) with `\TabICL{}`{=latex}. ::: # Experiments and Results {#app:experimental_results} ## Embedding similarity {#app:embedding_similarity} We additionally report the similarity between embeddings from different layers of each model across benchmarks. As shown in Figure `\ref{app:fig:embedding_similarity}`{=latex}, the patterns are largely consistent between benchmarks.

Layer-wise embedding similarity for each model across different benchmarks. Higher values indicate greater similarity between embeddings from the corresponding layers.
```{=latex} \FloatBarrier ``` ## Separation gap {#app:separation_gap} Here we provide more details about our separation gap. **Definition.** Let $h_\ell(x_i) \in \mathbb{R}^d$ denote the representation of sample $x_i$ at layer $\ell$, and $y_i$ its class label. We define the sets of within-class and between-class pairs as $$\mathcal{P}^{\mathrm{within}}_{\ell} = \{(x_i,x_j) : y_i = y_j, i \neq j\}, \quad \mathcal{P}^{\mathrm{between}}_{\ell} = \{(x_i,x_j) : y_i \neq y_j\}.$$ Let $d(u,v)$ denote a distance metric between two embeddings. Then, the average within-class and between-class distances at layer $\ell$ are $$D^{\mathrm{within}}_{\ell} = \frac{1}{|\mathcal{P}^{\mathrm{within}}_{\ell}|} \sum_{(x_i,x_j) \in \mathcal{P}^{\mathrm{within}}_{\ell}} d\big(h_\ell(x_i), h_\ell(x_j)\big), \quad D^{\mathrm{between}}_{\ell} = \frac{1}{|\mathcal{P}^{\mathrm{between}}_{\ell}|} \sum_{(x_i,x_j) \in \mathcal{P}^{\mathrm{between}}_{\ell}} d\big(h_\ell(x_i), h_\ell(x_j)\big).$$ The *separation gap* is then defined as $$\Delta_\ell = D^{\mathrm{between}}_{\ell} - D^{\mathrm{within}}_{\ell}.$$ For example, if $d(u,v)$ is the cosine distance $d_\mathrm{cos}(u,v) = 1 - \frac{u \cdot v}{\|u\|\|v\|}$, and for Euclidean distance $d_\mathrm{E}(u,v) = \|u - v\|_2$. notably a larger $\Delta_\ell$ indicates stronger separation between classes. **Separation Gap Across Layers and Benchmarks**. Figures `\ref{app:fig:no_pca_cosine_separation_gap}`{=latex} show the results for cosine distance.

Separation gap between embeddings without PCA using cosine distance. Higher values indicate stronger separation between classes.
Since the embeddings have very high dimensionality, using distance metrics (especially the Euclidean distance) can be sensitive to noise. To mitigate this, we apply PCA to the embeddings. To fit the PCA, we randomly select $5{,}000$ samples from embeddings across all layers. We then project $h_\ell(x)$ onto the top principal components retaining $95\%$ of the variance, which reduces noise while preserving most of the information in the representation. Figures `\ref{app:fig:cosine_separation_gap}`{=latex} and `\ref{app:fig:Euclidean_separation_gap}`{=latex} show the results for different distance metrics after applying PCA. As observed, applying PCA helps reduce noise in the embeddings, leading to a clearer signal and more visible separation between classes.

Separation gap after applying PCA, using cosine distance.

Separation gap after applying PCA, using Euclidean distance.
```{=latex} \FloatBarrier ``` ## Probing Classifier {#app:probing_classifiers} As illustrated in Figure `\ref{app:fig:probing_classifier_diagram}`{=latex}, the schematic shows the probing classifier experiment in the ICL setup. Embeddings $h_l$ are extracted from the hidden states at layer $l$ using the training portion of the query set (indicated in blue). A probing classifier, (for example, logistic regression) is trained on these embeddings. For evaluation, embeddings are extracted from the validation portion of the query set at the same layer or other layers (indicated in orange), and the classifier's performance is measured. This setup allows us to assess the extent to which label information from the support set is encoded in each layer's representation. ![Query set consists of both training and validation samples for probing classifier. Blue corresponds to the training set, and orange corresponds to the validation set.](./figures/app/probing_classifier_diagram.png){#app:fig:probing_classifier_diagram width="0.6\\linewidth"} ```{=latex} \FloatBarrier ``` We evaluate multiple probing classifiers, including logistic regression, k-Nearest Neighbors (KNN), and Linear Discriminant Analysis (LDA), as well as a fine-tuned original decoder. For each classifier, we report the normalized ROC-AUC on the `\PMLB{}`{=latex} and `\TabArena{}`{=latex} benchmarks. Figures `\ref{app:fig:probing_classifiers_KNN}`{=latex},  `\ref{app:fig:probing_classifiers_logistic_regression}`{=latex} and `\ref{app:fig:probing_classifiers_LDA}`{=latex} show the results for KNN, logistic regression, and LDA probing classifiers, respectively, trained on embeddings extracted from different layers of the models. Figure `\ref{app:fig:probing_classifiers_finetuned_decoder}`{=latex} presents the results for the fine-tuned original decoder used as a probing classifier.

Normalized AUC for logistic regression as a probing classifier trained on embeddings at different layers of the models for each benchmark.

Normalized AUC for k-Nearest Neighbors (KNN) as a probing classifier trained on embeddings at different layers of the models.

Normalized AUC for Linear Discriminant Analysis (LDA) as a probing classifier trained on embeddings at different layers of the models.

Normalized AUC for fine-tuned original decoder as a probing classifier trained on embeddings at different layers of the models.
```{=latex} \FloatBarrier ``` ## Tabular Logit Lens {#app:early_exit} Here, we present the results of the early-exit strategy. In this approach, after each layer, the embeddings are passed to the decoder to produce a prediction. Figure `\ref{app:fig:early_exit}`{=latex} shows the performance of each model at different early-exit points, while Figure `\ref{app:fig:early_exit_predition_entropy}`{=latex} reports the corresponding prediction entropy. Higher entropy indicates that the model is less confident, with output probabilities being more evenly distributed across classes. Furthermore, in Figure `\ref{app:fig:early_exit_auc_acc}`{=latex}, we report ROC-AUC and balanced accuracy across layers. From the middle to the final layers, although the ROC-AUC of the original decoder matches that of the corresponding individual decoders, its balanced accuracy remains substantially lower. This discrepancy indicates that the original decoder's predictions are poorly calibrated: while ranking performance (ROC-AUC) is preserved, decision-threshold--dependent metrics such as balanced accuracy degrade. This need for calibration is consistent with our observations based on prediction entropy in Figure `\ref{app:fig:early_exit_predition_entropy}`{=latex}.
Effect of the tabular logit lens experiment on the tabular foundation model’s performance.
Effect of the tabular logit lens experiment on prediction entropy of the tabular foundation model.
Comparison of ROC-AUC and balanced accuracy across layers on and , illustrating prediction calibration through tabular logit lens.
```{=latex} \FloatBarrier ``` ## Layer ablation {#app:layer_interventions} We do layer ablations as illustrated in Figure `\ref{fig:layer_experiments}`{=latex} and report detailed results of the layer ablation experiment. As shown in Figure `\ref{app:fig:layer_interventions}`{=latex}, the effects of layer ablation vary slightly across benchmarks. In particular, `\TabArena{}`{=latex} exhibits greater sensitivity to layer ablation compared to `\PMLB{}`{=latex}. Figure `\ref{app:fig:layer_interventions_prediction_entropy}`{=latex} shows the corresponding prediction entropy, providing insight into the model's confidence at each layer. As observed, for `\TabPFNtwohalf{}`{=latex} the final layers are primarily responsible for the model's predictive certainty, whereas this effect is less pronounced in the other models. Furthermore, to facilitate a clearer comparison between benchmarks, we provide Figure `\ref{app:fig:comparing_benchmarks_layer_interventions}`{=latex}.
Effect of the layer ablation on prediction on the tabular foundation model’s performance.
Effect of the layer ablation on prediction entropy for the tabular foundation model’s performance.
Skipping layer
Repeating layer
swapping layers
Effect of the layer ablation across benchmarks.
```{=latex} \FloatBarrier ``` ## Self-repair {#app:self_repair} We provide more detailed results illustrating the self-repair phenomenon. Figures `\ref{app:fig:skipping_layer_self_repair}`{=latex}, `\ref{app:fig:repeating_layer_self_repair}`{=latex} and `\ref{app:fig:swapping_layer_self_repair}`{=latex} show layer-wise performance with and without layer ablation. For the top plots, the solid black line represents performance without any intervention. After ablating a layer, the change in performance is indicated by dashed lines, and it continues with the colored lines, from blue (early) to orange (late), that show intermediate performance following the layer ablations. This visualization highlights how performance drops after an intervention and recovers in subsequent layers, indicating self-repair. In the bottom plots, we compare the change in performance between the final prediction and the immediate prediction after a layer ablation. Each dot corresponds to a dataset in the benchmark, and colors indicate the layer index, from blue (early) to orange (late). We observe instances where the final performance remains largely unchanged despite a significant drop in immediate performance, indicating self-repair (e.g., the green points in `\TabPFNtwohalf{}`{=latex}). Conversely, in `\TabPFNtwohalf{}`{=latex}, early-layer interventions (blue) typically do not recover, suggesting that self-repair does not occur in these cases.



Layer-wise self-repair following a skipped layer.



Layer-wise self-repair following a repeated layer.



Layer-wise self-repair following swapped layers.
```{=latex} \FloatBarrier ``` # Further Discussion {#app:further_discussion} In this section, we provide a conceptual interpretation of the internal inference process of TFMs by drawing an analogy with the well-studied inference stages of Large Language Models (LLMs). Figure `\ref{app:fig:stages}`{=latex} and Table `\ref{app:tab:llm_tfm_comparison}`{=latex} summarize this comparison by aligning each inference stage with the supporting experimental evidence, highlighting both shared structural principles and key differences induced by the tabular prediction setting. ![Internal inference process of TFMs.](./figures/stages_intro.png){#app:fig:stages height="6cm"} ```{=latex} \small ``` ```{=latex} \setlength{\tabcolsep}{6pt} ``` ::: {#app:tab:llm_tfm_comparison} +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | **LLMs** | **TFMs** | | | | +:==============================:+:======================+:==========================================================================================+:=================================+:====================================================================================+:============================================================================================================================================================================================================================================================================================================================================================================================================================================================================+ | 2-3 `\cmidrule`{=latex}(lr)4-6 | Stage | Function | Stage | Function | Evidences | | | | | | | | | Stage | | | | | | +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | Detokenization | Use local contextual information to map raw tokens into coherent representations | Latent Mapping | Extension of the input encoder; maps input embeddings into coherent representations | 1\) Low performance on tabular logit lens (Figure `\ref`{=latex}) and high sensitivity to layer ablations (Figure `\ref`{=latex}).`\newline`{=latex}) Robustness of skipping early layers in models with better input encoders (Figure `\ref`{=latex}). | +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 2 | Feature Engineering | Iteratively build feature representations depending on token context | Feature Engineering and Labeling | Iteratively build feature representations depending on labels | 1\) Enhancing the distinction between intra-class and inter-class feature distributions (Figure `\ref`{=latex}).`\newline`{=latex}) Sharp rise in early-exit accuracy espcially for individual decoders (Figure `\ref`{=latex}), due to internal label embeddings and their iterative adjustment (see red line in Figure `\ref`{=latex}). | +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3 | Prediction Ensembling | Convert semantic features into plausible next-token predictions via an iterative ensemble | Prediction Ensembling | Ensemble and transform internal label embeddings into decoder-ready predictions. | In early-exit, individual decoder performance saturates while the original decoder continues to improve, indicating feature alignment with the decoder (Figure `\ref`{=latex}). | +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4 | Residual Calibration | Eliminate obsolete features and form the final output distribution | Prediction Calibration | Calibrate the prediction for the original decoder | 1\) Prediction entropy continues to decrease (or change) despite stable performance (Figure. `\ref`{=latex}.), particularly for the original decoder, even after both individual and original decoders have reached high predictive performance. `\newline`{=latex}) For the original decoder, balanced accuracy exhibits a delayed improvement, with a noticeable jump occurring only after ROC-AUC has already reached a strong performance level (Figure `\ref`{=latex}) | +--------------------------------+-----------------------+-------------------------------------------------------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ : Comparison of inference stages: LLMs versus TFMs ::: ```{=latex} \FloatBarrier ``` # Multiclass Experiments {#app:multiclass} In this section, we extend our analysis to multiclass classification tasks on the `\TabArena{}`{=latex} benchmark. Our main experiments focus on binary classification; here, we verify whether the observed inference dynamics generalize to settings with more than two classes. We report balanced accuracy and multiclass ROC-AUC as the metrics. ## Datasets {#app:multiclass:datasets} For the multiclass experiments, we use all multiclass classification tasks from `\TabArena{}`{=latex} that satisfy the model constraints ($\leq 10{,}000$ samples and $\leq 100$ features), resulting in $6$ datasets. We apply the same cross-validation protocol as in the binary setting (see Appendix `\ref{app:benchmarks}`{=latex}). ## Embedding Similarity {#app:multiclass:embedding_similarity} Figure `\ref{app:multiclass:fig:embedding_similarity}`{=latex} shows the layer-wise embedding similarity for each model on multiclass `\TabArena{}`{=latex} tasks. The block structure observed in the binary setting (Section `\ref{sec:experiments}`{=latex}) is similarly visible here.

Layer-wise embedding similarity (upper triangular: linear CKA; lower triangular: cosine similarity) for multiclass tasks.
```{=latex} \FloatBarrier ``` ## Separation Gap {#app:multiclass:separation_gap} Figures `\ref{app:multiclass:fig:cosine_separation_gap}`{=latex} and `\ref{app:multiclass:fig:euclidean_separation_gap}`{=latex} show the separation gap for the multiclass setting using cosine and Euclidean distances, respectively. Consistent with the binary case, the separation gap increases with depth across all models, providing further evidence for the iterative inference hypothesis. The general trend of models incrementally increasing the inter-class distance relative to the intra-class distance holds in the multiclass regime.

Separation gap (cosine distance, after PCA) across layers for multiclass tasks. Bold lines indicate the average across tasks; thin lines represent individual datasets.

Separation gap (Euclidean distance, after PCA) across layers for multiclass tasks.
```{=latex} \FloatBarrier ``` ## Probing Classifiers {#app:multiclass:probing_classifiers} Figures `\ref{app:multiclass:fig:probing_lr}`{=latex} report the normalized balanced accuracy of probing classifiers trained on embeddings at different layers of each model, for multiclass `\TabArena{}`{=latex} tasks. The asymmetric transfer pattern identified in the binary setting (Takeaway of Experiment `\numcircledmod{\ref{exp:prob_cls}}`{=latex}) persists: a classifier trained on layer $i$ generally transfers better to later layers $j > i$ than the reverse, confirming that each layer cumulatively enriches the representation.

Normalized balanced accuracy for logistic regression as a probing classifier, multiclass .
```{=latex} \FloatBarrier ``` ## Tabular Logit Lens {#app:multiclass:logit_lens} Figure `\ref{app:multiclass:fig:early_exit_auc_acc}`{=latex} reports the performance of individual decoders at each layer for multiclass `\TabArena{}`{=latex} tasks and compares ROC-AUC and balanced accuracy across layers. Figure `\ref{app:multiclass:fig:early_exit_entropy}`{=latex} shows the corresponding prediction entropy.

Comparison of ROC-AUC and balanced accuracy across layers for multiclass tasks, illustrating prediction calibration via the tabular logit lens.

Prediction entropy across layers for multiclass tasks.
```{=latex} \FloatBarrier ``` ## Layer Ablation {#app:multiclass:layer_ablation} Figures `\ref{app:multiclass:fig:layer_interventions}`{=latex} and `\ref{app:multiclass:fig:layer_interventions_entropy}`{=latex} show the effect of layer ablations and the corresponding prediction entropy for multiclass `\TabArena{}`{=latex} tasks. The sensitivity profile is consistent with the binary setting: early layers are the most sensitive to ablation, middle and later layers exhibit robustness, and repeating certain layers provides marginal improvement. The overall pattern confirms that the observed dynamics are not an artifact of the binary classification objective.

Effect of layer ablation on model performance for multiclass tasks.

Prediction entropy under layer ablation for multiclass tasks.
```{=latex} \FloatBarrier ``` ## Self-Repair {#app:multiclass:self_repair} Figure `\ref{app:multiclass:fig:self_repair_skip}`{=latex} shows the self-repair analysis for the layer-skipping ablation in the multiclass setting. As in the binary case, early-layer ablations are not recovered in subsequent layers, whereas middle and later layers exhibit clear self-repair. Figures `\ref{app:multiclass:fig:self_repair_repeat}`{=latex} and `\ref{app:multiclass:fig:self_repair_swap}`{=latex} present the analogous results for layer repetition and swapping, respectively.

Layer-wise self-repair following a skipped layer, multiclass .

Layer-wise self-repair following a repeated layer, multiclass .

Layer-wise self-repair following swapped layers, multiclass .
```{=latex} \FloatBarrier ``` ## Is One Layer Enough? {#app:multiclass:looped_transformer} Figure `\ref{app:fig:looped_multiclass_combined}`{=latex} extends the looped transformer evaluation to the multiclass `\TabArena{}`{=latex} setting. The `\loopedNanoTabPFN{}`{=latex} model achieves performance comparable to the six-layer `\nanoTabPFN{}`{=latex} baseline, consistent with the binary results. This supports the conclusion that iterative reuse of a single transformer block is a viable strategy for parameter-efficient TFM design beyond binary classification. We also add $20$ multiclass tasks from the OpenML CC18 suite that satisfy the model constraints ($\leq 10{,}000$ samples and $\leq 100$ features).
Multiclass evaluation of 6l, 1l, and looped.
```{=latex} \FloatBarrier ``` # Regression Experiments {#app:regression} In this section, we extend our analysis to regression tasks on the `\TabArena{}`{=latex} benchmark to verify that our main findings are not specific to the classification setting. We adapted the experiments as follows: we include all TFMs that support regression, namely `\LimixS{}`{=latex}, `\LimixL{}`{=latex}, `\TabPFNtwo{}`{=latex}, and `\TabPFNtwohalf{}`{=latex}; we use negative RMSE as the primary performance metric and additionally report Spearman's rank correlation, which is invariant to calibration; the separation gap experiment is extended to regression by treating it as a classification problem (discretizing regression values into $K=10$ bins); we adapt `\TabICL{}`{=latex} priors to regression tasks for pretraining individual decoders; and we include all six regression tasks in `\TabArena{}`{=latex} with fewer than $100$ features and $10{,}000$ samples, using the repetitions and folds specified by `\TabArena{}`{=latex}. Overall, the results on regression tasks are consistent with and support our main findings. ## Embedding Similarity {#embedding-similarity} Figure `\ref{app:reg:fig:embedding_similarity}`{=latex} shows the layer-wise embedding similarity on regression tasks. As in the classification setting, TFMs form blocks in which the embeddings remain similar across adjacent layers.

Layer-wise embedding similarity (upper triangular: linear CKA; lower triangular: cosine similarity) for regression tasks.
```{=latex} \FloatBarrier ``` ## Separation Gap {#separation-gap} We extend the separation gap metric to regression by discretizing the regression targets into $K=10$ bins and computing the difference between inter-bin and intra-bin distances. Figures `\ref{app:reg:fig:sep_gap_cosine}`{=latex} and `\ref{app:reg:fig:sep_gap_euclidean}`{=latex} show the results using cosine and Euclidean distances, respectively. Note that in the regression setting, a larger separation gap does not necessarily correspond to better predictive performance, as the discretization into bins introduces an approximation. Nevertheless, the separation gap generally increases with depth, consistent with our classification findings.

Separation gap (cosine distance, after PCA) across layers for regression tasks. Bold lines indicate the average across tasks; thin lines represent individual datasets.

Separation gap (Euclidean distance, after PCA) across layers for regression tasks.
```{=latex} \FloatBarrier ``` ## Probing Experiment For regression tasks, we replace the logistic regression probing classifier with a random forest regressor, evaluating performance using normalized RMSE. Figure `\ref{app:reg:fig:probing}`{=latex} shows that the asymmetric transfer pattern observed in classification persists: a probe trained on layer $i$ generalizes better to later layers $j > i$ than the reverse. This confirms that each layer cumulatively enriches the representation by adding new features while preserving those from earlier layers.

Normalized RMSE for a random forest regressor used as a probing model, trained on embeddings at different layers for regression tasks. Lower values indicate better performance.
```{=latex} \FloatBarrier ``` ## Tabular Logit Lens {#tabular-logit-lens} We adapt `\TabICL{}`{=latex} priors to regression tasks and pretrain individual decoders following the same protocol as in the classification setting (Appendix `\ref{app:logit_lens}`{=latex}). Figure `\ref{app:reg:fig:early_exit}`{=latex} reports performance at each layer using both negative RMSE (sensitive to calibration) and Spearman's rank correlation (invariant to calibration). The results indicate the same inference stages as discussed for classification: a rapid performance improvement in early-to-middle layers, followed by a calibration stage in the final layers where rank correlation saturates while RMSE continues to improve. This suggests that the final layers primarily refine calibration rather than introducing fundamentally new predictive features, analogous to the prediction calibration stage identified in classification.

Tabular logit lens results for regression tasks. We report Spearman’s rank correlation (invariant to calibration) and negative RMSE (sensitive to calibration) across layers.
```{=latex} \FloatBarrier ``` ## Layer Ablation {#layer-ablation} Figure `\ref{app:reg:fig:layer_interventions}`{=latex} shows the effect of layer ablations on regression tasks. The results for the `\LimixS{}`{=latex} and `\LimixL{}`{=latex} families are consistent with the classification setting: early layers are the most sensitive, while middle and later layers are robust to ablation. For the `\TabPFN{}`{=latex} family, however, the final layer is also found to be important. A possible explanation is that the regression decoder must differentiate between up to $5{,}000$ output bins, making the final layers more critical for the decoding stage. This pattern is analogous to residual sharpening observed in LLMs.

Effect of layer ablation (skipping, repeating, swapping) on model performance for regression tasks.
```{=latex} \FloatBarrier ``` ## Self-Repair {#self-repair} Figure `\ref{app:reg:fig:self_repair}`{=latex} shows the self-repair analysis under layer skipping for regression tasks. Consistent with the classification setting, early-layer ablations (with the exception of `\LimixS{}`{=latex}) cannot be recovered by subsequent layers. In contrast, skipping middle or later layers is followed by performance recovery, indicating that self-repair occurs and that these layers perform overlapping computations. ![Self-repair analysis under layer skipping for regression `\TabArena{}`{=latex} tasks. The solid black line shows intermediate performance without intervention. Colored lines (blue to orange, early to late) show intermediate performance after each layer ablation; a drop followed by recovery indicates self-repair.](./figures/app/regression/c2_skipping_layer_TabArena.png){#app:reg:fig:self_repair width="0.5\\linewidth"} ```{=latex} \FloatBarrier ``` ## Is One Layer Enough? {#is-one-layer-enough-1} We evaluate our proof-of-concept looped transformer on regression tasks. Figure `\ref{app:reg:fig:looped}`{=latex} shows the early-exit performance curves. The `\loopedNanoTabPFN{}`{=latex} achieves performance comparable to `\nanoTabPFN{}`{=latex}$_{6l}$ across all datasets, while the single-layer `\nanoTabPFN{}`{=latex}$_{1l}$ consistently underperforms. This confirms that the benefits of iterative layer reuse extend beyond classification to regression tasks. ![Early-exit performance of `\nanoTabPFN{}`{=latex}$_{6l}$, `\nanoTabPFN{}`{=latex}$_{1l}$, and `\nanoTabPFN{}`{=latex}$_{looped}$ on regression `\TabArena{}`{=latex} tasks.](./figures/app/regression/looped_transformer_early_exit_TabArena_c0.png){#app:reg:fig:looped width="0.5\\linewidth"} [^1]: We adapt and run all experiments for both multiclass and regression tasks (see Appendix `\ref{app:multiclass}`{=latex} and `\ref{app:regression}`{=latex}). Overall, the results are consistent with and support our main findings. [^2]: To reduce noise in such high-dimensional spaces, we apply PCA while retaining $95\%$ of the variance, and then compute distances in the projected space. [^3]: We note that fluctuations may arise from nonlinearities in the representations, which our metric cannot fully capture. [^4]: We use the codebase provided at .