ICLR: In-Context Learning of Representations

Source

• Raw Markdown: paper_in-context-learning-of-representations-2025.md
• PDF: paper_in-context-learning-of-representations-2025.pdf
• Preprint: arXiv 2501.00070v2
• ICLR proceedings: ICLR 2025 conference paper
• OpenReview: pXlmOmlHJZ
• Author announcement: Core Francisco Park’s X thread. Local X API snapshot and media: papers/in-context-learning-of-representations-2025/x_thread_corefpark_1875929881856573905.json and x_media/.
• Later mechanistic scrutiny: In-context learning of representations can be explained by induction circuits and reproduction/ablation code.

Status And Credibility

The paper is an accepted ICLR 2025 conference paper by Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. arXiv records the first submission on 2024-12-31 and v2 on 2025-05-08. The accepted tier-1 venue, controlled synthetic tasks, multi-model appendix, explicit limitations, and preliminary interventions make it credible important evidence for in-context representation dynamics.

The interpretation still needs calibration. The paper establishes context-dependent graph-like activation geometry and correlates it with next-token behavior; it does not identify the full circuit that creates the geometry, prove that the geometry is the mechanism used for prediction, or demonstrate a general-purpose spatial world model. A later 2026 reproduction argues that induction circuits can solve the graph-tracing task and that previous-token mixing can produce the observed geometry as a byproduct.

Core Question

Can a frozen language model replace or supplement pretraining semantics with a new relational structure specified only through the current context?

The paper constructs a controlled in-context graph-tracing task. Nodes are common single-token concepts such as apple, bird, and math; edges are imposed by a square grid, ring, or hexagonal lattice unrelated to those words’ ordinary semantics. A token sequence is generated from a random walk or neighboring-node samples. The model must assign output probability to valid neighbors of the final node.

flowchart LR
  G[Hidden graph over familiar tokens] --> W[Random-walk event stream]
  W --> M[Frozen LLM]
  M --> H[Context-conditioned activations]
  M --> P[Next-token distribution]
  H --> Geo[Graph-like representation geometry]
  P --> Acc[Probability mass on valid neighbors]

The task separates two objects:

• Activation space: residual-stream states for occurrences of each token under the current context.
• Behavior space: next-token probability distributions, summarized by probability mass on graph-valid neighboring tokens.

This makes the paper a precursor to later work that compares internal representation manifolds with output-behavior manifolds, but this paper’s primary quantitative geometry is in activation space.

Experimental Construction

Let the hidden graph be

$$\mathcal{G}=(\mathcal{T},E),$$

where $\mathcal{T}$ is the set of token-labeled nodes. For a context

$$\mathcal{C}=[c_0,\dots ,c_{N-1}],c_t\in \mathcal{T},$$

consecutive tokens follow graph edges. At layer $\ell$, activations for occurrences of token $\tau$ in a recent window are averaged into

$${\mathbf{h}}_{\tau }^{\ell }\in {\mathbb{R}}^d.$$

The matrix of class means is

$${\mathbf{H}}^{\ell }(\mathcal{T})\in {\mathbb{R}}^{|\mathcal{T}|\times d}.$$

The main experiments use Llama-3.1-8B and inspect late residual-stream layers, with appendices extending the observation to Llama-3.2-1B, Llama-3.1-8B-Instruct, Gemma-2-2B, and Gemma-2-9B.

Main Findings

Context-dependent graph geometry

As context length increases, PCA of the mean token activations changes from a geometry dominated by pretrained token semantics toward one resembling the graph used to generate the event stream:

• square-grid tasks yield grid-like low-dimensional layouts;
• ring tasks yield circular layouts;
• hexagonal tasks yield the corresponding lattice structure;
• later layers generally expose the task-specific geometry more clearly than earlier layers.

The irregular width of the recovered square grid is informative: interior nodes have higher visitation frequency under the random walk, and the activation geometry reflects both graph adjacency and event frequency.

Pretraining semantics and in-context structure can coexist

When nodes are days of the week and the context imposes a conflicting ring ordering, the pretrained weekday ring remains dominant in the first two principal components. The context-defined ordering appears in principal components three and four. Behavior eventually follows the new ordering even though it is not the leading activation geometry.

This is stronger than a simple overwrite story. Context can add a task-specific relational subspace while a pretrained semantic subspace remains present.

Dirichlet energy tracks local graph smoothness

For adjacency matrix $\mathbf{A}$, the paper measures

$$E_{\mathcal{G}}({\mathbf{H}}^{\ell })=\sum _{i,j}{\mathbf{A}}_{ij}{\Vert {\mathbf{h}}_i^{\ell }-{\mathbf{h}}_j^{\ell }\Vert }^2.$$

Low energy means graph neighbors have similar activations. Across the tested tasks, energy decreases with context length and reaches its minimum shortly before rule-following accuracy rises rapidly. Standardized-energy controls reduce the trivial constant-representation concern.

The paper connects constrained Dirichlet-energy minimization to graph Laplacian eigenvectors. Under the theorem’s exact minimizer and non-degenerate singular-value assumptions, PCA directions coincide with leading non-trivial spectral-embedding coordinates. Empirically, the reported absolute cosine similarities between the first two activation PCs and corresponding spectral coordinates are approximately:

Graph	PC1 / spectral coordinate	PC2 / spectral coordinate
Grid	0.950	0.954
Ring	0.942	0.930
Hexagonal lattice	0.745	0.755

This explains why PCA can recover graph-like layouts under the stated energy-minimization model, but it does not by itself prove that the model literally runs an optimizer over the declared energy.

Abrupt context-length transitions

Rule-following accuracy exhibits a slow phase followed by a faster ascent. The fitted transition point moves to longer contexts with graph size and follows an empirical power-law trend over the tested sizes. The authors propose bond percolation as a possible analogy: sufficient observed transitions connect a large component of the hidden graph.

They explicitly describe this as a loose hypothesis. The graph sizes are too small to establish a percolation scaling theory, and later induction-circuit explanations provide a more direct account of why repeated transitions improve next-token predictions.

Preliminary causal evidence is positive but incomplete

The appendix rescales two or three principal-component coordinates of a source-node activation toward a target node and then measures whether the model predicts neighbors of the target. The intervention outperforms null and random-vector controls across ring, grid, and hexagonal tasks. Three-component interventions are stronger than two-component interventions.

This shows that the discovered directions have some causal effect on output behavior. It does not show that the whole PCA layout is the mechanism used by the unperturbed model; the paper calls the intervention rudimentary and the causal relation incomplete.

What The Paper Does And Does Not Establish

Claim	Evidence level
Context changes late-layer activation geometry	strong controlled evidence
Recovered local geometry reflects graph adjacency	supported by PCA, Dirichlet energy, and spectral-coordinate comparisons
Output behavior changes toward valid graph transitions	strong within-task behavioral evidence
Graph-like directions can causally redirect predictions	preliminary intervention evidence
The model internally performs literal Dirichlet-energy optimization	explanatory hypothesis, not mechanism identification
The graph geometry is necessary for task performance	not established
The model has learned a reusable 2D world model or planner	not established

Later Mechanistic Scrutiny

The 2026 ICLR Blogposts analysis by Andy Arditi reproduces the grid result and proposes a simpler decomposition:

1. Previous-token heads mix the predecessor token into the current residual state. Because every predecessor in a random walk is a graph neighbor, averaging activations by current token performs a form of neighbor mixing and can create graph-like PCA geometry.
2. Induction heads retrieve successors observed after previous occurrences of the current token. This bigram-recall mechanism can assign probability to valid neighbors without requiring explicit spatial planning.
3. Ablating previous-token heads disrupts both the geometry and learning curve, while ablating induction heads strongly harms task accuracy but can leave the visible grid geometry largely intact.

The safest synthesis is therefore:

The context induces a graph-structured latent geometry that is causally editable, but the graph-tracing benchmark does not yet separate a reusable inferred transition model from induction-style retrieval and neighbor-mixing artifacts.

This is a productive tension rather than a null result. It identifies the controls required for stronger in-context world-model claims: compositional graph queries, counterfactual edges, multi-step routes, unseen-node combinations, transition-rule changes, and causal tests that break induction shortcuts.

Foundation TSFM Relevance

The paper is not a numeric time-series model, but its setup maps cleanly to online latent-state inference from an event stream.

Agenda slot	Verdict	Evidence	Missing pieces
Context-conditioned latent state	adjacent	A frozen model’s internal state reorganizes as relational transition evidence accumulates in context.	Need multivariate time-series and mixed numeric/event-stream experiments.
Online system identification	adjacent	Hidden graph topology must be inferred from observed transitions without parameter updates.	Bigram retrieval can solve the current task; stronger controls must require a reusable transition model.
Representation geometry	adjacent	Dirichlet energy tests whether graph-neighbor states become locally smooth.	PCA and energy need calibrated nulls, matched frequency controls, and functional probes.
Behavior-space alignment	adjacent	Next-token mass on valid neighbors provides an external behavior measure and activation interventions alter it.	No full comparison of geodesics or local metrics between activation and behavior spaces.
World-model capability	insufficient evidence	The task contains latent transition structure and context-dependent state.	No action-conditioned rollouts, planning, counterfactuals, uncertainty calibration, or long-horizon state prediction.

For time-series work, the direct analogue is an event stream or multivariate observation stream generated by a hidden regime-transition graph. A model should update an internal latent state as evidence arrives. The important test is not whether PCA draws the graph, but whether the state supports unseen transition queries, multi-step forecasts, regime edits, counterfactual interventions, and OOD transfer.

Benchmark Design Lessons

A stronger temporal version SHOULD report:

1. graph or regime recovery beyond a two-dimensional PCA visualization;
2. block- or structure-preserving nulls that keep frequency and autocorrelation while breaking the target relation;
3. matched visitation-frequency controls, because node degree changes random-walk frequency;
4. probe results across named layers and context lengths;
5. causal edits tied to forecast or rollout changes;
6. induction/retrieval ablations and contexts where copied bigrams are insufficient;
7. multi-step and counterfactual queries over unobserved transitions;
8. separation between activation geometry, behavior geometry, and downstream decision utility.

Limitations And Gotchas

• PCA can expose a visually appealing graph even when the structure is created by averaging predecessor-mixed activations; visualization alone is not a functional-state test.
• Low Dirichlet energy has a constant-representation degeneracy. The paper addresses it with visible node separation, task accuracy, and standardized-energy analysis, but energy remains a smoothness diagnostic rather than a complete quality metric.
• Node degree and visitation frequency confound geometry on random walks.
• The simple one- and two-visit memorization baselines do not test modern induction-circuit retrieval, so beating them does not rule out a retrieval solution.
• The spectral theorem is conditional on idealized exact minimization and singular-value assumptions; empirical agreement is supportive rather than a proof of internal optimization.
• The causal intervention is performed in a low-dimensional PCA basis and only partially redirects behavior.
• Evidence is from language-model event streams, not numeric features, continuous time, actions, interventions, or exogenous variables.

Links Into The Wiki

• Graph Structure As Transformer Context
• Inference Dynamics
• Intermediate-Layer Representations
• Representational Similarity Calibration
• Time-Series Benchmark Hygiene
• Latent-State Time-Series Modeling
• Foundation Time-Series Model Research Agenda
• Contradictions And Open Tensions

Open Questions

• Which context-controlled tasks require an explicit reusable transition model rather than induction-style bigram retrieval?
• Can activation- and behavior-space metrics be compared directly while context length changes the inferred graph?
• Do graph-like activation coordinates support compositional route planning or only local next-neighbor prediction?
• What null model preserves token frequency, random-walk autocorrelation, and layer width while destroying graph topology?
• Can the same transition geometry be induced from multivariate time series with irregular sampling, missing values, actions, and exogenous variables?
• Does a causal edit along an inferred latent-state coordinate produce calibrated multi-step forecast changes rather than only local next-token shifts?