LT2: Linear-Time Looped Transformers

Source

• Raw Markdown: paper_lt2-2026.md
• PDF: paper_lt2-2026.pdf
• Preprint: arXiv 2605.20670v2
• Official code: chili-lab/LT2
• Official model: chili-lab/Ouro-hybrid-1.4B
• Official X announcement: Chunyuan Deng thread (compact discovery note stored at papers/lt2-2026/x_chunyuandeng_announcement_note.md)
• Gonzo ML discussion: Telegram post 5415 (local extract stored at papers/lt2-2026/telegram-post-gonzo-ml-5415.md)
• Gonzo-linked review: ArXivIQ review
• Podcast pointer: Gonzo ML Podcasts 3714

Status And Credibility

arXiv v1 was submitted on 2026-05-20 and v2 was revised on 2026-05-22. The paper is a current 2026 preprint from Rice University, Apple, UC Santa Cruz, and Carnegie Mellon authors. Credibility signals include an official BSD-3-Clause code repository, an official Hugging Face checkpoint, a primary-author X announcement, and enough implementation detail to reproduce the main architecture and distillation recipe. It is still preprint evidence, not a peer-reviewed venue result at ingest time.

The Hugging Face release describes Ouro-hybrid-1.4B as a research model distilled from ByteDance/Ouro-1.4B; its model card uses license: other, so downstream reuse needs a separate license check.

Core Claim

LT2 argues that the main serving bottleneck in modern looped Transformers is not parameter count but repeatedly applying full quadratic attention inside each loop. It replaces the token mixer inside the repeated block with linear, sparse, or hybrid attention so recurrent depth can remain parameter-efficient without paying a full KV-cache and attention cost at every loop.

Mechanism

LT2 studies two pure variants and several hybrids:

• LT2-linear uses linear-attention / DPLR-style recurrent-state mixers such as GDN, KDA, Mamba2-style SSD, RetNet, and related variants.
• LT2-sparse uses sparse or sliding-window attention variants.
• LT2-hybrid mixes full, linear, and sparse attention either across depth or across loop iterations.

The paper’s conceptual contribution is that looping is not only cheaper after replacing attention; it can also make the cheaper mixer more expressive:

$$A_t^{\mathrm{eff}}=\prod _{\tau =1}^T{A}_t^{(\tau )}$$

For DPLR linear attention, $T$ loop iterations can turn a rank-1 recurrent-state correction into an effective rank-$T$ correction when loop-specific keys are sufficiently independent. For sparse or sliding-window attention, looping expands the combinatorial receptive field from a window of size $w$ toward $O(Tw)$ reach, although the appendix warns that residual connections can still attenuate the effective horizon.

Evidence

The main from-scratch experiments train 0.6B and 1.3B models on FineWeb-Edu with a 100B-token budget and $T=4$ loops. The paper reports that looped GDN, KDA, and DSA variants come within roughly one average zero-shot point of the full-attention loop while avoiding quadratic attention. At 1.3B, Looped Hybrid (GDN+DSA) contains no full attention yet matches the full-attention Looped Transformer reference on perplexity closely (9.50 vs. 9.87) and improves average downstream score (60.73 vs. 59.27). Looped Hybrid (Full+GDN) is the strongest reported configuration, reaching 62.89 average zero-shot score versus 59.27 for the full-attention loop, while retaining only a small fraction of full-attention layers.

The long-context efficiency study measures prefill and decode throughput from 1k to 32k tokens on a single H100 80GB. LT2 reports that GDN and hybrid variants hold decode throughput roughly flat across context length; at batch size 1 and 32k, they decode about 3x faster than the full-attention loop, and at batch size 8 the full-attention loop runs out of memory by 8k while the GDN and GDN+Full variants reach 32k.

For pretrained conversion, the paper distills ByteDance/Ouro-1.4B into Ouro-hybrid-1.4B through linear pre-alignment, hybrid logit distillation, per-loop supervision, and long-context continuation. The abstract summarizes this as about 1B tokens of continued training; the method section describes 100M tokens for pre-alignment, 600M tokens for hybrid logit distillation, and 600M tokens for 32k continuation. The released checkpoint makes the conversion recipe more useful than a paper-only architecture proposal.

Relevance To This Wiki

This is a direct update to the looped-depth branch. It changes the cost model for recurrent-depth Transformers: if the repeated block uses full attention, loop count multiplies KV-cache and attention cost; if the repeated block uses linear or sparse mixers, loop count can become a practical dynamic-compute axis.

For the foundation time-series agenda, LT2 is most useful as an upstream architecture and serving-budget analogy. Always-on multivariate systems need long histories, bounded memory, and cheap state updates, but LT2’s evidence is language modeling, synthetic recall/state-tracking, and long-context language throughput rather than numeric telemetry, event streams, control inputs, interventions, or action-conditioned world models.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Dynamic compute under a serving budget	adjacent	Shows recurrent depth can be paired with linear or sparse mixers and reports long-context throughput and OOM-frontier improvements under $T=4$ loops.	Needs numeric time-series or trajectory experiments with matched latency, memory bandwidth, and expected-FLOPs budgets.
Streaming latent state / long context	adjacent	Linear-attention variants maintain compact recurrent state, and looping can increase DPLR update rank.	No always-on streaming benchmark, cross-window state-retention test, or rare-regime retention audit.
Native multivariate encoding	insufficient evidence	Architecture could be adapted to channel/time tokens, but all main evidence is language or synthetic token tasks.	Needs high-channel numeric series, channel-dependency probes, and dense numeric fidelity checks.
Control and counterfactuals	insufficient evidence	No action, control input, treatment, or intervention channel is evaluated.	Needs action-conditioned rollouts and intervention-aware evaluation.
Benchmark hygiene	warning	The paper reports meaningful throughput, memory, and zero-shot axes, and the model card labels the checkpoint as research-only.	Wiki synthesis should not convert long-context language throughput into TSFM readiness without domain tests.

Limitations

The paper’s own limitations are important: it studies depth-level hybridization and simple loop-level schedules but does not fully explore loop-level hybridization with distinct attention families per iteration; it also does not design explicit cross-loop recurrent-state carry mechanisms. The appendix says fixed loop count is used in the main pretraining runs because adaptive computation time remains unstable and difficult to implement efficiently at scale.

The sparse-attention theory distinguishes combinatorial receptive-field growth from effective signal propagation; residual paths can keep the effective horizon much shorter than the topological reach. Treat LT2 as evidence that looped efficient mixers are promising, not as proof that sparse loops preserve arbitrary long-range numeric state.

Links Into The Wiki

• LT2
• Looped Transformers And Test-Time Memory
• Efficient Recurrent Sequence Models
• Time-Series Scaling And Efficiency
• Time-Series Benchmark Hygiene
• GPU Inference Optimization
• Foundation Time-Series Model Research Agenda
• Universal Transformers
• Huginn
• Language Models Need Sleep
• Parcae
• Sparse Layers are Critical to Scaling Looped Language Models
• Hyperloop Transformers
• Mamba-2
• Mamba-3
• RWKV-TS

Open Questions

• For numeric time series, does looped linear attention preserve dense numeric detail and rare events better than one-pass SSM/RWKV/Mamba state under the same memory and latency budget?
• Can loop-level hybrid schedules become adaptive over windows, regimes, channel groups, or uncertainty rather than fixed across all inputs?
• Does DPLR rank growth across loops translate into better multivariate state tracking, or does it mainly improve language-style token associations?
• What is the right TSFM benchmark for comparing full attention, linear attention, sparse attention, and looped hybrids when channel count, context length, intervention history, and serving cost all vary?
• How should Ouro-hybrid-style post-hoc conversion be adapted for an existing time-series foundation model without erasing calibration, rare regimes, and action effects?