A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Source

• Raw Markdown: paper_hola-2026.md
• PDF: paper_hola-2026.pdf
• Preprint: arXiv 2607.02303v1
• Local source archive: papers/hola-2026/arXiv-2607.02303v1.tar.gz
• Telegram discovery provenance: gonzo_ML/5675; local snapshot: papers/hola-2026/telegram-post-gonzo_ML-5675.md.
• Telegram review provenance: gonzo_ML_podcasts/4307; local snapshot: papers/hola-2026/telegram-post-gonzo_ML_podcasts-4307.md, including the ordered textual continuation through posts 4308—4310.
• Official code or model release: none found at ingest time.

Status And Credibility

arXiv lists this as a cs.AI preprint by Wanyun Cui, first submitted as v1 on 2026-07-02. The paper identifies the affiliation as Shanghai University of Finance and Economics. It is current architecture evidence, and its main HOLA-versus-GDN comparisons use matched backbones and explicit component ablations.

The evidence is still preliminary. No peer-reviewed venue, official project page, code repository, or model checkpoint was found during this ingest. The principal 170M and 340M results are single-seed, several non-anchor baseline rows are borrowed from recipe-matched prior work, and the largest model is 340M parameters trained on 15B tokens. Treat HOLA as a strong mechanism proposal with controlled internal comparisons, not as a settled production architecture or frontier-scale result.

The Telegram posts are discovery and commentary provenance. Technical claims below are checked against the paper rather than inherited from the social summary.

Core Intuition

A fixed-size linear-attention state is efficient because it compresses the entire prefix, but that same compression overwrites exact key—value associations. HOLA keeps the recurrent state for compressible structure and adds a small exact cache for tokens that the state had to change most to absorb.

The central rule is:

ordinary/predictable history -> compressed recurrent state
large committed state update -> bounded exact KV memory

This is a two-resource memory contract rather than a claim that compression alone becomes lossless.

Mechanism

State update and write score

For a delta-rule layer, the residual innovation is the part of the value not predicted by the current state. In HOLA’s Gated DeltaNet instantiation,

$$e_t=v_t-{\alpha }_t{k}_t^{\top }{S}_{t-1},$$

where ${\alpha }_t$ is the GDN decay gate. The rank-one state edit has magnitude

$${\Delta }_t={\beta }_t{k}_t{e}_t^{\top },m_t=\Vert {\Delta }_t{\Vert }_F={\beta }_t\Vert e_t\Vert$$

because the state-path key is unit-normalized. HOLA uses $m_t$ as a parameter-free cache score. Each layer retains the exact key—value pairs for the top-$w$ scores seen so far, with $w=64$ in the reported models.

The paper calls this score “surprise,” but it is more precisely the committed delta-rule update magnitude. It is not calibrated predictive uncertainty and does not directly encode downstream utility.

Semiparametric read

HOLA frames memory as test-time regression. The recurrent state is a fixed-size parametric estimator, while the bounded exact cache is a non-parametric correction:

$$o_t=q_t^{\top }{S}_t+{\lambda }_t{g}_t(q_t).$$

The cache read $g_t$ uses softmax attention over the retained exact pairs. Reusing the unit-L2 query/key vectors from the state path makes these logits too flat, so HOLA applies a separate Qwen3-style RMSNorm-$\gamma$ to the cache-path queries and keys. The state path remains unit-normalized for stability; only the cache path gets the sharper, near-argmax read.

The persistent cache contains $w=64$ selected pairs. The implementation also exposes the current block of $C=256$ tokens and one null sink, so the maximum visible exact-memory set is $w+C+1\approx 321$ entries per layer.

flowchart LR
  Token[current token] --> Residual[delta-rule residual e]
  Residual --> State[update recurrent state]
  Residual --> Score[score beta times norm e]
  Score -->|top-w| Cache[bounded exact KV cache]
  State --> StateRead[compressed state read]
  Cache --> SharpRead[decoupled RMSNorm-gamma cache read]
  StateRead --> Mix[learned mixture]
  SharpRead --> Mix
  Mix --> Output[layer output]

Evidence

Matched scaling comparison

HOLA reports a consistent Wikitext-103 perplexity reduction against a same-scale GDN anchor:

Scale	Training data	GDN	HOLA	Relative reduction
46M	0.5B FineWeb-Edu tokens	71.0	59.5	about 16%
170M	6.22B SlimPajama tokens	35.98	30.51	about 15%
340M	15B SlimPajama tokens	27.32	22.92	16.1%

At 340M, HOLA also reports LAMBADA perplexity 30.26 versus 30.95 for its GDN anchor. Its Wikitext value is below the reported recipe-matched Transformer++ row, 26.88, but this does not establish general superiority over full attention: the exact-extraction FDA score remains far below Transformer++ (20.1 versus 46.1), and the models are small research-scale checkpoints.

Exact retrieval and long context

• On 340M in-context retrieval, HOLA improves FDA from 11.7 to 20.1 and SWDE from 29.0 to 35.9 against its same-backbone GDN anchor.
• On RULER S-NIAH-1 at 32k tokens, HOLA reports 0.58 accuracy versus 0.24 for a matched recency-cache control and 0.14 for GDN.
• The model was trained at 2k context, so 32k is a $16\times$ extrapolation test.
• The full-attention Transformer++ row is a 2k-trained RoPE checkpoint that drops to zero beyond 2k in the reported table. It is a useful in-range ceiling, not a modern length-extrapolating full-attention baseline.
• HOLA remains imperfect: 0.58 recall means the bounded cache still misses many items in long or needle-dense contexts.

Ablation evidence

The paper isolates both design choices:

1. What to retain. At matched memory and read path, top-$w$ selection by $\beta \Vert e\Vert$ beats a recent-token cache on Wikitext perplexity (22.92 versus 25.04) and 32k S-NIAH-1 (0.58 versus 0.24).
2. How to read. At 46M, replacing the unit-L2 cache read with decoupled RMSNorm-$\gamma$ lowers Wikitext perplexity from 70.10 to 59.5 and improves multi-key recall.

These ablations support the internal HOLA mechanism. They do not replace a matched large-scale comparison against learned-eviction systems such as LTE, which the paper lists as missing.

Parameter and memory cost

At 340M, HOLA adds 12,480 trainable scalars, under 0.004% of the model. The cache is inference state rather than parameters: the paper estimates about 31 MB of bf16 K/V state for its 24-layer configuration and reports peak allocation of 0.75 GB versus 0.72 GB for GDN at 32k and 128k contexts.

Thus memory is asymptotically constant in sequence length under the fixed cache budget, but not free. The reported approximately 5% overhead is configuration- and implementation-specific; no edge-device latency, energy, batching, or kernel benchmark is provided.

Comparison With Nearby Memory Designs

Design	Compressed memory	Exact-memory selection	Budget behavior	Main distinction
Gated DeltaNet	Fixed recurrent associative state	None	Constant state	Efficient but exact associations can be overwritten.
Hybrid Associative Memories	DeltaNet-style recurrent state	Prediction-error threshold or learned router	Data-dependent cache fraction	Routes selected tokens into a KV scratchpad and exposes cache-growth control.
HOLA	GDN recurrent state	Top-$w$ committed update magnitude $\beta \Vert e\Vert$	Fixed bounded cache	Uses the existing delta update as a parameter-free score and sharpens the exact-cache read.
LTE	GDN plus exact cache	Learned CNN eviction scorer	Bounded evictable cache	Closest paper-cited analogue, but no matched large-scale HOLA comparison is reported.

HAM and HOLA should not be ranked from their headline numbers because their model sizes, token budgets, cache contracts, and evaluations differ. The durable distinction is threshold/learned routing with data-dependent growth versus deterministic top-$w$ retention with a fixed cache budget.

HOLA is instantiated on the original GDN update. It does not test whether Gated DeltaNet-2’s decoupled erase/write gates change the best score or reduce the need for exact memory.

Relevance To Time Series And World Models

HOLA is upstream language-model architecture evidence, not a time-series or action-conditioned world-model result. Its transfer hypothesis is nevertheless concrete: a compact recurrent state could model predictable numeric dynamics, while a bounded exact memory retains rare events, change points, unusual cross-channel bindings, exogenous variables, or action/control-input windows that would otherwise be overwritten.

The key risk is objective mismatch. A large state update need not mean operational importance, and a predictable event can still be decision-critical. Scheduled interventions, periodic safety checks, known future exogenous variables, or stable-but-rare topology changes may receive low $\beta \Vert e\Vert$ even though the downstream controller must remember them.

For streaming systems, global top-$w$ retention creates another question: old high-score events can occupy the cache indefinitely, while a regime shift may change what should count as valuable. A time-series adaptation may need age, uncertainty, downstream value, intervention relevance, or per-regime quotas in addition to update magnitude.

Relation To Fixed-Budget Hierarchy

HOLA supports the intuition in Hierarchical Modeling with a Fixed FLOPs Budget: not every observation deserves exact high-resolution memory. It provides a particularly simple local allocation signal because the recurrent update already exposes one measure of compression difficulty.

It does not yet implement the requested hierarchy. HOLA has one flat top-$w$ cache per layer, a fixed cache size, and no global controller that allocates memory or FLOPs across layers, temporal scales, channels, or modalities. A hierarchical extension would need to prove that local surprise scores compose into a global budget rather than merely duplicating salient events at every layer.

Social-Claim Audit

• “O(1) inference memory” is correct asymptotically in sequence length under fixed recurrent-state and cache sizes, but the exact cache adds about 31 MB and approximately 5% peak allocation in the reported setup.
• “Practically zero overhead” is accurate for trainable parameters, not literally for memory traffic or cache-attention compute.
• “Closes the gap with full attention” is too broad. HOLA beats one recipe-matched full-attention row on Wikitext perplexity, but remains far behind it on FDA exact extraction and is not compared with a length-extrapolating full-attention model at 32k.
• “Ready for edge devices” is an extrapolation from bounded memory. The paper contains no edge-hardware, latency, throughput, power, or quantization evaluation.
• The paper has one author, despite plural wording in parts of the Telegram review.

Limitations

• Fresh single-author arXiv v1; no peer-reviewed venue, official code, or checkpoint found at ingest time.
• Main 170M/340M results are single-seed; only selected smaller diagnostics include multi-seed averaging.
• Largest experiment is 340M parameters and 15B tokens, below contemporary production-model scale.
• The bounded cache cannot retain every useful item; 32k single-needle recall is 0.58 rather than complete.
• Exact token extraction still trails full attention substantially on FDA.
• No matched large-scale comparison against LTE’s learned eviction module.
• No direct evidence for numeric features, multivariate time series, event streams, topology, exogenous variables, actions, control inputs, interventions, or closed-loop world-model use.
• Reported memory allocation does not establish production latency, throughput, batching stability, or edge deployment.

Links Into The Wiki

• Hippocampal Linear Attention
• Gated DeltaNet
• Gated DeltaNet-2
• Hybrid Associative Memories
• Efficient Recurrent Sequence Models
• Extra-Long Context For Time Series
• Streaming Latent-State Updates
• Looped Transformers And Test-Time Memory
• Time-Series Scaling And Efficiency
• Hierarchical Modeling with a Fixed FLOPs Budget
• Contradictions And Open Tensions

Open Questions

• Does update-magnitude retention outperform learned routing at matched cache size, latency, and training tokens?
• Can a HOLA cache adapt to regime shifts without allowing old high-score events to monopolize memory?
• Should numeric-stream retention combine update magnitude with uncertainty, downstream loss, action relevance, event type, or age?
• How should exact-memory slots be allocated across layers, temporal scales, channels, entities, and topology nodes under one global budget?
• Does Gated DeltaNet-2’s erase/write separation change the best cache-selection score?
• Which preservation probes can distinguish surprising-but-disposable observations from predictable-but-decision-critical state?