Alex Open Research Wiki

Hippocampal Linear Attention

2 min read

Hippocampal Linear Attention

Summary

Hippocampal Linear Attention (HOLA) is the bounded exact-memory extension to Gated DeltaNet introduced by A Hippocampus for Linear Attention. It preserves the recurrent state for compressible sequence structure and adds a per-layer exact KV cache for tokens with the largest committed delta-rule update magnitude .

HOLA also separates the memory interfaces: state-path keys remain unit-normalized for stable recurrent updates, while cache-path queries and keys use RMSNorm- to make exact-cache retrieval sharp rather than an almost uniform average.

Role In The Wiki

Use this page as the model card for HOLA. The source page owns the evidence, social-claim audit, comparison with Gated DeltaNet, Hybrid Associative Memories, and LTE, and the mapping to time-series and world-model research.

For this wiki, HOLA is upstream architecture evidence for bounded dual memory:

compressed recurrent state + selective exact event memory

It does not directly evaluate numeric time series, event streams, exogenous variables, actions, control inputs, interventions, or world-model rollouts.

Main Configuration

  • Backbone: Gated DeltaNet.
  • Persistent cache: top 64 exact key—value pairs per layer by .
  • Current-block exposure: 256 tokens plus one null sink, for at most about 321 visible exact-memory entries.
  • Largest reported model: 340M parameters, 24 layers, four 256-dimensional heads, trained on 15B SlimPajama tokens at 2k context.
  • Added trainable scalars: 12,480, less than 0.004% of the 340M model.
  • Reported peak allocation: 0.75 GB versus 0.72 GB for GDN at 32k/128k decoding.

Evidence

Official Artifacts

  • Preprint: arXiv 2607.02303v1
  • Official code/model/project page: none found at ingest time.
  • Local paper bundle: papers/hola-2026/
  • Local Telegram provenance: papers/hola-2026/telegram-post-gonzo_ML-5675.md and papers/hola-2026/telegram-post-gonzo_ML_podcasts-4307.md.

Graph View

Backlinks