Gated Delta Networks: Improving Mamba2 with Delta Rule

Source

• Raw Markdown: paper_gated-deltanet-2025.md
• PDF: paper_gated-deltanet-2025.pdf
• Preprint: arXiv 2412.06464
• OpenReview / venue page: ICLR 2025 submission
• Official NVIDIA page: NVIDIA LPR publication page
• Official code: NVlabs/GatedDeltaNet
• Local source archive: papers/gated-deltanet-2025/arXiv-2412.06464.tar.gz
• Local provenance: papers/oryx-2026/x_thread_kevinyli_atrost3122_2067658710810603549.md mentions Gated DeltaNet as one of Oryx’s linear mixer backbones.

Status And Credibility

arXiv lists Gated Delta Networks as a cs.CL / cs.LG preprint first submitted on 2024-12-09 and revised to v3 on 2025-03-06. The arXiv page and official repository identify it as an ICLR 2025 camera-ready conference paper by Songlin Yang, Jan Kautz, and Ali Hatamizadeh.

Although the paper is now older than one year relative to the 2026-06-30 ingest date, it is still important rather than stale background: it is an ICLR 2025 paper, has official NVIDIA code, is the direct predecessor of Gated DeltaNet-2, and is used as a backbone in Oryx. The official repository says Gated DeltaNet was incorporated into Olmo Hybrid and Qwen-family hybrid models after publication.

GitHub API at ingest time reported NVlabs/GatedDeltaNet with 615 stars, 32 forks, SPDX NOASSERTION, and a 2026-06-30 update timestamp. The repository page describes the license as NVIDIA Source Code License-NC, so reuse needs license review.

Core Claim

Gated DeltaNet argues that two memory-control mechanisms are complementary in linear recurrent attention:

• gating can rapidly erase or decay irrelevant state;
• the delta rule can make targeted key-value updates.

The paper combines them into a gated delta rule, then derives a hardware-efficient chunkwise training algorithm for modern GPUs.

Mechanism

The proposed update is:

$$S_t=S_{t-1}\left({\alpha }_t(I-{\beta }_t{k}_t{k}_t^{\top })\right)+{\beta }_t{v}_t{k}_t^{\top }.$$

Here ${\alpha }_t\in (0,1)$ is a data-dependent decay gate and ${\beta }_t$ is a data-dependent write/update strength. In the paper’s framing, this is a fast-weight associative memory update: the hidden state stores key-value associations, gating clears state when context changes, and the delta update edits a particular association.

flowchart LR
  Prev[previous recurrent state] --> Gate[decay gate alpha_t]
  Key[key k_t] --> Delta[delta erase/update]
  Value[value v_t] --> Delta
  Gate --> Delta
  Delta --> State[updated Gated DeltaNet state]
  Query[query q_t] --> Read[read from state]
  State --> Read
  Read --> Output[output]

The paper extends prior DeltaNet chunkwise algorithms with decay terms, preserving a parallel training path that uses matrix multiplications and a triangular inverse inside chunks.

Evidence

The paper evaluates language modeling, commonsense reasoning, in-context retrieval, length extrapolation, and long-context understanding. The most agenda-relevant evidence is the long-context associative-memory behavior:

Probe from paper	Baseline contrast	Reported Gated DeltaNet behavior
S-NIAH-1 pass-key retrieval	Mamba2 loses long retention by 8K; DeltaNet stays near-perfect.	Gated DeltaNet remains much stronger than Mamba2 at 4K/8K, though not always as high as pure DeltaNet on the simplest repeated-context case.
S-NIAH-2 number-in-haystack	DeltaNet degrades under real-world essay context from poor memory clearance.	Gated DeltaNet reports 92.2 at 4K versus 56.2 for Mamba2 and 18.6 for DeltaNet in the table excerpt.
S-NIAH-3 UUID-in-haystack	Mamba2 struggles with complex memorization; DeltaNet lacks enough filtering.	Gated DeltaNet reports 84.2 at 2K versus 47.6 for Mamba2 and 47.0 for DeltaNet in the table excerpt.

The paper also reports that hybrid variants combining Gated DeltaNet layers with sliding-window attention or Mamba2 layers improve performance and efficiency relative to the tested baselines.

Relevance To This Wiki

Gated DeltaNet is upstream architecture evidence for compact recurrent associative memory. It matters because later sources in this wiki build on it:

• Gated DeltaNet-2 decouples erase and write beyond Gated DeltaNet’s scalar-gate formulation.
• Oryx uses Gated DeltaNet as one of its sequence-axis linear mixer instantiations.
• Hybrid Associative Memories uses GDN as a core baseline and RNN component for selective KV-cache routing.

For time-series and world-model work, the transferable idea is selective memory editing under a bounded state. A streaming model may need to clear stale relationships at context changes while preserving rare or action-relevant associations.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Streaming latent state / long context	adjacent	Fixed-size recurrent state with gated memory management and improved long-context retrieval versus Mamba2/DeltaNet on language probes.	Needs numeric time-series, event-stream, high-channel, and action-history evaluations.
Dynamic compute / serving efficiency	adjacent	The update remains in the linear recurrent family and includes a chunkwise parallel training algorithm.	Needs wall-clock TSFM serving comparisons, state-size accounting, and hardware/kernel availability under time-series workloads.
Benchmark hygiene	warning	NIAH/retrieval probes isolate memory behaviors better than aggregate loss alone.	NIAH is not dense numeric fidelity, rare-regime preservation, or control-utility evidence.
Native multivariate encoding	insufficient evidence	Hidden key/value state can be read as an associative-memory analogy.	Hidden coordinates are not observed variables, sensors, channels, graph nodes, or topology.
Control and counterfactuals	insufficient evidence	Memory gates could preserve action-relevant state if adapted.	No actions, control inputs, interventions, treatments, rewards, or counterfactual rollouts.

Limitations

• Evidence is language-model and retrieval-centric, not numeric time-series or action-conditioned world-model evidence.
• The scalar decay/write formulation is exactly what Gated DeltaNet-2 later argues is too coarse for some memory-editing problems.
• The official repository does not release pretrained weights according to its FAQ; it releases code and points users to FLA for easier integration.
• The code license is non-permissive for some reuse contexts.
• Retrieval tasks are prompt- and protocol-sensitive; repository guidance warns not to treat generic lm-eval-harness runs as enough for recall-intensive tasks.

Links Into The Wiki

• Gated DeltaNet
• Gated DeltaNet-2
• Oryx
• Hybrid Associative Memories
• Mamba-2
• Efficient Recurrent Sequence Models
• Extra-Long Context For Time Series
• Streaming Latent-State Updates
• Time-Series Scaling And Efficiency
• Time-Series Benchmark Hygiene
• Foundation Time-Series Model Research Agenda

Open Questions

• When is scalar gated-delta memory enough, and when does the decoupled erase/write interface from Gated DeltaNet-2 matter?
• Can gated delta recurrence preserve rare time-series regimes better than Mamba-style decay under the same state size?
• What is the numeric analogue of a memory collision in linear attention: cross-channel interference, regime overwrite, action-effect forgetting, or topology-binding loss?
• Can Gated DeltaNet-style state updates be combined with explicit sparse memory without losing their serving advantage?