MiniMax Sparse Attention

Source

• Raw Markdown: paper_minimax-sparse-attention-2026.md
• PDF: paper_minimax-sparse-attention-2026.pdf
• Preprint: arXiv:2606.13392
• Official code: MiniMax-AI/MSA. Local metadata: papers/minimax-sparse-attention-2026/official_artifacts_metadata.json.
• Official model: MiniMaxAI/MiniMax-M3 and MiniMax-AI/MiniMax-M3.
• Gonzo ML discussion: t.me/gonzo_ML/5579. Local public-post snapshot: papers/minimax-sparse-attention-2026/telegram-post-gonzo-ml-5579.md.
• Review link from the post: ArXivIQ
• GQA background linked from the post: arXiv:2305.13245
• Podcast pointer from the post: Gonzo ML Podcasts 4056

Status And Credibility

This is an arXiv v2 preprint submitted on 2026-06-11 and revised on 2026-06-12. The listed affiliations are MiniMax, Peking University, NVIDIA, Zhejiang University, Huazhong University of Science and Technology, Nanjing University, and Hangzhou Dianzi University.

Treat it as current and important but not yet peer-reviewed evidence. Positive credibility signals are the official MiniMax code repository, an open-weight MiniMax-M3 model release, a concrete 109B-parameter MoE evaluation setting, native multimodal training evidence, explicit ablations, and kernel-level implementation details. Caveats: the paper is a company technical report; the public code repository describes an SM100 kernel package while the paper reports H800 speedups; the MiniMax-M3 weights use the MiniMax Community License rather than a standard permissive license; and independent reproduction of the 1M-context speed/quality tradeoff is not yet visible in the KB.

Core Claim

MiniMax Sparse Attention (MSA) argues that long-context LLMs can keep exact softmax attention on a learned, content-dependent block subset rather than paying dense attention over the full sequence. A lightweight Index Branch scores key-value blocks for each Grouped-Query Attention (GQA) group. The Main Branch then computes exact block-sparse attention only over the selected blocks.

The deployed configuration emphasized in the paper uses block size $B_k=128$ and selects $k=16$ blocks, so each query/GQA group attends to at most $k{B}_k=2,048$ tokens even at 1M context.

flowchart LR
  X[Hidden states] --> IQ[Index Q/K projections]
  IQ --> Score[Block scores]
  Score --> TopK[Top-k KV block selection per GQA group]
  X --> MainQ[Main Q/K/V projections]
  TopK --> Sparse[Exact block-sparse softmax attention]
  MainQ --> Sparse
  Sparse --> Out[Layer output]
  MainQ --> Teacher[Main-branch attention distribution]
  Teacher --> KL[KL alignment loss]
  IQ --> KL

Mechanism Notes

• Selector shape: the Index Branch adds one index query head per GQA group and one shared index key head, then performs block-level max-pooling before Top-$k$ selection.
• Training signal: because Top-$k$ selection is non-differentiable, the indexer is trained with a KL alignment loss against the group-averaged Main Branch distribution on the selected support.
• Gradient routing: the paper stops the KL gradient at the Index Branch input so the auxiliary loss updates only the index projections rather than reshaping the backbone through a side objective.
• Warmup: sparse training starts with a full-attention warmup for the indexer before the sparse selector controls routing. The same warmup is used when converting a pretrained GQA checkpoint.
• Local context: the local block containing the current position is always included, which protects short-range modeling and stabilizes early training.

Kernel Co-Design

The paper’s main systems point is that sparsity must be matched to GPU execution to become wall-clock speed. MSA uses:

• exp-free Top-$k$ selection, relying on score ordering rather than a full softmax before selection;
• a small-$k$ TopK kernel tuned around $B_k=128$ and $k=16$;
• KV-outer sparse attention, where selected KV blocks gather associated queries so the kernel can fill tensor-core matrix multiplies better than query-outer sparse execution;
• pre-scheduled chunking and two-phase combine to handle popular KV blocks without atomics;
• fused LSE handling for the sparse KL loss and persistent load balancing in backward.

The reported wall-clock result is $14.2\times$ prefill and $7.6\times$ decode speedup versus dense GQA at 1M context on H800, with $28.4\times$ lower per-token attention FLOPs in the model configuration they instantiate.

Evidence And Results

Claim	Paper evidence	Wiki interpretation
MSA can be trained natively, not only applied at inference	MSA-PT trains a 109B total-parameter / 6B activated-parameter MoE model from scratch on 3T tokens with 40B-token indexer warmup.	Stronger than inference-only sparsification because the model can adapt representations to sparse routing.
Dense-to-sparse conversion is viable	MSA-CPT starts from a 2.6T-token full-attention checkpoint, runs 40B-token indexer warmup, then sparse continued pretraining for 400B tokens.	Practical route for existing GQA checkpoints, but still expensive and not a drop-in runtime flag.
Quality remains close to GQA	The paper reports broadly comparable general, code, math, image, video, RULER, HELMET, and agent-oriented perplexity metrics.	Useful current evidence, but should be read as vendor-reported benchmark parity until independently replicated.
Sparse attention can preserve long-context behavior under tight budgets	After long-context training, MSA-CPT remains close to the Full-Attention baseline on HELMET-128K and RULER-128K while attending to 2,048 selected tokens per query/GQA group.	The strongest modeling claim: content-dependent sparse blocks can preserve much of long-context retrieval with a fixed selected-token budget.
Kernel work matters	The paper reports TopK-kernel wins over torch.topk and TileLang and large prefill/decode speedups at 1M context.	This belongs in GPU inference optimization, not only model architecture: the algorithm and kernel are inseparable.

MiniMax-M3 Release Context

The linked MiniMax-M3 artifact is a production/open-weight model powered by MSA. The Hugging Face model card describes it as a native multimodal model with about 428B parameters and about 23B activated parameters, 1M-context support, image/video/text input, coding and agentic emphasis, and thinking modes. The model repository and card are evidence that MSA is not only a paper operator; it is part of a released model stack.

The model license is a MiniMax Community License with commercial-use conditions. Treat MiniMax-M3 as an open-weight release, not as a standard permissively licensed artifact.

Relation To Existing Long-Context Methods

MSA sits between fixed sparse patterns and fully dense attention:

• Compared with local/sliding-window attention, MSA’s selected blocks are content-dependent and per-GQA-group.
• Compared with inference-time KV pruning such as H2O/SnapKV/Quest-style methods, MSA trains the selector and sparse attention path as part of the model.
• Compared with recurrent or state-space approaches, MSA keeps exact softmax attention over a selected subset rather than compressing the entire past into a fixed recurrent state.
• Compared with learned context compression, MSA does not compress selected tokens into a smaller latent; it sparsifies which blocks are read.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Dynamic compute and long context	adjacent	MSA allocates attention compute to content-selected blocks and keeps the selected budget fixed as context grows.	Need numeric time-series or telemetry tests where rare events, delayed exogenous variables, and action history must survive sparse selection.
Native multivariate encoding	adjacent	Per-GQA-group block selection is a useful analogy for group-specific sparse retrieval.	GQA groups are hidden attention groups, not observed sensors, channels, graph nodes, or typed event streams.
Serving efficiency	adjacent	MSA reports large prefill/decode speedups and released kernels/model artifacts.	Need full serving benchmarks under vLLM/SGLang-style batching, KV cache pressure, bursty workloads, and heterogeneous hardware.
Dense-detail preservation	warning	Exact attention over selected tokens avoids lossy compression inside the selected support.	The unselected support is invisible to the layer, so preservation probes must test rare, off-window, low-salience events.
Control and counterfactuals	insufficient evidence	Long agentic context is a motivating workload.	No action-conditioned time-series rollouts, interventions, treatments, control inputs, or counterfactual futures are evaluated.

Limitations And Gotchas

• This is a 2026 arXiv preprint/company technical report, not a peer-reviewed venue result at ingest time.
• Evidence is LLM/multimodal/coding-agent centric, not numeric time-series evidence.
• The public code repository targets NVIDIA SM100 in its README, while the paper reports H800 results; use the paper numbers and public artifact as related but not automatically identical reproducibility claims.
• The MiniMax-M3 model license has commercial-use conditions and prohibited-use clauses; do not describe it as Apache/MIT open weights.
• Sparse attention is not free compression. It reduces attention work by hiding unselected blocks from the layer. Any TSFM transfer must audit whether rare regimes, event timing, exogenous variables, actions, and topology-dependent state are dropped.
• The reported parity is under MiniMax’s own training and evaluation suite. The KB should prefer independent long-context, multimodal, and serving benchmarks when they appear.

Links Into The Wiki

• MiniMax Sparse Attention
• MiniMax-M3
• GPU Inference Optimization
• Time-Series Scaling And Efficiency
• Extra-Long Context For Time Series
• Unified Multimodal Models
• Vision-Language Models

Open Questions

• Can MSA-style per-group block selection preserve rare numeric events and intervention history in multivariate time-series streams, or does it over-select high-salience local/sink patterns?
• What is the right analogue of a GQA group for time-series: channel groups, topology regions, latent factors, event types, or learned sensor clusters?
• Can sparse block selection be composed with learned context compression, recurrent state, or KV-cache quantization without compounding preservation failures?
• Which serving metrics remain favorable when MSA is evaluated under bursty multi-tenant workloads with batching, prefix caching, and output-length variation?