# A Few Gaussians Is All You Need: SSOG

## Provenance

- Source type: author technical blog, X launch thread, and public code repository.
- Method name: Separable Sum of Gaussians (SSOG).
- Blog title: A Few Gaussians Is All You Need: SSOG-Attention That Steers Instead of Scores.
- Blog author: Raphael Pisoni.
- Blog timestamp: 2026-08-15 18:30, as displayed by the official page.
- Official blog: <https://pisoni.ai/posts/ssog/>
- Original X status: <https://x.com/i/status/2089002002185384395>
- Canonical X status: <https://x.com/ml_4rtemi5/status/2089002002185384395>
- X root timestamp: 2026-08-16T14:51:03Z.
- Official code: <https://github.com/4rtemi5/ssog>
- Audited code commit: [`c6f00afb7affec06735d94aa00b7064b9df4c266`](https://github.com/4rtemi5/ssog/commit/c6f00afb7affec06735d94aa00b7064b9df4c266), timestamped 2026-08-16T08:47:46Z.
- Snapshot date: 2026-08-16.
- Local artifacts include the complete authenticated X API responses, a normalized X thread, blog HTML and Markdown snapshots, official visual assets, repository metadata and README, a source archive pinned to the audited commit, and a separate code audit.

## Source Status

This is a fresh author blog/code/X package, not a paper, formal technical report, or peer-reviewed result. The primary artifacts agree on the mechanism and reported benchmark values, and the public JAX/Flax code makes the core token mixer inspectable. The evidence nevertheless remains author-reported: no ImageNet training pipeline, checkpoints, raw logs, complete seed-level records, or optimized-kernel benchmark is released at the audited commit. The blog reports three-seed CIFAR runs and standard deviations for selected ImageNet SSOG variants, while the ImageNet SDPA and other bare rows are single runs. Treat SSOG as a technically concrete research prototype, not established SOTA.

## Launch-Thread Claim

The ten-post same-author X thread argues that pairwise scaled dot-product attention is a poor inductive bias for vision because each patch scores every other patch. It proposes replacing the query-key score matrix with a small number of learned Gaussian spatial fields that move with each query and receive bounded content-conditioned adjustments. The thread claims:

- factorization into vertical and horizontal one-dimensional passes;
- $O(N\sqrt{N}d)$ spatial mixing rather than $O(N^2d)$ on square grids with fixed atom count;
- a 17-point-class gain over a matched ViT on CIFAR-100;
- smaller ImageNet-1k models with lower reported FLOPs and slightly better top-1 accuracy;
- zero-shot transfer from $224^2$ to larger image resolutions after resizing the learned positional embedding.

The exact posts, timestamps, canonical URLs, media, and the small public conversation are preserved in `x_thread_ml_4rtemi5_2089002002185384395.md`, `x_post_ml_4rtemi5_2089002002185384395.json`, and `x_conversation_2089002002185384395.json`. Engagement counts are unstable and are not technical evidence.

## Method Snapshot

For a square feature map with query position $p=(p_y,p_x)$ and key position $q=(q_y,q_x)$, each head has $R$ Gaussian atoms. Atom $r$ has a learned center offset $\mu_r=(\mu_{r,y},\mu_{r,x})$, positive widths $\sigma_r=(\sigma_{r,y},\sigma_{r,x})$, and mixture weight $\lambda_r$.

The blog describes a log-mixture score:

$$
s(p,q)=\operatorname{logsumexp}_{r=1}^{R}
\left[
\log \lambda_r
+\log\mathcal N(p_y-q_y;\mu_{r,y},\sigma_{r,y})
+\log\mathcal N(p_x-q_x;\mu_{r,x},\sigma_{r,x})
\right],
$$

followed by:

$$
A(p,q)=\operatorname{softmax}_q\!\left(\frac{s(p,q)}{\tau}\right),
\qquad
Y(p)=\sum_q A(p,q)V(q).
$$

Two operating modes are described:

- **fixed:** atom centers, widths, and weights depend on layer/head parameters but not on token content;
- **steered:** the query token predicts bounded residual changes to the atom center, log-width, and mixture logits.

The steering path is deliberately residual:

$$
\mu(p)=\mu_0+\alpha\tanh(\Delta\mu(p)),
$$

with analogous bounded changes for $\log\sigma$ and mixture logits. The intended inductive bias is therefore a learned geometric routing field plus small content-dependent correction, not unrestricted pairwise content matching.

## Separable Application

Each two-dimensional atom factorizes:

$$
\mathcal N(\Delta_y,\Delta_x;\mu,\sigma)
=
\mathcal N(\Delta_y;\mu_y,\sigma_y)
\mathcal N(\Delta_x;\mu_x,\sigma_x).
$$

The released code uses this property to apply a vertical kernel and then a horizontal kernel to the value grid. For fixed atom count $R$ and square $N=HW$, the spatial term is $O(RN\sqrt{N}d)$ rather than $O(N^2d)$. The steered path also stores per-query one-dimensional kernels, so batch size, heads, atoms, grid width/height, memory traffic, and implementation quality still matter.

```mermaid
flowchart LR
    X[query token and coordinates] --> B[head-specific Gaussian base field]
    X --> S[bounded steering residuals]
    B --> G[separable vertical and horizontal kernels]
    S --> G
    V[value grid] --> G
    G --> M[mixture over a few atoms]
    M --> Y[mixed output token]
```

## Author-Reported Evidence

All rows below are transcribed from the official blog and figures and remain author-reported.

| Dataset / setup | Mixer | Parameters | Reported FLOPs | Top-1 accuracy | What is actually shown |
|---|---:|---:|---:|---:|---|
| CIFAR-100, matched 90-epoch toy ViT | SDPA, learning rate $2\times10^{-3}$ | not tabulated | not tabulated | 53.2% | Three-seed chart with an unlabelled numeric error bar. |
| CIFAR-100, matched 90-epoch toy ViT | SDPA, learning rate $10^{-3}$ | not tabulated | not tabulated | 54.3% | Best reported SDPA setting; still a weak small-data baseline. |
| CIFAR-100, matched 90-epoch toy ViT | SSOG fixed, axial / factorized | not tabulated | not tabulated | 70.4% / 70.0% | Three-seed chart; geometry alone supplies nearly all of the gain. |
| CIFAR-100, matched 90-epoch toy ViT | SSOG $+\mu\delta$, axial / factorized | not tabulated | not tabulated | 70.4% / 70.4% | Steering is effectively null at the chart's precision. The gap is 17.2 points against the author's main 53.2% SDPA row and 16.1 points against the better 54.3% row. |
| ImageNet-1k, $d=256$, 6 layers, 4 heads | SDPA | 3.66M | about 1.5G | 64.34% | Single run. |
| ImageNet-1k, $d=256$, 6 layers, 4 heads | fixed SSOG, axial / factorized | 2.88M | not separately tabulated | 63.21% / 63.39% | Single runs, roughly one point behind SDPA. |
| ImageNet-1k, $d=256$, 6 layers, 4 heads | factorized SSOG $+\mu\delta$ | 2.92M | not separately tabulated | $64.48\%\pm0.13$ | Standard deviation over three seeds. |
| ImageNet-1k, $d=256$, 6 layers, 4 heads | factorized SSOG $+\mu\delta+\sigma\delta+\lambda$ | 3.00M | about 1.0G | $65.28\%\pm0.02$ | Standard deviation over three seeds; +0.94 points over the single-run SDPA row. |
| ImageNet-1k, $d=384$, 12 layers, 6 heads | SDPA | 14.94M | about 6.3G | 71.84% | Scale-up comparison; no uncertainty reported. |
| ImageNet-1k, $d=384$, 12 layers, 6 heads | factorized SSOG $+\mu\delta+\sigma\delta+\lambda$ | 11.96M | about 4.4G | 72.02% | +0.18 points; about 19.9% fewer parameters and 30.2% fewer tabulated FLOPs. |

The blog reports matched 90-epoch training on one RTX 5090 and emphasizes faster early convergence. It also reports single-seed CIFAR ablations over atom count, steering components, offset radius, cold start, and layer start, plus qualitative plots of learned atom geometry, steering gates, and net attention-mass export.

The headline “about 20% fewer parameters and roughly 30% less compute” is consistent with both tabulated model scales. It remains a symbolic parameter/FLOP claim, not a released, reproducible end-to-end throughput or peak-memory benchmark.

## Resolution Transfer

The blog reports evaluating one $224^2$-trained, $d=384$ factorized SSOG checkpoint at larger resolutions after bilinear resizing of the learned positional embedding. Reported top-1 is 72.0% at $224^2$, 73.7% at $288^2$, 71.6% at $384^2$, and 63.4% at $512^2$.

This supports the narrow claim that a coordinate-parameterized SSOG field can be evaluated on a larger grid if the rest of the model, especially the position embedding, is adapted correctly. The improvement is not monotone and reverses beyond $288^2$; the blog provides no matched SDPA resolution-transfer curve, and the released code does not implement the required position-embedding resize.

## Public-Code Audit

The pinned repository contains `SSOGAttention`, a dot-product baseline, a minimal ViT, and one CIFAR-100 training script. Same-resolution forward passes succeeded for both mixers.

Material gaps and drift:

1. The repository has no ImageNet pipeline, checkpoints, raw logs, multi-seed records, figure-generation scripts, tests, CI, tags/releases, or detected license.
2. The CIFAR entrypoint defaults to 100 epochs, while the blog describes a matched 90-epoch recipe.
3. The public ViT does not implement the positional-embedding resize needed for resolution transfer. Initializing at `8×8` and applying at `12×12` raised `ScopeParamShapeError` for both SSOG and dot-product models because the stored position embedding had shape `(1,16,32)` while the larger input required `(1,36,32)`.
4. The blog equation and released factorized operator are not generally identical. The code normalizes every atom's vertical and horizontal kernels separately, applies them to values, and only then mixes the atom outputs with $\lambda$. Finite-grid boundary normalization and non-unit temperature make this differ from one global softmax over a log-sum-exp mixture. A deterministic `3×3`, two-atom check produced a maximum absolute attention-weight difference of `0.29178012726036207` for a non-degenerate parameter setting while both rows remained normalized.

These findings are documented in `code_artifact_audit.md` and backed by the retained audit scripts and outputs.

## Prior-Art Boundary

SSOG's individual ingredients have substantial precedent:

- [Synthesizer (ICML 2021)](https://proceedings.mlr.press/v139/tay21a.html) learned or fixed synthesized attention matrices without query-key dot products.
- [Attention Free Transformer (2021 preprint)](https://arxiv.org/abs/2105.14103) used learned positional biases, locality, and convolutional sharing without ordinary dot-product attention.
- [LambdaNetworks (ICLR 2021 Spotlight)](https://openreview.net/forum?id=xTJEN-ggl1b) replaced pairwise attention with linear transformations of structured context.
- [DRAW (ICML 2015)](https://proceedings.mlr.press/v37/gregor15.html) used recurrent-state-conditioned, differentiable Gaussian filterbanks for selective image read/write; its two-axis filterbank is an especially close precedent for content-controlled separable Gaussian geometry outside ViT token mixing.
- [Spatial Transformer Networks (NeurIPS 2015)](https://papers.nips.cc/paper/5854-spatial-transformer-networks) made feature-map spatial transformation conditional on the feature map itself, while [Deformable ConvNets (ICCV 2017)](https://openaccess.thecvf.com/content_iccv_2017/html/Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.html) learned task-conditioned offsets for sampling locations.
- [Peripheral Vision Transformer (NeurIPS 2022)](https://proceedings.neurips.cc/paper_files/paper/2022/hash/cf78a15772ec1a6aee9bbee2d2b382c3-Abstract-Conference.html) combined position-based and content-based visual attention and explicitly studied Gaussian-like learned spatial regions.
- [Vision Transformers Provably Learn Spatial Structure (NeurIPS 2022)](https://proceedings.neurips.cc/paper_files/paper/2022/hash/f69707de866eb0805683d3521756b73f-Abstract-Conference.html) showed a position-only attention mechanism can be competitive on several vision benchmarks.
- [Gaussian Attention Bias (BMVC 2023)](https://proceedings.bmvc2023.org/214) explicitly injected a Gaussian attention bias into ViTs.
- [GAFlow (ICCV 2023)](https://openaccess.thecvf.com/content/ICCV2023/html/Luo_GAFlow_Incorporating_Gaussian_Attention_into_Optical_Flow_ICCV_2023_paper.html) combined Gaussian-constrained and deformable Gaussian-guided attention for optical flow.
- [Locality-Attending Vision Transformer (ICLR 2026)](https://arxiv.org/abs/2603.04892) adds a query-specific Gaussian locality term to ordinary ViT attention.
- [Deformable DETR (ICLR 2021)](https://openreview.net/forum?id=gZ9hCDWe6ke), [DAT (CVPR 2022)](https://openaccess.thecvf.com/content/CVPR2022/html/Xia_Vision_Transformer_With_Deformable_Attention_CVPR_2022_paper.html), and earlier deformable sampling/filtering work establish content-dependent spatial offsets.

The plausible novelty is not “Gaussian attention” or “steering” alone. It is the specific combination of a small mixture of head-level geometric priors, bounded query-conditioned residual steering, removal of query-key pair scores, and separable value aggregation as a drop-in ViT mixer. A formal paper must position this combination against the closest structured/Gaussian/deformable alternatives under matched training and kernels.

## Local Interpretation

SSOG's durable research question is broader than its present benchmark: **how much pairwise content matching is actually needed when the domain already supplies a strong coordinate system?** The prototype suggests an intermediate point between static convolution and unrestricted self-attention: a small learned geometric field carries most routing, while content only steers it locally.

For multivariate time-series and world models, the transfer is adjacent rather than direct. Time has a coordinate system, but many dependencies are not spatially local or stationary. A useful temporal analogue would need asymmetric/causal atoms, multi-scale lag fields, irregular timestamp support, event-conditioned steering, and separate channel/topology routing. It must preserve delayed effects, rare regimes, exogenous variables, action history, control inputs, and interventions rather than assuming all useful memory is Gaussian in lag.

## Limitations

- No peer review, formal paper, independent reproduction, or full bibliography.
- One toy CIFAR comparison and two ImageNet model scales do not establish a general ViT replacement.
- Standard deviations are reported only for selected SSOG rows; complete run records, uncertainty for the ImageNet SDPA and other single-run rows, hyperparameter-search accounting, and a matched strong modern backbone suite are absent.
- Classification-only evidence does not test detection, segmentation, tracking, video, or long-range correspondence where content-based matching may matter more.
- The strongest CIFAR gain may reflect a weak or poorly regularized small-data dot-product baseline; the blog does not compare against convolutional, local/windowed, positional-only, AFT/Synthesizer, or Gaussian/deformable alternatives.
- Theoretical FLOPs do not establish wall-clock speed, peak memory, throughput, or hardware efficiency without optimized kernels and matched measurement protocol.
- The public code does not reproduce the ImageNet or resolution-transfer claims and has no license.
- The factorized code path and displayed mixture-softmax equation are materially different operators.
- Interpretability claims are based on parameter plots, not causal interventions or faithfulness tests.
- Time-series/world-model relevance is a testable architecture hypothesis, not evidence from this source.