Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Source

• Raw Markdown: paper_nape-2026.md
• PDF: paper_nape-2026.pdf
• arXiv: https://arxiv.org/abs/2608.19863
• DOI: https://doi.org/10.48550/arXiv.2608.19863
• Official project page: https://umbertocappellazzo.github.io/nape
• Official code: https://github.com/umbertocappellazzo/nape
• Audited code commit: https://github.com/umbertocappellazzo/nape/commit/42ff6c0ef48f196a5122fbcd5d44b87b101c46be
• Discovery X post: https://x.com/Umberto_Senpai/status/2090855967881679257

Status And Credibility

This is a fresh arXiv v1 preprint submitted on 2026-08-20 by Umberto Cappellazzo, Xubo Liu, Stavros Petridis, and Maja Pantic. The official project page identifies Imperial College London and the University of Surrey as the affiliations. Umberto Cappellazzo’s publication page labels the work under review; no accepted venue is claimed as of the 2026-08-23 ingest.

The evidence package is stronger than a social announcement alone: the arXiv source and PDF are public, the authors maintain an official project page, the code repository is public, and three checkpoint endpoints were reachable. The team has an established audio/speech research record, but this specific result is only three days old, has not been peer reviewed, and has no independent replication. Treat it as a credible, artifact-rich preprint rather than settled state of the art.

Core Claim

NAPE asks whether audio self-supervised learning can use the same basic interface as causal language modeling without reconstructing acoustic inputs or discretizing them into tokens. It turns a log-mel spectrogram into patch embeddings, linearizes the time-frequency grid, and predicts each next patch embedding from the visible prefix.

The main claim is narrow but useful: causal next-patch-embedding prediction with target stop-gradient can train competitive audio encoders with a much simpler pre-training objective than decoder-, tokenizer-, teacher-student-, or auxiliary-regularizer-heavy recipes. The strongest support is the combination of mechanism ablations, monotonic downstream gains from Small to Large, and competitive fine-tuning across six audio/speech benchmarks.

Method Contract

Let $x$ be a log-mel spectrogram, $f$ the patch embedding layer, $\pi$ a scanning order over the 2D time-frequency patch grid, $h$ a causal Transformer, and $g$ a predictor head. NAPE constructs

$$z_t=f\left(x_{\pi (t)}\right),{\hat{z}}_{t+1}=g\left(h(z_{\le t})\right),$$

and minimizes negative cosine similarity to a detached next-patch target:

$${\mathcal{L}}_{\mathrm{NAPE}}=-\frac{1}{N-1}\sum _{t=1}^{N-1}\frac{\mathrm{sg}(z_{t+1})}{\Vert z_{t+1}{\Vert }_2}\cdot \frac{{\hat{z}}_{t+1}}{\Vert {\hat{z}}_{t+1}{\Vert }_2}.$$

sg denotes stop-gradient. The causal mask prevents access to later sequence positions, and the one-position prediction shift prevents a trivial copy of the current patch. The default target is the output of the independent Conv2d patch embedding layer, not a contextual target-encoder state.

flowchart LR
  X[log-mel spectrogram] --> P[non-overlapping patches]
  P --> F[patch embedding f]
  F --> O[scan time-frequency grid]
  O --> H[causal Transformer h]
  H --> G[predictor g]
  G --> ZH[predicted next embedding]
  O --> SG[stop-gradient next patch embedding]
  ZH --> C[negative cosine similarity]
  SG --> C

Scanning order is part of the causal model

The spectrogram uses 128 mel bands and, after cropping, 1008 time frames. With $16\times 16$ patches, it becomes an $8\times 63$ grid, or 504 tokens. The paper evaluates four ways to convert that grid to a sequence:

• Raster: advance in time across one frequency row, then move to the next frequency band.
• Time-major: traverse frequencies within a time column, then advance in time.
• Zigzag: alternate raster direction between adjacent frequency rows.
• Diagonal: traverse anti-diagonals that mix time and frequency progression.

Raster and diagonal perform best in the reported ablation; time-major is weakest. This is a useful causal-design result: once attention is masked, linearization is not a harmless implementation detail because it defines which time-frequency patches count as past context.

Encoder and downstream interface

The encoder is a pre-norm Vision Transformer with causal self-attention during pre-training, 2D RoPE, LayerScale, and query-key normalization. The released sizes are approximately 19M, 85M, and 303M parameters. Fine-tuning removes the causal mask by default and uses mean pooling, LayerNorm, and a classification head. Linear probing freezes the backbone and selects a useful intermediate layer rather than assuming the final predictive layer is the best representation.

Evidence

Mechanism ablations

The controlled AudioSet-20K ablation supports the need for all three core constraints in the tested recipe:

Variant	Validation mAP	Test mAP	Interpretation
Full NAPE	27.9	31.3	causal mask + next-position shift + target stop-gradient
No prediction shift	0.0	0.5	current-position identity target destroys the learning signal
No stop-gradient	18.9	22.9	substantial degradation, though not literal zero performance
No causal mask	19.3	22.5	target leakage weakens the predictive task
No causal mask + 6-layer causal predictor	25.7	29.0	a stronger predictor recovers some, not all, of the gap

The target ablation is equally important for the wiki’s target-construction thread:

Target	Validation mAP	Test mAP
Independent patch embedding	27.9	31.3
Raw log-mel patch	16.7	20.8
Encoder-layer representation	13.1	17.2

The paper attributes the deep-target failure to collapse when both target and prediction depend on the same encoder. The public implementation repeats that warning. This evidence supports a conditional claim: stop-gradient is effective for the reported shallow patch-embedding target, not evidence that target detachment alone stabilizes every contextual or own-hidden target.

Scaling across the released family

All three models are pre-trained on AudioSet-2M. Fine-tuning scores increase monotonically with model size in the paper’s raster-scan comparison:

Model	Parameters	AS-2M mAP	AS-20K mAP	ESC-50 acc.	KS1 acc.	KS2 acc.	IEMOCAP acc.
NAPE-S	19M	42.6	33.4	91.6	96.3	97.9	60.5
NAPE-B	85M	49.4	39.0	94.6	97.5	98.6	67.1
NAPE-L	303M	50.2	40.5	96.0	97.9	98.8	68.0

This is favorable within-family scaling across three points, not a fitted data/parameter/compute scaling law or a matched-compute comparison. Model sizes, checkpoint naming, and effective training cost must be reported separately in any reproduction.

Competitive performance, with a narrower SOTA reading

The authors’ comparison table supports a strong but mixed leaderboard result:

• NAPE-L reaches 68.0% on IEMOCAP, 3.5 points above the strongest listed SSL baseline with an IEMOCAP result.
• It reaches 50.2 mAP on AudioSet-2M, tying the best listed score.
• It is below the best listed result on AudioSet-20K, ESC-50, Speech Commands V1, and Speech Commands V2.
• NAPE-B diagonal is competitive with similarly sized models but does not dominate the whole table.

Accordingly, the X-post shorthand SOTA performance should not be read as best on every benchmark. The defensible claim is state-of-the-art or tied performance on specific reported tasks, with broadly competitive transfer elsewhere.

Frozen features

Linear-probe performance also improves from Small to Large on every reported dataset, and the best probe usually comes from the middle of the encoder rather than its top. That supports the claim that useful representations survive strict feature freezing, while also warning that the final layer is specialized for next-embedding prediction.

What The Minimalism Claim Does And Does Not Cover

NAPE’s pre-training objective uses no reconstruction decoder, acoustic tokenizer, EMA teacher, contrastive negatives, or auxiliary distribution regularizer. That is a real simplification.

It is not a completely heuristic-free end-to-end benchmark pipeline. The encoder still uses a predictor head, stop-gradient, causal scanning order, RoPE, LayerScale, query-key normalization, model-specific training schedules, and downstream adaptation. Fine-tuning uses task-dependent augmentation and applies EMA for AudioSet. The correct comparison is therefore objective and architecture complexity during pre-training, not “no training heuristics anywhere.”

Code And Checkpoint Audit

The public code at commit 42ff6c0ef48f196a5122fbcd5d44b87b101c46be implements the default causal mask, one-step shift, detached patch-embedding target, and negative cosine loss. Public directory indexes expose NAPE-S/B/L model.safetensors files.

Reproduction is not turnkey at the audited commit:

• no release tags, tests, or CI are present;
• the README’s pre-training filenames and config paths differ from the tree;
• committed pre-training launchers call missing run_audio_nape.py instead of the present run_nape.py;
• the README license badge says MIT while the actual license is Apache-2.0;
• the Hugging Face link is a placeholder;
• checkpoint directory names and the paper/scripts disagree about whether Small/Large are 25- or 30-epoch artifacts;
• AudioSet is not redistributed and must be supplied separately.

These are artifact-maturity caveats, not evidence that the reported scores are wrong. They do mean that “all open-sourced” is not equivalent to an independently reproduced, one-command release.

Limitations

• The work is an under-review arXiv v1 preprint with no independent replication.
• All pre-training is on AudioSet; cross-corpus pre-training transfer and data-scaling behavior are untested.
• The inputs are fixed-length, regularly sampled log-mel grids. Irregular event streams, variable-rate sensors, missing channels, and native multivariate numeric features are outside scope.
• The scaling claim uses only three encoder sizes and does not fit a scaling law or compare matched training FLOPs, wall-clock time, or serving cost.
• Benchmark tables compare methods with different objectives, training budgets, augmentations, and sometimes missing task results. They are not a fully compute-matched study.
• The paper does not report multi-seed uncertainty for the headline scaling and transfer table.
• Attention maps and embedding-similarity visualizations are qualitative evidence; they do not prove causal use of acoustic structure.
• Collapse is inferred primarily through learning curves and downstream performance. Rank, variance, covariance, eigenspectrum, and rare-state preservation diagnostics are not the main evidence.
• NAPE is passive representation learning. It has no action, control input, intervention, treatment, reward, or counterfactual rollout interface.

Relevance To Time-Series Modeling And World Models

NAPE is audio evidence, not a multivariate time-series foundation model or an action-conditioned world model. Its strongest transferable lessons are architectural and diagnostic:

1. Causal linearization is an inductive bias. A time-channel patch grid cannot be flattened arbitrarily once causal masking is applied. Time-major, channel-major, diagonal, and event-aware orders define different visible histories.
2. Shallow target embeddings are a distinct target family. NAPE works best when the target is an independent patch embedding; raw observations and contextual encoder targets are much worse. This directly informs Next-Embedding Prediction and LeNEPA target-family ablations.
3. Stop-gradient evidence is conditional. It is sufficient in this shallow-target audio recipe, while LeNEPA shows a no-stop-gradient temporal-SIGReg alternative in a time-series setting. Neither result establishes a universal anti-collapse mechanism.
4. Intermediate layers may carry the reusable state. A predictive objective can make the final layer task-specialized even when middle layers remain broadly discriminative.
5. Prediction is not yet a world model. To become relevant for planning, the interface would need typed actions/control inputs/interventions, uncertainty over future latent states, and probes for event timing, dense numeric fidelity, rare regimes, and counterfactual consequences.

Links Into The Wiki

• NAPE
• Next-Embedding Prediction
• Latent-Space Predictive Learning
• Self-Supervised Representation Learning
• Representation Collapse
• JEPA
• LeNEPA
• LeNEPA source
• Intermediate-Layer Representations
• Dynamic Curriculum Learning For JEPA

Open Questions

• Do raster/diagonal gains survive matched experiments on multivariate time-series channel-time grids, or are they specific to spectrogram geometry?
• Can temporal SIGReg or VISReg replace target stop-gradient for NAPE’s shallow target without hurting audio transfer?
• Can a contextual target be stabilized without the severe degradation reported for the encoder-layer target?
• Does the mid-layer linear-probe advantage persist under dense event-timing, localization, and rare-sound probes rather than clip-level classification?
• How much of the gain comes from the causal objective versus RoPE, LayerScale, query-key normalization, and the SimSiam predictor under matched compute?
• Can the released artifacts reproduce the paper after launcher/path inconsistencies are corrected, and what are the actual Small/Large checkpoint training epochs?
• What action- or intervention-conditioned version would make next-audio-patch prediction useful for interactive acoustic world modeling rather than passive representation learning?