VISReg: Variance-Invariance-Sketching Regularization for JEPA training

Source

• Raw Markdown: paper_visreg-2026.md
• PDF: paper_visreg-2026.pdf
• Preprint: https://arxiv.org/abs/2606.02572
• DOI: https://doi.org/10.48550/arXiv.2606.02572
• Official project page: https://haiyuwu.github.io/visreg/
• Official code: https://github.com/HaiyuWu/visreg
• Official Hugging Face checkpoints: https://huggingface.co/BooBooWu/visreg
• Official X post by Haiyu Wu: https://x.com/HaiyuWu1/status/2070866534462087626
• X discussion by co-author Randall Balestriero: https://x.com/randall_balestr/status/2072088917348630648

Auxiliary snapshots are preserved under papers/visreg-2026/: github_readme_snapshot.md, project_page_snapshot.md, huggingface_model_card_snapshot.md, x_post_haiyuwu1_2070866534462087626.*, x_post_randall_balestr_2072088917348630648.*, and research-context-links.md.

Status And Credibility

This is an arXiv v1 preprint submitted on 2026-06-01 by Haiyu Wu, Randall Balestriero, and Morgan Levine. The authors list Altos Labs and Brown University affiliations. The paper has official code, project page, and Hugging Face checkpoint artifacts; the repository states that code and pretrained weights are released under CC BY-NC 4.0.

Credibility signal: the work is current, by a credible SSL/JEPAs team that includes Randall Balestriero, and ships official code plus ImageNet-pretrained ViT-B/16 and ViT-L/14 checkpoints. Caveat: the arXiv page does not report peer-reviewed venue acceptance; treat claims as preprint evidence until a venue or independent replication appears.

Core Claim

VISReg argues that SIGReg-style distribution sketching can scale for JEPA-like self-supervised vision training, but that SIGReg itself has a collapse-stage weakness: the Epps-Pulley sketching gradient can vanish when embeddings collapse. VISReg keeps the LeJEPA/SIGReg goal of an isotropic Gaussian-like embedding distribution, but decouples scale and shape:

• a VICReg-like variance term provides a strong corrective gradient when variance collapses;
• a Sliced-Wasserstein shape term matches normalized random 1D projections to standard Gaussian quantiles;
• a center term stabilizes the embedding mean;
• the invariance/prediction term follows the LeJEPA multi-view JEPA recipe.

The source claims this combination keeps the heuristic-free regularization story while improving collapse recovery, low-quality-data robustness, distributed scaling, and OOD visual transfer.

Method Contract

VISReg is a visual self-supervised learning regularizer used with a JEPA-style multi-view invariance loss. For embeddings $Z\in {\mathbb{R}}^{N\times D}$, the regularization part is:

$${\mathcal{L}}_{\mathrm{Reg}}={\lambda }_{\mathrm{scale}}{\mathcal{L}}_{\mathrm{scale}}+{\lambda }_{\mathrm{shape}}{\mathcal{L}}_{\mathrm{shape}}+{\lambda }_{\mathrm{center}}{\mathcal{L}}_{\mathrm{center}}.$$

The scale loss constrains per-dimension standard deviation:

$${\mathcal{L}}_{\mathrm{scale}}=\frac{1}{D}\sum _{j=1}^D(1-{\sigma }_j(\hat{Z}){)}^2.$$

The shape loss normalizes scale with a stop-gradient standard deviation, projects normalized embeddings onto $K$ random directions, sorts each projected sample, and matches those order statistics to Gaussian quantiles:

$${\mathcal{L}}_{\mathrm{shape}}=\frac{1}{K}\sum _{k=1}^K{\Vert \mathrm{sort}(\tilde{Z}{w}_k)-q_{\mathcal{N}}\Vert }_2^2.$$

flowchart LR
  X[augmented image views] --> E[ViT encoder + projector]
  E --> Z[embedding batch]
  Z --> Inv[LeJEPA-style invariance / prediction loss]
  Z --> Scale[variance scale loss]
  Z --> Norm[normalize with stop-grad std]
  Norm --> Proj[random 1D projections]
  Proj --> Sort[sort projected samples]
  Sort --> Shape[SWD shape loss to Gaussian quantiles]
  Z --> Center[center loss]
  Inv --> Total[VISReg objective]
  Scale --> Total
  Shape --> Total
  Center --> Total

Important boundary: VISReg still uses image augmentations and multi-view invariance. Its “heuristic-free” claim concerns avoiding EMA, teacher-student asymmetry, stop-gradient targets as collapse-prevention heuristics, frozen layers, and similar stabilization machinery. The shape term does use sg(std) for objective decomposition, so wiki summaries should not overstate this as literally no stop-gradient anywhere.

Evidence And Results

• Collapse-gradient simulation: the paper reports that VISReg keeps a strong gradient under embedding collapse, while SIGReg’s gradient vanishes in the collapse-stage simulation.
• Scaling analysis: the regularizer is framed as $O(NDK)$ plus sorting, with random slices distributable across GPUs. The paper reports that multi-GPU slice distribution closes the accuracy gap from using fewer slices per GPU.
• Low-quality datasets: on ImageNet-LT and Galaxy10, VISReg, SIGReg, SWD, and VICReg avoid complete collapse better than DINO in the paper’s from-scratch stress tests; VISReg with stronger shape weighting is best in the long-tailed ImageNet-LT table.
• ImageNet-1K OOD evaluation: ImageNet-1K-pretrained VISReg reports the best average OOD linear-probe accuracy among the compared methods on DTD, Galaxy10, AID, ChestXRay, RetinaMNIST, and OrganAMNIST.
• ImageNet-22K data efficiency claim: VISReg ViT-L/14 pretrained on ImageNet-22K is reported to match DINOv2-LVD142M average OOD performance despite DINOv2 using the much larger LVD-142M data source.
• Transfer and generation: the source reports stronger fine-tuning transfer than DINO in its selected datasets and useful generation guidance for SiT-B/2 under an iREPA-style setup.

Social Source Notes

Haiyu Wu’s X post frames VISReg as relevant to “world model or SSL” work, emphasizing strong collapse prevention, scaling friendliness, LeJEPA-like heuristic-free training, OOD performance, data efficiency, and robustness to long-tailed/sparse data. The same post says the results indicate that SIGReg-type methods can scale up.

Randall Balestriero’s quoted X post is more directly relevant to this wiki’s SIGReg thread: he asks whether regularization-based JEPA such as SIGReg can scale and compete with DINO, answers yes, and describes VISReg as a slight variation of SIGReg that competes with DINOv2-LVD142M while training only on ImageNet-22K.

These posts are useful author/co-author framing, but the paper remains the source of truth. The posts’ “world model” language should be treated as motivation and ecosystem positioning, not as evidence that VISReg itself learns action-conditioned world models.

Relation To LeNEPA And SIGReg

VISReg is close to LeNEPA because both sit on the LeJEPA/SIGReg branch of anti-collapse regularization:

• LeNEPA uses temporal SIGReg to stabilize no-augmentation next-latent-token prediction for time-series representation learning.
• VISReg argues that a SIGReg-like sketching prior can scale in visual SSL, but that SIGReg’s monolithic shape signal can be weak at collapse; it adds a separate variance/scale term plus SWD shape matching.

For future LeNEPA work, VISReg suggests a concrete ablation: compare temporal SIGReg against a temporal VISReg-style regularizer that decouples token-set scale, shape, and center losses. The key question is not only whether the representation avoids collapse, but whether the stronger scale signal preserves dense numeric state, rare regimes, channel relationships, event timing, and action/control-input histories.

Limitations And Gotchas

• VISReg is vision SSL evidence, not direct time-series evidence. Its ImageNet/OOD image results should be transferred to LeNEPA only as a regularizer-design hypothesis.
• The method’s OOD evidence is classification-heavy. It is not evidence for numeric forecasting, latent-state accessibility, intervention response, or counterfactual rollout.
• The “10x less data” comparison is against DINOv2-LVD142M on selected OOD metrics; it should not be generalized to all foundation-model data/compute regimes without matched training and evaluation budgets.
• VISReg still depends on projector dimensions, slice counts, batch statistics, augmentation views, and loss weights. These are part of the objective contract, not implementation trivia.
• The non-commercial CC BY-NC 4.0 license on code and weights matters for artifact reuse.
• A stronger anti-collapse gradient can be a double-edged sword for time series: it may rescue collapsed dimensions, but it may also impose a Gaussian-like prior that distorts naturally long-tailed, bounded, periodic, or policy-shaped latent states.

Foundation TSFM Relevance

Agenda slot	Verdict	Evidence	Missing pieces
Anti-collapse regularization	adjacent	Provides current vision evidence that a SIGReg-family regularizer can be made more robust under collapse by decoupling scale and shape.	Needs temporal token-set tests, dense-state probes, and matched LeNEPA ablations.
Data diversity, curriculum, and long tail	adjacent	Tests long-tailed ImageNet-LT and low-rank Galaxy10 and reports stronger robustness than DINO in those stress tests.	Needs time-series rare-regime, intervention-window, and useful-signal-poor corpora rather than image-classification proxies.
Representation quality	warning	OOD and transfer results suggest useful visual features, but aggregate classification can hide which state factors are preserved.	Need probes for dense numeric values, event timing, multivariate channel relationships, exogenous variables, and action histories.
Latent-state prediction	insufficient evidence	VISReg regularizes representations; it is not itself a next-state or next-latent time-series objective.	Combine with LeNEPA/NEPA-style targets and test whether state prediction improves.
Control and counterfactuals	insufficient evidence	No action, control-input, intervention, or candidate-action rollout interface is evaluated.	Add action-conditioned or intervention-conditioned benchmarks before making world-model claims.

Links Into The Wiki

• VISReg
• LeNEPA
• LeNEPA idea
• LeJEPA
• Joint Embedding Predictive Architecture
• Distribution Priors In Self-Supervised Learning
• Representation Collapse
• Next-Embedding Prediction
• Foundation Time-Series Model Research Agenda

Open Questions

• Does a temporal VISReg-style scale/shape/center regularizer outperform temporal SIGReg in LeNEPA under matched compute and target-family choices?
• Does VISReg’s stronger collapse-stage gradient preserve rare regimes and dense numeric detail, or does it simply force more active dimensions under a mismatched Gaussian-like prior?
• How should slice count, projection dimension, and batch construction scale when the batch axis is temporal tokens, multivariate channels, patients/users/assets, or mixed-domain windows?
• Can VISReg-style regularization combine safely with own-hidden-state targets such as NextLat, or does it over-regularize internal belief states?
• Which evaluation protocol separates regularizer scaling from backbone scale, data scale, augmentation recipe, and layer-selection budget?