LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

Sign up for free to create and run prompts on this paper using GPT-5.

A simple explanation of “LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives”

What this paper is about (overview)

This paper is about teaching computers to understand images and text together. Most popular methods today (like CLIP) do this by comparing which image goes with which caption and which ones don’t—using “negatives” (wrong pairs) to push them apart. The authors introduce a new way, called LeVLJEPA, that learns from image–caption pairs without using any negatives. Even better, it creates image features that are especially useful for tasks that look at every part of an image, not just the whole picture at once.

What questions the paper asks

The researchers focus on four simple questions:

• Can we train an image–text model without using “negative” examples?
• Will this training be simple and stable, even at large scale?
• Does this give better “dense” features—information at every patch or region of the image—that modern systems actually use?
• How does it compare to standard methods on common tests like zero-shot classification and segmentation?

How the method works, in everyday terms

Two-way prediction instead of “finding the wrong pair”

Imagine you have an image and its caption. Instead of scoring “this image matches this caption and doesn’t match those other captions,” LeVLJEPA does something else:

• The image side learns to predict the text side’s representation.
• The text side learns to predict the image side’s representation.

Think of it like two students solving the same riddle from different clues, then checking if their answers line up.

“Stop-gradient”: freeze the target so training doesn’t go in circles

When the image tries to predict the text, the text representation is treated like a frozen snapshot—it doesn’t get updated from that prediction step. This “stop-gradient” is like saying “compare your answer to the teacher’s answer, but don’t change the teacher’s answer while you’re learning.” This prevents both sides from collapsing to the same meaningless output.

Avoiding “collapse” with SIGReg: keeping the feature shape healthy

“Collapse” is when the model starts producing almost the same representation for everything—like giving every image nearly the same vector. To avoid this, LeVLJEPA uses a regularizer called SIGReg. In plain terms:

• Picture all image features as dots in space. SIGReg nudges these dots to form a round, evenly spread cloud, not a squished line or tiny blob.
• It does this by checking the shape along many random directions and encouraging a balanced spread.

This keeps features informative and different from each other.

What’s not needed anymore

Because of the prediction + stop-gradient + SIGReg recipe, LeVLJEPA:

• Doesn’t need negatives (no “push away the wrong pairs”).
• Doesn’t need tricky temperature settings.
• Doesn’t need a “momentum” or teacher network. It’s a simpler training recipe.

What data and models they used

• Images and captions from mid-size (CC12M, ~12 million pairs) and larger (DataComp-L, ~92 million pairs) datasets.
• A Vision Transformer (ViT-B/16) for images and a GPT-2–style model for text.
• They especially care about “patch tokens”—small tile-like pieces of the image (a 14×14 grid) that together represent the whole picture. Many modern systems use these dense tokens directly instead of a single pooled “whole-image” vector.

How they tested it

The authors look at two kinds of tests:

1. Global features (single-vector summary of the image)

• Zero-shot classification: Can the model pick the correct class by comparing the image’s vector to text prompts like “a photo of a dog” without extra training?
• Linear probing: Train a simple linear classifier on top of the frozen image vector to see how separable the classes are.

2. Dense features (information at every image patch)

• Semantic segmentation: Predict a label for each image patch (e.g., sky, road, person) using only a tiny linear layer on the frozen patch tokens.
• As a frozen backbone for a vision–LLM (VLM): Keep both the image encoder and a LLM frozen, and train only a small connector (bridge) that feeds patch tokens into the LLM. Then evaluate on:
  • GQA (reasoning about objects and relations),
  • VQAv2 (general visual question answering),
  • POPE (checking for object hallucinations—saying things are in the image when they aren’t).

They also check “background robustness”—whether predictions rely too much on the background instead of the actual object.

Main findings (what they discovered and why it matters)

1) Zero-shot vs. linear probe (global features)

• Zero-shot: At large scale, LeVLJEPA is a bit worse than contrastive methods, which is expected because zero-shot directly rewards contrastive-style alignment with negatives.
• Linear probe: LeVLJEPA is on par with contrastive methods. That means its “whole-image” features are just as informative when you don’t rely on text alignment.

Why this matters: If you only look at zero-shot, you might think LeVLJEPA is weaker. But linear probing shows the global visual quality is similar.

2) Better background robustness

LeVLJEPA is less fooled by background changes. Its accuracy drops less when the object is placed on different backgrounds. This suggests it focuses more on the object itself, not background “hints.”

3) Stronger dense features (the big win)

• Semantic segmentation with just a linear layer on frozen patch tokens: LeVLJEPA beats contrastive methods by a clear margin. This means its patch-by-patch features carry richer, more usable information.
• As a frozen visual backbone for a VLM: With two different frozen LLMs (Llama-1B and Qwen-1.5B), LeVLJEPA leads on GQA, VQAv2, and POPE. It also shows less bias toward always answering “yes” in POPE, reducing hallucinations.

Why this matters: Many modern systems don’t use a single image vector; they look at all the patches. That’s exactly where LeVLJEPA shines. So even if it’s not best at zero-shot, it’s better for the way these models are actually used today.

4) Stable and simpler training, and not tied to huge batches

• Contrastive methods often need very large batch sizes to find many good “negatives.” LeVLJEPA doesn’t depend on that; it works similarly across different batch sizes.
• With fewer moving parts (no negatives, no momentum teacher), training is simpler.

So what’s the takeaway?

• If you want an image–text model to be a good “global label matcher” without extra training, contrastive methods still have an edge in zero-shot classification.
• But if you care about what most modern multimodal systems actually do—reading many image patches and using the image encoder as a frozen backbone—LeVLJEPA’s non-contrastive training gives you stronger, denser, more reliable features.
• It’s simpler to train and more robust to background changes.

Looking ahead

The authors suggest combining the strengths: keeping LeVLJEPA’s strong dense features while closing the remaining gap in zero-shot alignment. They also encourage the community to evaluate models more on dense, patch-level tasks, since that’s how these models are increasingly used in practice.

Knowledge gaps, limitations, and open questions

Below is a consolidated list of concrete gaps and open questions that remain unresolved by the paper, phrased to guide follow-up work.

Scaling and generality

• Model scale remains untested beyond ViT-B/16 and a GPT-2 text encoder; it is unknown whether the dense-feature advantages persist (or grow) with larger and different backbones (e.g., ViT-L/H, Swin, ConvNeXt) and stronger/modern text encoders (e.g., T5, LLaMA, Qwen, multilingual encoders).
• Data scale is limited to CC12M and a link-rot-reduced DataComp-L (~92M); behavior on larger/higher-quality or more diverse corpora (e.g., LAION-400M/2B, DataComp-1B, curated captions) is untested.
• Domain and language generalization are unassessed: no evaluations in specialized domains (medical, satellite, scientific) or multilingual settings; robustness when captions are noisy, short, or misaligned remains unclear.
• Patch size, input resolution, and positional encoding choices (e.g., ViT-B/32 vs B/16 vs B/14) were not varied; their impact on dense token quality is unknown.

Objective design and training mechanics

• Zero-shot alignment remains weaker at scale; it is unresolved how to combine LeVLJEPA’s dense-feature benefits with competitive alignment (e.g., auxiliary contrastive term, cosine regression, post-hoc alignment, retrieval-aware fine-tuning) without reintroducing batch-size dependence.
• The reliance on a projection MLP to circumvent LayerNorm’s interference with SIGReg suggests architectural sensitivity; it is unknown how robust SIGReg is across normalization schemes (pre-/post-LN transformers, RMSNorm), other backbones, or convnets.
• SIGReg hyperparameters and compute trade-offs are underexplored: number of random projections, weighting ($\lambda_v,\lambda_t$), and projection distributions vs. stability, throughput, and memory were not quantified or systematically ablated.
• Predictor design is narrow (MLPs in both directions); the effect of alternative predictors (cross-attention, residual adapters, tied/shared heads), single-direction vs. bi-directional prediction, or asymmetric weighting between modalities is unknown.
• Batch-size invariance was shown only on CC12M for B∈{1024, 2048, 4096}; behavior under very small batches, extreme large batches, gradient accumulation, and mixed precision regimes is untested.
• The loss uses MSE in latent space; it is unclear whether alternative objectives (cosine/Mahalanobis regression, whitening/whitened MSE, Huber, or contrast-free mutual information surrogates) improve alignment or dense features.
• Interaction between per-modality isotropy (SIGReg) and cross-modal alignment geometry is not analyzed; when and why isotropic marginals hinder/help zero-shot remains an open question.

Evaluation scope and baselines

• Dense-task coverage is limited to linear semantic segmentation; performance on other dense tasks (object detection, instance/panoptic segmentation, depth/normal estimation, keypoints, open-vocabulary detection/segmentation) is unknown.
• VLM-backbone results use frozen, small LLMs (Llama-1B, Qwen-1.5B) and an MLP bridge; it is unknown whether gains persist with larger LLMs, stronger bridges (e.g., Q-Former, cross-attention adapters), or full end-to-end instruction tuning (LLaVA/BLIP-2-style training).
• No comparison to vision-only SSL backbones (e.g., DINO, I-JEPA, LeJEPA) in the same frozen-dense protocols; it is unclear whether the VL objective adds value over strong vision-only pretraining for dense features.
• Image–text retrieval (ZS retrieval, Recall@K) and captioning are not reported; alignment quality outside zero-shot classification is unknown.
• Stronger or more diverse contrastive baselines (e.g., VL-JEPA with InfoNCE, EVA-CLIP, LiT, ALIGN, SLIP) and broader hyperparameter scans at scale are absent, limiting comparative conclusions.
• Limited dataset coverage in VQA-style evaluation (GQA, VQAv2, POPE only); generalization to more multimodal tasks (e.g., TextCaps, OK-VQA, ScienceQA, MM-Vet, MME, MMStar) is untested.

Robustness, bias, and interpretability

• Robustness beyond IN-9 background shifts is not assessed (e.g., ImageNet-C/-R, ObjectNet, ImageNet-A/O, occlusions, patch/dropout, adversarial or photometric corruptions); failure modes under natural distribution shifts remain unclear.
• Safety, bias, and fairness are unaddressed (e.g., demographic bias, stereotype amplification, toxicity propagation from web captions); no audits or mitigation strategies are presented.
• The claim of more “object-centric” features is supported by IN-9 and segmentation, but lacks deeper analyses (e.g., token anisotropy metrics, objectness scores, attention/saliency visualization, part/attribute localization, probing of spatial relations).
• Calibration and hallucination are only evaluated on POPE; broader hallucination and calibration diagnostics (e.g., CHAIR, annotation artifacts, answer distribution biases) are missing.

Efficiency, ablations, and reproducibility

• Training efficiency is not quantified (throughput, wall-clock, energy) vs. contrastive methods; the practical cost of SIGReg and predictors per sample is unknown.
• Data and text augmentation strategies (multi-view crops, caption perturbations, multi-caption per image, text dropout) are not explored, leaving potential gains untapped.
• Sensitivity to optimizer choice, learning-rate schedules, and weight decay at large scale is not reported; large-scale hyperparameters were not tuned to avoid overfitting, which may understate absolute performance and obscure best practices.
• Variance across runs is reported for a subset of experiments only; broader statistical variability (seed sensitivity, convergence stability across runs) remains to be characterized.

Overview

Based on the paper’s findings—non-contrastive, end-to-end vision–language pretraining (LeVLJEPA) that delivers stronger dense per-token visual features, better background robustness, and stable training without negatives or massive batch sizes—below are actionable applications across sectors. Each item notes potential tools/workflows and key dependencies or assumptions.

Immediate Applications

• Stronger frozen backbones for vision–LLMs (VLMs)
  • What: Swap CLIP-like backbones with LeVLJEPA in existing VLMs (e.g., LLaVA, MiniGPT-4). Train only a lightweight MLP “bridge” to the LLM (as in the paper), improving VQA accuracy and reducing hallucination bias.
  • Sectors: Software, education, customer support, e-commerce, media.
  • Tools/workflows:
  • “Dense-VQA bridge” module that maps patch tokens to LLM embeddings.
  • Drop-in Hugging Face checkpoints; bridging scripts for Llama/Qwen.
  • Assumptions/dependencies:
  • Zero-shot retrieval/classification may be weaker; favor applications where dense reasoning or spatial cues dominate.
  • LLM choice and prompt engineering remain important (shown robust across Llama-1B and Qwen-1.5B).
• Lightweight semantic segmentation for rapid bootstrapping
  • What: Train a single linear head over frozen LeVLJEPA patch tokens for quick semantic segmentation (better mIoU than contrastive baselines).
  • Sectors: Software (data annotation), robotics (scene parsing), retail (planogram compliance), agriculture (field segmentation), media (content editing).
  • Tools/workflows:
  • “Auto-annotation” pipelines that generate high-quality pseudo-labels to reduce labeling costs.
  • Plug-in segmentation heads for existing CV stacks (e.g., Detectron2, MMDetection).
  • Assumptions/dependencies:
  • For specialized domains, some domain adaptation or fine-tuning may be needed.
  • Evaluation shows linear heads on general data; extreme domain shift requires validation.
• Background-robust visual classifiers and QA systems
  • What: Deploy models that are less dependent on spurious background context, improving reliability in real-world imaging conditions.
  • Sectors: Autonomous systems, retail, manufacturing QC, public safety analytics.
  • Tools/workflows:
  • “Background-robustness audits” in MLOps CI pipelines using ImageNet-9-style checks.
  • Model selection criteria emphasizing token-level feature quality for deployment.
  • Assumptions/dependencies:
  • Gains are measured on broad benchmarks; validate in specific environments (e.g., factory floors, stores).
• Smaller-scale training and adaptation for labs and startups
  • What: Pretrain or adapt vision–language encoders without huge batch sizes or negative sampling, reducing hardware/infrastructure needs.
  • Sectors: Academia, startups, public sector labs.
  • Tools/workflows:
  • Batch-size-invariant training recipes; DDP-friendly training without momentum encoders/teachers.
  • Fine-tuning/adaptation scripts using CC12M-like corpora or domain-specific paired data.
  • Assumptions/dependencies:
  • Access to sufficiently large paired image–text data (or synthetic pairs) remains necessary.
  • For commercial deployments, verify data licensing/compliance.
• Visual assistants for education and knowledge work
  • What: Classroom or workplace tools that answer questions about images, diagrams, and slides with improved spatial reasoning and reduced “yes” bias.
  • Sectors: Education, enterprise productivity, publishing.
  • Tools/workflows:
  • Browser extensions or LMS integrations that route uploaded visuals through LeVLJEPA + frozen LLM bridges.
  • Assumptions/dependencies:
  • Content moderation and safety layers remain necessary.
  • For math/diagram-heavy content, dataset coverage and instructions affect quality.
• Rapid prototyping of robotics perception with language grounding
  • What: Use LeVLJEPA as the visual backbone for language-conditioned policies (e.g., CLIPort-style) to improve object-centric features and reduce reliance on background cues.
  • Sectors: Robotics (warehousing, logistics, home/service robots).
  • Tools/workflows:
  • Replace the vision encoder in existing imitation/RL pipelines; map patch tokens to the policy’s latent space.
  • Assumptions/dependencies:
  • Control policies still need training; perception gains translate best when policies exploit token-level structure.
  • Real-world deployments require safety validation.
• More calibrated multimodal customer support and e-commerce tools
  • What: Product Q&A and troubleshooting from user-uploaded images with reduced object hallucination and better spatial grounding.
  • Sectors: Retail, consumer electronics, automotive aftermarket.
  • Tools/workflows:
  • Fine-tune only the bridge on product-specific QA datasets; integrate into support chat systems.
  • Assumptions/dependencies:
  • Domain mismatch may require small domain adapters.
  • If zero-shot retrieval is critical (e.g., catalog matching), consider hybrid encoders.

Long-Term Applications

• Foundation-scale non-contrastive vision–LLMs for dense reasoning
  • What: Scale LeVLJEPA to larger backbones/datasets to produce general-purpose visual feature encoders optimized for dense token use (instruction tuning, video QA, grounding).
  • Sectors: Software, media, autonomous systems, research.
  • Tools/products:
  • Next-generation VLM backbones emphasizing token quality over pooled embeddings; standardized bridges for LLM families.
  • Assumptions/dependencies:
  • Requires validation at larger model scales and on video/temporal data.
  • May need complementary alignment components for strong zero-shot retrieval.
• Language-conditioned robotics with enhanced spatial semantics
  • What: Deploy perception modules that improve grasping/placement and compositional reasoning (“pick the red cup to the left of the bowl”) by leveraging stronger dense features.
  • Sectors: Industrial automation, logistics, domestic robotics.
  • Tools/products:
  • Unified “vision-to-action” stacks where LeVLJEPA backs perception; sample-efficient policy learning via better representations.
  • Assumptions/dependencies:
  • Bridging from tokens to control needs task-specific training and safety assurance.
  • Real-time constraints and latency optimization for on-robot inference.
• Safety-critical medical and scientific imaging assistants (with caution)
  • What: Assistive tools that combine segmentation and visual question answering for reports, measurements, and quality checks, benefiting from object-centric, background-robust features.
  • Sectors: Healthcare, life sciences, pharma R&D.
  • Tools/products:
  • Assistive radiology/pathology viewers: token-level segmentation + Q&A overlays; lab instrument QA via visual reasoning.
  • Assumptions/dependencies:
  • Requires domain-specific pretraining or adapters; regulatory-grade validation and bias audits.
  • Web-image pretraining alone is insufficient for clinical use.
• AR/VR multimodal agents with spatially-grounded understanding
  • What: On-device agents that interpret scenes and answer queries with strong token-level spatial grounding (e.g., “What’s the safest route around this obstacle?”).
  • Sectors: Consumer AR, enterprise field service, training/simulation.
  • Tools/products:
  • Real-time token-processing pipelines; compressed backbones and low-latency bridges.
  • Assumptions/dependencies:
  • Efficient inference, quantization, and edge compute support required.
  • Robustness to motion blur/lighting; possible need for video-adapted training.
• Document and form understanding with dense multimodal reasoning
  • What: Token-centric parsing and question answering over complex documents (tables, forms, diagrams), improving over pooled-embedding methods.
  • Sectors: Finance, insurance, government services, legal tech.
  • Tools/products:
  • “Doc-VLM” pipelines combining layout-aware token features with LLMs for field extraction and compliance checks.
  • Assumptions/dependencies:
  • Pretraining on document-centric image–text pairs or adapters substantially improves results.
  • Privacy and data governance constraints on sensitive documents.
• Retail/warehouse digital twins and analytics
  • What: Shelf/product segmentation + language queries at scale, robust to background changes across stores and seasons.
  • Sectors: Retail, supply chain, CPG analytics.
  • Tools/products:
  • Store-agnostic visual analytics: stock-out alerts, planogram verification, interactive Q&A over camera feeds.
  • Assumptions/dependencies:
  • Domain adaptation to store-specific layouts; camera calibration and lighting variability handling.
• Public-sector AI standards and procurement guidelines
  • What: Update evaluation and procurement frameworks to include dense-token metrics (segmentation linear probes, background robustness) rather than only zero-shot.
  • Sectors: Policy, standards bodies, public procurement.
  • Tools/workflows:
  • Benchmark suites emphasizing patch-token quality; robustness scorecards for adoption decisions.
  • Assumptions/dependencies:
  • Community consensus and reproducibility infrastructure; clear ties between metrics and societal impacts.
• Energy- and compute-efficient training regimes
  • What: Reduce dependence on massive batch sizes and momentum/teacher architectures, enabling greener training and broader access.
  • Sectors: Academia, SMEs, public research infrastructure.
  • Tools/workflows:
  • Best-practice recipes for non-contrastive pretraining; carbon accounting integrated into MLOps.
  • Assumptions/dependencies:
  • Scaling laws for non-contrastive VL pretraining at very large sizes require further evidence.
• Hybrid objectives for unified alignment and dense feature quality
  • What: Combine non-contrastive cross-modal prediction (for dense features) with moderate alignment modules (e.g., lightweight contrastive heads or retrieval adapters) to recover strong zero-shot while preserving token quality.
  • Sectors: General-purpose AI platforms, enterprise multimodal systems.
  • Tools/workflows:
  • Dual-head training frameworks; retrieval adapters attached post hoc to LeVLJEPA features.
  • Assumptions/dependencies:
  • Research needed to avoid regressing dense-feature quality when adding alignment losses.

Cross-cutting assumptions and dependencies

• Data availability and licensing: Domain-specific applications need paired image–text data or adapters; ensure legal use for commercial deployments.
• Safety and compliance: For regulated domains (healthcare, AV, finance), rigorous validation, interpretability, and bias/robustness audits are required.
• Zero-shot reliance: If your product hinges on zero-shot retrieval/classification, consider hybridizing LeVLJEPA with alignment modules or using contrastive baselines.
• Engineering constraints: Real-time systems (robotics, AR) require optimized inference on patch tokens and careful systems engineering.
• Model scaling: The paper validates on ViT-B/16 and mid-scale datasets; large-scale and video settings need further research.

• Anti-collapse regularizer: A mechanism to prevent representations from collapsing to trivial solutions by explicitly discouraging degenerate distributions. "could in principle be replaced by a sample-independent anti-collapse regularizer"
• Background robustness: The resilience of a model’s predictions to changes in image background content that should be irrelevant to the target. "We therefore evaluate background robustness on the ImageNet-9 backgrounds challenge"
• Batch size invariance: A property where training performance is largely insensitive to the minibatch size used during optimization. "Non-contrastive cross-modal pretraining is therefore largely insensitive to batch size, removing a constraint that ties contrastive performance to training infrastructure."
• Batch-level negative sampling: Using other samples in the same minibatch as negatives to enforce discrimination in contrastive learning. "inherits the batch-level negative sampling and batch-size dependence of contrastive pretraining"
• BYOL: A non-contrastive self-supervised method that uses an online network to predict a stop-gradient target from a momentum-averaged network to avoid collapse. "BYOL~\citep{grill2020bootstrap} prevents collapse without negatives through architectural asymmetry: an online network equipped with a prediction head regresses the representation produced by a momentum-averaged target network, with a stop-gradient applied to the target."
• Characteristic-function-based normality test: A statistical test leveraging characteristic functions to assess whether a distribution is Gaussian. "applying a characteristic-function-based normality test to each projection"
• CLS token: A special token in transformer architectures representing a pooled summary of the input (e.g., an image) used for classification tasks. "a ViT~\citep{dosovitskiy2020image} image encoder producing a CLS token embedding"
• CLIP: A contrastive vision-language pretraining approach that aligns images and text by pulling matched pairs together and pushing unmatched pairs apart. "CLIP~\citep{radford2021learning} trains $f_\theta$ and $g_\phi$ jointly by pulling matched image-caption pairs together and pushing unmatched pairs apart, using a symmetric InfoNCE objective over the batch,"
• Contrastive objectives: Losses that explicitly compare positive (matched) and negative (unmatched) pairs to learn discriminative representations. "Vision-language pretraining remains dominated by contrastive objectives"
• Cosine similarities: A similarity measure between two vectors based on the cosine of the angle between them, often used for embedding comparisons. "When cosine similarities are needed for zero-shot evaluation or retrieval, LeVLJEPA embeddings are normalized only at evaluation time."
• Cross-attending: A mechanism where one modality (e.g., text) attends to another modality’s tokens (e.g., image patches) in a multimodal model. "or a LLM cross-attending to the visual tokens during instruction tuning."
• Cross-modal prediction: Predicting the embedding of one modality (e.g., text) from another (e.g., image) to align representations without explicit negatives. "LeVLJEPA learns image-text alignment through cross-modal prediction"
• Curse of dimensionality: The phenomenon where data analysis becomes challenging as dimensionality increases, often requiring dimension-reduction strategies. "avoiding the curse of dimensionality of direct density matching."
• Datacomp-L: A large-scale image-text dataset used for training and evaluation in vision-language research. "Datacomp-L is distributed as a list of image URLs,"
• Dense prediction: Tasks that require predictions at many spatial locations (e.g., per-pixel), such as segmentation, rather than a single global output. "dense prediction methods such as DenseCLIP~\citep{rao2022denseclip}"
• Distributed data parallelism: A training paradigm that distributes mini-batches across multiple devices/nodes to parallelize and speed up training. "and is compatible with distributed data parallelism."
• DINO: A self-supervised vision method using knowledge distillation without labels to learn from images without negatives. "Methods such as DINO~\citep{caron2021emerging}, I-JEPA~\citep{assran2023self}, and LeJEPA~\citep{balestriero2025lejepa} learn strong visual representations without negative pairs, relying instead on predictive or distributional objectives."
• Effective rank: A measure of the dimensionality (rank) of an embedding matrix indicating how many directions carry significant variance; low values suggest collapse. "the effective rank of $Z^v$ and $Z^t$ drops rapidly during training"
• Epps--Pulley: A specific statistical test for normality used within SIGReg to assess Gaussianity of projected embeddings. "applies a characteristic-function-based statistical test (Epps--Pulley) to each projection"
• Frozen visual backbone: A pretrained vision encoder whose parameters are kept fixed while training downstream components. "as the frozen visual backbone of a vision-LLM"
• GPT-2: A transformer-based LLM used here as the text encoder to produce text embeddings. "and a GPT-2~\citep{radford2019language} text encoder using the final-token hidden state."
• I-JEPA: A non-contrastive image self-supervised method that predicts latent representations of masked image regions from visible ones. "I-JEPA~\citep{assran2023self} predicts the representations of masked image regions from visible ones in latent space, avoiding both negatives and pixel-level reconstruction."
• ImageNet-9 backgrounds challenge: An evaluation protocol assessing sensitivity to background by compositing objects onto different backgrounds. "We therefore evaluate background robustness on the ImageNet-9 backgrounds challenge"
• In-batch denominator: The normalization term in contrastive losses that sums over other batch samples, effectively creating negatives. "through the in-batch denominator, pushes it away from the other captions in the batch."
• InfoNCE: A contrastive loss that encourages alignment of matched pairs while contrasting against all other pairs in the batch. "using a symmetric InfoNCE objective over the batch,"
• Isotropic Gaussian: A Gaussian distribution with equal variance in all directions; targeted by SIGReg for embedding distributions. "to keep each modality's marginal embedding distribution close to isotropic Gaussian."
• JEPA (Joint-Embedding Predictive Architecture): A family of methods that predict one embedding from another within a shared latent space instead of predicting pixels. "VL-JEPA~\citep{chen2025vljepa} recasts vision-language learning in the language of joint-embedding predictive architectures"
• Layer Normalization: A normalization technique applied across features within a layer that can interact with distributional regularizers like SIGReg. "apply a Layer Normalization~\citep{ba2016layernormalization} which prevents the SIGReg anti-collapse objective from being optimized effectively"
• LeJEPA: A JEPA-based method using a distributional regularizer (SIGReg) to enforce isotropic Gaussian embeddings and avoid collapse. "LeJEPA~\citep{balestriero2025lejepa} provides a simple and theoretically grounded alternative within the JEPA family by replacing such mechanisms with an explicit distributional regularizer."
• LeVLJEPA: A non-contrastive end-to-end vision-language pretraining method combining cross-modal prediction with SIGReg. "We introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method."
• Linear probing: Evaluating feature quality by training a linear classifier atop frozen embeddings without fine-tuning the encoder. "Under linear probing, by contrast, the objectives are on par at both the CC12M and Datacomp-L scales"
• MLP bridge: A small neural mapping that projects visual tokens into a LLM’s input space in a frozen-backbone setup. "only a lightweight MLP bridge mapping the patch tokens into the language-model input space is trained"
• Momentum encoder: A slowly updated target network in self-supervised learning (e.g., BYOL) used to provide stable targets. "without negatives, temperature, momentum encoder, or teacher-student schedule"
• Momentum-averaged target network: The target network in methods like BYOL updated via momentum from the online network. "produced by a momentum-averaged target network"
• Negative pairs: Unmatched examples used in contrastive learning to push apart representations and avoid collapse. "without negative pairs"
• Non-contrastive objective: A training objective that avoids explicit negatives, often relying on prediction and regularization instead. "establishing that the non-contrastive objective matches contrastive pretraining"
• Patch tokens: Tokenized representations of image patches emitted by a vision transformer for dense, spatially localized features. "consume the full grid of patch tokens"
• Pooled embedding: A single vector summary of an image (or sequence) obtained by pooling tokens, used for global tasks like classification. "rather than a single pooled embedding."
• Semantic segmentation: A dense prediction task assigning a class label to each pixel (or patch), measuring spatial-semantic structure. "outperforms contrastive baselines on semantic segmentation"
• SIGReg (Sketched Isotropic Gaussian Regularization): A distributional regularizer that enforces isotropic Gaussian embeddings via projected normality tests. "SIGReg (Sketched Isotropic Gaussian Regularization) is the objective that enforces this distribution in a scalable way."
• SigLIP: A contrastive vision-language method replacing softmax normalization with independent sigmoid classification for stability. "SigLIP~\citep{zhai2023sigmoid} replaces the softmax with an independent sigmoid classification over every pair"
• SimCLR: A canonical contrastive self-supervised learning baseline relying on augmented views and in-batch negatives. "SimCLR~\citep{chen2020simpleframeworkcontrastivelearning} remains a canonical contrastive baseline,"
• SimSiam: A non-contrastive SSL method showing that a momentum encoder is not necessary when using a predictor and stop-gradient. "which later was simplified by SimSiam~\citep{chen2021exploring}---which shows the momentum encoder is not required---"
• Stop-gradient: An operation that detaches targets from gradient flow to stabilize prediction-based training and prevent collapse. "with stop-gradient targets"
• Teacher-student schedule: A training setup with a teacher and student network (often momentum-averaged) used in some SSL methods. "without negatives, temperature, momentum encoder, or teacher-student schedule"
• Temperature parameter: A scaling factor in contrastive losses (e.g., InfoNCE) controlling the sharpness of the similarity distribution. "without negatives, temperature, momentum encoder, or teacher-student schedule"
• ViT (Vision Transformer): A transformer-based vision model that processes images as sequences of patch tokens. "a ViT~\citep{dosovitskiy2020image} image encoder producing a CLS token embedding"
• Vision-LLM: A multimodal model that processes both visual and textual inputs, often via cross-attention. "as the frozen visual backbone of a vision-LLM"
• View-prediction loss: A JEPA-style loss predicting one view’s latent representation from another to encourage invariant features. "with a view-prediction loss averaged across the $B$ samples in the batch:"
• Zero-shot transfer: Evaluating a model on unseen tasks by matching image representations to text prompts without task-specific fine-tuning. "At the smaller CC12M scale, LeVLJEPA is competitive with CLIP and SigLIP on both zero-shot transfer and linear probing"

Sign up for free to add this paper to one or more collections.