OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

Sign up for free to create and run prompts on this paper using GPT-5.

What this paper is about (overview)

This paper introduces OpenVision 2, a simpler and faster way to teach computers to understand images and work with text at the same time (that’s called “multimodal” learning). The big idea: instead of training with a complicated “matching game” between images and sentences, the authors make the model learn just by writing captions for images. Even with this simpler “tell a story about the picture” training, the model stays just as good (or better) on many tests, while training much faster and using less memory.

What questions the researchers asked (objectives)

The authors wanted to find out:

• Can a vision model learn well using only caption generation (writing a caption for an image) without the usual “contrastive” matching between images and text?
• If we remove extra parts (like a separate text encoder) to make training lighter, will performance on real tasks stay strong?
• Will this simpler setup make training cheaper and allow bigger models?

How they did it (methods, in everyday language)

Think of two training styles:

• Contrastive learning = a matching game. The model sees a bunch of images and a bunch of sentences and learns to match which sentence goes with which image.
• Generative learning = storytelling. The model looks at an image and tries to write a good caption for it.

OpenVision 2 chooses the storytelling approach only.

What changed:

• They removed the text encoder (the part that reads text during training) and the contrastive loss (the matching game). Now there are just two parts: 1) An image encoder that turns the picture into a sequence of “tokens” (little pieces of information). 2) A text decoder that reads those tokens and writes a caption.
• They train the model only by asking it to write the caption correctly (caption loss). This is like practicing writing instead of playing a matching game.

Helpful tricks to make it efficient:

• Better training captions: They use high-quality, AI-generated captions (called ReCap-DataComp-1B v2). These are cleaner and more detailed than noisy web text.
• Token masking: Before the text decoder writes the caption, they randomly hide about two-thirds of the image tokens. This makes the text decoder do less work and teaches it to focus on the most useful bits—like summarizing from fewer clues.
• CLIPA-style schedule: Train first on smaller images (faster and cheaper), then briefly fine-tune on larger images for extra quality. Think: learn the basics with small pictures, polish with bigger ones.

How they tested it:

• They plugged their vision encoders into popular multimodal systems (like LLaVA and Open-LLaVA-Next) and evaluated on many benchmarks: reading text in images (OCR), answering questions about images (TextVQA, ScienceQA, GQA), charts (ChartQA), and general perception tests (MME, SEED).

What they found (results) and why it matters

Even with the simpler “caption-only” training, OpenVision 2 keeps up with or beats the older, more complex version on many tasks—especially ones that involve reading text inside images (OCR). At the same time, it’s much faster and lighter to train.

Here are a few headline numbers:

• Training speed: About 1.5× faster on a ViT-L/14 model (83 hours down to 57 hours) and about 2× faster on a larger model (241 hours down to 121 hours).
• Memory savings: About 1.8× less memory needed (for example, from 24.5 GB down to 13.8 GB per device), letting them use much bigger batch sizes.
• Scaling up: Because it’s cheaper to train, they could build a vision encoder with over 1 billion parameters—something that was hard before with the heavier setup.
• Strong performance: On benchmarks like TextVQA, ScienceQA, SEED-Bench, OCR tests, and more, OpenVision 2 matches or slightly improves over the original. It’s especially good on OCR-heavy tasks, which means it’s better at “reading” inside images.

Why this matters:

• It shows you don’t need the usual contrastive matching game to get a strong, general-purpose vision encoder.
• Lower cost means more researchers (even with fewer resources) can train and improve these models.
• Training now looks more like how the model is actually used later (image tokens feeding a text decoder), which can make learning smoother.

What this could change (implications)

• Simpler recipe, wider access: With fewer parts and cheaper training, more labs can build powerful, open-source vision encoders. This helps the whole community, not just big companies.
• Better fit for modern multimodal systems: Many current AI assistants connect an image encoder to a LLM. OpenVision 2 trains in that exact style, which may make it easier to fine-tune and improve downstream.
• Stronger on text-in-image tasks: The caption-first approach and token masking seem to help models read and reason about text inside images, which is useful for documents, charts, receipts, and signs.
• Faster scaling: Since it’s more efficient, pushing to larger models and bigger datasets becomes practical, opening doors to better performance in the future.

In short, OpenVision 2 suggests a clear message: teaching a vision model to “tell the story of an image” can be enough—and can even be better—than training it with the classic image–text matching game. The team also releases their code, models, and data so others can build on this work.

Knowledge gaps, limitations, and open questions

Below is a focused list of what remains missing, uncertain, or unexplored in the paper. Each point is framed to be actionable for future research.

• Absence of standard zero-shot image–text retrieval and classification evaluations (e.g., COCO/Flickr30k retrieval, ImageNet zero-shot, ImageNet ReaL), making it unclear how a generative-only encoder compares to contrastive encoders on embedding-based alignment tasks.
• No linear-probe or frozen-feature evaluation on core vision benchmarks (e.g., ImageNet, VTAB), leaving the standalone quality of the vision encoder (without an LLM) unassessed.
• Lack of spatial grounding, detection, and segmentation benchmarks (e.g., RefCOCO/+/g, COCO det/seg), so the impact of caption-only supervision on localization-heavy tasks is unknown.
• Unclear support for retrieval-style downstreams where a text encoder is typically required; practical pathways for using OpenVision 2 in retrieval systems (e.g., via pooled image features, decoder states, or auxiliary text encoders) are not explored.
• The claim of “objective alignment” between pretraining and multimodal finetuning is not directly validated; controlled experiments isolating the effect on data efficiency, convergence speed, and transfer (vs. contrastive pretraining) are missing.
• Dependency on fully synthetic captions raises questions about bias, factuality, and hallucination in supervision; there is no systematic audit of synthetic caption quality (e.g., bias profiling, factuality checks, and human evaluation).
• No study on mixing synthetic and human captions (or web alt-text) to assess whether hybrid corpora outperform purely synthetic training for generalization and robustness.
• Multilingual generalization is not evaluated; it remains unknown whether OpenVision 2 transfers to non-English captions (training and inference) or multilingual OCR tasks.
• Distribution-shift robustness (e.g., corrupted images, out-of-domain datasets, adversarial perturbations) is not assessed; compare against contrastive baselines under common robustness suites.
• Safety, fairness, and content moderation aspects are not analyzed beyond POPE; study demographic bias, toxicity propagation from synthetic captions, and safety failure modes across subgroups.
• Token-masking design space is underexplored: only random masking is tested; evaluate spatially structured masking (blockwise/strided), saliency- or confidence-guided masking, and curriculum schedules (progressive keep ratios) across scales and resolutions.
• Fusion strategy choice (simple concatenation) lacks a head-to-head comparison with alternatives (e.g., cross-attention connectors, Q-Former, gated fusion); quantify trade-offs in accuracy, compute, and scalability.
• Decoder architecture sensitivity is not characterized; analyze effects of decoder type, size, freezing vs. finetuning, and different LLM backbones on the learned visual representations and downstream performance.
• Inference-time efficiency is not reported; measure end-to-end latency and throughput for common multimodal tasks, and evaluate encoder-only inference paths for applications that cannot afford autoregressive decoding.
• Hardware portability and reproducibility are unclear; results are on TPU v4 pods—provide GPU benchmarks (A100/H100/L40S), throughput/memory profiles, and guidance for smaller-scale training setups.
• Scaling laws and compute–data–model trade-offs are not studied; derive scaling curves (loss vs. compute/data/model size), compute-optimal training regimes, and batch-size/optimizer stability at billion-parameter scale.
• Data composition is insufficiently dissected; quantify the proportion of text-rich vs. natural images, domain coverage, duplicates, and their influence on OCR-centric gains; include decontamination analysis relative to evaluation sets.
• Caption-generation strategy (weighted top-k sampling, alt-text conditioning) lacks a systematic study; vary sampling temperatures, top-k, conditioning sources, and prompt templates to measure downstream impacts and caption quality.
• Pretrain–finetune masking mismatch is not examined; evaluate whether maintaining or removing visual-token masking during downstream finetuning affects transfer and sample efficiency.
• Generalization to video, temporal reasoning, and multi-image inputs is not addressed; extend OpenVision 2 to video encoders, measure on temporal benchmarks, and test multi-image fusion tasks.
• Fine-grained recognition capabilities (e.g., CUB, Aircraft, Cars) are untested; verify whether descriptive caption supervision suffices for subtle category distinctions often underrepresented in captions.
• Stability and variance across runs are not reported; include multiple seeds, confidence intervals, and sensitivity analyses (learning rates, batch sizes, token keep ratios) to assess reproducibility.
• Pathways for exporting robust image-only embeddings (without a text encoder) are not detailed; investigate projection heads or contrastive auxiliary heads to enable CLIP-like plug-and-play use without sacrificing the generative gains.

Immediate Applications

The following applications can be deployed now by leveraging the released models, dataset, and training code of OpenVision 2. Each item notes sectors, potential tools/products/workflows, and feasibility considerations.

• Drop-in open alternative to CLIP/SigLIP for multimodal assistants
  • Sectors: software, productivity, education, customer support
  • What to build: Replace CLIP-like encoders in LLaVA/Open-LLaVA-Next pipelines with OpenVision 2 checkpoints to power image captioning, VQA, chart/table understanding, and visually grounded chat; build domain-specific assistants (e.g., lab-report explainer, classroom tutor for diagrams).
  • Why now: Comparable or better downstream VQA/OCR performance with significantly lower training cost; code, models, and dataset are publicly released.
  • Dependencies/assumptions: Fine-tuning still needed for target tasks; inference stacks must be compatible with chosen VLM framework; synthetic-caption training data biases can propagate.
• OCR-centric document and chart understanding in enterprise workflows
  • Sectors: finance, logistics, insurance, retail/e-commerce, legal
  • What to build: Receipt/invoice/contract triage; KYC/AML document intake; chart/slide QA for analysts; product label parsing; e-commerce listing enrichment.
  • Why now: Stronger OCR-related benchmark results (e.g., OCR-Bench gains) translate to practical improvements in text-in-image tasks.
  • Dependencies/assumptions: For regulated settings, require domain fine-tuning and human-in-the-loop QA; document privacy and PII handling; multilingual coverage may need additional training.
• Cost- and compute-constrained pretraining for labs and startups
  • Sectors: academia, startups, ML platforms
  • What to build: Train task- or domain-specific vision encoders with limited compute (TPUs/GPUs) using OpenVision 2’s generative-only objective, CLIPA curriculum, and token masking (25–35% keep ratio performed best in ablations).
  • Why now: 1.5–2× faster training and ~1.8× lower memory enable larger batches (up to 8k on v4-64), reducing wall-clock and energy costs.
  • Dependencies/assumptions: Original training used TPU v4; porting to GPUs or other accelerators may require engineering; data scale and quality still drive outcomes.
• Synthetic recaptioning and data quality upgrading for internal image corpora
  • Sectors: enterprise data/ML teams, content platforms, search
  • What to build: Adopt the ReCap-DataComp-style pipeline to recaption internal image collections; use enhanced sampling strategies (e.g., weighted top-k) and conditioning on existing metadata to produce richer captions; feed into OpenVision 2 pretraining/fine-tuning.
  • Why now: Paper shows large gains from synthetic caption quality; tooling and recipes are open.
  • Dependencies/assumptions: LLM captioners incur inference costs; ensure data licensing compliance and avoid leakage of proprietary or sensitive content.
• Visually grounded search, tagging, and moderation
  • Sectors: media platforms, DAM/CMS, ecommerce marketplaces
  • What to build: Multimodal search with natural language queries; fine-grained auto-tagging; content safety filters assisted by better text-in-image understanding.
  • Why now: Encoder aligns well with LLM decoders in multimodal stacks, facilitating high-quality descriptions and reasoning over image text (signage, packaging, screenshots).
  • Dependencies/assumptions: Contrastive retrieval may require adaptations since pretraining removed contrastive loss; add retrieval heads or small contrastive fine-tunes if needed.
• Accessibility features for blind and low-vision users
  • Sectors: consumer apps, public sector services, NGOs
  • What to build: On-device or cloud captioning and VQA for everyday scenes, documents, menus, and signage; context-aware guidance in public spaces; extended chart/diagram explanations in courseware.
  • Why now: OpenVision 2’s strengths on OCR and multimodal reasoning improve real-world utility; integration with LLaVA-like assistants is straightforward.
  • Dependencies/assumptions: Safety and calibration to avoid hallucinations; latency and on-device constraints may require model distillation or quantization.
• Academic benchmarking and reproducibility
  • Sectors: academia, open science, research consortia
  • What to build: New controlled studies comparing contrastive vs. generative supervision; scaling laws for generative-only vision encoders; curriculum studies for token masking and CLIPA schedules; shared, reproducible baselines for multimodal benchmarks.
  • Why now: Fully open code, checkpoints, and dataset variants enable low-friction replication and extension.
  • Dependencies/assumptions: Continued community maintenance of datasets and evaluation harnesses; careful documentation of training compute for comparability.
• Greener AI and cost reporting in ML operations
  • Sectors: ML Ops, sustainability teams, policy-compliant R&D
  • What to build: Integrate OpenVision 2 training into internal cost/energy dashboards; include efficiency metrics in model cards; set organizational targets for per-epoch FLOPs and peak memory budgets.
  • Why now: Quantified reductions in FLOPs and memory present actionable levers for cost and emissions reduction.
  • Dependencies/assumptions: Actual energy savings depend on hardware, datacenter PUE, and job scheduling; transparency requires consistent reporting frameworks.

Long-Term Applications

These applications are promising but require additional research, scaling, domain data, or engineering to be production-ready.

• Domain-specialized medical and scientific vision encoders
  • Sectors: healthcare, biotech, scientific publishing
  • What to build: Encoders trained with radiology/pathology images and paired reports (synthetic recaptioning over de-identified data); assistants for interpreting scans, lab results, or scientific figures; safety-critical chart QA for clinical trials.
  • Rationale: Generative-only pretraining aligns with text-report supervision and may reduce reliance on contrastive pairs in scarce domains.
  • Dependencies/assumptions: Regulatory compliance (HIPAA/GDPR), rigorous validation, domain-expert curation; bias and safety audits; possibly multimodal privacy-preserving training.
• Multilingual and multimodal retrieval without contrastive pretraining
  • Sectors: global search, cross-border e-commerce, public information portals
  • What to build: Retrieval systems that rely on generative alignment (e.g., generative scoring or re-ranking) rather than contrastive embeddings; multilingual recaptioning to expand coverage.
  • Rationale: If generative supervision scales, retrieval may be reframed around generation-first pipelines.
  • Dependencies/assumptions: Current retrieval still favors contrastive representations; expect hybrid training (small contrastive adapters) or rerankers; need high-quality multilingual captions.
• Video, 3D, and sensor-fusion encoders via caption-only objectives
  • Sectors: robotics, AV, XR/AR, geospatial
  • What to build: Extend token-masking and generative captioning to video clips, point clouds, and multi-sensor data; instruction-following encoders for embodied agents; scene narration for 3D/XR.
  • Rationale: Architectural simplicity and alignment with LLM decoders could scale across modalities, with masking improving efficiency and robustness.
  • Dependencies/assumptions: Large-scale, high-quality paired captions for video/3D are scarce; needs temporal grounding, streaming inference optimizations, and robust safety.
• Enterprise-scale auto-recaptioning and governance pipelines
  • Sectors: MLOps, data governance, legal/compliance
  • What to build: Production systems to continually recaption evolving image corpora, track lineage, watermark synthetic content, and enforce data licenses; policy checks for synthetic data reuse.
  • Rationale: Synthetic captions demonstrably improve supervision; enterprises will need auditable pipelines.
  • Dependencies/assumptions: Legal clarity on derivative captions; watermarking standards; monitoring for bias amplification by captioner models.
• On-device multimodal assistants via compact generative encoders
  • Sectors: mobile, wearables, IoT
  • What to build: Distilled or quantized OpenVision 2 variants paired with small LLM decoders for low-latency captioning, OCR, and VQA on edge devices.
  • Rationale: Training-time efficiencies suggest pathways to smaller, better-aligned models that are easier to compress.
  • Dependencies/assumptions: Inference efficiency is not guaranteed by training efficiency; requires model compression, kernel-level optimizations, and memory-aware token handling.
• Safety, bias, and robustness frameworks for generative-only pretraining
  • Sectors: policy, standards bodies, responsible AI
  • What to build: Evaluation suites tailored to generative-pretrained vision encoders (hallucination under occlusion/masking, text-in-image equity, adversarial robustness); standards and disclosures for synthetic-caption reliance.
  • Rationale: The supervision shift (contrastive → generative) changes failure modes and audit needs.
  • Dependencies/assumptions: Community consensus on metrics; diverse stakeholder input; open testbeds with licensed data.
• Unified multimodal foundation stacks and tooling
  • Sectors: ML platforms, cloud providers, enterprise AI
  • What to build: Reference stacks where pretraining, fine-tuning, and inference all use the same two-module architecture (vision encoder + text decoder), minimizing objective/architecture mismatch; one-click domain adaptation workflows using masking and CLIPA curriculum.
  • Rationale: OpenVision 2 reduces architectural mismatch with downstream VLMs, simplifying engineering and increasing reuse.
  • Dependencies/assumptions: Broad library support across JAX/PyTorch and multiple accelerators; stable APIs and model zoos; continued benchmark leadership.
• Carbon-aware scheduling and training policy informed by efficiency gains
  • Sectors: cloud infrastructure, public sector R&D funding
  • What to build: Policies and schedulers that prefer generative-only pretraining recipes when accuracy is comparable, to minimize emissions for funded projects; cost/emission caps per pretraining run.
  • Rationale: Demonstrated FLOPs and memory reductions create a policy lever for greener AI.
  • Dependencies/assumptions: Reliable carbon accounting; alignment of incentives for researchers and providers; exceptions for tasks where contrastive objectives are necessary.

Notes on feasibility and assumptions across applications:

• Reported gains are on multimodal benchmarks (e.g., TextVQA, ChartQA, OCR-Bench); direct transfer to production tasks requires task-specific fine-tuning and evaluation.
• Retrieval-heavy use cases may still benefit from contrastive components; consider hybrid or adapter approaches.
• Synthetic captions inherit biases and errors of the captioning MLLMs; governance and auditing are essential.
• Training stacks in the paper use TPUs; porting to GPUs may need reimplementation or framework changes.
• Token masking ratios (25–35% keep) are empirically effective in this paper; optimal values may vary by domain and model size.

• AIMv2: A multimodal pretraining approach that blends image reconstruction and text generation in an autoregressive framework. "CapPa~\cite{tschannen2024image} and AIMv2~\cite{fini2024multimodal}"
• Alt-text: Raw descriptive text associated with web images, often noisy or incomplete. "Alt-text: raw web alt-text;"
• Autoregressive prediction: Sequential generation where each token is predicted conditioned on previously generated tokens. "an additional text decoder is required to autoregressively predict the synthetic caption"
• CapPa: A caption-only pretraining approach advocating generative supervision without contrastive loss. "CapPa~\cite{tschannen2024image}"
• Caption loss: A generative training objective that maximizes likelihood of the target caption given image features. "by training solely with the caption loss."
• ChartQA: A benchmark for question answering over charts and plots. "ChartQA~\cite{masry-etal-2022-chartqa}"
• CLIP: A contrastive vision–LLM aligning images and texts via a dual-encoder objective. "CLIP~\cite{radford2021clip}"
• CLIPA: A training strategy that pretrains at low resolution then briefly fine-tunes at high resolution for efficiency. "CLIPA-style training \cite{li2023clipa}"
• CLIPS: An approach introducing dual contrastive and caption losses to better leverage synthetic captions. "CLIPS introduces two additional objectives:"
• Contrastive loss: An objective that brings matched image–text pairs closer while pushing mismatched pairs apart. "remove the text encoder---and therefore the contrastive loss---"
• Cross-attention: A mechanism that allows one modality to attend to another for fusion. "CapPa fuses modalities with cross-attention."
• DataComp-1B: A large-scale image–text dataset used for pretraining vision–LLMs. "DataComp-1B collection"
• Dual contrastive loss: A contrastive objective pairing each image with multiple captions to enhance alignment. "(i) a dual contrastive loss that pairs each image with both web-crawled and generated captions"
• FLOPs: A measure of computational cost in floating-point operations. "FLOPs / Image"
• GQA: A benchmark for compositional reasoning and question answering over images. "GQA~\cite{hudson2018gqa}"
• Generative pretraining: Training that learns representations by generating text (e.g., captions) from images. "generative pretraining of vision encoders for multimodal learning."
• Keep ratio: The fraction of visual tokens retained (unmasked) as conditioning for caption generation. "with different image token keep ratios."
• LLaMA-3: A LLM used to power synthetic caption generation within LLaVA. "LLaMA-3-powered LLaVA"
• LLaVA: A multimodal framework that connects a vision encoder to a LLM for visual instruction following. "modern multimodal designs like LLaVA~\cite{llava}"
• LLaVA-1.5: An updated LLaVA framework used for evaluating multimodal encoders. "under both the LLaVA-1.5~\cite{liu2023improved}"
• MME: A benchmark suite assessing multimodal perception and understanding. "MME~\cite{fu2023mme}"
• Multimodal autoregressive decoder: A decoder that jointly models image and text tokens in an autoregressive manner. "a multimodal autoregressive decoder that simultaneously (i) reconstructs image patches through pixel-level regression and (ii) generates text tokens"
• Multimodal foundation models: Large models that integrate vision and language to perform a wide range of tasks. "multimodal foundation models"
• Objective mismatch: A discrepancy between pretraining and finetuning objectives that can hinder transfer. "eliminates the objective mismatch between pretraining and downstream finetuning"
• OCR-Bench: An evaluation focused on optical character recognition within multimodal systems. "327 \vs 315 on OCR-Bench"
• Open-LLaVA-Next: An open implementation of a next-generation LLaVA framework for evaluation. "Open-LLaVA-Next~\cite{chen2024open}"
• Out-of-memory (OOM): A hardware memory exhaustion failure during training or inference. "out-of-memory (OOM)"
• POPE: A benchmark for probing object hallucinations in vision–LLMs. "POPE~\cite{Li-hallucination-2023}"
• Prefix-ViT: A ViT variant using prefix tokens with special attention masks for multimodal autoregression. "prefix-ViT~\cite{el2024scalable}"
• Q-Former: A learnable connector module used to interface image encoders with LLMs. "LLM connectors (Q-Former)"
• ReCap-DataComp-1B v2: A recaptioned dataset using image and alt-text conditioning for higher-quality synthetic captions. "ReCap-DataComp-1B v2: synthetic captions generated by MLLM with both image and alt-text as input."
• ReCap-DataComp-1B: A recaptioned dataset where captions are generated without conditioning on alt-text. "ReCap-DataComp-1B: synthetic captions generated by MLLM without conditioning on alt-text;"
• SEED: A benchmark evaluating various multimodal understanding capabilities. "SEED~\cite{li2023seed}"
• SigLIP: A vision–LLM using a sigmoid loss for image–text alignment. "SigLIP~\cite{zhai2023sigmoid}"
• SoViT-400M/14: A scaled Vision Transformer backbone variant used as a vision encoder. "SoViT-400M/14"
• SQA: ScienceQA, a benchmark requiring multimodal scientific reasoning. "SQA~\cite{lu2022learn}"
• Synthetic captions: Model-generated descriptions used as training supervision instead of noisy web text. "synthetic captions generated by MLLM"
• Text decoder: The generative module that produces captions conditioned on visual tokens. "a text decoder"
• Text encoder: The encoder that processes text for contrastive alignment; removed in the proposed design. "remove the text encoder"
• TextVQA: A benchmark for question answering that requires reading text in images. "TextVQA \cite{singh2019towards}"
• Token masking: Randomly dropping a subset of visual tokens to reduce compute and improve robustness. "validating the effectiveness of our synthetic captioning and token masking strategies"
• TPU v4-512: A Google Cloud TPU configuration (512-chip pod) used to measure training time. "TPU v4-512"
• Two-stage curriculum: Training first on low-resolution images then fine-tuning briefly at high resolution. "adopts this two-stage curriculum to accelerate training."
• ViT-L/14: A Vision Transformer model with large size and 14-pixel patches. "ViT-L/14"
• Vision encoder: The image-processing backbone that produces visual token representations. "the vision encoder"
• Visual tokens: Sequence tokens derived from images that condition text generation. "producing a sequence of visual tokens."
• Weighted top-k sampling: A decoding strategy encouraging diverse yet grounded caption generation. "applying weighted top-$k$ sampling, which encourages more diverse yet grounded captions."

Sign up for free to add this paper to one or more collections.