- The paper introduces a pre-alignment stage using black-box on-policy distillation with a Mixture-of-Experts discriminator to correct multimodal drift.
- It demonstrates consistent performance gains across various RL algorithms and benchmarks by decoupling perception and reasoning feedback.
- Empirical results validate that the three-stage pipeline mitigates SFT-induced drift and effectively readies models for robust downstream RL.
Pre-alignment via Black-box On-policy Distillation for Multimodal Reinforcement Learning: An Expert Analysis
Context and Motivation
The dominant post-training approach for large multimodal models (LMMs) is a two-stage paradigm: initial supervised fine-tuning (SFT) on curated demonstrations, followed by reinforcement learning with verifiable rewards (RLVR). While SFT provides a necessary capability bootstrap, it introduces significant distributional drift—misalignments between the policy and the target supervision distribution—which subsequent RLVR cannot reliably correct, particularly in highly capable foundation models. This problem is magnified in multimodal settings where perception (visual grounding) and reasoning (cognitive chains of logic) drift in qualitatively different ways, and a single RLVR objective cannot effectively counteract this heterogeneity.
Technical Contributions
PRISM (Pre-alignment via Black-box On-policy Distillation for Multimodal Reinforcement Learning) introduces a three-stage post-training pipeline for LMMs by incorporating a dedicated pre-alignment stage between SFT and RLVR. The alignment is formulated as a black-box, response-level adversarial on-policy distillation (OPD) game between the current model policy and a Mixture-of-Experts (MoE) discriminator, specializing in independent correction of perception and reasoning drifts. This process proceeds without access to teacher logits, utilizing only sample-level comparisons to the supervision pool. Key technical elements include:
- MoE Discriminator: The discriminator is architected with separated perception and reasoning experts. Each expert specializes in evaluating the visual description or the logical chain-of-thought trace, providing disentangled corrective signals. This modular design is essential for handling the distinct error distributions characteristic of multimodal policy drift.
- Adversarial Alignment via OPD: Alignment proceeds as a minimax game. The policy is optimized to generate rollouts indistinguishable from high-quality supervision, while each discriminator expert is trained to distinguish policy responses from references using the Bradley-Terry loss.
- Data Curation: Beyond leveraging over 1.26M public demonstrations for SFT, the authors distill an additional 113K challenging, high-fidelity multimodal reasoning cases from Gemini 3 Flash, specifically targeting difficult and unsolved problems, providing dense supervision for both SFT and alignment.
- Algorithm Flexibility: The final RLVR stage is validated using a spectrum of RL algorithms (GRPO, DAPO, GSPO), demonstrating pipeline-agnostic improvement.
Empirical Results
Main Findings
- Consistent Performance Improvements: The introduction of PRISM’s alignment stage yields substantial quantitative improvements on Qwen3-VL-4B and 8B models across seven benchmarks (Math Vista, Math Verse, Math Vision, WeMath, MMMU, MMMU-Pro, HallusionBench). For example, the PRISM+GRPO configuration outperforms the SFT->GRPO baseline by +4.4 and +6.0 average points on 4B and 8B settings, respectively.
- Algorithm-Agnostic Gains: Similar, robust improvements are observed across DAPO and GSPO, indicating that the effectiveness of the alignment stage is not tied to any specific downstream RLVR algorithm.
- Drift Correction, Not Immediate Accuracy: The immediate effect of the alignment stage on accuracy is neutral—its value is in reshaping the model distribution to be more RL-ready, not in directly increasing pre-RL accuracy. Downstream RLVR leverages this improved distribution for stronger gains.
- Scaling Effects: Distributional drift introduced by SFT worsens as model scaling increases. Notably, the performance degradation in 8B models post-SFT is only partially recoverable with standard RLVR; PRISM’s alignment can fully compensate and surpass the original foundation performance by over 5 points.
Ablation Studies
- MoE vs. Monolithic Discriminator: Replacing the MoE architecture with a single dense discriminator results in a significant performance drop (−3.4 average points), empirically establishing the necessity of decoupled perception and reasoning feedback for robust alignment.
- Necessity of the Three-Stage Pipeline: Removing SFT or the alignment stage causes severe performance regression; each stage is shown to be indispensable and complementary.
- Vision-Language Discriminator: A text-only discriminator fails to enforce faithful visual grounding, producing “parrot alignment” where textual styles are mimicked without true perception, confirming the need for vision-language signals.
- SFT Data Scale: Reducing SFT data impairs performance, and although the alignment stage can partially compensate, high-quality, large-scale SFT remains crucial for optimal initialization.
Theoretical and Practical Implications
PRISM reframes on-policy distillation within multimodal learning as an explicit, standalone correction of SFT-inducted distribution drift. This underscores a critical theoretical position: strong LMMs cannot be reliably improved with SFT-RLVR pipelines lacking distribution-matching alignment, especially when RLVR is initialized from policies distorted by token-level imitation. The MoE discriminator paradigm introduces modularity in reward engineering, paving the way for more granular and interpretable multimodal policy shaping.
Practically, PRISM directly addresses a bottleneck in scalable multimodal RL: ensuring that the starting point for RLVR is appropriately matched to high-fidelity supervision, thus maximizing sample efficiency, convergence speed, and final task performance. The black-box nature of alignment decouples it from teacher logit dependency, facilitating alignment to proprietary or inaccessible distributions.
Limitations and Directions for Future Research
PRISM's additional alignment stage introduces computational and memory overhead due to the joint training of generator and discriminator. The current approach depends on a structured output (caption, reasoning, answer), potentially constraining applicability to tasks lacking decomposable responses. Direct distributional metrics for alignment are absent, with current analysis depending on interpretable proxies such as reasoning step counts and descriptive item density.
Future directions include automating or learning the response decomposition required for the MoE architecture, scaling to larger and more heterogeneous base models, developing model-agnostic distribution alignment metrics, and amortizing alignment overhead across multiple RL objectives.
Conclusion
PRISM (2604.28123) establishes the necessity of explicit, disentangled distribution alignment in the post-training pipeline for large multimodal models. By interleaving a black-box adversarial on-policy distillation stage with a Mixture-of-Experts discriminator, it effectively remediates the heterogeneous drift induced by SFT and creates policy initializations that enable stronger, more stable downstream RLVR optimization. These results have immediate significance for scalable multimodal RL, principled post-training calibration, and future directions in decoupled reward modeling.
