Structured 4D Latent Predictive Model for Robot Planning

Sign up for free to create and run prompts on this paper using GPT-5.

What is this paper about?

This paper is about teaching robots to think ahead in 3D. Instead of predicting future 2D videos (flat pictures over time), the authors build a model that predicts how the full 3D world will change over time—think of it as a 4D “mental movie” where 4D = 3D space + time. The robot uses this future 3D “movie” to plan what to do next, so it can act more precisely and handle changes in viewpoint, lighting, and clutter.

What questions did the researchers ask?

They focused on a few simple questions:

• Can a robot learn an internal 3D map of the scene that changes over time, not just a flat video?
• Can this 3D “future forecasting” be guided by a text instruction like “stack the red block on the blue block”?
• Will planning in 3D make the robot’s actions more reliable and accurate, especially when the camera moves or things get occluded?
• Does this work better than popular video-based planners on tough manipulation tasks?

How did they study it?

A 3D memory of the scene

• The robot looks at the world from several cameras around it (multi-view). From these views, it builds a compact 3D memory called a “latent.”
• Imagine the 3D world as a Minecraft-like grid of tiny cubes (called voxels—3D pixels). Only the cubes that matter are kept (“sparse” voxels), and each active cube stores a short feature vector that captures local geometry and appearance.
• This 3D latent can be “decoded” back into different 3D formats, such as:
  • Point clouds (a set of 3D points)
  • 3D Gaussians (think of many small fuzzy blobs that can be rendered into images from any viewpoint)
• Why this helps: a single 3D memory gives a consistent view of the scene from any camera angle, unlike separate 2D videos that can disagree with each other.

Predicting the future (the “4D” part)

• The model predicts how that 3D memory will change over time, guided by a text instruction (the goal).
• It does this in two steps: 1) Single Dynamics Model (coarse changes): predicts which voxels (3D cubes) will be active next—like sketching the rough shape of the next scene. 2) Latent Generator (fine details): fills in the detailed features (color/texture) for those voxels—like coloring in the sketch.
• Training method: They use a technique similar to diffusion models called “flow matching.” You can think of it as learning how to turn noisy guesses into clean, realistic 3D futures step by step.

Turning futures into actions

• Predicting a future scene isn’t enough; the robot needs exact moves.
• They use an inverse dynamics module: given the current 3D scene and the next predicted 3D scene, it figures out the short sequence of robot joint positions to move toward that next scene.
• Two ways to do this:
  • Learned version: a small neural network learns to map from the current and goal point clouds to robot actions.
  • Learning-free version: align a 3D model of the gripper to the predicted goal using standard 3D alignment tools (RANSAC + ICP) and then use a motion planner. Think of it like matching shapes and then planning a path.

What did they find, and why is it important?

Here are the main results, explained simply:

• More consistent 3D futures: Compared to video-based planners, their method produces future scenes that stay consistent across different camera views and over time. This is crucial for precise manipulation (like stacking or inserting pegs).
• Better planning performance: On several robot tasks (e.g., stacking blocks, pulling with a tool, inserting a peg), their 3D planner outperformed strong video-based planning baselines and was competitive with top imitation-learning methods.
• Strong generalization: The planner kept working well even when lighting changed, backgrounds changed color, or cameras moved to new angles. That’s because the model reasons in 3D, not just in 2D pixels.
• Real-world success: They tested on a real robot (placing a block in a basket) and achieved higher success than a popular baseline method.

Why this matters: For robots to work in everyday places, they need reliable 3D understanding to handle occlusions, moving cameras, and detailed spatial tasks. This approach gives them that “3D common sense.”

What could this change in the future?

• More reliable home and warehouse robots: Planning in 3D could help robots handle messy, changing environments and perform fine-grained tasks more safely and accurately.
• Easier transfer across setups: Because the model uses a shared 3D state, it can adapt to different camera placements or viewpoints without retraining everything.
• Flexible outputs: The same 3D latent can be turned into point clouds, images, or 3D Gaussian scenes, making it useful for many downstream tools and planners.
• Next steps: Make it work from fewer cameras (or even a single camera), speed it up for real-time use, and broaden to more complex, dynamic environments.

In short, this paper shows that giving robots a structured 3D “mind’s eye” of the future—not just a 2D video—leads to better, more general, and more dependable planning and actions.

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of unresolved issues that future work could address.

• Dependence on calibrated multi-view RGB-D: The method assumes known intrinsics/extrinsics and depth at inference; robustness to single-view or RGB-only inputs, calibration drift, moving cameras, and depth noise is not evaluated.
• Domain dependence of the 3D encoder/decoder: The pipeline relies on a pre-trained TRELLIS encoder/decoder trained with RGB reconstruction losses; the impact of domain shift (sim→real, unseen materials/lighting) and whether fine-tuning is required is not studied.
• Latent capacity and spatial scale: The sparse voxel latent (e.g., N≈64, L≈8k, d=8) may limit workspace size and geometric precision; scalability to larger scenes, higher resolution, and dense clutter is untested.
• Runtime and real-time feasibility: No measurements of inference latency, FPS, or GPU memory for SD/LG, 3DGS decoding, and rendering; ability to operate at control rates on real robots remains unclear.
• Two-stage SD/LG design: There is no ablation isolating the benefit of separate structure (SD) and feature (LG) predictors versus joint models; sensitivity of LG to SD errors and potential joint training remains unexplored.
• Autoregressive error accumulation: The work lacks analysis of compounding rollout errors, drift, and recovery strategies beyond optional closed-loop updates; no uncertainty quantification or use of multiple stochastic rollouts for plan selection.
• Physics and contact modeling: The model imposes no explicit physical constraints (contact, friction, stability) and does not evaluate contact-rich tasks or non-rigid interactions; physical plausibility of generated futures is unverified.
• Baseline coverage: Comparisons are mainly to 2D/2.5D video planners; head-to-head evaluation against explicit 3D dynamics/world models (e.g., Gaussian/NeRF/point-cloud dynamics) is missing.
• Inverse dynamics generalization: The learned ID is trained per task (≈1k demos); its cross-task/embodiment transfer and how well the 3D latent decouples action spaces in practice are untested.
• Learning-free registration limits: The registration-based ID assumes visible, rigid gripper geometry; robustness under occlusion, symmetries, specular surfaces, or non-rigid end-effectors is not assessed.
• Language grounding: Instructions are encoded with CLIP, but the breadth and compositionality of language is unclear; performance on ambiguous, multi-step, or long-horizon textual commands is not evaluated.
• Planning horizon and decomposition: Training uses 4–10 intermediate subgoals; performance on longer, multi-stage tasks and principled subgoal selection remains an open question.
• Viewpoint robustness at inference: Although training uses many views, inference uses 4 fixed cameras; behavior with 1–2 cameras, handheld/moving viewpoints, or severe occlusions is not reported.
• Sensor noise robustness: While some RGB perturbations are tested, sensitivity to depth noise, time desynchronization, and systematic calibration errors is unquantified.
• Real-world scope: Real experiments are limited to one simple task and one robot; generalization to different robots, grippers, workspaces, and more complex tasks is untested.
• Action representation and control: The approach uses joint position commands without torque/impedance control; handling of latency, tracking errors, and compliance is not discussed.
• Safety and constraints: The learned ID lacks explicit collision/constraint handling; the planning pipeline omits safety guarantees and does not verify collision-free execution except for the learning-free variant (details minimal).
• Data efficiency: The model requires ≈1k demonstrations per task; sample efficiency versus baselines and low-data regimes are not analyzed.
• Clutter and fine tolerances: There is no evaluation under dense clutter, small objects, or sub-centimeter precision requirements; geometric accuracy thresholds for successful manipulation are not established.
• Deformable/articulated objects: The latent’s ability to capture topology changes, non-rigid dynamics, or articulated kinematics is not tested.
• Occlusion completion: How well the model infers unobserved geometry behind occluders and the impact on planning is not evaluated with targeted benchmarks.
• Metric–success correlation: Beyond a robot mask IoU proxy, there is limited analysis linking multiview consistency metrics to downstream task success; thresholds and failure correlations are missing.
• Camera self-calibration: The pipeline assumes fixed, known camera poses; online calibration or self-correction from data is not considered.
• Training objective details: Flow-matching schedules/noise and architectural specifics are sparse; no comparison to diffusion or autoregressive alternatives, nor sensitivity analyses to objective/hyperparameters.
• RGB-D reliance in encoding: The method depends on depth for unprojection; feasibility with monocular RGB (self-supervised depth) or passive stereo at train/test time is not assessed.
• Dynamic backgrounds/humans: Robustness to moving distractors or human presence in the scene is untested.
• Computational footprint: No report of training cost, convergence time, or scaling with number of cameras and scene complexity.
• Failure mode analysis: The paper lacks a taxonomy or quantitative breakdown of common errors (e.g., mislocalized gripper, object hallucinations, geometry drift) and corresponding mitigation strategies.
• Planner integration: There is no mechanism to rank/select among multiple candidate latent futures or to integrate cost functions/MPC; how to combine the model with trajectory optimization or task-level search remains open.

Immediate Applications

Below are actionable use cases that can be deployed with the paper’s current capabilities (structured 4D latent predictive model with calibrated multi-view RGB-D input, decoding to point clouds/3D Gaussians, and inverse dynamics via a learned module or learning-free registration + motion planning).

• Text-guided manipulation planners for cells and workstations (Robotics, Manufacturing, Warehousing)
  • Use case: Deploy the model as a 3D-aware planner for tasks like bin-picking, tool pulling, stacking, and peg insertion in fixed workcells with multi-camera rigs.
  • Tools/products/workflows:
  • ROS/MoveIt integration that converts predicted 3D subgoals to actions (via diffusion-based inverse dynamics or ICP-based gripper registration).
  • A “4D Subgoal Planner” API that takes multi-view RGB-D + text instruction and returns subgoal point clouds and joint commands.
  • Operator UI showing predicted 3D rollouts for validation before execution.
  • Assumptions/dependencies: Calibrated multi-view RGB-D; known gripper geometry for learning-free option; moderate task variability; mostly rigid objects; sufficient training demos per task; compute for real-time inference.
• Teleoperation with predictive 3D previews (Robotics, Industrial Ops, Remote Ops)
  • Use case: Show operators 3D rollouts of likely future scenes under a high-level instruction to reduce trial-and-error and improve safety.
  • Tools/products/workflows:
  • HMI overlay that renders decoded 3D Gaussians/point clouds for the next subgoals from multiple camera viewpoints.
  • “What-if” previews conditioned on alternative textual goals.
  • Assumptions/dependencies: Multi-view calibration; real-time rendering; human-in-the-loop oversight; integration with existing teleop stacks.
• Mobile manipulation with varying viewpoints (Robotics, Logistics, Service Robots)
  • Use case: Planning and execution on robots whose sensors move (eye-in-hand or mobile base), leveraging the model’s multiview consistency and viewpoint generalization.
  • Tools/products/workflows:
  • Online latent updates from changing camera poses; replanning loop as per Algorithm 1.
  • Lightweight point-cloud decoding for fast control cycles.
  • Assumptions/dependencies: Accurate camera pose tracking; online re-encoding at each step; reliable depth in cluttered scenes.
• Low-data deployment via learning-free inverse dynamics (Robotics, SMEs/SMBs)
  • Use case: Execute simple grasp-move-place tasks without action-labeled data using geometric registration of the gripper to predicted subgoal geometry.
  • Tools/products/workflows:
  • Registration pipeline (FPFH, RANSAC, ICP) + motion planner packaged as a “No-Train Subgoal Executor.”
  • Assumptions/dependencies: Known end-effector mesh; good point clouds; mostly rigid scenes; fewer occlusions.
• 3D-consistent planning benchmarks and evaluation (Academia, Standards/Benchmarking)
  • Use case: Use MVGBench-style metrics, robot-mask IoU, and multiview render consistency as standardized tests for 3D-aware planners.
  • Tools/products/workflows:
  • Open-source evaluation suite with cPSNR/cSSIM/cLPIPS, Chamfer distance, depth error, mask IoU (SAM3-based).
  • Reproducible protocols on ManiSkill3/RLBench with multiview capture.
  • Assumptions/dependencies: Access to simulation datasets and multi-view GT; shared camera parameters; reproducible seeds.
• Data augmentation and policy debugging via 4D rollouts (Academia, Software Tools, Robotics R&D)
  • Use case: Stress-test policies by generating multiview-consistent future scenes, perturb lighting/cameras, and evaluate robustness.
  • Tools/products/workflows:
  • Plugins to ManiSkill3/RLBench for “latent rollouts” and synthetic multiview re-rendering.
  • Visual analyzers to compare predicted vs. observed 3D states over time.
  • Assumptions/dependencies: Simulation integration; GPU capacity for batch rollouts; consistent domain randomization.
• Training and education content for robot manipulation (Education, Training)
  • Use case: Visualize 3D subgoals and state transitions to teach spatial reasoning, calibration, perception-to-action pipelines, and viewpoint invariance.
  • Tools/products/workflows:
  • Classroom modules using decoded point clouds and 3D Gaussians; lab exercises on inverse dynamics and motion planning.
  • Assumptions/dependencies: Access to low-cost RGB-D cameras; simplified lab setups; curated demos.
• AR-assisted procedure previews for technicians (AR/VR, Field Service, Maintenance)
  • Use case: Show 3D-consistent predicted object/tool motion sequences before executing a maintenance step.
  • Tools/products/workflows:
  • AR overlays from decoded Gaussians rendered to the technician’s viewpoint; text-conditioned task steps.
  • Assumptions/dependencies: On-site multi-view capture or SLAM-driven alignment; latency constraints; safety validation for suggestions.
• Cloud robotics “planner-as-a-service” (Software, Cloud Robotics)
  • Use case: Upload multiview frames and an instruction; receive subgoal point clouds and action chunks.
  • Tools/products/workflows:
  • REST/gRPC APIs; containerized model inference; caching of encoders/decoders.
  • Assumptions/dependencies: Bandwidth for frame upload; privacy/security; stable calibration data sent alongside frames.

Long-Term Applications

These opportunities require additional research, scaling, or engineering (e.g., single-/few-view input, deformables, long horizons, safety-critical constraints, regulatory approvals).

• General-purpose household robots with text-guided 3D planning (Robotics, Consumer)
  • Potential product: Home assistant that uses a 3D world model to execute tasks like tidying, sorting, and fetching under natural language.
  • Dependencies: Robust single- or dynamic multi-view perception (SLAM), deformable-object modeling, long-horizon credit assignment, reliable failure recovery, cost-effective sensors.
• Surgical and interventional robotics planning (Healthcare)
  • Potential product: 4D predictive planning in minimally invasive procedures with endoscopic multi-view cues and precise tool pose estimation.
  • Dependencies: Soft-tissue dynamics and safety margins, submillimeter accuracy, domain-specific datasets, explainability, FDA/CE approvals, sterile integration.
• Construction and infrastructure manipulation (Construction, Civil/Utilities)
  • Potential product: Robots performing drilling, fastening, wiring, or valve operations with 3D-consistent predictive subgoals in evolving sites.
  • Dependencies: Outdoor lighting robustness, dust/occlusions, large-scale mapping, changing geometry, compliance with site safety protocols.
• Energy sector inspection and maintenance (Energy, Oil & Gas, Nuclear)
  • Potential product: Predictive planning to operate valves, handle tools, or manipulate fixtures in plants where viewpoint shifts are common.
  • Dependencies: Radiation-/hazard-hardened sensors, remote calibration, compliance and safety certifications, operator oversight, robust registration in clutter.
• Agricultural manipulation (Agriculture)
  • Potential product: Fruit picking, pruning, and gentle handling guided by 4D scene predictions under variable lighting and foliage.
  • Dependencies: Modeling deformables, partial/occluded views, domain adaptation to outdoor conditions, seasonal variability.
• Multi-robot shared 4D world models for coordination (Robotics, Logistics, Warehousing)
  • Potential product: Fleet coordination using a shared 3D latent state for collision-free, cooperative manipulation.
  • Dependencies: Latent synchronization over networks, scalable SLAM fusion, latency-tolerant planning, standardized state schemas.
• Facility-scale digital twins with learned 4D dynamics (Industrial Digital Twins)
  • Potential product: Predictive digital twins for planning and simulating maintenance/upgrades with multiview video feeds and text-defined scenarios.
  • Dependencies: Efficient scaling of sparse voxel latents to large spaces, streaming inference, integration with CAD/BIM and physics, security and data governance.
• Autonomous-vehicle or drone scene forecasting with action semantics (Autonomy, Aerial/Surface)
  • Potential product: 3D-consistent scene evolution forecasting for navigation with natural-language-instructed maneuvers (“approach dock, avoid crane”).
  • Dependencies: Integration with LiDAR/radar, moving-actor modeling, safety verification, domain-specific constraints, real-time compute on embedded hardware.
• 4D scene-editing and content creation (Media/Software, AR/VR)
  • Potential product: “4D scene editor” where users describe edits in text (e.g., “move the vase to the shelf”) and receive multiview-consistent video/3D outputs.
  • Dependencies: Photorealistic decoding, artist-friendly controls, IP/content rights, performance at high resolution.
• Physics-aware learned simulators for planning (Software Tools, Research)
  • Potential product: Hybrid engines that fuse fast 4D latent predictions with physics constraints for quick, approximate planning.
  • Dependencies: Stability guarantees, error bounds, composability with physics simulators, interpretability for debugging.
• Policy and standards for 3D-consistent predictive planning (Policy, Standards Bodies)
  • Potential deliverables: Benchmarks and certification protocols for 3D consistency, interpretability requirements for predicted subgoals, calibration standards for multi-view systems, telemetry schemas for sharing latent states.
  • Dependencies: Multi-stakeholder consensus (academia, industry, regulators), open datasets, safety case frameworks, privacy compliance.
• Workforce upskilling tools with predictive feedback (Education, Workforce Development)
  • Potential product: Training platforms that visualize predicted 3D manipulation steps and allow learners to compare attempts to model rollouts.
  • Dependencies: Affordable sensor packs, simplified curricula, institutional adoption, accessible compute.

Notes on Key Assumptions and Dependencies Across Applications

• Multi-view calibration: Current system assumes calibrated multi-view RGB-D to reconstruct the initial latent; reducing to single-/few-view with SLAM or learned depth is an active research dependency.
• Scene characteristics: Results are strongest on rigid or quasi-rigid objects; deformable-object dynamics and contact-rich physics require further model and data advances.
• Data needs: Inverse dynamics benefits from task-specific demonstrations; the learning-free route requires accurate gripper models and clean point clouds.
• Compute and latency: Flow-matching models and 3D decoding require GPUs for real-time/near-real-time operation; optimization (pruning/quantization/edge deployment) may be needed.
• Safety and reliability: For high-stakes domains (healthcare, energy), formal verification, runtime monitors, and fail-safes are necessary.
• Generalization: Robustness to lighting, background, and viewpoint shifts is improved relative to video-only planners, but broader domain shifts (materials, specularities, heavy occlusion) may still degrade performance.
• Tooling ecosystem: Smooth adoption depends on ROS/MoveIt plugins, camera calibration toolkits, dataset loaders for ManiSkill3/RLBench, and visual debuggers for 3D latents/subgoals.
• Governance: Applications that stream multi-view RGB-D to the cloud require privacy/security controls and compliance with data regulations.

• 3D Gaussian Splatting: A point-based 3D representation that renders scenes using anisotropic Gaussian primitives for fast, photorealistic views. "Photorealistic representations like Neural Radiance Fields (NeRFs)~\citep{mildenhall2021nerf} or 3D Gaussian Splatting~\citep{kerbl3Dgaussians} better capture appearance, but are computationally intensive and not easily amenable to dynamic modeling."
• 3DGS: Abbreviation for 3D Gaussian Splatting, a Gaussian-based scene representation used for rendering and dynamics. "A parallel line of research learns dynamics over structured 3D representations such as point clouds, meshes, NeRF-like fields, or 3DGS, enabling physical simulation or relational reasoning for manipulation tasks."
• Autoregressive: A generation strategy where future states are predicted sequentially, each conditioned on previous outputs. "The generator operates autoregressively $g(z_{t+1} | z_t, c)$, and future states are obtained by iterative rollout."
• cLPIPS: Consistency LPIPS; perceptual similarity metric adapted to evaluate multi-view consistency across generated views. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench} (Chamfer Distance, depth error, cPSNR, cSSIM, cLPIPS)."
• cPSNR: Consistency PSNR; signal-to-noise metric measuring multi-view consistency of reconstructions across viewpoints. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench} (Chamfer Distance, depth error, cPSNR, cSSIM, cLPIPS)."
• cSSIM: Consistency SSIM; structural similarity metric adapted for assessing multi-view coherence. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench} (Chamfer Distance, depth error, cPSNR, cSSIM, cLPIPS)."
• Chamfer Distance: A distance measure between two point sets, commonly used to compare 3D shapes. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench} (Chamfer Distance, depth error, cPSNR, cSSIM, cLPIPS)."
• CLIP: A vision-LLM that aligns images and text via contrastive learning for conditioning generative models. "Text instructions are encoded with a pre-trained CLIP model~\citep{radford2021learning}."
• Closed-loop execution: A control mode that replans using new observations after each action, enabling feedback-based corrections. "At test time, we consider both open-loop execution (execute the predicted action chunk) and closed-loop execution (replan after each action step using new observations)."
• Conditional flow matching: A generative training approach that learns continuous-time velocity fields to transform noise into data conditioned on inputs. "We use conditional flow matching~\citep{lipman2022flow} for generative modeling"
• Condition augmentation: Robustness technique that perturbs conditioning inputs (e.g., dropouts, noise) during training to handle partial observations. "To improve robustness to partial observations, we use condition augmentation, randomly dropping out voxel features from the input latent condition $z_{t}$ and adding Gaussian noise to its features $\{\bm{f}_i\}$."
• Cross-attention: An attention mechanism that conditions one sequence on another, enabling information flow between modalities or tokens. "Both conditions share positional encodings with the voxel tokens, enabling the model to capture correlations, and are injected in each transformer block through cross-attention."
• D-SSIM: A dissimilarity score derived from SSIM, used as a reconstruction loss for image supervision. "We use the pre-trained 3D encoder $\mathcal{E}$ and decoder $\mathcal{D}$ from TRELLIS~\citep{xiang2025structured}, which was trained using RGB reconstruction losses (L1, D-SSIM, LPIPS) to supervise the 3D Gaussians."
• DINOv2: A self-supervised vision transformer producing patch-level embeddings useful for 3D feature lifting. "For each image, we use a pre-trained DINOv2 encoder to extract patch-level embeddings."
• Diffusion head: A neural head that predicts actions by denoising noisy action samples within a diffusion framework. "encode them with a pyramid convolutional backbone~\citep{ze2024humanoid_manipulation}, concatenate the resulting features with the robot proprioceptive state, and use a diffusion head to predict $a_{1:H}$."
• FPFH: Fast Point Feature Histograms; 3D point descriptors for establishing correspondences in point cloud registration. "Specifically, we compute FPFH features for coarse correspondence matching, use RANSAC to obtain an initial alignment, and refine the resulting transformation with ICP."
• ICP: Iterative Closest Point; an algorithm for refining rigid alignment between two point clouds. "Specifically, we compute FPFH features for coarse correspondence matching, use RANSAC to obtain an initial alignment, and refine the resulting transformation with ICP."
• Inverse dynamics (goal-conditioned): Mapping from current state and desired next state (subgoal) to the control commands that realize it. "This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module."
• IoU: Intersection over Union; an overlap metric used here to compare generated and ground-truth robot masks. "We evaluate the IoU between ground truth robot mask and generated robot mask."
• Latent Diffusion Models: Diffusion models operating in compressed feature space to improve efficiency and quality. "Following the success of Latent Diffusion Models~\citep{rombach2022high} which utilize spatially-aware 2D feature maps rather than unstructured 1D global latents, we adopt a structured 3D latent representation"
• Latent space: A compressed representational space where high-dimensional data (e.g., 3D scenes) are encoded for modeling and generation. "We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene’s 3D structure in a structured latent space conditioned on observations and textual instructions."
• LPIPS: Learned Perceptual Image Patch Similarity; a perceptual metric used as a loss and evaluation measure. "We use the pre-trained 3D encoder $\mathcal{E}$ and decoder $\mathcal{D}$ from TRELLIS~\citep{xiang2025structured}, which was trained using RGB reconstruction losses (L1, D-SSIM, LPIPS) to supervise the 3D Gaussians."
• Model predictive control (MPC): A receding-horizon control method that optimizes actions using a system model at each step. "enabling effective planners like model predictive control (MPC) to operate on top of them~\citep{garcia1989model,mayne2000constrained,qi25ral-gpc}."
• Multi-view coherence: Consistency of generated content across different camera viewpoints over time. "Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners."
• MVGBench: A benchmark suite for evaluating multi-view generative consistency and geometry fidelity. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench} (Chamfer Distance, depth error, cPSNR, cSSIM, cLPIPS)."
• Neural Radiance Fields (NeRFs): Neural volumetric representations modeling view-dependent appearance and density for photorealistic rendering. "Photorealistic representations like Neural Radiance Fields (NeRFs)~\citep{mildenhall2021nerf} or 3D Gaussian Splatting~\citep{kerbl3Dgaussians} better capture appearance"
• Occlusion: The phenomenon where parts of the scene are hidden from a particular viewpoint, causing prediction ambiguity. "which improves interpretability but lacks explicit 3D structure, leading to inconsistencies under occlusion or viewpoint change."
• Open-loop execution: A control mode that executes predicted actions without replanning from new observations. "At test time, we consider both open-loop execution (execute the predicted action chunk) and closed-loop execution (replan after each action step using new observations)."
• Proprioceptive state: Internal robot sensing (e.g., joint positions/velocities) used alongside perception for control. "concatenate the resulting features with the robot proprioceptive state, and use a diffusion head to predict $a_{1:H}$."
• PSNR: Peak Signal-to-Noise Ratio; a pixel-wise fidelity metric for image/video evaluation. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench}"
• RANSAC: Random Sample Consensus; a robust estimator for model fitting in the presence of outliers. "Specifically, we compute FPFH features for coarse correspondence matching, use RANSAC to obtain an initial alignment, and refine the resulting transformation with ICP."
• SE(3): The Special Euclidean group in 3D, representing rigid-body poses (rotation and translation). "The final transformation defines the target end-effector pose $T_{\text{ee}, t+1} \in SE(3)$ in the robot base coordinate frame."
• Segment Anything Model 3 (SAM3): A segmentation model used here to extract robot masks for evaluation. "we use Segment Anything Model 3 (SAM3)~\citep{carion2025sam} to segment the robot shape mask in each generated image"
• SDFs: Signed Distance Fields; scalar fields giving the signed distance to surfaces, used as a geometric representation. "Traditional representations, such as meshes, point clouds, or SDFs, often lack photorealism"
• Sparse voxel grid: A 3D grid where only occupied voxels are stored, providing efficient structured scene encoding. "we encode the scene into a sequence of sparse voxel grids where each active voxel holds a compact feature vector."
• SSIM: Structural Similarity Index; a perceptual image quality metric comparing luminance, contrast, and structure. "We evaluate both standard image quality metrics (PSNR, SSIM, LPIPS) and 3D consistency metrics from MVGBench~\citep{xie2025MVGBench}"
• Transformer denoiser: A transformer-based network that predicts noise/velocities in a diffusion/flow-matching model. "A transformer denoiser then operates in this compressed space."
• Unproject: Back-project 2D depth pixels into 3D space using camera intrinsics/extrinsics. "We first unproject the depth map using camera geometry and merge them into a 3D point cloud"
• Voxelization: Discretizing 3D space into voxels, typically to convert point clouds into grid structures. "which is then voxelized into a sparse grid $\{\bm{p}_i\}_{i=1}^L$ ."
• Zero-shot generalization: Performing well under novel conditions (e.g., lighting/viewpoints) without task-specific fine-tuning. "Zero-shot generalization with visual and viewpoint changes."

Sign up for free to add this paper to one or more collections.