PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation

PhysMani: Physics-Principled 3D World Modeling for Dynamic Object Manipulation

Introduction and Motivation

Dynamic object manipulation in unstructured 3D environments is a fundamental challenge for embodied AI, particularly under physically realistic, high-speed, and complex motion regimes. Prevailing visual-language-action (VLA) and world models exhibit severe deficiencies in explicit 3D geometric knowledge, physical plausibility in temporal predictions, and computational latency, which compromise efficacy in time-sensitive manipulation scenarios. The PhysMani framework directly addresses these limitations by introducing a physics-principled 3D Gaussian world model that learns divergence-free velocity fields via online optimization and couples this with a future-aware action policy conditioned explicitly on predicted future 3D scene dynamics.

Figure 1: The overall framework of PhysMani, comprising a physics-principled 3D Gaussian world model and a future-aware action policy.

PhysMani introduces PhysMani-Bench, a diverse benchmark of 16 dynamic tasks, and demonstrates clear empirical superiority over contemporary baselines in both synthetic and real-robot settings.

Physics-Principled 3D Gaussian World Model

At the core of PhysMani is a 3D scene representation utilizing thousands of Gaussian kernels, where each kernel captures local geometry, color, and density, as well as its own physically grounded velocity state. Initialization is performed via back-projection of multiview RGB-D inputs into a sparse 3D point cloud, with fine-tuning of all Gaussian parameters through differentiable rendering. The world model's progression is governed by a Gaussian velocity module parameterized by MLPs and designed for divergence-free velocity field learning, encoding both linear and angular motion for each Gaussian particle.

A distinctive contribution is the reformulation of the FreeGave pipeline into a lightweight and efficient online optimizer that forgoes complex deformation networks, enabling real-time dynamics tracking and prediction at robot-control timescales.

Figure 2: Left: Canonical 3D Gaussian module. Top-right: Physics-principled Gaussian velocity module. Bottom-right: Online optimization schema for real-time future dynamics.

PhysMani thereby continuously updates the scene model as a stream of RGB-D frames is received, iteratively optimizing both the velocity field and the geometry for accurate multistep forecasting of physically meaningful 3D states.

Future-Aware Policy with Dynamics-Conditioned Action Prediction

Action policy in PhysMani is grounded on the FlowMatch Actor backbone but expanded to tightly couple policy inference with future scene dynamics. The policy leverages a multi-modal encoding of current visual observations (lifted to 3D tokens) and natural language instructions via cross-attention mechanisms. Critically, for each visual feature, a local neighborhood of k-nearest Gaussians is queried for proximity and future velocity predictions—these are processed and fused with a learnable token in attention blocks to yield future-aware scene tokens for end-to-end action inference.

Figure 3: Left: Encoding of visual observations and language. Right: Incorporation of 3D scene future dynamics for action prediction through attention mechanisms.

This tight coupling ensures that the policy not only reacts to the present scene but anticipates its physically plausible evolution, providing robustness and precision in high-speed and complex manipulations.

PhysMani-Bench: Dynamic Manipulation Benchmark

PhysMani-Bench offers 16 tasks (8 dynamic scenarios, each at normal and high speeds) incorporating manipulation primitives such as reaching, grasping, placing, inserting, and tool-use under complex object trajectories (linear, circular, rotational). The scenarios strictly require non-reactive, predictive control, as target motion is commensurate with robot actuation limits.

Figure 4: Diverse and challenging dynamic tasks in PhysMani-Bench, showcasing complex target motions.

Each task is evaluated with a single model jointly trained on all scenarios, highlighting the generalization required for scalable real-world deployment.

Experimental Evaluation

Quantitative Performance

PhysMani surpasses all baselines in average success rate (SR) across all 16 PhysMani-Bench tasks as well as all four real-world evaluation tasks. The method gives an 8.1-point mean SR improvement over the next-best approach (3DFA backbone), confirming that action policy conditioned on accurate future 3D dynamics is a critical factor in high-performance dynamic manipulation. Attempts to "patch" baselines with 2D motion cues (optical flow) provide no significant improvement, demonstrating that only 3D, physically plausible future knowledge substantially benefits policy.

Future Frame and Trajectory Prediction

PhysMani delivers significantly superior future frame prediction results, sustaining high PSNR and low trajectory error over temporal horizons (mean error 0.008 m at 50 ms, 0.074 m at 500 ms). In contrast, ManiGaussian—designed for dynamic prediction—exhibits substantially higher error after a single step. PhysMani also achieves a $3\times$ lower per-frame latency than FreeGave while maintaining higher predictive fidelity.

Figure 5: Qualitative comparison of future frame prediction; PhysMani accurately predicts dynamic target trajectories (red circles).

Figure 6: PSNR for future frame prediction; PhysMani achieves higher quality and operates with lower per-frame latency than FreeGave.

Real-World Robotic Manipulation

PhysMani consistently outperforms 3DFA in all real-robot dynamic manipulation tasks, underscoring the transferability and robustness of the world model-policy architecture under realistic perception and actuation constraints.

Figure 7: Physical robot setup and real-world dynamic tasks with Astribot S1 arm.

Ablation Studies and Analysis

Ablations confirm that removal of 3D scene velocity conditioning drops mean success substantially, validating its essentiality. Removing the learnable token for attention fusion also impairs performance, indicating nontrivial benefits in local dynamics-aware action generation. Experiments across different world model optimization iteration budgets reveal that PhysMani maintains high robustness for reasonable iteration ranges, enabling control-rate operation (<300 ms per inference-action step).

Figure 8: Mean PhysMani SR over all 16 tasks as a function of world model optimization iterations.

Visualization of learned velocity fields demonstrates high selectivity to moving objects and robot arms, confirming that the velocity representation is effectively capturing actionable scene motion cues.

Figure 9: Visualizations of the learned six basic Gaussian velocity components during dynamic task execution.

Practical and Theoretical Implications

PhysMani establishes a framework where real-time, physics-consistent world models can effectively drive predictive and robust policies for dynamic manipulation. The divergence-free 3D Gaussian representation yields both computational efficiency and physical plausibility, offering a scalable and generalizable path beyond low-level reactive control or template-based approaches.

The explicit fusion of future dynamics into policy via local attention and learnable tokens is experimentally motivated as crucial for sample efficiency, success, and transferability.

Practically, these contributions enable adaptive manipulation in open, dynamic conditions—essential for automating real-world environments with unpredictable object behaviors. Theoretically, it advances the integration of physical world modeling into closed-loop control and highlights the utility of structured, physically meaningful generative models for embodied AI.

Conclusion

PhysMani provides a high-fidelity, computationally tractable solution for dynamic 3D object manipulation by integrating a divergence-free, physics-principled world model with a future-aware action policy. The method achieves state-of-the-art performance on both synthetic benchmarks and real robots, even as tasks scale in complexity and speed, and demonstrates the centrality of 3D future dynamics prediction for high-level embodied intelligence. Future directions include scaling to even larger and more diverse tasks (leveraging PhysMani-Bench) and extending the approach to multi-agent and multi-object interaction regimes, suggesting broad utility for embodied AI in unconstrained environments.