Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models

Sign up for free to create and run prompts on this paper using GPT-5.

What is this paper about?

This paper introduces Deform360, a giant collection of real videos and touch data showing how soft, bendy, and squishy objects move when robots handle them. The goal is to help robots “imagine” what will happen next when they poke, squeeze, fold, or twist everyday soft things like ropes, cloth, and plush toys.

What questions did the researchers ask?

To make their work clear, the authors focused on a few simple questions:

• How can we collect enough high-quality, real-world data so robots can learn the tricky physics of soft objects?
• Which kind of “world model” works better for predicting the future of deformable objects: models that think in 2D pictures (videos) or models that think in 3D shapes (particles in space)?
• Can vision (cameras) and touch (tactile sensors) together help track object motion more accurately—especially when parts are hidden by the robot’s hands?
• Can models trained on this data actually help a real robot plan and do tasks?

How did they do it?

To answer these questions, the team built a large dataset and a new way to turn videos and touch into detailed 3D motion.

The dataset: many objects, many views, with touch

They recorded 1,980 robot–object interactions with:

• 198 real, everyday deformable objects (big variety: ropes/cables, cloth/bags/paper, plush/foam).
• 41 cameras around the scene, so the object is visible from all sides (360°).
• Two robot grippers with “fingertip” pressure sensors to feel contact.
• Over 215 hours and 23.3 million frames of synchronized video and touch data.

Think of the cameras like a stadium full of spectators, each with a clear view, while the robot’s fingertips report exactly when and where they press on the object.

Turning raw sensing into 3D motion (explained simply)

Predicting soft-object motion is hard because the object can bend in countless ways, and the robot’s fingers often block the view. The team built a markerless tracking pipeline that works in three steps:

1. Rebuild each frame in 3D with “tiny paint blobs”

• They use a method called 3D Gaussian Splatting. Imagine the object as a cloud of tiny, colored, see‑through blobs. From any camera angle, the blobs render a realistic image. This gives a high-quality 3D snapshot for each video frame.

2. Track points in 2D, then lift to 3D

• They track many points on the object in each camera view (like following dots on a moving shirt).
• Using the 3D snapshot, they “lift” these 2D tracks into 3D positions.
• They combine information from many cameras to get stable 3D trajectories.

3. Use touch to enforce physically sensible motion

• Touch sensors tell when and where the robot is pressing.
• The tracker adjusts the motion so it stays smooth, locally rigid (nearby points don’t stretch unrealistically), and consistent with where the robot actually pressed.
• This is like telling the system, “These points should move together,” and “Here is where a push really happened.”

What did they test with this data?

To make the dataset useful for others, the team set up three tasks:

• Contact prediction: Can a model look at video and guess when/where the robot is in contact (pressed) with the object?
• World-model benchmarking: Compare two big ideas for predicting the future:
  • 2D video models (like advanced video generators that imagine future frames).
  • 3D particle models (think of the object as a 3D puppet made of many tiny beads connected by soft springs).
• Robot planning: Use a learned model to help a robot choose actions (Model Predictive Control, or MPC) to reach a goal shape/state.

What did they find?

Here are the main results, explained clearly:

• The reconstructions are sharp and reliable. The method produces high-quality 3D per-frame geometry across ropes, cloth, and plush objects.
• Touch helps a lot under occlusion. When the gripper blocks the view, adding tactile data made 3D tracking much more accurate (about 5× lower error than vision only).
• Vision can predict contact surprisingly well. A model trained on videos predicted touch/no-touch with about 89% F1-score, showing that visible surface changes reveal hidden contact events.
• 3D models vs. 2D video models: there’s a trade-off.
  • In low-data situations (like learning from just one short interaction), 3D particle models with built-in physics assumptions did better. Their structure helps them stay physically correct.
  • With more data and more variety, large 2D video models generalized better to new objects and preserved visual details more vividly. But sometimes they didn’t strictly follow the robot’s exact commands over long sequences.
• Real robot planning worked. Using a 3D model (PhysTwin) in an MPC planner, they showed early demos of a different robot in a different lab manipulating deformable objects toward goals. The 3D shape representation made it easier to measure progress and define rewards.

Why does this matter?

Soft objects are everywhere—clothes, cables, packaging, plush toys, food items. Teaching robots to handle them safely and reliably is essential for household help, warehouse sorting, laundry folding, and more. This work matters because:

• It provides a rich, real-world benchmark at a scale that didn’t exist before (many objects, many views, with touch), so researchers can test and improve their models under realistic conditions.
• It clarifies when to use which kind of world model. If you have little data, structured 3D models with physics assumptions shine. If you have lots of data and variety, big 2D video models can generalize visually to new situations.
• It shows how combining vision and touch leads to better understanding of hidden interactions, which is crucial for safe and precise manipulation.
• It takes a step toward practical planning with soft objects by using learned models in real robots.

Limitations and what’s next

There are still challenges: heavy occlusions can break tracking, very “plasticky” materials may stretch in ways the model doesn’t expect, and the touch sensors mainly feel pressure (not sideways slip). Future work could add richer touch sensing, reduce action-mismatch in video models, and blend the strengths of both 2D and 3D approaches—scalability plus physical structure—for even better world models of the soft, squishy real world.

Knowledge gaps, limitations, and open questions

Below is a focused list of what remains missing, uncertain, or unexplored in the paper, phrased to guide concrete follow-up work:

• Tactile modality limitations: current sensors measure only normal-axis pressure; lack of shear/torque sensing prevents slip detection and force direction estimation—add shear-sensitive or 6-axis sensing and annotate slip events.
• Interaction coverage gap: slip is intentionally avoided; frictional sliding, stick–slip, and deliberate slip-rich tasks remain unrepresented—design protocols with controlled slip and friction variation.
• Missing ground-truth contact/wrench data: absence of end-effector force/torque or interaction wrenches limits quantitative contact dynamics learning—integrate calibrated F/T sensing and align with visuotactile streams.
• Topological change and plasticity: tracking assumes local rigidity/smoothness (ARAP), which fails for tearing, cutting, puncturing, yarn-level slippage, and plastic/viscoplastic deformation—expand data and tracking objectives to support topology changes and permanent deformations.
• Material parameter supervision: no ground-truth material properties (e.g., Young’s modulus, damping, density) are provided—add offline mechanical tests per object to enable parameter estimation and physics-grounded evaluation.
• Reconstruction assumptions: per-frame 3DGS emphasizes rendering fidelity without temporal consistency; remaining drift/identity switches are possible—investigate temporally consistent 4D reconstruction and physically constrained dynamic splatting.
• Challenging appearance regimes: robustness to transparent, glossy, dark, reflective, or highly textureless materials is not evaluated—add objects and protocols targeting these failure modes and quantify reconstruction/segmentation errors.
• Severe occlusion handling: residual failures when large regions are obscured for long durations—evaluate additional sensors (e.g., in-hand cameras, depth, NIR), active viewpoint control, or occlusion-aware priors.
• Temporal resolution: 30 Hz capture may miss fast transients and high-frequency vibrations—assess higher frame rates or event cameras and quantify impact on tracking and model learning.
• Action representation sparsity: only 6D wrist pose and gripper openness are used; no velocities, torques, or force commands—study richer action spaces and their effect on action-conditioned models’ faithfulness.
• Foundation-model fairness: 2D video model benefits from large-scale pretraining while 3D models do not—explore pretraining for 3D world models or compare to similarly pretrained 2D/3D baselines for fairer conclusions.
• Action-following reliability: Cosmos sometimes ignores commanded actions during long-horizon prediction—develop and evaluate stronger action conditioning, alignment losses, or action tokenization schemes; report action-following metrics.
• Evaluation metrics: reliance on PSNR/SSIM/LPIPS and Chamfer/track error omits physics-centric metrics—add measures of contact consistency, deformation/strain fields, energy/momentum conservation, and adherence to commanded actions.
• Long-horizon stability: no quantitative analysis of error accumulation over extended horizons—benchmark stability, drift, and compounding error in minute-scale rollouts.
• Contact prediction scope: only binary contact is predicted; no localization, magnitude, or temporal persistence—extend labels and models to predict per-taxel contact, pressure magnitude, and contact patches.
• Synchronization quality: contact prediction uses 36 “synchronization-filtered” views, suggesting broader sync issues—quantify sync accuracy across all views and provide standardized procedures/benchmarks for synchronization robustness.
• Environment interactions: episodes focus on prehensile interactions; contact with supporting surfaces (e.g., table friction, edge catches) and multi-object interactions are underrepresented—include tasks with surface contacts and multi-object coupling.
• Computational scalability: per-frame 3DGS and multi-view tracking cost/time and hardware requirements are not reported—provide processing budgets, throughput, and guidance for labs with limited compute; explore lighter pipelines.
• Sensitivity and ablation analyses: effects of hyperparameters (e.g., λ weights, tactile radius r, number of tracked points M) and segmentation quality on tracking/model performance are unreported—conduct systematic ablations.
• Annotation reliability: 3DGS-derived geometry is used as de facto ground truth—validate annotations on a subset with independent measurements (e.g., marker-based motion capture, structured light scans) to quantify bias and noise.
• Cross-domain generalization: robustness to new labs, lighting, backgrounds, cameras, and different tactile hardware is untested—evaluate domain shifts and provide domain adaptation baselines.
• Planning evaluation: the MPC demonstration lacks quantitative success rates, baselines, latency analysis, and comparisons to alternative planners or models—report standardized planning benchmarks and real-time feasibility.
• Video-model planning: video-based models are not used for planning due to reward design and OOD appearance—investigate 3D proxy extraction from video latents, goal-conditioned diffusion rewards, and appearance-robust conditioning.
• Benchmark breadth: only one action-conditioned video model (Cosmos) is evaluated—add comparisons to other open models (e.g., Vid2World, PAN) and to recent 3D world models as they become available.
• View sparsity robustness: models are trained/evaluated with 41 views; performance with sparse or single-view setups common in practice is unknown—release curated few-view splits and benchmark cross-view sparsity.
• Data split standardization: precise train/test partitions for episode/object generalization are not fully specified in the main text—publish fixed splits, seeds, and protocols for reproducibility.
• Category gaps: dataset excludes fluids, gels, and granular media and most soft–rigid coupled phenomena—extend to non-solid deformables and hybrid systems with appropriate sensing/annotation.

Immediate Applications

The following items describe use cases that can be deployed now using the Deform360 dataset, the markerless visuotactile 3D tracking pipeline, and the benchmarking results reported in the paper.

• Benchmarking and model selection for deformable world models — sectors: robotics, software/AI (R&D), academia
  • Use Deform360 as a standardized benchmark to compare 2D action-conditioned video models versus 3D particle models for deformable dynamics, across frame/episode/object generalization.
  • Actionable guidance: pick 3D particle models when data are scarce (strong structural priors) and video models when large training data are available (better zero-shot visual generalization).
  • Tools/workflows: “Deform360 Benchmark Suite” with tasks for contact prediction, future prediction, and resimulation; reproducible evaluation pipelines with metrics (PSNR/SSIM/LPIPS, Chamfer, track error).
  • Dependencies/assumptions: dataset license availability; compute for training/evaluation; action conditioning for video models may require post-training.
• Markerless visuotactile 3D tracking for lab perception — sectors: robotics, computer vision, academia
  • Apply the paper’s pipeline (3D Gaussian Splatting + multi-view 2D tracking + tactile-regularized 3D lifting) to recover dense 3D trajectories of deformable objects without markers.
  • Use cases: dataset creation for new objects; ground-truth trajectory generation for model training; analyzing contact-induced local deformations under occlusion.
  • Tools/workflows: “Visuotactile Reconstruction Toolkit” combining 3DGS, CoTracker3, and tactile fusion; object-only masking; RANSAC-based multi-view fusion; ARAP and Laplacian regularization.
  • Dependencies/assumptions: multi-view calibration; sufficient camera coverage (the paper uses 41 cameras); GPU compute for 3DGS; tactile sensors measure primarily normal pressure; no-slip assumption near contacts.
• Contact-from-vision models for grippers without tactile sensors — sectors: robotics (automation, logistics), academia
  • Train contact classifiers that infer local contact/no-contact events from RGB and robot actions; reported mean accuracy 88.67% and F1=0.89.
  • Use cases: retrofitting robots lacking tactile sensors with visual contact estimation for skill triggering (e.g., tightening grasp, switch from reaching to manipulation).
  • Tools/workflows: transformer-based encoder taking synchronized video and 6D pose/gripper openness as inputs.
  • Dependencies/assumptions: camera viewpoints with adequate visibility; generalization to new visual domains may require fine-tuning.
• Rapid prototyping of deformable manipulation policies in research settings — sectors: robotics (R&D), academia
  • Use learned 3D world models (e.g., PhysTwin/ParticleFormer/PGND trained on Deform360) within MPC to plan actions for cloth/rope/bag/plush manipulation; the paper demonstrates zero-shot planning on a different robot (xArm) in a different lab.
  • Use cases: in-lab validation for folding, stretching, twisting primitives; testing material-agnostic planning strategies.
  • Tools/workflows: MPC with geometric costs (Chamfer distance) computed on predicted particle states; rendering via 3DGS for visual verification.
  • Dependencies/assumptions: planning stability depends on accurate state estimation; reward design is simpler for 3D models than for video; domain shift can degrade performance.
• Material- and category-aware data curation for foundation model fine-tuning — sectors: software/AI, robotics
  • Curate episodes by material type (1D/2D/3D deformables) to post-train action-conditioned video models or 3D dynamics models for targeted tasks (e.g., rope management vs cloth folding).
  • Tools/workflows: category split loaders; per-episode/ per-object splits for systematic generalization studies.
  • Dependencies/assumptions: action conditioning for video models requires architectural support and consistent action encoding.
• Education and curriculum development in deformable object modeling — sectors: education, academia
  • Course modules on visuotactile perception, deformable tracking, and model-based planning using Deform360 as a turnkey, real-world dataset.
  • Tools/workflows: labs around multi-view calibration, 3DGS reconstruction, contact prediction, and MPC.
  • Dependencies/assumptions: access to compute; simplified subsets of the dataset can reduce resource demands.
• Quality assurance in soft-goods R&D prototypes (lab-scale) — sectors: manufacturing/soft goods, retail R&D
  • Use the pipeline to analyze deformation under prototyped grippers/clamps; measure repeatability and local strain in fabrics/foams at lab scale.
  • Tools/workflows: per-frame 3D recon + particle tracking to report deflection and local metric distortion; visual contact event detection to tie QA metrics to contact phases.
  • Dependencies/assumptions: multi-view coverage and calibration; adaptation needed to industrial lighting/backgrounds.
• VFX/animation research for data-driven deformable motion — sectors: media/entertainment (R&D), academia
  • Leverage high-fidelity multi-view footage and tracked point clouds to train/validate learned deformation rigs or to bootstrap mesh-based animations from real captures.
  • Tools/workflows: lift tracked particles onto meshes via skinning; refine using ARAP constraints.
  • Dependencies/assumptions: licensing for media; adaptation from particles to production mesh pipelines.
• Sensor fusion benchmarks for visuotactile research — sectors: robotics sensors, academia
  • Use synchronized tactile streams and videos to develop/benchmark multimodal fusion architectures; ablation studies on tactile vs visual contributions under occlusions.
  • Tools/workflows: standardized train/test splits; contact/no-contact labels; occlusion partitions for stress testing.
  • Dependencies/assumptions: tactile modality in the dataset measures normal pressure; slip events intentionally minimized.
• Practical guidance on dataset/rig design for new labs — sectors: academia, robotics R&D
  • Replicate a scaled version of the 360° rig and synchronization to build domain-specific datasets (e.g., medical textiles, packaging).
  • Tools/workflows: camera calibration with ArUco grids; time synchronization; lens undistortion workflow.
  • Dependencies/assumptions: budget and space for multi-view arrays; careful calibration and maintenance.

Long-Term Applications

These use cases require further research, scaling, integration, or engineering to be deployable at production scale.

• Commercial deformable manipulation in logistics and services — sectors: logistics/warehousing, retail, home robotics
  • Robots that robustly fold/pack garments, bag groceries, handle bubble wrap, cables, and soft packaging, trained on visuotactile world models and evaluated on Deform360-like benchmarks.
  • Products/workflows: “Deformable Manipulation Skill Library” pre-trained on Deform360; on-robot fine-tuning with limited visuotactile data; autonomous bagging/packing stations.
  • Dependencies/assumptions: domain adaptation from lab to warehouse/home; reduced-sensor rigs (few cameras, no tactile or alternative tactile) must retain performance; safety and throughput requirements.
• Tactile-augmented home/assistive robots — sectors: healthcare, eldercare, assistive tech
  • Dressing assistance (shirts, jackets), bandaging, towel handling using learned deformable world models with visuotactile feedback.
  • Products/workflows: low-profile tactile grippers; on-robot MPC using 3D dynamics; contact prediction from egocentric cameras for safety.
  • Dependencies/assumptions: compliance/safety standards; generalization to varied garments and body shapes; robust tracking under heavy occlusion.
• Video-world-model-based planning for deformables — sectors: robotics (autonomy), software/AI
  • Use action-conditioned video models directly in model-based planning once reward design in image space is reliable (e.g., learned reward models or video-to-3D alignment).
  • Products/workflows: “Video-MPC” with learned perceptual/goal rewards; hybrid pipelines that lift video latents to coarse 3D for geometry-aware constraints.
  • Dependencies/assumptions: improved action adherence and long-horizon temporal consistency; robust reward surrogates; stable post-training on in-domain data.
• Digital twins of deformable products for design and QA — sectors: manufacturing/soft goods, consumer products
  • Create digital twins of fabrics/foams with learned material properties for non-destructive testing, packaging layout optimization, and assembly planning.
  • Products/workflows: material property estimators from visuotactile sequences; parametric simulators calibrated with Deform360-like data; end-to-end CAD-to-robot workflows.
  • Dependencies/assumptions: estimation accuracy under varied lighting/textures; linking particle-space estimates to CAD/mesh material models; scalable capture with fewer cameras.
• Field robotics handling hoses/cables/flexible fixtures — sectors: infrastructure, energy, construction
  • Robots that route cables/hoses, connect soft lines, or manipulate protective wraps in maintenance tasks, leveraging learned deformable dynamics.
  • Products/workflows: zero-shot policies adapted via small on-site datasets; contact-from-vision in low-visibility conditions.
  • Dependencies/assumptions: weather and lighting variability; ruggedized sensors; fine-tuned generalization to long, heavy, and dirty deformables.
• Sim-to-real bridges for deformables using real-world priors — sectors: software/AI, robotics
  • Transfer learned priors from Deform360 to improve differentiable simulators and synthetic data realism, reducing the gap for training policies at scale.
  • Products/workflows: simulator parameter auto-tuning using visuotactile trajectories; learned residual corrections for particle/mesh simulators.
  • Dependencies/assumptions: consistent mappings between tracked particles and simulator states; coverage of target material regimes in training data.
• Standardization and policy/benchmarking frameworks for visuotactile datasets — sectors: standards bodies, research policy
  • Establish data standards for visuotactile collection (calibration, synchronization, annotation quality) and safety/ethics guidelines for human-in-the-loop data.
  • Products/workflows: open protocols for calibration/QA; public leaderboards for deformable world modeling.
  • Dependencies/assumptions: community adoption; sustainable hosting and governance.
• Real-time, online visuotactile tracking with minimal sensors — sectors: robotics products, embedded systems
  • Port the tracking pipeline to fewer cameras and run on embedded GPUs for on-robot state estimation of deformables.
  • Products/workflows: monocular or stereo variants with learned depth priors; sparse tactile arrays; online ARAP tracking.
  • Dependencies/assumptions: substantial algorithmic advances to replace 41-view 3DGS; robustness to motion blur and occlusion.
• Cross-domain foundation models for 3D deformables — sectors: software/AI, robotics
  • Pretrained 3D world models (PointWorld-like) that generalize zero-shot to new deformables and tasks, informed by Deform360-scale real data.
  • Products/workflows: large-scale pretraining on mixed real-sim corpora; adapters for different grippers/robots; unified visuotactile encoders.
  • Dependencies/assumptions: availability of more large-scale datasets; standardized action encodings; compute budgets for large models.
• Autonomous tool design and gripper optimization for soft objects — sectors: robotics hardware, manufacturing
  • Use data-driven deformable simulations to automatically design finger geometries, skins, and compliance to achieve target manipulation performance.
  • Products/workflows: differentiable hardware-in-the-loop optimization using learned dynamics; design-to-print cycles.
  • Dependencies/assumptions: differentiable objectives that correlate with real performance; accurate contact modeling (including tangential slip, currently underrepresented).

Notes on assumptions and dependencies common across applications:

• The dataset uses 41 synchronized cameras and tactile-equipped UMI grippers; practical deployments will need to reduce sensor count or substitute modalities without significant performance loss.
• Tactile signals measure primarily normal pressure and assume localized no-slip; micro-slip is underobserved and data collection avoided slip, which may limit training for tasks with significant sliding.
• 3DGS-based per-frame reconstruction is compute-intensive; real-time or on-edge operation requires algorithmic simplification.
• Action adherence in video world models remains a challenge; robust conditioning and reward design are active research areas.
• Domain shifts (lighting, backgrounds, object appearances) can degrade generalization; fine-tuning or robust architectures may be necessary.

• 3D Gaussian Splatting (3DGS): A rendering-based representation that models a 3D scene with many Gaussian primitives to reconstruct geometry and appearance efficiently. "using 3D Gaussian Splatting (3DGS)~\cite{kerbl_3D_2023}"
• 3D particle models: Dynamics models that represent deformable objects as sets of particles with learned or simulated interactions and transitions. "comparing 2D video models against 3D particle models."
• Action-conditioned: Conditioned on the agent’s actions so predictions depend on planned controls or inputs. "Action-conditioned 2D Video Models."
• Anisotropic Gaussians: Elliptical (direction-dependent) Gaussian primitives with covariance defining different spread along axes. "a set of $K$ anisotropic Gaussians."
• ArUco markers: Fiducial markers used for camera calibration and pose estimation in multi-view setups. "multi-view calibration using ArUco grids"
• As-Rigid-As-Possible (ARAP): A constraint/regularizer that encourages local deformations to be as rigid as possible while allowing global nonrigid motion. "enforces As-Rigid-As-Possible (ARAP) constraints"
• Bimanual: Involving two robot manipulators or grippers simultaneously. "bimanual tactile-equipped UMI grippers"
• Chamfer distance: A bidirectional distance between two point sets used to evaluate geometric reconstruction and tracking accuracy. "the bidirectional Chamfer distance"
• Diffusion Transformer (DiT): A Transformer architecture used to model the diffusion denoising process in latent space for video generation. "modeled by a Diffusion Transformer (DiT)"
• Differentiable simulators: Physics simulators that allow gradients to flow through simulation steps for learning and optimization. "utilize differentiable simulators"
• Extrinsic matrix: A 4x4 matrix describing a camera’s pose (rotation and translation) in the world coordinate system. "extrinsic matrix $\mathbf{E}_n \in \mathbb{R}^{4 \times 4}$"
• Homogeneous coordinates: Projective coordinates (adding a scale component) used to map image points through camera models into 3D. "the homogeneous coordinate of $\mathbf{u}_{n,t}$"
• Intrinsic matrix: A 3x3 matrix encoding a camera’s internal parameters (focal lengths, principal point). "intrinsic matrix $\mathbf{K}_n \in \mathbb{R}^{3 \times 3}$"
• Laplacian regularization: A smoothness prior that penalizes deviations from the neighborhood average, encouraging spatially smooth fields. "we apply a Laplacian regularization to encourage spatial smoothness"
• Latent Diffusion Model (LDM): A generative paradigm that performs diffusion in a compressed latent space for efficiency and fidelity. "follow the Latent Diffusion Model (LDM) paradigm"
• Linear blend skinning: A technique to render/deform geometry by blending transformations (e.g., from particles or bones) to vertices. "via linear blend skinning"
• LPIPS (Learned Perceptual Image Patch Similarity): A perceptual metric that compares images using deep features to assess visual similarity. "Learned Perceptual Image Patch Similarity (LPIPS)"
• Model Predictive Control (MPC): A planning/control method that optimizes a sequence of future actions over a horizon using a predictive model. "Model Predictive Control (MPC)"
• No-slip assumption: A contact assumption that nearby material points move with the contacting surface without tangential slip. "assumes localized no-slip only around activated taxels"
• Point cloud: A set of 3D points representing sampled object surfaces or geometry. "warped point cloud"
• Proprioception: Internal sensing of a robot’s body state (e.g., joint poses, gripper opening), used for action and state estimation. "the robot's proprioception (6D pose and openness)"
• RANSAC: A robust estimation algorithm that fits models while rejecting outliers in noisy data. "using the RANSAC algorithm."
• SO(3): The special orthogonal group of 3×3 rotation matrices representing 3D rotations. "remains in $SO(3)$."
• solvePnP: A pose-estimation method (Perspective-n-Point) that computes camera-to-object pose from 2D–3D correspondences. "using solvePnP with sub-pixel corner refinement."
• Spherical harmonics: Basis functions on the sphere used here to model view-dependent color in Gaussian primitives. "spherical harmonic coefficients encoding view-dependent color"
• SSIM (Structural Similarity Index): An image metric that compares structural similarity between images, often used to evaluate reconstructions. "Structural Similarity Index (SSIM)"
• Tactile sensing: Sensing of contact forces/pressures using sensors on robot hands/grippers to observe interactions. "tactile sensing to observe occluded interactions"
• Taxel: A tactile sensor element (analogous to a pixel for touch) measuring local pressure/force. "activated taxels"
• Unimanual: Involving a single robot manipulator or gripper. "5 unimanual and 5 bimanual episodes per object"
• Visuotactile: Combining visual (camera) and tactile sensing modalities for richer perception. "a massive multi-view visuotactile dataset"
• World model: A predictive model of environment dynamics that forecasts future states given actions for planning and control. "World models enable robots to learn environmental dynamics and plan actions through imagination"
• Zero-shot generalization: The ability of a model to perform on new, unseen objects/tasks without additional training. "its zero-shot generalization capabilities"

Sign up for free to add this paper to one or more collections.