TACO: TActile World Model as a Self-COrrector forScalable VLA Post-Training

TACO: Tactile-Aware World Model as a Self-Corrector for Scalable VLA Post-Training

Motivation and Problem Statement

Contact-rich manipulation tasks remain a notable limitation for Vision-Language-Action (VLA) models in robotics. While recent VLA architectures have realized significant generalization capacities, their performance on contact-sensitive stages is impaired by the limited observability of physical interactions from vision alone. Minor perturbations during contact transitions—such as insufficient grip force, slippage, or torque failures—can cause unrecoverable failures that are both localized and difficult to detect with visual modalities. Existing approaches relying on vision-only world models or exclusive human corrective supervision are fundamentally limited: vision-based models synthesize physically implausible rollouts during contact events, and large-scale human interventions are not cost-effective.

The TACO framework directly addresses these problems, introducing a tactile-aware world model to facilitate scalable, autonomous, post-deployment improvement of VLA policies for contact-rich manipulation. By modeling tactile signals as part of the environment dynamics, TACO generates high-fidelity visuo-tactile correction segments that are leveraged for policy refinement, enabling robots to recover from and correct localized contact failures without repeated human annotation.

Framework Overview

TACO employs an iterative Recognize–Imagine–Label loop operating over real-world rollouts (Figure 1). The pipeline involves the following core components:

• Recognition: Identification of failure-adjacent states based on dense progress estimation from both vision and tactile modalities.
• Imagination: Generation of visuo-tactile correction trajectories using a joint video-force denoising diffusion model with temporal RoPE alignment to maintain physical consistency.
• Labeling: Relabeling the imagined corrections with executable actions and progress scores via a unified progress-action model.
• Policy Updation: Incorporation of the generated corrective supervision into VLA post-training using knowledge-insulated tactile adaptation and advantage-conditioned training.

  Figure 1: The TACO framework utilizes an iterative Recognize–Imagine–Label loop for automatic correction data generation in real-world rollouts via a tactile-aware world model and supervised advantage-conditioned post-training.

Tactile-Aware World Model

The core innovation of TACO is the tactile-aware world model, which synthesizes physically plausible correction trajectories from failure-adjacent states. The architecture comprises two principal components: (1) a joint visuo-tactile generation model for future observation synthesis, and (2) a unified progress-action model for dense progress estimation and action labeling.

• Visuo-Tactile Generation Model: Extending a DiT-based backbone initialized with Wan2.2-TI2V-5B, the world model processes both video latent tokens and synchronized 12-D tactile signals (6-DoF force-torque for each finger). Video and force are tokenized, temporally aligned using a task-specific RoPE, and concatenated for bidirectional self-attention, allowing the system to jointly denoise both modalities. Objective function enforces consistent denoising trajectories across video and force channels, facilitating generation of local corrections that are physically viable.
• Unified Progress-Action Model: This module ingests RGB frames and tactile signals, producing 7-DoF action vectors and normalized progress scores per timestep. It leverages a DINOv2-derived visual encoder with spatial grounding and an MLP-based tactile branch. The architecture design allows tactile feedback to guide both corrective action generation and temporal progress estimation, enabling accurate identification of failure-adjacent states and optimal correction strategies.

  Figure 2: The Tactile-Aware World Model generates visuo-tactile correction data via joint denoising with temporal RoPE and produces executable actions from imagined rollouts.

Iterative Correction, Training, and Knowledge Insulation

TACO employs an iterative refinement loop. In each iteration, the policy generates rollouts, which are analyzed to detect failure-adjacent states using progress deltas. At these anchors, the tactile-aware world model imagines corrective trajectories conditioned on vision, force, and language. The unified progress-action model labels every imagined trajectory with actions and progress, and binary advantage labels separate corrective segments from failures. All generated data is pooled with original demonstrations for post-training.

Crucially, TACO integrates knowledge-insulated tactile adaptation: the tactile-action loss is backpropagated exclusively through the action expert and tactile adaptation layers, while the high-level vision-language backbone (e.g., PaliGemma or DINOv2) is frozen. Stop-gradient is applied at the connection, thus protecting pretrained perceptual and semantic priors from deleterious catastrophic forgetting during tactile learning. This is further augmented with advantage-conditioned offline RL, which structurally differentiates between positive recovery behaviors (imagined corrections, demonstrations) and negative failure segments.

Experimental Results

Extensive evaluations were conducted across six contact-rich manipulation tasks—Insert Flower, Wipe Whiteboard, Twist Bottle Cap, Play Xylophone, Toast Bread, and Move Hanoi Rings—on an FR3-based robot platform (Figure 3).

Figure 3: Physical real-world setup used for evaluation: FR3 arm, parallel-jaw gripper with Xense tactile sensors, and an RGB-D camera for workspace observation.

Key empirical findings:

• After two post-training iterations, TACO achieved an average absolute success rate improvement of 44% over the base policy and 32% over a variant lacking knowledge insulation.
• The inclusion of tactile feedback in both imagination and action labeling phases is essential; ablative variants that exclude tactile signals show marked degradation (e.g., success rates drop to 28% and 65%, respectively).
• Scaling the ratio of imagined corrections linearly increased performance, up to 97% success with a real-to-imagined ratio of 1:8, indicating the high utility of imagined recovery data.
• Incorporating imagined corrections exposes the policy to a broader distribution of successful action behaviors, expanding beyond the demonstration manifold and enhancing robustness to execution perturbations (Figure 4).

  Figure 4: Action distribution analysis for Insert Flower task; TACO iteratively expands the policy support, enabling robust recovery and successful generalization.

Generalization studies demonstrate strong adaptation to out-of-domain test conditions (unseen backgrounds, novel objects, shifted object positions), with TACO-trained policies recovering up to 80.5–85.5% success after one adaptation, compared to 13.5–35% for the base model (Figure 5).

Figure 5: Generalization comparison under OOD conditions, showing consistent superiority of TACO-trained policies after a single adaptation cycle.

Qualitative inspection of imagined corrections (Figure 6) confirms that the tactile-aware world model generates physically consistent and effective correction segments at contact transitions, precisely recovering from failed real-world contact interactions.

Figure 6: Visualization of imagined correction segments generated by the tactile-aware world model; bottom: corrections recover contact failure (top) trajectories.

Further failure-case analyses pinpoint failure modes addressed by tactile correction and knowledge insulation (Figure 7). TACO uniquely succeeds in completing contact-rich tasks where baselines either stall or degrade grounding due to overfitting.

Figure 7: Comparative failure case analysis on representative tasks: only TACO achieves robust contact transition recovery.

Theoretical Implications and Future Directions

The TACO paradigm demonstrates that visuo-tactile imagination, coupled with knowledge insulation and offline RL, can autonomously and scalably improve VLA policy performance in contact-rich regimes. The explicit modeling of tactile feedback aligns imagined and real interaction dynamics, circumventing the brittleness of vision-only imagination or demonstration-only supervised learning. The knowledge-insulated adaptation protocol prevents catastrophic forgetting and enables targeted refinement on contact transitions, creating a modular path from foundation models toward robust, real-world deployable agents.

Practically, TACO obviates large-scale manual intervention for labeling or correction, delivering scalable data efficiency and improved policy support with minimal additional annotation, and can be deployed as a real-to-imagine-to-real correction loop in deployed systems.

However, the current implementation of TACO imagines corrections offline rather than at runtime. Integration of online correction generation or closed-loop communication between world model and policy would further increase reactivity, adaptation, and real-world robustness in highly stochastic or rapidly changing settings. Additionally, further work is needed to extend the method to deformable object tasks and highly dynamic interaction regimes.

Conclusion

TACO introduces an effective, tactile-aware world-model framework for scalable post-training of VLA policies targeting contact-rich manipulation tasks. By synthesizing failure-adjacent visuo-tactile corrections and leveraging knowledge-insulated, advantage-conditioned adaptation, TACO surpasses prior methods in sample efficiency, in-distribution and out-of-distribution robustness, and autonomy in real-world correction. These results advance the state of offline world model-based improvement for embodied AI systems and inform future developments in scalable robot policy optimization.