TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models

TAP-VLA: Tactile Annotation Prompting for Vision-Language-Action Models

Introduction

Vision-Language-Action (VLA) models have become the foundation for generalizable robotic manipulation due to their integration of large-scale visual and language pre-training. However, these systems remain fundamentally limited in contact-rich environments; physical interaction features such as contact force, object mass, and friction are not readily deducible from visual input alone. Tactile data, though directly informative of these properties, introduces significant distribution shift when supplied as a novel modality absent in the original VLA training regime. Prior tactile fusion strategies frequently require architectural changes or dedicated encoders, finetuned with limited tactile data, which often degrades performance due to the loss of compatibility with the pre-trained policy.

TAP-VLA presents a minimal-assumption framework for endowing pre-trained VLA models with tactile awareness while maintaining architectural and distributional alignment with pre-training. Tactile shear fields are extracted from visuo-tactile sensors (GelSight) and directly rendered as visual overlays onto multi-view RGB observations, integrating salient physical feedback into the model's existing observation space without increasing input dimensionality or compute.

Figure 1: The TAP-VLA architecture overlays tactile shear fields as spatially-grounded vector annotations on multi-view camera input, here demonstrated for a peg insertion task.

Methodology

Tactile Annotation Pipeline

TAP-VLA operationalizes tactile-visual integration by spatially annotating per-finger shear data onto robot camera views. The annotation procedure comprises:

1. Shear Field Extraction: Optical flow (Farneback) is computed between a reference tactile image (captured per-grasp) and the current sensor image for both gripper fingers, generating a pixel-wise 2D shear field. This field isolates deformations due to external object interaction (removing the reference state eliminates internal grip forces).
2. Mean-Pooling and Lifting: The dense shear field is mean-pooled to a coarse grid, then each 2D vector is mapped to the 3D geometry of the fingertip, enabling geometric projection into camera frames.
3. Annotation Rendering: Vectors are projected from the fingertip coordinate frame into each calibrated camera view and rendered as colored arrows. Color encodes relative shear magnitude, allowing the model to visually parse contact force information in spatial context.

This pipeline is computationally inexpensive, incurs no additional input channels, and maintains strict architectural invariance relative to the pre-trained VLA backbone.

Figure 2: Visual overlay of shear fields for the 'medicine' task with a full bottle, highlighting annotated tactile feedback.

Figure 3: Shear field annotation for the 'medicine' task with an empty bottle, where shear patterns inform the model of differing mass and contact dynamics.

Experimental Protocol

Experiments employ a 7-DoF Franka Emika Panda robot with parallel-jaw gripper and dual GelSight tactile sensors. Three calibrated RGB cameras (two over-the-shoulder D435, one wrist-mounted D405) provide multi-view input. Tasks evaluated include:

• medicine: Mass classification by placement (bottle full/empty).
• balance: Stable placement requiring center-of-mass reasoning.
• gear: Peg-in-gear insertion with alignment via extrinsic forces.
• plug: Socket insertion with prong alignment.

Demonstrations (100 per task) are collected via teleoperation, with model performance compared against three baselines: vision-only, tactile images as auxiliary views, and a dedicated tactile encoder. Success rates are measured over 30 trials per task.

Results

TAP-VLA achieves an overall success rate of 78% across four diverse manipulation tasks, outperforming all baselines (vision-only and other tactile-fusion strategies achieve ≤50%). Notably, for tasks where physical properties are visually ambiguous—bottle mass (medicine) and center of mass (balance)—baseline methods perform at chance, while TAP-VLA demonstrates robust classification and placement capabilities.

This result establishes that direct visual rendering of tactile feedback into the model's observation space is more usable by pre-trained VLA architectures than dedicated tactile encoders, even when the underlying tactile signal is equivalent. On peg/gear and plug insertion, TAP-VLA not only maximizes successful completions but also exhibits error-corrective adaptations through dynamic tactile annotation interpretation.

Analysis and Implications

This work demonstrates that distributionally aligned visual augmentation enables effective deployment of tactile inputs without architectural modification or large-scale tactile pre-training. The primary insight is that for models trained on vast corpora of vision-language data, the mode of tactile integration (visual overlay versus new branch and encoder) is determinant for downstream efficacy. Rendering sparse shear vectors as interpretable, spatially grounded cues leverages existing visual-language backbone capabilities for contact-rich manipulation.

Practically, TAP-VLA’s scheme admits straightforward extension to any 2D-plausible sensory signal, including other tactile modalities, force/torque signals (following projected overlays), or even audio spectrograms. This opens a pathway for rapid exploitation of new sensing signals by LLM-based VLA systems while native tactile datasets and training protocols lag in scale and diversity.

Qualitative analysis supports the claim that the rendered shear fields serve as meaningful cues for both contact location and force direction, guiding both per-task reasoning and error correction strategies.

Limitations and Future Directions

While TAP-VLA is effective for the chosen tasks and sensor setup, several constraints persist:

• Dimensionality Reduction: Mean-pooling and vector overlay discard higher-order tactile features (e.g., slip, surface compliance, local texture).
• Occlusion and Overlap: In visually complex or highly cluttered scenes, overlays risk occlusion of task-relevant visual input.
• Scalability: Policies with multi-fingered hands or dense tactile arrays will require more advanced or hierarchical overlay rendering for interpretability.

Future work should consider adaptive overlay schemes, multimodal attention fusion, or dynamic view selection to mitigate visual masking, as well as systematic studies on transferability to other domains (audio, kinematics, proprioception). Deeper integration with large-scale tactile pre-training remains a critical avenue, especially as such datasets expand.

Conclusion

TAP-VLA provides a demonstrably effective, computationally efficient framework for supplying tactile feedback to pre-trained VLA models through native visual augmentation. By fully sidestepping architectural integration pitfalls, TAP-VLA sets a strong precedent for leveraging multimodal signals in generalist robot learning, with implications for rapid, scalable adaptation to new physical and sensory affordances in autonomous systems operating in contact-rich environments.