Heterogeneous Tactile Transformer

Sign up for free to create and run prompts on this paper using GPT-5.

Overview

This paper is about helping robots “feel” better. Robots use different kinds of touch sensors to understand what they’re holding or pressing. The problem is that these sensors are very different from each other, so a model trained on one sensor usually doesn’t work on another. The authors introduce a system called the Heterogeneous Tactile Transformer (HTT) and a big dataset to teach it. HTT learns a shared “language of touch” so it can understand and use data from many kinds of touch sensors and help robots do tricky, contact-heavy tasks (like screwing in a toy bolt or lifting soft tofu without squishing it).

Key Questions

The paper asks:

• Can we teach one model to understand touch from very different sensors?
• Will this shared understanding help on many tasks (like recognizing objects, estimating force, or detecting slipping)?
• Does training with paired sensors (two sensors feeling the same event at the same time) make the model better?
• Will these touch features help robots actually perform better on real tasks?
• Can the model handle new sensors it hasn’t seen before?

How It Works (Methods) — In Simple Terms

Robots can have two main types of touch sensors:

• Optical/vision-based sensors (like tiny cameras under soft rubber). These produce detailed images of the contact area but run at camera speeds.
• Array/pressure-based sensors (grids of tiny pressure spots). These produce fast, force-like signals over time, but with less spatial detail.

These two types “feel” differently. HTT is designed to handle both.

Here’s the idea:

• Sensor-specific encoders: Think of these as custom plugs that convert each sensor’s unique signal (images or pressure time series) into a common format.
• Shared transformer trunk: This is a shared “brain” that looks for patterns, no matter which sensor the data came from.
• Pretraining with two self-supervised tricks:
  • Masked reconstruction: Like a jigsaw puzzle. The model hides parts of the input and tries to fill in the missing pieces. For images, it fills missing image patches; for pressure time series, it fills missing time chunks. This teaches it to understand fine details.
  • Cross-modal alignment: Imagine two friends describing the same event in different languages. The model learns to predict hidden parts of one sensor using the other sensor’s data, aligning them into a shared understanding of the same touch event.

To train this well, the authors built a big dataset called HPT (Heterogeneous Paired Tactile):

• 1.6 million synchronized “paired frames” across four sensors: two optical (GelSight Mini, 9DTact) and two array-based (Xela, TAC-02).
• The data comes from a robot doing motions like press, twist, and slide on everyday objects, and from controlled setups that measure real forces and slip.

After pretraining, the model keeps only the useful parts (the encoders and the shared trunk) and uses them for real tasks.

What They Found

The authors tested HTT on several tasks and compared it to other methods.

1. Object classification (which object is being touched?):

• HTT and its simple version (just masked reconstruction) beat training from scratch on all sensors.
• HTT did especially well on the optical sensors, showing that aligning with array sensors can add helpful force-related cues.
• On one array sensor, alignment slightly hurt classification, likely because the optical data is richer and can “pull” the shared space in a way that doesn’t always help the simpler signal. Still, overall performance improved.

2. Force estimation and slip detection (physics-heavy tasks):

• HTT clearly improved over baselines across most sensors.
• It beat optical-only methods, which sometimes got worse than training from scratch, showing that learning with paired force-rich data matters.
• The biggest gains showed up in slip detection, a hard, imbalanced problem where small cues matter. Aligning the two sensor types helped the model catch both spatial details and subtle force changes.

3. Real robot tasks without cameras:

• Toy screw: The robot had to grip and rotate a toy screw through multiple cycles. Using only joint positions failed. Adding raw force (wrench) helped. But using HTT touch embeddings was best, reaching about 95% success. The richer touch features helped the robot regrip reliably through cycles.
• Grasp tofu: The robot had to lift a soft tofu block without crushing or dropping it. HTT reduced slips and improved success over raw force, suggesting it captures contact quality better than a simple 6-D force number.

Importantly, the robot used fingertip sensors not seen during pretraining. HTT still worked, showing it can generalize to new sensors.

4. Simulation tasks (peg insertion and bulb installation):

• HTT improved success rates compared to other methods, again showing its touch features help with precise, contact-rich actions.

Why This Matters

• Shared touch “language”: HTT learns a common way to represent touch across very different sensors. That makes it easier to reuse what a robot learns on one sensor with another sensor.
• Better transfer: The model’s features work across different tasks (recognizing objects, estimating force, detecting slip) and different setups (real and simulated).
• Real-world impact: On camera-free, contact-rich tasks, HTT’s touch embeddings helped robots succeed much more often, especially where precise grip and regrip matter.
• Future-proofing: Because HTT generalizes to new sensors, it can serve as a backbone for future robot hands without retraining everything from scratch.

Final Thoughts and Future Directions

HTT shows a practical way to build general touch understanding for robots by training with paired, heterogeneous sensors. This could speed up progress toward robots that handle fragile, slippery, or complex objects more safely and reliably.

The authors note some next steps:

• Add more sensor types (like magnetic or fluid-based sensors).
• Study pairing sensors within the same family (optical-optical or array-array).
• Link touch signals more precisely in space (which image patch matches which pressure spot), for even tighter alignment.

Overall, HTT and the HPT dataset are promising building blocks for smarter, touch-aware robots.

Knowledge gaps, limitations, and open questions

The following list distills what remains missing, uncertain, or unexplored in the paper and can guide future research:

• Sensor family coverage: pretraining and evaluation only include optical-based and array-based tactile sensors; extension to magnetic-, fluidic-, capacitive-, resistive-, and piezoelectric-based sensors is untested.
• Same-family pairing: HPT collects only cross-family pairs (optical ↔ array); effects of pairing within the same family (e.g., optical ↔ optical, array ↔ array) on alignment and transfer are unknown.
• Spatial registration: paired data are synchronized in time/contact but not geometrically registered; no method is provided to map optical patches to corresponding taxels, limiting fine-grained cross-modal supervision.
• Interaction diversity: pretraining interactions (press, twist, slide) may not cover impacts, rapid transients, multi-contact transitions, complex tool-object interactions, or tasks involving fluids, dust, or deformable objects beyond simple probes.
• Environmental robustness: robustness to lighting changes (for optical sensors), gel wear/aging, hysteresis, temperature/humidity, and long-term drift is not evaluated or modeled.
• Slip labeling imbalance and noise: slip labels are highly imbalanced (incipient slip ≈1.2%) and derived via CUSUM from force signals; how label imbalance and potential label noise affect representation and downstream slip detection remains unclear.
• Force supervision scope: force estimation focuses on 3D force MAE; the role of full 6D wrench (forces + torques), high-force regimes beyond 40 N/14 N, and generalization to different contact compliances is not analyzed.
• Temporal context limits: a fixed 0.2 s window may be too short for long-horizon or slow contact phenomena (e.g., creep, viscoelastic effects); multi-scale temporal modeling and longer contexts are not explored.
• Alignment objective choice: cross-modal alignment uses L2 regression on masked target embeddings; comparisons to contrastive, InfoNCE, CCA-style, cycle-consistency, or optimal-transport objectives—and their impact on stability and transfer—are missing.
• Asymmetric alignment effects: alignment improves optical sensors but slightly degrades Xela performance in classification; mechanisms and mitigations (e.g., modality-specific regularization, curriculum, confidence-weighted alignment) are not studied.
• Gradient blocking design: alignment gradients are blocked at encoder outputs; the consequences of alternative gradient flows (e.g., partial/unblocked, adapter-based) and schedules are not ablated.
• Scalability with many sensors: the use of per-pair predictors P_ij scales quadratically with sensor count; architectures for scalable multi-sensor alignment (e.g., shared predictors, anchor latents, mixture-of-experts) are not investigated.
• Learning from unpaired data: HTT relies on paired data; strategies to leverage abundant unpaired tactile data (e.g., cycle-consistency, domain-adversarial or teacher-student alignment) are not explored.
• Handling asynchronous data: alignment assumes synchronized windows; robustness to sensor-specific latencies, clock drift, and missing frames is untested.
• Sensor instance variability: generalization across different units of the same model (manufacturing variance, calibration differences) is not evaluated.
• Zero-shot/new-sensor adaptation: Sharpa fingertips are handled by reusing a 9DTact encoder; systematic evaluation of zero-shot vs. few-shot adaptation pipelines (e.g., lightweight adapters, LoRA, prompt tokens) across multiple unseen sensors is absent.
• Compute and latency: inference-time latency and resource usage (onboard vs. edge/GPU) are not reported; suitability for high-rate control with array sensors is unclear.
• Normalization assumptions: subtracting a non-contact reference frame per window may be brittle under drift or bias shifts; alternatives (online baseline tracking, robust normalization) are unexamined.
• Interpretability/physical grounding: what physical quantities (e.g., normal/shear force, contact area, friction coefficient, slip rate) the embeddings linearly encode is not probed.
• Contact-geometry coverage: transfer to sharp edges, textured surfaces with fine asperities, very soft/viscoelastic materials, or thin objects is not characterized.
• Multi-finger/multi-contact learning: HPT pairs two opposing sensors; learning coordinated representations with more than two simultaneous contacts or across multi-finger hands is not studied.
• Multimodal fusion: integration of HTT with vision, audio, or language for richer policies is not tested, despite prior works showing benefits of visuo-tactile fusion.
• Policy-learning breadth: real-world evaluation covers two tasks with modest demonstrations; broader benchmarks, comparisons to stronger visuo-tactile baselines, and robustness tests (distractors, variable friction) are missing.
• Safety-aware tactile control: embedding use to prevent damage (e.g., crushing tofu) is anecdotal; explicit safety objectives or constraints leveraging tactile features are not explored.
• Augmentation strategies: tactile-specific augmentations (for optical: illumination, blur, gel deformation; for arrays: noise, dropout, temporal jitter) and their impact on pretraining are not reported.
• Hyperparameter ablations: mask ratios, patch sizes, temporal tokenization choices, and alignment-schedule sensitivity are not systematically ablated.
• OOD generalization: performance on out-of-distribution objects/materials vs. those seen in pretraining interactions is not quantified.
• Public release and reproducibility: dataset, code, and checkpoints are slated for release upon publication; independent validation and reproduction are not yet possible.
• Canonical latent targets: instead of predicting raw masked embeddings, aligning to a canonical physics-aware latent (e.g., latent force/pressure fields) is not explored.
• Beyond pairwise alignment: jointly aligning more than two sensors in the same interaction episode (multiway co-training) and its benefits vs. pairwise training are unexplored.
• Physics priors: explicit contact mechanics or differentiable simulators are not used to regularize or structure the latent space; potential improvements from physics-informed pretraining remain open.

Overview

Based on the paper “Heterogeneous Tactile Transformer (HTT),” the following outlines practical applications that leverage its findings, methods, and innovations (HTT model, HPT dataset, cross-modal pretraining with MAE + alignment, and demonstrated transfer to unseen sensors and manipulation tasks).

Immediate Applications

• Sensor-agnostic tactile perception modules for robots
  • Sectors: Robotics, Manufacturing, Logistics, Consumer Robotics
  • What: Drop-in modules for object classification, slip detection, and force estimation that work across different tactile sensors using HTT embeddings instead of raw signals.
  • Tools/Products/Workflows:
  • A ROS node/SDK that subscribes to tactile devices (optical or array-based), publishes HTT embeddings, and exposes APIs for slip alerts and force estimates.
  • Pretrained HTT checkpoints integrated into imitation learning (e.g., ACT) or reinforcement learning pipelines.
  • Assumptions/Dependencies: Access to the released HTT weights and code; supported sensor drivers/encoders; real-time inference capability on target hardware.
• Tactile policy boosters for contact-rich manipulation
  • Sectors: Robotics, Automation, Warehousing
  • What: Improve success rates for tasks like screwing, cap tightening, insertion, and gentle pick-and-place by replacing/augmenting raw 6-DoF wrench input with HTT embeddings.
  • Tools/Products/Workflows:
  • Integration of HTT embeddings into existing policies (e.g., ACT), as demonstrated by improvements on toy screw and grasping soft objects.
  • Fine-tune downstream policies on small task-specific datasets rather than training from scratch.
  • Assumptions/Dependencies: Availability of per-finger tactile feeds; policy stack that accepts auxiliary inputs; compute budget for real-time embedding extraction.
• Rapid cross-sensor deployment and hardware SKU unification
  • Sectors: Robotics OEMs, System Integrators
  • What: Deploy the same manipulation policies across different tactile hardware (e.g., swapping an optical fingertip for an array-based one) with minimal re-training.
  • Tools/Products/Workflows:
  • Use HTT’s shared trunk + sensor-specific encoders to standardize tactile inputs into a common latent space across hardware variants.
  • A test harness that verifies performance parity after sensor swaps.
  • Assumptions/Dependencies: The new sensor is reasonably similar in signal characteristics to those in HPT or can be encoded via available encoders; minor fine-tuning may still be required.
• Slip detection and prevention in production lines
  • Sectors: Manufacturing, Food & Beverage, Electronics Assembly
  • What: Use HTT embeddings to achieve more robust slip detection under class imbalance and noisy conditions; trigger immediate corrective actions (regrasp, pressure modulation).
  • Tools/Products/Workflows:
  • Deploy an HTT-based slip monitor within a PLC/ROS control loop to adjust grip thresholds or initiate regrip strategies.
  • Assumptions/Dependencies: Sensors placed at relevant contact points; latency acceptable for closed-loop control; calibrated thresholds for different objects.
• Force estimation in occluded or vision-denied environments
  • Sectors: Logistics, Field Robotics, Maintenance
  • What: Improve force estimation (normal/shear) from tactile streams where cameras are unreliable (dust, occlusion, glare).
  • Tools/Products/Workflows:
  • On-arm HTT inference producing per-contact force estimates to inform compliant control and prevent damage.
  • Assumptions/Dependencies: Pretrained encoders suitable for the deployed sensor; controller can consume continuous force estimates.
• Education and prototyping kits for tactile learning
  • Sectors: Academia, EdTech, Research Labs
  • What: Ready-to-use backbone for labs to explore tactile learning without extensive sensor-specific pretraining.
  • Tools/Products/Workflows:
  • Teaching modules using HPT + HTT to illustrate cross-modal alignment, MAE-style pretraining, and policy transfer.
  • Assumptions/Dependencies: Availability of sensor simulators or low-cost tactile sensors; GPU resources for fine-tuning.
• Dataset collection workflow using paired-sensor rigs
  • Sectors: Research, Robotics R&D
  • What: Reproduce or extend HPT-like paired datasets with a Universal Manipulation Interface (UMI) or similar rig to capture synchronized optical and array tactile data.
  • Tools/Products/Workflows:
  • Open-source plans for 3D-printed gripper shells and synchronization scripts; standardized protocols for press/twist/slide interactions.
  • Assumptions/Dependencies: Access to two heterogeneous tactile sensors; time synchronization; careful labeling for downstream tasks.
• Quality control and soft-material handling
  • Sectors: Food Handling, Pharma, Precision Manufacturing
  • What: Maintain grip without crushing delicate items (e.g., tofu, pastries, vials) using HTT-informed controllers that infer contact distribution and slip risks beyond what a single wrench vector can provide.
  • Tools/Products/Workflows:
  • Real-time HTT embedding streams feeding into compliant controllers that modulate force and contact area.
  • Assumptions/Dependencies: Sufficient tactile coverage on grippers; tuning for each product category; hygiene and material compatibility.
• Simulation-to-real tactile transfer
  • Sectors: Robotics, Software Tools
  • What: Use HTT embeddings to align simulated tactile RGB/force fields with real sensors, improving policy transfer for insertion and assembly tasks (as shown in ManiFeel).
  • Tools/Products/Workflows:
  • Pre-train TacFF/TacRGB encoders with HTT’s shared trunk and MAE on simulated data; fine-tune minimally on real.
  • Assumptions/Dependencies: Access to simulation environments producing tactile fields (e.g., ManiFeel); sim-real gap manageable with fine-tuning.
• Multi-finger contact state aggregation
  • Sectors: Robotics, Prosthetics R&D
  • What: Fuse embeddings from multiple fingertips into a consistent contact-state vector for whole-hand control and dexterous manipulation.
  • Tools/Products/Workflows:
  • Embedding fusion layer feeding a high-level controller for grasp stability, regrasp planning, and finger gaiting.
  • Assumptions/Dependencies: Time-synchronized streams; limited inter-sensor latency; calibrated per-finger encoders.

Long-Term Applications

• Tactile foundation models for generalist robot policies
  • Sectors: Robotics, Software Platforms
  • What: Scale HTT-like pretraining across many tactile families (optical, array, magnetic, fluidic) and integrate into vision-language-action (VLA) models or generalist policies (e.g., OpenVLA, Octo).
  • Tools/Products/Workflows:
  • Unified tactile backbones that feed into multi-task, multi-robot policy training pipelines.
  • Assumptions/Dependencies: Larger, more diverse paired datasets across sensor families; standardized APIs; substantial compute.
• Interoperability standards and procurement guidelines for tactile sensing
  • Sectors: Policy, Standards Bodies, Industry Consortia
  • What: Establish standards for tactile data formats, time-synchronization, and cross-sensor alignment tests to ensure plug-and-play compatibility across vendors.
  • Tools/Products/Workflows:
  • Benchmark suites derived from HPT-like datasets; certification processes for “HTT-compatible” sensors and drivers.
  • Assumptions/Dependencies: Broad industry participation; open licensing; alignment on evaluation metrics.
• Surgical and rehabilitation robotics with sensor-agnostic tactile feedback
  • Sectors: Healthcare, Medical Devices
  • What: Safer tissue handling, suturing, and prosthetic control by combining high-rate force cues with high-resolution contact geometry in a unified representation usable across devices.
  • Tools/Products/Workflows:
  • Tactile backbones integrated into teleoperation and shared autonomy systems; slip/force-aware safeguards.
  • Assumptions/Dependencies: Regulatory approvals; sterile-compatible sensors; extensive validation on medical-grade tactile hardware beyond optical/array families.
• Visuo-tactile-LLMs for material/property reasoning
  • Sectors: Robotics, E-commerce Fulfillment, Inspection
  • What: Link tactile embeddings with vision and language to infer material properties (e.g., slipperiness, compliance) and guide manipulation strategies or SKU-specific handling policies.
  • Tools/Products/Workflows:
  • LLMs conditioned on HTT embeddings for planning and instruction following; tactile-language datasets.
  • Assumptions/Dependencies: High-quality tactile-language annotations; robust multimodal alignment; compute for training.
• Cross-sensor virtualization and “touch-to-touch” translation
  • Sectors: Software Tools, Simulation
  • What: Predict equivalent tactile signals across different sensor types so a policy trained with one tactile modality can run with another (or in sim) without retraining.
  • Tools/Products/Workflows:
  • Cross-modal predictors refined into robust translators; developer tools that emulate absent hardware during development.
  • Assumptions/Dependencies: Larger paired datasets with spatial correspondences; improved cross-attention and registration.
• Tactile digital twins and predictive maintenance
  • Sectors: Manufacturing, Asset Management
  • What: Use standardized tactile embeddings to monitor tool wear, sensor drift (e.g., elastomer degradation), and process anomalies, enabling predictive maintenance and QA.
  • Tools/Products/Workflows:
  • Embedding drift trackers; unsupervised anomaly detection on tactile streams integrated into MES systems.
  • Assumptions/Dependencies: Longitudinal tactile data; fleet-scale monitoring infrastructure; thresholds tuned to processes.
• Agricultural robotics for delicate crop handling
  • Sectors: AgriTech
  • What: Gently harvest fruits/vegetables, avoid bruising, and maintain grip in variable outdoor conditions using robust slip- and force-aware tactile policies.
  • Tools/Products/Workflows:
  • HTT-powered grippers capable of modulating force based on contact pattern shifts and incipient slip cues.
  • Assumptions/Dependencies: Outdoor-rated tactile sensors; domain adaptation to dirt/moisture; object variability.
• Assistive and household robots operating without reliable vision
  • Sectors: Assistive Tech, Consumer Robotics
  • What: Perform chores (e.g., dishwashing, laundry, tidying) in clutter/occlusion using tactile-backbone policies that detect slip, infer forces, and generalize across gripper upgrades.
  • Tools/Products/Workflows:
  • Tactile-first manipulation stacks that degrade gracefully when cameras fail (steam, low light).
  • Assumptions/Dependencies: Cost-effective, durable tactile sensors; integrated safety constraints; on-device inference.
• High-DoF dexterous hands and in-hand manipulation
  • Sectors: Advanced Robotics
  • What: Enable reliable in-hand regrasping, finger gaiting, and tool use by fusing multi-fingertip and palm tactile streams into a coherent latent space.
  • Tools/Products/Workflows:
  • Whole-hand controllers operating over unified tactile embeddings for real-time reorientation and slip prevention.
  • Assumptions/Dependencies: Dense tactile coverage; low-latency fusion; improved spatial registration across sensors.
• Safety and compliance enforcement via tactile safeguards
  • Sectors: Policy, Workplace Safety, Insurance
  • What: Require tactile-based slip/force monitoring for robots interacting with humans or fragile goods to reduce accidents and losses.
  • Tools/Products/Workflows:
  • Compliance tests based on benchmark tasks (e.g., slip detection thresholds, force limits) using standardized tactile backbones.
  • Assumptions/Dependencies: Consensus on safety thresholds; field validation; regulatory alignment.

Notes on feasibility across applications:

• The strongest immediate gains occur in robotics domains that already deploy optical or array-based tactile sensors and can consume HTT embeddings without hard real-time constraints.
• Long-term applications generally require broader sensor families, spatially registered cross-modal datasets, larger-scale pretraining, and deeper integration with vision/LLMs, as well as standardization and regulatory processes.

• 9DTact: An optical (vision-based) fingertip tactile sensor that images surface deformation to infer contact. "Pair A (Xela $\leftrightarrow$ 9DTact)"
• ACT policy: An imitation-learning policy that outputs action chunks for robot control. "and train an ACT policy~\cite{Zhao-RSS-23}."
• Array-based tactile sensors: Tactile sensors composed of discrete sensing elements (taxels) that output high-rate force/pressure time series. "array-based tactile sensors \cite{xela,bhirangi2021reskin,tacniq,huang20243dvitac} provide high-rate temporal signals and more direct force or pressure measurements."
• Bidirectional cross-sensor prediction: A training strategy where the model predicts masked representations of one sensor from another in both directions. "During pretraining, it combines masked reconstruction with bidirectional cross-sensor prediction."
• Contact-rich manipulation: Robot manipulation that involves sustained or complex physical contact with objects. "learning contact-rich manipulation policies from diverse tactile data at scale."
• Cross-attention transformer: A transformer module that conditions predictions on another modality via cross-attention. "Cross-attention transformer that predict masked target embeddings from source embeddings and visible target embeddings."
• Cross-modal alignment: Aligning representations from different sensor modalities into a shared space. "per-modality masked reconstruction together with cross-modal alignment between paired sensors."
• Elastomer deformation: Deformation of a soft polymer layer observed by optical tactile sensors to infer contact geometry. "capture detailed spatial and geometric information from elastomer deformation."
• Force/torque (F/T) sensor: A device that measures forces and torques, typically in six degrees of freedom. "A tactile sensor and a $6$-D F/T sensor are mounted on a robot arm;"
• Friction-coefficient time series: A sequence of estimated friction coefficients over time used to detect slip events. "by computing a friction-coefficient time series"
• GelSight Mini: A compact optical tactile sensor from the GelSight family that uses vision to sense contact. "optical sensors (GelSight Mini, 9DTact)"
• Heterogeneous Paired Tactile (HPT) dataset: A large-scale dataset of synchronized, paired tactile signals across heterogeneous sensors for self-supervised learning. "Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors."
• Heterogeneous Tactile Transformer (HTT): A self-supervised framework with sensor-specific encoders and a shared transformer trunk that learns shared tactile representations across sensors. "we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors."
• Incipient slip: The onset stage of slipping at the contact interface, before gross sliding occurs. "yielding three classes: static, incipient, and gross slip"
• Macro-F1: The unweighted mean of per-class F1 scores, used to evaluate performance on imbalanced classification tasks. "slip detection reports the macro-F1 (\%, higher is better) as the dataset is highly imbalanced."
• Masked Autoencoder (MAE): A self-supervised model that learns representations by reconstructing masked parts of the input. "Self-supervised methods based on Masked Autoencoders (MAE) \cite{mae} can learn powerful features from GelSight-style images"
• Mean Absolute Error (MAE): A regression metric measuring the average absolute difference between predictions and ground truth. "Force estimation reports the overall 3D force MAE (in N, lower is better);"
• Page CUSUM change-point detector: A statistical method that detects change points using cumulative sums, used here to label slip events. "apply a two-sided Page CUSUM change-point detector~\cite{page1954continuous}, yielding three classes: static, incipient, and gross slip"
• Patchified: Converted into non-overlapping patches for transformer-based processing. "Data from each sensor is patchified, fed into a sensor-specific encoder, and forwarded to the shared transformer trunk."
• Proprioception: Internal sensing of a robot’s body state (e.g., joint positions), independent of external sensors. "the policy must rely entirely on proprioception and finger-tip tactile sensing."
• Self-supervised pretraining: Training with objectives derived from the data itself (without labels) to learn general representations. "designed to learn a shared, transferable representation from heterogeneous tactile sensors via self-supervised pretraining."
• Sensor-agnostic manipulation policies: Control policies designed to work across different sensors without being tailored to a specific device. "This mismatch makes it difficult to train reusable tactile representations or sensor-agnostic manipulation policies."
• Shared latent space: A common representation space where different sensor modalities are mapped to facilitate transfer and alignment. "aligns sensor representations in a shared latent space"
• Shared transformer trunk: The modality-agnostic central transformer module that processes embeddings from all sensor-specific encoders. "HTT consists of sensor-specific encoders and a shared transformer trunk,"
• Slip detection: The task of identifying whether contact is static, incipient, or sliding using tactile signals. "Slip Detection. Following the same setup as force data collection, we use probes to slide on tactile sensors and derive per-frame slip labels"
• Stop-gradient: An operation that blocks gradient flow through a target tensor to prevent representation collapse during training. "To prevent representation collapse, a stop-gradient $\mathrm{sg}[\cdot]$ is applied to the regression target."
• TAC-02: An array-based tactile sensor with distributed sensing elements for force/pressure measurement. "Pair B (TAC-02 $\leftrightarrow$ GS Mini)."
• TacFF (tactile force field): A tactile modality representing spatial distributions of forces as fields for learning policies. "we evaluate Peg Insertion and Bulb Installation with two tactile modalities: tactile RGB (TacRGB) and tactile force field (TacFF)."
• TacRGB (tactile RGB): Color images from optical tactile sensors used as visual tactile observations. "we evaluate Peg Insertion and Bulb Installation with two tactile modalities: tactile RGB (TacRGB) and tactile force field (TacFF)."
• Taxel-based tactile data: Data from tactile sensors composed of individual tactile pixels (taxels), typically forming an array. "optical-based and taxel-based tactile data"
• Teleoperated demonstrations: Human-operated demonstrations collected to train policies via imitation learning. "We collect teleoperated demonstrations ($20$ for screw, $50$ for tofu)"
• Universal Manipulation Interface (UMI): A modular platform for collecting synchronized robot manipulation data across sensors. "collected using a Universal Manipulation Interface (UMI) setup \cite{chi2024universal,YuS-RSS-25}."
• Vision Transformer (ViT): A transformer architecture that operates on image patches for visual representation learning. "following standard Vision Transformer (ViT)~\cite{dosovitskiy2021vit}."
• Xela: An array-based tactile sensing system (Xela uSkin) with distributed taxels for force/pressure measurement. "Xela uSkin~\cite{xela}"
• Zero-shot adaptation: Applying a pretrained model to a new sensor or domain without additional training. "for this zero-shot adaptation we apply the 9Dtact encoder directly."

Sign up for free to add this paper to one or more collections.