OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies

Sign up for free to create and run prompts on this paper using GPT-5.

What this paper is about (in simple terms)

Robots are getting good at “seeing” and copying what they watch in videos, but they still mess up when they need to feel their way through tasks that involve tight fits or careful force, like plugging in a charger or opening a bottle. This paper shows a way to take a robot’s existing “visual plan” and add a small “touch-based correction” on top, so the robot can finish tricky, contact-heavy steps reliably. The approach is called OmniTacTune.

The main questions the paper asks

• How can a robot that learned from videos or demos use touch to fix its mistakes during the moment of contact?
• Can we add touch in a way that works with many different kinds of existing visual policies (not just one specific model)?
• Can the robot learn these touch-based fixes quickly in the real world, without needing a big, special dataset of touch examples?

How the method works (with everyday analogies)

The system treats vision and touch like a human would:

• Vision: Learn “the big picture” by watching—what to do and in what order.
• Touch: During contact, make small, precise adjustments based on what you feel.

OmniTacTune adds touch in two stages:

Stage 1: Warm-up practice

Think of this as the robot getting familiar with a new tool.

• The robot runs its existing visual policy (the plan it learned from videos/demos) to collect examples of what its camera and touch sensor feel like during the task.
• A “tactile encoder” (like a translator) learns to turn raw touch images into useful signals for decision-making.
• A “critic” (like a coach that scores how good an action is) is trained so the robot has good feedback for learning later.
• To save time, they also create extra, realistic touch data from each real attempt (like making extra practice drills from one play), so the robot learns faster.

Stage 2: Learn small touch-based corrections

Now the robot keeps the original visual plan fixed and learns only small “residual” corrections—like tiny steering nudges while driving straight.

• Final action = base visual action + small touch-based correction.
• The correction policy looks at:
  • the robot’s current state,
  • a simple representation of the goal/object motion (like a map of where parts of the object should move),
  • the touch features,
  • and the base policy’s intended short-term moves,
  • then decides how to adjust.
• A safety scheduler limits how big the corrections can be at first, so exploration stays safe.

How the robot knows it’s improving: a multi-sensory reward

Robots learn by getting scores for their actions (reinforcement learning). Designing a good score is hard, so they combine:

• Visual guidance: a generated “object flow” that acts like arrows showing how the object should move, broken into small sub-goals.
• Touch cues:
  • Reward for making stable, proper contact (like a good grip),
  • Penalties for unsafe forces (to avoid damage).
• A final success reward when the task is done.

This mix gives the robot frequent, meaningful feedback, so it can learn quickly with real hardware.

What they tested and what they found

They used a robot arm with a camera and a fingertip touch sensor on four contact-heavy tasks:

• Peg-in-hole: pick up a peg and insert it precisely.
• Charger insertion: plug a small charger into a power strip.
• Cap opening: use a tool to open a bottle cap.
• Box opening: use a lever-style opener to pop open a box cap.

Key results (why this matters):

• Starting from visual-only policies that succeeded only 5–40% of the time, OmniTacTune boosted success to 85–100% after about 40–80 minutes of real-world practice per task.
• It worked across different visual policy types (for example, policies trained from human videos or from teleoperated robot demos).
• It also worked with different kinds of touch inputs (fancy touch-image encoders and simple marker-based signals).
• Compared to other methods that try to learn vision+touch all at once from extra demonstrations, OmniTacTune performed better with the same or less time, because it learns the “last-mile” touch corrections through trial-and-error instead of collecting lots of new paired demos.

Why this is important and what comes next

Implications:

• Practical recipe: Learn general skills from big video datasets (which scale well), then add touch through short real-world practice to handle precise contact. This is more efficient than trying to gather huge touch datasets.
• More reliable robots: Tasks like assembly, plugging, fastening, and tool use become more dependable because the robot can feel and correct itself during contact.
• Flexible and future-proof: The method is “policy-agnostic,” meaning it can plug into many different existing visual policies and different touch sensors.

Limitations and future ideas:

• Real-world practice still needs time, resets, and can wear out hardware—especially delicate touch sensors.
• Future work could automate resets, make training safer, and use world models to create even better practice data.
• Extending to more robot types and touch sensors should be straightforward.

In short, OmniTacTune is a neat way to give robots “good hands” to go with their “good eyes”—they watch to learn the plan, then practice with touch to nail the tricky parts.

Below is a single, concrete list of the key knowledge gaps, limitations, and open questions that remain unresolved in the paper. Each point is phrased to be actionable for future research.

• Quantify and mitigate dependence on generated object-centric flow quality: How do inaccuracies or drifts in keypoint tracking and flow generation (DINOv2/SAM/CoTracker) affect reward shaping and residual learning, especially for small, occluded, or highly deformable objects? Explore uncertainty-aware flow rewards, confidence-weighted shaping, and flow correction loops.
• Robustness under severe visual occlusions and lighting changes: The system relies on a third-view camera and tracked keypoints; evaluate performance and adaptation with wrist-mounted cameras, varying viewpoints, cluttered scenes, and partial/total occlusion of the manipulated object.
• Interface-level policy agnosticism beyond position-space actions: The residual actor assumes additive corrections on 6D delta poses with grasp. Test adaptation to base policies with different action spaces (torque/force control, hybrid impedance, velocity control, variable-length action chunks, language-conditioned or hierarchical policies), and design residuals that handle heterogeneous control interfaces.
• Effects of freezing vs co-adapting the base policy: The base policy is frozen during residual RL. Systematically study joint fine-tuning (e.g., constrained or low-rate updates) versus freezing to understand stability, sample efficiency, and convergence, including catastrophic interference between base and residual.
• Tactile gating design and failure modes: Contact gating uses marker displacement thresholds. Analyze false positives/negatives across sensors, sliding contacts, micro-contacts, and dynamic interactions; compare binary vs continuous gating; explore learned gating from uncertainty or slip detectors.
• Safety under real-world exploration: The approach uses a simple safety penalty and manual resets. Investigate formal safe RL with explicit constraints (force limits, contact duration, slip risk), automatic recovery/reset strategies, and risk-sensitive exploration to reduce hardware wear, especially with fragile visuotactile sensors.
• Automated success detection and reduced human supervision: Terminal success rewards are operator-assigned. Develop robust, task-agnostic success classifiers using multisensory cues to eliminate manual labeling and support long unattended training runs.
• Generalization across embodiments and end-effectors: Results are limited to xArm7 with a two-finger gripper and GelSight Mini. Validate cross-robot transfer (different arms/controllers), different grippers (parallel, suction, soft hands), bimanual manipulation, and wrist/finger-mounted tactile arrays.
• Cross-sensor tactile adaptation at scale: While multiple tactile encoders are tested, all are visuotactile representations. Extend to non-vision tactile modalities (BioTac, force-torque, capacitive arrays, high-frequency pressure sensors), mixed-rate fusion, and cross-sensor normalization without task-specific tuning.
• Online tactile encoder adaptation under large representation shift: The warm-start stage uses critic loss + reconstruction regularization. Compare alternative self-supervised objectives (contrastive, predictive coding, temporal consistency), domain-adversarial adaptation, and explicit drift monitors; quantify their impact on sample efficiency and stability.
• Validity of tactile data augmentation: ControlTac-style augmentation synthesizes tactile images conditioned on contact force. Rigorously validate physical plausibility, calibration to specific sensors, and potential label leakage or overfitting; measure the augmentation’s actual contribution across tasks and sensors.
• Reward shaping sensitivity and auto-tuning: The multisensory reward depends on manual weights and normalization constants. Develop automated weight tuning (Bayesian optimization, meta-gradient methods), task-agnostic normalization, and reward learning from preferences; characterize sensitivity/robustness across tasks.
• Handling base policies with near-zero success: Warm-start relies on autonomous rollouts of weak base policies (as low as 5% success). Investigate bootstrapping when base policy fails to reach near-contact states—e.g., curriculum schedules, auxiliary goal-reaching skills, or exploration priors—and quantify failure cascades.
• Sample efficiency at larger scales and longer horizons: Training times of 40–80 minutes are promising but untested for longer-horizon or multi-stage tool-use tasks. Study scalability to complex, sequential manipulation, multi-contact coordination, and tasks that require hours of practice; compare against model-based RL and world models for planning.
• Theoretical guarantees for residual RL with frozen priors: Provide formal analysis of stability, convergence, and performance bounds when learning residuals over fixed policies under partial observability and non-stationary contact dynamics.
• Robustness to sensor drift and degradation: Visuotactile sensors can change over time (gel wear, illumination shifts). Develop online calibration, drift detection, and robust adaptation strategies; quantify performance degradation and recovery.
• Evaluation breadth and metrics: Current measurement focuses on success rate. Add quantitative contact metrics (peak force, impulse, slip rates), insertion tolerances (millimeter-level alignment), safety/impact metrics, cycle time/latency, and energy consumption; report statistical significance across diverse object instances and materials.
• Multitask and continual residual learning: Residuals are trained per task. Explore shared residuals across tasks, meta-learning for rapid adaptation, and continual learning without catastrophic forgetting; measure transfer to novel tasks with minimal additional practice.
• Flow generator generality and data requirements: The flow generator is fine-tuned per task using human or robot demos. Investigate pretraining a task-general flow model, few-shot adaptation, data-efficient fine-tuning, and robustness to domain shifts (backgrounds, objects, human styles).
• Partial observability and memory: The residual policy appears feedforward. Evaluate recurrent architectures or belief-state critics to handle tactile/visual delays, hysteresis in contact, and short-term memory needs in dynamic tool-use.
• Incorporating force/impedance modulation: Residuals are additive pose corrections. Study residuals that modulate stiffness/impedance, contact compliance, and micro-motions (dithering) to improve delicate insertions and dynamic levering.
• Comparative baselines breadth: Baselines exclude strong model-based or MPC methods with tactile feedback, sim-to-real with contact-augmented simulators, and constraint-aware planners. Include these to contextualize gains and identify regimes where residual RL is most advantageous.
• Object-level generalization and tolerance variation: Peg-in-hole generalization is mainly spatial. Test across varying peg diameters, hole tolerances, materials, surface friction, and chamfers; characterize limits and required adaptation time for each.
• Compute and deployment constraints: Detail and evaluate real-time inference/training latency, GPU/CPU requirements, and edge deployment feasibility; quantify how latency affects contact correction quality.
• Language-conditioned goals and task specification: Residual interface currently uses flow-generated subgoals. Investigate conditioning residuals on language, symbolic goals, or 3D affordances to broaden applicability to open-world tasks and reduce reliance on per-task flow fine-tuning.
• Failure mode taxonomy and diagnostics: Provide a systematic categorization of observed failures (misalignment, premature contact, slip, overforce), link them to sensor/policy causes, and propose targeted remedies (adaptive gating, impedance changes, local search).
• Autonomous resets and self-recovery policies: Beyond safety penalties, develop self-recovery behaviors (back-off, re-approach, keypoint re-localization) and automatic reset routines to minimize human intervention and improve training throughput.

Practical Applications of OmniTacTune

Below are actionable, real-world applications derived from the paper’s findings and methods. Each item lists the sectors it benefits, the product/workflow it enables, and key dependencies or assumptions that affect feasibility.

Immediate Applications

• OmniTacTune “last‑millimeter” calibration for insertion and fastening on factory lines
  • Sectors: robotics, manufacturing, electronics
  • What it does: Rapidly boosts success of existing visual policies for contact-rich steps (connector/plug insertion, press-fit assemblies, screw alignment) via on-robot residual RL in 40–80 minutes.
  • Tools/products/workflows: “Tactile Residual Adapter” add‑on for industrial arms; on-line warm-start routine; visuo‑tactile reward shaping presets per station; safety scheduler for residual magnitude.
  • Dependencies/assumptions: Availability of tactile sensors (e.g., GelSight/DIGIT-class or marker-based), a competent visual base policy that reaches near-contact reliably, safe reset procedure, and a third-view camera or equivalent perception for object-centric flow goals.
• Retrofit upgrade for deployed visual policies in brownfield automation cells
  • Sectors: robotics, system integration
  • What it does: Plug‑and‑play residual correction layer over ACT/Diffusion/vision‑language‑action policies to reduce scrap and downtime when tolerances drift or parts vary.
  • Tools/products/workflows: Policy‑agnostic residual SDK; API wrapper ingesting base action chunks and subgoal keypoints; automated warm-start rollout + tactile encoder adaptation.
  • Dependencies/assumptions: Interface-level access to base policy actions, calibrated visuo‑tactile alignment, brief supervised success labeling for terminal rewards.
• Precision packaging, kitting, and box opening in logistics operations
  • Sectors: logistics, e-commerce fulfillment
  • What it does: More reliable flap opening, tab release, seal manipulation, and tight-fit item insertion using tactile residuals for alignment and slip detection.
  • Tools/products/workflows: Task templates (box opening, blister pack handling); parameterized safety thresholds to prevent over-force; quick per‑SKU calibration routine.
  • Dependencies/assumptions: Robust tactile coverage at contact interface; safe exploration limits; standard reach-to-contact visual behavior in place.
• Tool-use steps in general-purpose manipulation cells
  • Sectors: manufacturing, consumer goods, maintenance
  • What it does: Stable tool engagement and force-reactive execution (cap opening, lever operations, latch release) by learning contact-aware corrections without new demos.
  • Tools/products/workflows: Library of object-centric flow goals for common tools; residual schedulers tuned for dynamic contacts; prebuilt tactile reward terms for “grasp/engage” and “safety.”
  • Dependencies/assumptions: Tool geometry represented in object-flow generator; consistent grasp affordances; safe reset process.
• Lab automation requiring delicate insertions and couplings
  • Sectors: biotech, pharma, research labs
  • What it does: Improves pipette tip attachment, microfluidic tubing insertion, vial cap handling without redesigning visual pipelines.
  • Tools/products/workflows: Lab-grade tactile modules for end-effectors; standardized RL calibration protocols between runs; contamination-safe, force-capped reward shaping.
  • Dependencies/assumptions: Cleanroom-compatible tactile skins/sensors; minimal-contact exploration policies; controlled, supervised training windows.
• Data center operations: connector insertion and cable management
  • Sectors: IT operations, infrastructure maintenance
  • What it does: Reduces bent pins and misalignment during plug insertion (network/power) by adding tactile-aware last-step corrections.
  • Tools/products/workflows: “Connector kit” with geometry priors and flow subgoals; low-force safety policies; roll-out warming at maintenance windows.
  • Dependencies/assumptions: Strict force limits and ESD-safe tactile skins; base navigation/visual reach competency; supervised terminal success flag.
• Quality assurance and probing tasks
  • Sectors: electronics, automotive
  • What it does: Reliable probe alignment and contact, tactile-threshold-based safe re-tries, and reduced false fails during contact verification.
  • Tools/products/workflows: Probe positioning residual modules; multi-sensory reward with gentle-contact bands and safety penalties; automatic replay warm-start.
  • Dependencies/assumptions: Fine-grained tactile depth/marker signal; repeatable fixturing; small action residual bounds.
• Research and education: standardized visuo‑tactile RL benchmarks and teaching modules
  • Sectors: academia, education
  • What it does: A reusable, policy-agnostic pipeline for hands-on courses and research on contact-rich manipulation, sensor fusion, and real-world RL.
  • Tools/products/workflows: Open-source scripts for warm-start, residual learning, and reward shaping; tasks like peg-in-hole and tool use; tactile augmentation utilities.
  • Dependencies/assumptions: Access to tactile sensors and a compatible base policy/checkpoints; lab safety procedures for real-world RL.
• Software packages for integrators and OEMs
  • Sectors: software, robotics
  • What it does: Developer SDKs and cloud/on-prem services to generate object-centric flow goals, run warm-starts, and manage multi-sensory rewards across diverse base policies.
  • Tools/products/workflows: “Visuo‑Tactile Reward Shaping Toolkit,” “Residual Policy Orchestrator,” and “Tactile Encoder Adapter” with AnyTouch2/Sparsh/T3 backends.
  • Dependencies/assumptions: GPU for on-robot training or nearby edge server; stable API from base policy; data governance for storing rollouts.
• Operational policy for on-robot RL in production
  • Sectors: policy/governance, safety engineering
  • What it does: Immediate SOPs for safe residual learning: force thresholds, reset criteria, curriculum schedules, and human-in-the-loop success labels.
  • Tools/products/workflows: Checklists for safety gating, reward bounds, and wear monitoring; KPIs for training time vs. gain.
  • Dependencies/assumptions: Organizational approval for controlled on-line learning; safety interlocks; audit trails for updates.

Long-Term Applications

• Fleet-level, continual tactile residual learning across sites
  • Sectors: robotics, software (MLOps), manufacturing
  • What it could do: Share residual policies and tactile encoder adapters across identical tasks/cells for rapid cross‑site adaptation with federated updates.
  • Tools/products/workflows: Centralized policy registry; telemetry-driven auto‑warm-start; cross-hardware alignment layers.
  • Dependencies/assumptions: Secure, privacy-preserving policy sharing; consistent sensor calibration; versioned safety governance.
• General-purpose home robots that master contact-rich tasks
  • Sectors: consumer robotics, daily life
  • What it could do: Plugging chargers, opening jars, operating appliances with on-device tactile learning layered over foundation visual policies.
  • Tools/products/workflows: Household “tactile practice mode” with safe constraints; recipe-based flow goals; user-approved training sessions.
  • Dependencies/assumptions: Rugged, low-cost tactile skins; autonomous resets and high-reliability safety; tolerant household variability.
• Surgical and medical manipulation with force/tactile adaptation
  • Sectors: healthcare, medical robotics
  • What it could do: Catheter insertion, tissue manipulation, suture handling with tactile‑aware residuals that respect strict safety envelopes.
  • Tools/products/workflows: Certified tactile modules, validated reward shaping with soft constraints; sim-to-real pretraining via visuo‑tactile world models.
  • Dependencies/assumptions: Regulatory approval, traceable safety guarantees, sterile and biocompatible sensors; high-fidelity ground truth for success labeling.
• Autonomous inspection/maintenance for critical infrastructure
  • Sectors: energy, utilities, transportation
  • What it could do: Tactile-guided valve turning, connector seating, panel latch operations in unstructured settings.
  • Tools/products/workflows: Mobile manipulation with on-site warm-start; seasonal re-calibration jobs; tactile thresholds tied to asset protection.
  • Dependencies/assumptions: Weatherproof tactile hardware; reliable base navigation and perception; remote reset/override mechanisms.
• EV charging and robot docking with self-calibrating tactile alignment
  • Sectors: energy, mobility
  • What it could do: Reliable plug/dock engagement across tolerances and wear, using tactile residual policies that adapt over time.
  • Tools/products/workflows: Public or depot chargers with robot arms; periodic re‑learning windows; shared residual artifacts across locations.
  • Dependencies/assumptions: Standardized connector geometries; safety-certified residual bounds; vandal-resistant sensors.
• High-yield battery and semiconductor assembly steps
  • Sectors: manufacturing (battery, semiconductor)
  • What it could do: Force-limited tab alignment, thin-foil/wafer contact tasks with minimal damage via tactile rewards that penalize over-force.
  • Tools/products/workflows: Cleanroom-compliant tactile skins; advanced flow generation with CAD priors; digital twin-informed reward designs.
  • Dependencies/assumptions: Extreme cleanliness and ESD constraints; ultra-low-contact-force sensing; tight integration with MES/QMS.
• Visuo‑tactile world models for safer, cheaper practice before real trials
  • Sectors: academia, software, robotics
  • What it could do: Pre-train tactile encoders and critics with generated visuo‑tactile rollouts, cutting real-world training time and wear.
  • Tools/products/workflows: World-model “pre‑warm‑start” service; model-based rollouts for curriculum design; counterfactual safety testing.
  • Dependencies/assumptions: Accurate contact and tactile simulation or learned generative models; validation pipelines to prevent sim‑to‑real drift.
• Standardization of tactile training protocols and certification
  • Sectors: policy/governance, standards bodies
  • What it could do: Industry-wide guidelines on contact thresholds, residual bounds, and audit requirements for on-robot RL in production settings.
  • Tools/products/workflows: Certification programs for tactile sensors and training procedures; conformance tests for reward shaping and safety.
  • Dependencies/assumptions: Multi-stakeholder buy-in; measurable risk and reliability metrics; harmonization with existing robot safety standards (e.g., ISO/TS).
• Novel tactile sensor products tuned for residual RL
  • Sectors: hardware, robotics
  • What it could do: Robust, modular tactile fingertips with native encoders and augmentation pipelines, optimized for residual learning.
  • Tools/products/workflows: “RL‑ready” tactile modules with onboard feature projection layers; drop-in calibration flows.
  • Dependencies/assumptions: Cost-effective, durable materials; standardized mechanical/electrical interfaces; supply chain maturity.
• Human–robot collaboration with touch-aware compliance
  • Sectors: manufacturing, warehousing
  • What it could do: Robots that adapt residual contact behavior in shared workspaces (e.g., handovers, co-manipulation) while honoring safety constraints.
  • Tools/products/workflows: Mixed-initiative training sessions; tactile gates that suppress residuals near humans; events-based reward signals.
  • Dependencies/assumptions: Reliable human detection; conservative force limits; rigorous HRI safety validation.

Notes on overarching dependencies and assumptions across applications:

• A competent visual base policy is assumed for reaching the near-contact regime; OmniTacTune addresses the last-mile contact correction.
• Training requires brief, controlled real-world interaction (tens of minutes) with safe resets and occasional success labeling.
• Access to tactile sensing (vision-based or marker-based) and object-centric flow generation is required; both must be calibrated to the robot frame.
• Compute availability (on-robot or edge GPU) and integration with base policy APIs are necessary for real-time residual learning.
• Safety policies (force thresholds, residual schedulers, emergency stops) are critical to mitigate hardware wear and risk during on-line learning.

• 6-DoF: Six degrees of freedom describing a 3D pose (3 translations, 3 rotations). "tracks the 6-DoF hand pose under its camera frame${}^{q}\mathbf{T}_{h}(t)$."
• Action Chunking Transformer (ACT): An imitation-learning policy that predicts chunks of future actions. "Action Chunking Transformer (ACT)~\cite{act}"
• AnyTouch2: A pretrained tactile image encoder for extracting features from vision-based tactile sensors. "such as AnyTouch2~\cite{anytouch2}"
• CoTracker 3: A point tracking model used to follow object keypoints across video frames. "CoTracker 3~\cite{cotracker3}"
• contact-rich manipulation: Manipulation tasks that rely on precise contact, forces, and friction. "contact-rich manipulation"
• ControlTac: A tactile data augmentation approach for synthesizing consistent tactile images. "We utilize ControlTac~\cite{controltac}-style SOTA augmentation method"
• DAgger: A dataset aggregation algorithm for imitation learning with expert interventions. "DAgger-style approaches~\cite{dagger}"
• delta poses: Incremental pose updates (relative transformations) used as action outputs. "a sequence of $6D$ delta poses"
• DINOv2: A vision backbone producing robust features for object/keypoint extraction. "DINOv2~\cite{dinov2}"
• Diffusion Policy (DP): A diffusion-based action-generative policy for robot manipulation. "Diffusion Policy (DP)~\cite{dp}"
• DrqV2: A data-augmentation-based reinforcement learning method improving sample efficiency. "inspired by DrqV2~\cite{drqv2}"
• embodiment gap: The difference between human and robot bodies/kinematics that complicates transfer. "reduces the embodiment gap"
• end-effector: The robot’s tool center point or gripper pose relative to the world/robot frame. "robot end-effector poses"
• flow generator: A model that predicts future object motion as flow fields or transformations. "we fine-tune a pretrained flow generator"
• flow-tactile critic: A Q-function conditioned on proprioception, flow features, and tactile features. "bootstrap a flow-tactile critic $$Q_{\eta}(\mathbf{q}_t,\mathbf{z}^f_t,\mathbf{z}^{\tau}_t,\mathbf{a}_t)$$"
• flow-based policy: A control policy guided by generated object flow to produce actions. "Our flow-based policy builds on top of Im2Flow2Act~\cite{im2flow2act}, GenFlowRL~\cite{genflowrl}, and Dex4D~\cite{dex4d}"
• GelSight Mini: A high-resolution vision-based tactile sensor used on the robot gripper. "GelSight Mini"
• human video retargeting: Mapping human-hand trajectories from videos/AR tracking into robot coordinates. "collects visual demonstrations via human video retargeting"
• object-conditioned motion guidance: Task guidance derived from predicted object motion used to steer policies. "an object-conditioned motion guidance for both base policy and residual policy"
• object-centric flow: A representation focusing on the motion of the manipulated object(s). "generative object-centric flow"
• object keypoints: Sparse points on objects tracked over time to compute motion and goals. "a sparse set of object keypoints"
• online residual RL: Real-world reinforcement learning that learns corrective residuals during execution. "online residual RL"
• policy-agnostic: Designed to work across different policy architectures/interfaces. "policy-agnostic real-world RL pipeline"
• proprioception: Robot-internal sensing (e.g., joint states) used as part of the policy input. "where $\mathbf{q}_t$ is proprioception"
• real-to-sim-to-real: A transfer paradigm that adapts policies via simulation between real-world phases. "Prior real-to-sim-to-real approaches~\cite{vtrefine, xsim, lodestar}"
• real-world RL: Reinforcement learning conducted directly on physical robot hardware. "real-world RL"
• replay buffer: A memory storing past transitions for off-policy learning and bootstrapping. "initialize the replay buffer"
• residual correction: Adding corrective actions on top of a base policy’s outputs. "through residual correction"
• residual policy: A policy that predicts adjustments to a frozen base policy’s actions. "tactile residual policy"
• retargeting: Transforming trajectories from one frame/embodiment (human) to another (robot). "retarget it to the robot base frame"
• SAC: Soft Actor-Critic, an off-policy maximum-entropy RL algorithm. "start SAC"
• scheduler: A mechanism that scales residual magnitudes over time for safe exploration. "we add residual scale with a scheduler"
• subgoal: An intermediate goal state that structures long-horizon tasks. "the next sparse subgoal"
• tactile depth image: A depth-like representation derived from tactile sensors indicating deformation/contact. "Here, $\mathbf{D}_t^\tau$ is the tactile depth image"
• tactile encoder: A model mapping tactile images/measurements to compact feature embeddings. "adapt a pretrained tactile encoder"
• tactile markers: Visual fiducials inside a gel-based sensor used to estimate surface deformation. "low-dimensional Tactile Markers with two-layer MLP"
• teleoperation: Human control of the robot (often via VR) to collect demonstrations. "VR teleoperation"
• visuo-tactile: Combining visual and tactile modalities for perception/control. "visuo-tactile feedback"
• vision-language-action models: Models conditioned on visual and language inputs to output actions. "vision-language-action models~\cite{pi05, pi06, openvla}"
• visual base policy: A pretrained visual policy providing motion priors for a task. "visual base policies"
• warm-start: An initialization phase using autonomous rollouts to seed training/critics. "warm-start training"
• world models: Predictive models of environment dynamics used for planning or training. "incorporating world models~\cite{vlaw,unisim,dreamerv3,letacmpc}"
• object-centric multi-sensory reward: A reward function combining object-flow guidance with tactile signals. "object-centric multi-sensory reward"

Sign up for free to add this paper to one or more collections.