GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Sign up for free to create and run prompts on this paper using GPT-5.

What is this paper about?

This paper introduces GaP, short for Graph-as-Policy. It’s a new way to teach robots to do everyday jobs that change a bit each time, like packing different groceries or plugging in cables at different angles. Instead of one big “black box” AI, GaP uses a team of AI helpers to build a flowchart-like plan made of small, reusable robot skills. The system practices in a simulator (like a safe video game for robots), improves itself, and then runs the refined plan on real robots.

What questions were the researchers asking?

They focused on “Variational Automation” (VA): tasks that are repeated over and over, but with real variation in object shape and position. Think of making many sandwiches with different breads and fillings, not just repeating the exact same motion every time.

In simple terms, they asked:

• Can a team of AI coding agents build reliable, understandable robot plans that handle variation?
• Can robots practice in simulation to improve their plans before doing the job for real?
• Does this approach beat popular end-to-end robot AIs and traditional planning systems on real tasks?

How did they do it?

They built GaP, which works like a coach guiding a team:

• A flowchart for robots: GaP represents a robot’s plan as a directed graph—a flowchart where each box is a small, well-defined “skill” (like “find the handle,” “plan a path,” or “close the gripper”) and arrows show what happens next. This makes the robot’s behavior clear and easy to debug.
• A library of skills: MORSL (Modular Open Robot Skill Library) is a toolbox of 51 ready-to-use skills for seeing, planning, and moving. Think of it like having LEGO pieces you can snap together to build many different robots’ behaviors.
• A team of AI helpers (multi-agent harness): Different AI agents take on different roles:
  • An Orchestration Agent breaks the big task (like “make popcorn”) into steps (“turn on stove,” “pick up pan,” “place pan,” “turn off stove”).
  • Skill Agents pick the right LEGO pieces (skills) and wire them into small sub-flowcharts.
  • GaP stitches these sub-flowcharts into one working plan.
• Practice and self-improvement (simulation rehearsal): Before trying on a real robot, GaP runs many practice sessions in a physics simulator (Isaac). It: 1) Generates many varied scenes (objects moved around, different angles, etc.). 2) Tries the current flowchart in parallel across these scenes. 3) Analyzes failures (for example, the gripper missed the handle, or the motion wasn’t safe). 4) Updates the flowchart: swap skills, tweak parameters, adjust connections. This loop repeats until performance levels off.
• How to think about the technical terms:
  • Variational Automation (VA): Repeating a job that changes in small but important ways (different positions, sizes, or orientations).
  • Computation graph (flowchart): A clear map of steps and data passing between them.
  • Skills: Small, reusable actions or tools (like functions) with defined inputs/outputs.
  • Agentic coding: Letting AI coding assistants write, test, and refine code under guidance.
  • Simulator: A realistic practice arena where robots can try risky things safely.
  • Throughput: How many successful tasks per hour (mixing success rate and speed).

What did they find?

They tested GaP on 8 benchmarks: four in simulation and four in the real world. Tasks included:

• Fulfilling grocery orders (pick the right item and place it in a basket)
• Packing multiple grocery items
• Making popcorn (turn on a stove, pick up a pan, place it, turn off stove)
• Plugging and unplugging USB-C cables
• Washing crates with two robot arms (simulation)

Here are some key results from many trials:

• Grocery tasks in simulation: GaP stayed strong even when objects were shuffled, moved within a 20×20 cm area, or baskets were swapped. It kept success around 95–99%, while popular end-to-end robot AI models often dropped to 10–43% under those variations. A classical planner (TipTop) also struggled on many instances.
• Helping other AIs do better: When GaP first moved the camera and arm to a good vantage point and then handed off to an end-to-end model, those models improved a lot. In other words, GaP “set up the shot,” bringing the scene back to what those models expect.
• Real-world grocery tasks:
  • Fulfill one item: GaP succeeded 25/25; a strong baseline planner did 8/25.
  • Pack multiple items: GaP 28/30.
  • Time per pick-and-place: about 67 seconds with GaP vs 95 seconds for the baseline planner.
• Make Popcorn:
  • In simulation, GaP improved from about 33% to 94% after self-learning.
  • In real tests, it achieved 18/20 successes (90%). Failures were due to occasional mis-grasps or motion issues, which are understandable and fixable.
• USB-C cable insertion (real):
  • 121/130 successes (93%) across different angles, positions, and goal instructions (insert in order, odd ports, etc.).
  • About 30 seconds per insertion.
  • GaP used a mix of vision, force sensing, and careful motion steps, and it plugged into ROS (a common robot software system) cleanly.
• Washing crates with two arms (simulation):
  • GaP: 143/150 successes (95%), average 179 seconds per cycle.
  • Hand-engineered expert flowchart: 148/150 (99%), 176 seconds per cycle.
  • Over 3 hours, GaP achieved nearly the same throughput as the expert design.

Why this is important:

• GaP’s plans are interpretable (flowcharts) and robust (handle variation).
• It can automatically improve itself before touching real hardware.
• It can mix classic tools (like ROS) with modern AI models.

What does this mean going forward?

GaP shows a practical path between two worlds:

• The reliability and clarity of traditional engineering (explicit flowcharts with safety checks).
• The flexibility of modern AI (learning, vision-LLMs, and code-generating agents).

This could reduce the human effort needed to set up and maintain robots for jobs that vary day to day, like warehouse picking, café prep, factory kitting, and datacenter cable handling.

Limitations and future work:

• Speed: It’s not yet at industry-level speed (often around 7 seconds per item). They need to cut down on vision calls and motion-planning time.
• Task diversity: Most tested tasks are “quasi-static” (not soft or fast-moving). More work is needed for deformable objects, dynamic targets, and complex forceful manipulation.
• Reliability: While GaP outperforms many baselines, there’s still room to reach “always-on” industrial reliability.

Overall, GaP provides a promising way to build robot solutions that are both smart and understandable: a team of AI helpers assembles a clear plan from tested skills, practices safely, and then performs reliably in the real world.

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of gaps and open questions that remain unresolved and could guide future work:

• Benchmark scope and diversity
  • Limited to 8 tasks with a strong bias toward quasi-static, tabletop pick-and-place; only one task (cable insertion) uses force sensing and none address deformables, moving targets, or dynamic interactions. How does GaP extend to deformable objects, non-prehensile manipulation, or high-speed/dynamic tasks?
  • Most real-world evaluations use a single-arm Franka and one UR5 setup; generalization to other robots (mobile manipulators, parallel jaw vs. suction vs. multi-fingered hands), end-effectors, and sensing suites remains untested.
  • Long-horizon, multi-stage industrial processes (dozens/hundreds of steps with recovery) are not evaluated. How does the graph scale with very long horizons, branching, and persistent state?
• Reliability and throughput
  • Reported success rates are promising but not at “industrial” levels; no statistically rigorous confidence intervals or stress tests under adverse conditions (lighting changes, partial occlusions, clutter, reflective/transparent objects).
  • Throughput is far below 500 units/hour; there is no detailed profiling of bottlenecks (e.g., LLM/VLM calls vs. motion planning vs. perception) or concrete strategies to reduce cycle time.
  • Real-world persistence is only evaluated in short runs (tens of trials); there’s no evidence of 24/7 operation, fault recovery, or maintenance over shifts/days.
• Assumptions about known workcells and objects
  • GaP assumes a known, fixed workcell and object set with 3D models and calibrated sensors. How robust is the system to calibration drift, sensor relocation, camera failure, or the introduction of new SKUs without CAD?
  • No mechanism is described for automatic belief space estimation from real data (e.g., logging object pose distributions over time) or for continual updating of the belief model $\mathcal{B}$.
• Simulation-to-real (sim2real) transfer
  • Internal rehearsal relies on Isaac simulation; no quantitative analysis of the fidelity required (contact/friction/compliance) to achieve reliable sim2real transfer across tasks.
  • No ablation of domain randomization levels, physics parameters, or simulator biases; unclear how sensitive graph refinements are to simulation inaccuracies.
  • Self-learning is demonstrated on one task (Make Popcorn) but without a systemic comparison against baselines “with vs. without” rehearsal across multiple benchmarks.
• Self-learning algorithm and optimization procedure
  • The process for Graph_Update (search over graph structure and parameters) is not formalized or reproducible (e.g., search space definition, proposal strategies, acceptance criteria, regularization). How to ensure monotonic improvement or avoid catastrophic regressions?
  • Stopping criteria are vaguely defined as a “plateau”; no convergence analysis, early stopping rules, or performance guarantees.
  • Failure analysis (Analyze_Failure) is underspecified: What metrics, thresholds, or heuristics are used to localize causes (perception vs. planning vs. control)? How robust is it under noise?
• Multi-agent harness design and robustness
  • The orchestration/skill agents and their message-passing protocols are not standardized or evaluated against alternative designs (e.g., number of agents, role specialization, scheduling policies).
  • No quantitative robustness analysis against LLM hallucinations, constraint violations, or “cheating” in multi-agent settings; no formal guardrails beyond type/edge checks.
  • Compute and monetary costs (LLM/VLM calls per task, wall-clock time to synthesize/iterate graphs) are not reported; sample and compute efficiency of self-learning remains unknown.
• MORSL (skill library) coverage and evolution
  • Only 51 initial skills are provided; there is no coverage analysis indicating which skill-gaps limit task classes (e.g., force-torque primitives beyond insertion, tactile/visual servoing, deformable manipulation, safety interlocks).
  • No method to autonomously acquire, verify, or benchmark new skills, or to retire/replace underperforming ones. How to curate and version-control MORSL as tasks grow and hardware diversifies?
  • Interfacing with non-ROS or proprietary industrial controllers is shown only in a specific case; general strategies for heterogeneous middleware integration are not laid out.
• Graph representation, verification, and runtime
  • Although type/edge validation is applied, there is no formal verification for safety, deadlocks, race conditions, or real-time constraints in the execution graph.
  • Parallel execution is mentioned but not evaluated: How are thread-safety, deterministic scheduling, and resource contention handled at runtime?
  • The edge-device interpreter is not characterized in terms of latency, determinism, portability, or failure modes.
• Safety and compliance in real operation
  • No formal safety layer is described (e.g., virtual fixtures, dynamic collision checking under perception uncertainty, force limits, or human-proximity rules), beyond the use of motion planners in some tasks.
  • No procedures for online anomaly detection, fault recovery, or safe fallback behaviors when graphs fail mid-execution.
• Integration with model-free policies
  • Handover strategies that “bring VLA inputs into distribution” are promising but ad hoc; there’s no systematic method for selecting handover points, gating conditions, or fallback between GaP and VLA policies across tasks.
  • Only two VLAs are tested; it’s unclear how general the gains are across other policy families, different observation/action spaces, and closed-source deployments.
• Evaluation fairness and completeness
  • Some baseline comparisons are acknowledged as unfair (e.g., CaP-X without environment knowledge); there’s no controlled study ensuring consistent priors and resources across methods.
  • TipTop failures are attributed to planner infeasibility; it remains unclear whether alternative configurations (e.g., different grasps/planners) would alter outcomes.
• Handling uncertainty and partial observability
  • Although a belief space $\mathcal{B}$ is defined, there is no explicit belief updating or POMDP-style reasoning during execution; how to make the graph robust to uncertain perception, occlusions, and delayed observations?
  • No use of probabilistic confidence scores (from detectors/segmenters) to branch control logic or adapt thresholds.
• Generalization and adaptability
  • How does GaP adapt when object sets expand, task goals change, or new constraints are introduced post-deployment? Is there a safe, on-robot online learning loop that respects industrial safety constraints?
  • No mechanism for detecting distribution shift (e.g., drift in object appearance or workspace) and triggering self-rehearsal or graph updates autonomously.
• Metrics, logging, and reproducibility
  • Detailed logging schemas, telemetry, and evaluation protocols (variance, CI, failure taxonomies) are not described, hindering reproducibility and long-term monitoring.
  • The paper references a project website/code but lacks a standardized recipe for reproducing benchmarks, including asset versions, seeds, and hardware configurations.
• Task decomposition quality
  • The orchestrator’s task segmentation quality is not measured; no analysis of how decomposition errors impact performance or of techniques to refine segment boundaries over time.
• Parameter sensitivity and tuning
  • Many thresholds (e.g., force/position limits in cable insertion) are fixed; there is no adaptive tuning or Bayesian optimization to adjust parameters per environment or over time.
• Comparison with alternative structured policies
  • No head-to-head evaluation against behavior trees, state machines, or other graph-based methods with LLM-assisted authoring to isolate the benefits of GaP’s specific graph formalism.
• Security and deployment concerns
  • There is no discussion of sandboxing, permissions, or cybersecurity for agent-generated code/graphs interfacing with industrial systems (e.g., ROS networks, PLCs).
• Human-in-the-loop tooling
  • The framework lacks interfaces for expert oversight: interactive editing, constraint injection, approval gates, and audit trails to manage risk during deployment and updates.

These gaps point to actionable next steps: broaden benchmarks and sensing modalities; formalize and measure the self-learning and verification loops; profile and optimize runtime; develop robust safety layers; quantify sim2real dependencies; and build mechanisms for continual adaptation, logging, and human oversight in industrial settings.

Immediate Applications

The following applications can be piloted or deployed now using GaP’s graph-structured multi-agent harness, MORSL skills, ROS-compatible execution, and self-learning rehearsal in simulation (e.g., Isaac):

• Logistics/retail: order fulfillment cells for variable SKUs
  • Use case: Pick target items from a variable workspace and place into bins (e.g., grocery order fulfillment and packing).
  • Sector: Logistics, retail, e-commerce micro-fulfillment.
  • Tools/workflow: GaP orchestration agent; MORSL perception (e.g., SAM/OWL-ViT), grasping (e.g., GraspGen), cuRobo motion; ROS-based graph interpreter; internal rehearsal with Isaac; optional “VLA-boosting” front-end that centers wrist camera and hands off to a VLA.
  • Products that might emerge: “GaP Order Cell” turnkey kits; graph authoring/validation UI; prebuilt skill packs for common SKUs and trays.
  • Assumptions/dependencies: Known workcell and calibrated cameras; object models or reliable open-vocabulary perception; singulated or sparse clutter; cycle times currently ~60–90 seconds per pick (throughput constraints); on-edge GPU for inference and planning.
• Electronics/datacenter assembly: connector insertion workflows
  • Use case: Align–touch–insert–extract for USB-C and similar press-fit connectors under pose variability.
  • Sector: Electronics assembly, datacenter operations, test/validation lines.
  • Tools/workflow: ROS-compatible nodes (align, touch, insert, extract); GaP graph generation and validation; force–torque thresholds and probing grids; self-learning to tune thresholds and approach vectors.
  • Products that might emerge: “GaP Connector Insertion Module” with force sensing; reusable ROS subgraphs for standard connector families.
  • Assumptions/dependencies: Force–torque sensing and compliant control; constrained tolerances/jigs; high-quality calibration; safety interlocks; limited occlusion during alignment; throughput ~30s/insert (current).
• Food and beverage: fixed-station appliance operation and utensil handling
  • Use case: Toggle knobs/buttons, place cookware on burners, basic station tasks in controlled kitchens (e.g., “Make popcorn” demo).
  • Sector: Food service, ghost kitchens, R&D kitchens.
  • Tools/workflow: MORSL knob-turning and grasping skills; perception for handle/knob localization; simulation rehearsal for placements and offsets; ROS execution.
  • Products that might emerge: “GaP Appliance Task Packs” (knob/button actuation, utensil placement) with safety gating.
  • Assumptions/dependencies: Safety-certified workspace; known appliance geometry; heat/fire safety protocols; slower cycle times acceptable; environment kept static and clean.
• Cooperative material handling: crate lift/flip/place cells (simulation-backed)
  • Use case: Bimanual cooperation for bin/crate handling in washing or staging workflows.
  • Sector: Food packaging, post-harvest, industrial cleaning.
  • Tools/workflow: Coordinated two-arm graphs; grasp planners for narrow slits/handles; Isaac rehearsal across pose distributions; ROS execution.
  • Products that might emerge: “GaP Bimanual Handling Cell” templates; throughput dashboards reporting success/cycle time.
  • Assumptions/dependencies: Two well-calibrated arms; reliable fixture geometry; mostly rigid objects; current cycle time ~3 minutes (simulation results); limited, bounded pose variation.
• Enhancing existing VLA deployments with GaP pre-grasp staging
  • Use case: Improve VLA success by first centering camera and moving to a standardized pre-grasp pose before handing over to the VLA.
  • Sector: Any VLA-enabled manipulation setting with positional off-distribution failures.
  • Tools/workflow: GaP graph stages (interactive perception + linear Cartesian motion) as front-end; VLA policy plug-in node; logging for handover conditions.
  • Products that might emerge: “GaP VLA-Stabilizer” middleware; diagnostics that flag when VLA is out of distribution.
  • Assumptions/dependencies: VLA supports API handoff; wrist camera and short-range perception; known workspace layout.
• Systems integration and commissioning: faster cell setup with formalized graphs
  • Use case: Replace bespoke scripts with interpretable graphs that pass static checks and validation; automate rehearsal loops to improve success/throughput before floor deployment.
  • Sector: Robotics integrators, OEMs, line builders.
  • Tools/workflow: GaP authoring harness; graph validators (type/edge checks); internal simulation for iterative graph updates; on-robot interpreter for deployment.
  • Products that might emerge: “GaP Studio” for graph authoring/versioning; CI/CD for robot graphs; MORSL registry and marketplace.
  • Assumptions/dependencies: Accurate digital workcell description; simulator assets; procedures for transferring tuned parameters to real cells.
• Academic research and education: reproducible VA benchmarks and curricula
  • Use case: Teach modular, interpretable robotics with multi-agent code generation and self-learning; replicate open VA benchmarks; expand MORSL.
  • Sector: Academia, teaching labs, robotics research groups.
  • Tools/workflow: Open benchmarks and skills; GaP harness; Isaac scenes; graph evaluation metrics (success, throughput).
  • Products that might emerge: Course modules, student projects, public MORSL contributions.
  • Assumptions/dependencies: Sim and lab robots; instructor familiarity with ROS and calibration; access to LLM/VLM APIs.
• Quality, audit, and safety documentation via interpretable graphs
  • Use case: Present human-readable computation graphs for safety review, change control, and compliance audits.
  • Sector: Regulated manufacturing, food processing, medical devices.
  • Tools/workflow: Graph export with node I/O signatures; change logs between versions; runtime logs mapped to graph nodes.
  • Products that might emerge: Audit-ready graph reports; approval workflows linked to graph diffs.
  • Assumptions/dependencies: Organizational processes for change control; traceable logging; role-based approvals.
• Daily-life/makerspaces: structured automation for hobbyist cells
  • Use case: Simple pick-and-place or panel actuation in maker labs or rehab/assistive environments with fixed setups.
  • Sector: Maker communities, assistive tech pilots.
  • Tools/workflow: Prebuilt GaP templates; low-cost cameras; small skill sets; ROS execution on compact arms.
  • Products that might emerge: Hobbyist “GaP Kits” for desktop arms.
  • Assumptions/dependencies: Very controlled environments; tolerance for slow cycle times; limited safety envelope.

Long-Term Applications

These applications require further research, scaling, or engineering to meet industrial throughput, robustness, or regulatory standards:

• High-throughput VA cells approaching 500 UPH
  • Use case: Achieve ~7-second cycle times in bin picking, kitting, or induction with significant variability.
  • Sector: Warehouse automation, parcel handling, electronics kitting.
  • Tools/workflow: Compiled/optimized graph execution; caching; minimal VLM calls; faster IK/planners; GPU acceleration; predictive pre-positioning.
  • Dependencies: Real-time ROS2 execution; deterministic timing; tight HW–SW co-design; ultra-fast perception with low-latency networks.
• Deformable and dynamic-object manipulation
  • Use case: Bagging, cable routing, textiles, food preparation, and tasks with moving conveyors or human–robot interaction.
  • Sector: Food processing, apparel, consumer goods, automotive wiring.
  • Tools/workflow: Tactile/force-rich skills; deformable-object simulators; online parameter adaptation; richer failure taxonomies.
  • Dependencies: High-fidelity sim-to-real for soft materials; robust compliance control; expanded MORSL for deformables.
• Mobile manipulation and multi-cell orchestration
  • Use case: Robots navigating between stations to perform VA tasks across areas (receiving→kitting→packing).
  • Sector: Warehouses, hospitals, flexible manufacturing.
  • Tools/workflow: Global task graphs coordinating mobile base and arm; scheduling; cross-cell calibration; fleet-level planning.
  • Dependencies: Reliable localization; dynamic obstacle handling; operations planning; standardized interfaces across cells.
• Automatic skill discovery and library growth
  • Use case: Continually learn reusable skills from rehearsal and real runs, expanding MORSL with verified skills.
  • Sector: All robotized industries; integrators.
  • Tools/workflow: Skill mining from execution traces; unit tests for new skills; semantic signatures; risk-scored publishing.
  • Dependencies: Data governance/MLOps; robust verification pipelines; community curation and versioning.
• Formal safety layers and verifiable guarantees
  • Use case: Enforce safety constraints at the graph level with runtime monitors and formal verification for critical tasks.
  • Sector: Medical devices, food processing, human-in-the-loop environments.
  • Tools/workflow: Contracts/types for safety envelopes; reachability analysis; runtime shields; anomaly detection aligned to nodes.
  • Dependencies: Formal methods integration; certified components; standardized safety schemas for skills and graphs.
• Cross-site template deployment and auto-calibration
  • Use case: Instantiate a “graph template” across facilities with semi-automated calibration and belief-space tuning.
  • Sector: Distributed manufacturing, retail chains.
  • Tools/workflow: Calibration subgraphs; automated belief distribution estimation; transfer-learning of parameters.
  • Dependencies: Consistent fixtures; robust calibration aids; site-specific sensing differences; shadow mode trials.
• Digital twin pipelines and enterprise integration
  • Use case: Link GaP graphs with PLM/MES/ERP and digital twins for change simulation and ROI impact analysis.
  • Sector: Enterprise manufacturing and supply chain.
  • Tools/workflow: Bidirectional connectors from graphs to digital twins; order-driven goal generation; throughput and cost dashboards.
  • Dependencies: Standards-based APIs; asset model alignment; data security and access control.
• Human-in-the-loop graph authoring and governance at scale
  • Use case: Multi-role workflows (author, reviewer, tester) with gated promotion to production; audit trails.
  • Sector: Large integrators and OEMs.
  • Tools/workflow: Approval workflows; rollback; differential testing; policy constraints embedded in harness.
  • Dependencies: Organizational processes; training; separation of duties; toolchain maturity.
• Regulatory standards for Variational Automation cells
  • Use case: Define VA-specific certification (known workcell, bounded belief space, interpretable graph) and procurement criteria.
  • Sector: Regulators, industry consortia, insurers.
  • Tools/workflow: Standardized graph schemas and reporting; benchmark suites; certification audits.
  • Dependencies: Industry consensus; compliance frameworks; third-party test labs.
• Robust sim-to-real self-learning loops with real-world feedback
  • Use case: Continuous improvement using real runs to refine graphs, skills, and thresholds, with safe deployment gates.
  • Sector: Any persistent automation setting.
  • Tools/workflow: Data capture mapped to graph nodes; failure root-cause analysis; confidence scoring; staged rollout.
  • Dependencies: Data infrastructure; change management; safety gating; high-fidelity simulators and domain randomization.
• Consumer/assistive robotics for daily living
  • Use case: Home assistance tasks (kitchen, laundry, restocking) with modular graphs and self-improvement.
  • Sector: Healthcare, eldercare, consumer robotics.
  • Tools/workflow: Compact skill sets; home digital twins; safe manipulation in proximity to people.
  • Dependencies: Affordable, safe hardware; robust perception in clutter; strong safety/regulatory compliance; end-user configuration tools.

Notes on feasibility across applications:

• Core assumptions (per VA): stationary, known workcell; bounded object sets and pose distributions; calibrated sensors and robots.
• Dependencies that commonly impact deployment: availability of accurate 3D models or strong open-vocabulary perception; simulator fidelity for rehearsal; on-robot compute; coverage and quality of MORSL skills; safety and throughput requirements; reliability of LLM/VLM agents (steered by harness and validators).

• Agentic coding: Using autonomous LLM/VLM agents to synthesize, run, and refine software or policies. "Agentic coding is new and advancing rapidly, but remains prone to hallucinations and constraint violations."
• Articulated entities: Objects with joints and kinematic degrees of freedom (e.g., doors, knobs) with bounded motion. "articulated entities $\{\kappa_j\}$ (e.g., drawers, knobs) with defined kinematic joint limits $[\theta_{\min}, \theta_{\max}]$."
• Belief space: The probability distribution over possible task/world states (e.g., object poses) used to sample task instances. "{$\mathcal{B}$, Belief Space:} Describes the distribution of objects and poses in instances of the task."
• Bimanual: Involving coordinated control of two manipulators/arms. "coordinated bimanual policies whose reliability and throughput approach those of a manually hand-tuned baseline."
• Blox-Net: A framework that leverages physical experiments to improve LLM-generated task plans. "Building on the structure of Blox-Net~\cite{goldberg2025bloxnet}, which uses physical experiments to improve LLM-generated task plans"
• Cartesian linear motion planning: Planning straight-line end-effector trajectories in Cartesian space. "(*) {Cartesian Linear Motion Planning with CuRobo};"
• Code-as-Policy (CaP): Approach where code generated by LLMs directly serves as the control policy for robots. "The use of LLM coding agents for robotics was explored as early as 2022 with a series of papers on Code-as-Policy (CaP) ~\cite{codeaspolicies2022, YinG-RSS-25}."
• CaP-X: An extended benchmark and agent suite for evaluating and improving Code-as-Policy methods. "The recent CaP-X paper provides an open suite of CaP benchmarks, agents, and encouraging results using coding agents that were state-of-the-art as of January 2026 ~\cite{fu2026cap}."
• Computation graph: A directed graph of modular nodes encoding data flow and control for robot perception, planning, and control. "introduces Graph-as-Policy (GaP), a multi-agent coding harness that generates directed computation graphs with perception, planning, and control nodes"
• Context window: The token-length limit of an LLM’s input that constrains how much context it can consider. "the context window can grow sufficiently large so that it is difficult for the agent to obey constraints and converge on reliable code."
• Control edges: Graph edges encoding conditional execution branches based on predicates over node outputs. "{Control edges} are condition branches and carry a predicate over node outputs."
• Contact GraspNet: A learned 6-DoF grasping model that predicts feasible contact-based grasps. "Contact GraspNet~\cite{sundermeyer2021contact}"
• cuRobo: A GPU-accelerated motion planning library for fast, feasible trajectory generation. "(cuRobo~\cite{sundaralingam2023curobo} and cuRobov2~\cite{sundaralingam2026curobov2}; 8 skills)"
• cuRobov2: A dynamics-aware successor/version of cuRobo for motion generation. "(cuRobo~\cite{sundaralingam2023curobo} and cuRobov2~\cite{sundaralingam2026curobov2}; 8 skills)"
• cuTAMP: A differentiable task-and-motion planning framework used within TipTop’s pipeline. "We configure this from the github code with the default configuration of 128 particles for cuTAMP~\cite{shen2024differentiable} with a 60-second planning timeout."
• DBSCAN: A density-based clustering algorithm used here for point cloud processing. "2D and 3D vision utilities with NumPy and OpenCV (e.g., DBSCAN for point cloud processing; 15 skills)"
• Directed edge: An ordered dependency from one node to another in a computation graph. "A directed edge $e = (n_i, n_j) \in E$ represents a data or logic dependency."
• Edge device: A deployed on-robot or on-premises computing unit executing the policy without cloud agents. "on an edge device for persistent execution over time."
• Edge-based interpreter: A runtime on the robot/edge that executes the optimized computation graph. "The optimized graph is then sent to an edge-based interpreter for repeated execution on the robot."
• Franka (robot arm): The Franka Emika Panda 7-DoF manipulator used in several benchmarks. "These 6 tasks use common kitchen items and one Franka robot arm from the LIBERO~\cite{liu2023libero} benchmark."
• GraspGen: A generative model-free grasp synthesis method used as a skill. "model-free policies (e.g. GraspGen~\cite{murali2025graspgen})."
• Grounding DINO: An open-vocabulary object detector used for perception skills. "Grounding DINO~\cite{liu2023grounding}"
• Hallucinations: LLM-generated content that is fabricated or incorrect relative to constraints or resources. "Agentic coding is new and advancing rapidly, but remains prone to hallucinations and constraint violations."
• Inverse kinematics (IK): Computing joint configurations that achieve a desired end-effector pose. "one inverse kinematics (IK) error during linear Cartesian motion"
• Isaac simulator: NVIDIA’s robotics simulator used for physics, rendering, and contact in rehearsal. "GaP uses Isaac simulator~\cite{NVIDIA_Isaac_Sim} as an internal simulation to render the visualization for the perception nodes, simulate physics, and compute contacts."
• Isaac-Lab physics simulator: A physics-based simulation suite for robotics training and evaluation. "(5) a self-learning approach to graph rehearsal and optimization using the Isaac-Lab physics simulator;"
• Kinematic joint limits: Bounds on joint variables restricting an articulated mechanism’s motion. "with defined kinematic joint limits $[\theta_{\min}, \theta_{\max}]$."
• LIBERO-Pro: A LIBERO evaluation variant introducing harder positional variations. "the {LIBERO-Pro}~\citep{zhou2025liberopro} evaluation suite swaps target and basket object positions at test time (object_swap)."
• Minkowski sum: A geometric operation used here to ensure non-overlapping placements (singulation). "we utilize Minkowski sum~\cite{minkowski1896geometrie} operations to ensure item singulation."
• Modular Open Robot Skill Library (MORSL): A catalog of atomic, reusable robotic skills with defined I/O and preconditions. "from a Modular Open Robot Skill Library (MORSL)."
• M2T2: A learned robotic skill/model used for grasping/manipulation within the library. "M2T2~\cite{yuan2023m2t2}; 5 skills"
• Molmo: An open-weight vision-LLM used as a perception component. "Molmo~\cite{deitke2024molmo}"
• MolmoAct2: A vision-language-action policy used as a baseline in benchmarks. "MolmoAct2~\cite{fang2026molmoact2}"
• Occupancy map: A representation of free/occupied space in the environment/workcell. "establishes world frame $\mathcal{W}$ and occupancy map $\mathcal{M}_E$"
• Orchestration Agent: The coordinating agent that decomposes tasks and directs skill agents to build subgraphs. "GaP starts with an Orchestration Agent that partitions a VA task into functional segments"
• Oriented bounding box: A rotated bounding box tightly aligned to an object for pose/grasp estimation. "using 14.1 seconds to generate oriented bounding boxes"
• OWL-ViT: An open-world, vision-transformer-based detector used for open-vocabulary object detection. "OWL-ViT~\cite{minderer2022simple}"
• Parallel rehearsal: Running multiple simulated executions concurrently to evaluate and improve a graph. "Then, a parallel rehearsal starts across these $N$ scenes to evaluate graph performance."
• Predicate: A Boolean-valued condition used to control branching in graphs. "carry a predicate over node outputs."
• Robot Operating System (ROS): A middleware framework for robot software integration via nodes and topics. "the Robot Operating System (ROS)\cite{macenski2022robot} which is based on a computation graph structure."
• SE(3): The group of 3D rigid-body poses (positions and orientations). "to the non-colliding subset of $SE(3)$."
• Skill nodes: Atomic, typed functional units (perception, planning, control) that compose the graph. "computation graphs that include ``skill'' nodes from a Modular Open Robot Skill Library (MORSL)"
• State space: The joint product of robot configuration, object poses, and articulated states. "{$\mathcal{X}$, State Space:} The product space of the robot joint configurations, the $SE(3)$ poses of all rigid objects, and the kinematic states of articulated entities"
• Task and Motion Planning (TAMP): Integrating high-level task sequencing with low-level motion feasibility. "Task and Motion Planning (TAMP)"
• Throughput: Units completed per time (e.g., successes per hour), often success rate divided by cycle time. "throughput (units per hour: success rate divided by cycle time)"
• TipTop: A modular open-vocabulary planning system built on TAMP for robotic manipulation. "We also evaluate TipTop~\cite{shen2026tiptop}, a modular open-vocabulary planning system for robotic manipulation based on TAMP."
• UR5: A 6-DoF industrial robot arm used in the cable insertion benchmark. "This benchmark task requires a UR5 robot arm with with the ZED Mini wrist camera"
• URDF: A robot description XML format specifying kinematics and visuals. "specifying the robot URDF, joint limits, gripper configuration, camera spec, and camera placement."
• Variational Automation (VA): Task class with persistent execution over varied object geometries and poses. "We focus on ``Variational Automation'' (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation."
• Vision-Language-Action (VLA): Models that map visual and textual inputs to robotic actions. "Vision-Language-Action (VLA) models"
• Vision-LLM (VLM): Models that jointly process images and text for perception/reasoning. "Vision-LLMs (VLM)"
• Visual Differencing: Using before/after visual feedback to guide code or policy refinement. "CaP-X~\cite{fu2026cap} used a VLM to provide feedback before and after execution (i.e. Visual Differencing)"
• World frame: The global coordinate frame for the workcell/environment. "establishes world frame $\mathcal{W}$ and occupancy map $\mathcal{M}_E$"
• Wrist camera: A camera mounted on the robot’s end-effector for close-up perception. "The real version of this benchmarks uses a Franka arm with wrist camera and real items from a grocery store."
• Zero-shot generalization: Performing new tasks or conditions without task-specific training. "and zero-shot generalization~\cite{ji2025robobrain}."

Sign up for free to add this paper to one or more collections.