Papers
Topics
Authors
Recent
Search
2000 character limit reached

KAT-Coder-V2.5 Technical Report

Published 6 Jul 2026 in cs.SE and cs.AI | (2607.05471v1)

Abstract: We present KAT-Coder-V2.5, a coding-focused agentic model trained to act autonomously inside real, executable repositories rather than as a single-turn code generator. Its capability is bottlenecked less by model scale than by the scarcity of reproducible environments, verifiable rewards, and high-value trajectories, which we address with an end-to-end agentic post-training framework. AutoBuilder reconstructs multilingual repositories into sandboxed environments with fail-to-pass and pass-to-pass verification at scale, from which we regenerate self-contained task specifications, recover near-miss trajectories, and distill supervision through process-aware filtering, while KwaiClawEnv synthesizes large-scale tool-use trajectories from executable services and real task seeds. We further scale reinforcement learning with harness randomization, a reliability-hardened sandbox, an asymmetric actor--critic PPO with hindsight-augmented value estimation, and a harness-oriented reward framework, and unify SWE, Agent-Claw, and WebCoding experts via Multi-Teacher On-Policy Distillation. Across six software-engineering and agentic benchmarks, KAT-Coder-V2.5 delivers the best agentic tool-use result on PinchBench and ranks second only to the frontier Opus 4.8 on repository-level software engineering. Our service is available at https://streamlake.com/product/kat-coder.

Summary

  • The paper introduces a verified, task-centric pipeline that boosts reproducible environment construction from 16.5% to 57.2% across 12 languages.
  • The paper employs process-aware trajectory curation and randomization to mitigate overfitting and enhance long-horizon reinforcement learning stability.
  • The paper integrates multi-teacher on-policy distillation with asymmetric PPO, achieving top benchmark results in agentic software engineering and tool-use tasks.

KAT-Coder-V2.5: Infrastructure-Centric Advances in Agentic Software Engineering and Tool-Use

Introduction

KAT-Coder-V2.5 (2607.05471) is positioned on the premise that further progress in agentic coding agents depends dominantly on infrastructure advances—not parameter count. The work systematically dissects the limiting factors for large code models in the transition from passive code completion to autonomous code agents, emphasizing: (1) scalable, verifiable, and reproducible code environments, (2) process-aware trajectory signal curation, and (3) stability and generalization in long-horizon RL. These are treated as systems-level challenges, each addressed with new post-training methodologies and RL architecture. The paper demonstrates substantial quantitative gains on diverse, difficult software engineering and tool-use benchmarks, supported by an infrastructure capable of mining, cleaning, and validating real-world tasks at scale.

Agentic Software Engineering: Task Design and Data Pipelines

KAT-Coder-V2.5 introduces a verifiable task-centric training infrastructure designed to overcome the inadequacies of raw issue and PR text as task specifications and the brittleness of repository environments at scale. The construction pipeline (Figure 1) employs task mining from merged PRs and commits, reconstructing each as a triplet: structured, context-independent problem/requirement/interface description; a deeply validated build environment (via AutoBuilder); and a thorough test oracle with both fail-to-pass and pass-to-pass verification signals. Figure 1

Figure 1: The agentic software-engineering data pipelines, illustrating the structured progression from raw code changes to validated, agent-ready tasks.

AutoBuilder integrates language- and build-system-specific templates, an extensible recipe database, and active agent feedback loops, substantially improving the success rate of reproducible environment construction from 16.5% to 57.2% across 12 languages.

Central to the data curation process is a process-aware signal pipeline: high-value near-miss trajectories are preserved and completed via hint-based regeneration, while trajectories with brittle, superficial, or shortcut-based pass strategies are filtered out using rule-based and heuristic scoring. Harness randomization swaps tool invocation protocols, context layouts, and injects noise, which robustifies downstream models against interface and environment-specific overfitting.

KwaiClawEnv: General Agentic Capabilities through Large-Scale Environment Synthesis

To support large-scale agentic tool-use, KAT-Coder-V2.5 trains on data from KwaiClawEnv—a modular environment generation framework (Figure 2). This system formalizes environments via composable, validated services (atomic units derived from both open-source skill libraries and LLM-templated APIs), scalable task derivation (with configurable complexity and tractable, semantically diverse interaction trajectories), and an Eval layer for trajectory formatting, multi-level filtering, and quality feedback. Figure 2

Figure 2: Pipeline of KwaiClawEnv, demonstrating modular construction from service definition to evaluation of agentic trajectories.

KwaiClawEnv can scale a small pool of high-quality primitive services into tens of thousands of reliable, functionally diverse tool-use environments. A progressive, closed-loop validation framework enforces cross-layer consistency and applies post hoc LLM-based trajectory judgment to maximize semantic, execution, and behavioral quality.

Reinforcement Learning Infrastructure and Harness Generalization

KAT-Coder-V2.5 implements RL advances that target long-horizon instability, training-environment brittleness, and overfitting to single harness schemas.

Harness diversity is systematically constructed along axes of tool protocol, context structure, and agent control flow, incorporating both clean “white-box” harnesses for core capability grounding and complex “black-box” harnesses for real-world deployment scenarios. The RL training loop is mediated by a Gateway Server (Figure 3), which decouples inference/interaction and training, ensures tokenization consistency, and isolates harness idiosyncrasies. Figure 3

Figure 3: Overall architecture of the Agentic RL training infrastructure, highlighting harness scaling and feedback-loop isolation.

Sandbox reliability is specifically optimized: container image management and environment variable isolation reduce reward corruption from environment errors from 16% to below 2%, allowing stable convergence and reliable credit assignment for PPO-based learning. Figure 4

Figure 4: RL training curve in the SWE scenario, indicating consistent and stable reward improvement throughout training.

An asymmetric PPO architecture is utilized: the critic accesses privileged hindsight context at training time (including real reward, verifier coverage, patch-level diffs), boosting sample efficiency and enabling turn-level credit assignment. The reward framework interleaves rule-based, process-oriented, and failed trajectory incentives with rubric-guided, model-based trajectory reward assessment, ensuring dense and functionally critical learning signals.

Multi-Teacher On-Policy Distillation and Long-Context Stability

The method fuses multiple specialized domain experts—covering SWE, tool use, terminal, web, and general knowledge—using a Multi-Teacher On-Policy Distillation (MOPD) strategy. MOPD performs function-space, reverse-KL distillation from teacher logits on student-generated policies, conditioned on active, potentially drifted prefixes. The system mitigates the instability of long-context distillation via an off-policy cold start (teacher rollouts supervise initial phase training) and drift-aware dynamic truncation (excluding unreliable, highly divergent supervision tokens based on top-kk overlap metrics).

This method effectively mitigates the typical see-saw effect of multi-domain merging, achieving robust cross-domain generalization without catastrophic interference.

Quantitative Results

On six major benchmarks (Figure 5), KAT-Coder-V2.5 demonstrates:

  • Repository-level software engineering: Second-highest accuracy (53.1, 65.2) on KAT Code Bench and SWE-Bench Pro, surpassed only by the general-purpose Opus 4.8, outperforming GLM-5 series and Kimi-K2.6.
  • Agentic long-horizon tool use: Top performance (94.9) on PinchBench and very strong results (85.5) on the business-critical KAT Claw Bench.
  • Terminal/scientific coding: Remains competitive, though larger general models show modest advantage. Figure 5

    Figure 5: Results of KAT-Coder-V2.5 on various SWE and agent benchmarks, establishing state-of-the-art tool-use and near-frontier engineering performance.

These results are robust to harness, environment, or protocol variance by construction and reflect infrastructure-driven improvements in reasoning, localization, patching, and complex multi-tool workflows.

Implications and Future Directions

KAT-Coder-V2.5 establishes that orchestrated infrastructure, trajectory quality filtering, and harness-oriented RL optimization can yield substantial and generalizable gains in agentic code-related tasks—sometimes exceeding what can be obtained by further model scaling. The developed verification, reward, and expert merger protocols form a blueprint for new agentic training systems.

Future developments could extend robust environment construction and process annotation to broader domains (including CLI interaction, scientific computing, and multimodal workflows), develop credit assignment architectures for even longer horizons and sparser reward scenarios, and unify agentic training with open-ended, dynamic environment feedback. Robustness on less-structured tasks (e.g., general AAA coding) is a salient open direction.

Conclusion

KAT-Coder-V2.5 demonstrates that systematic, verifiable environment and trajectory construction, coupled with robust and harness-general RL protocols and multi-domain expert fusion, can produce highly performant code agents, especially in realistic engineering and agentic tool-use scenarios. The technical methods established serve as a foundation for future iterations of autonomous software and agentic systems.

Whiteboard

Explain it Like I'm 14

What is this paper about?

This paper describes KAT-Coder-V2.5, a next‑generation “coding agent.” Think of it as a careful, tool‑using robot programmer that can read instructions, explore a big code project, change files, run tests, and keep fixing things until the tests pass. The team’s main idea is that building a strong coding agent isn’t just about a bigger model. It’s about giving the agent a solid “workshop” (reliable environments), good “homework and grades” (clear tasks and fair scoring), and safe “practice arenas” (stable training systems).

What questions were the researchers trying to answer?

In simple terms, they asked:

  • How can we give coding agents realistic, reproducible projects to work on, so passing tests really means the code is correct?
  • How do we train agents to follow good engineering habits, not just “cheat” to pass a test?
  • How do we make training stable for long, multi‑step problems where rewards are rare and feedback can be noisy?
  • How can we teach agents to use many different tools and interfaces so they don’t get stuck on just one way of working?

How did they do it?

They built a full training system with several parts. Here are the most important ones, explained with everyday language:

1) AutoBuilder: building real, testable coding tasks

  • Real code projects are messy: different languages, dependencies, and test setups. AutoBuilder is like an automated lab technician. It sets up each project in a clean, isolated “sandbox,” installs what’s needed, and runs tests.
  • It only accepts an environment if the tests are truly detected and reproducible (not just “the command ran”). This prevents fake passes.
  • From real pull requests (PRs), they reconstruct each task with:
    • A clear problem statement (what’s broken or missing),
    • Requirements (what should happen, based on test changes),
    • Interface constraints (APIs and rules to follow).
  • Result: over 100,000 tasks across 12 programming languages, with the success rate of building working environments improved from 16.5% to 57.2%.

2) Process‑aware training data: not just final scores

  • A “trajectory” is the step‑by‑step record of how the agent solved a task.
  • Some passes are low‑quality (like hard‑coding a value), and some failures are close and instructive.
  • They:
    • Recover “near miss” failures by giving temporary, non‑spoiler hints, then regenerate the solution without hints so training stays fair.
    • Filter out “bad habits” even if they pass, like bypassing project rules or tampering with tests.
    • Grade not just the result, but the process: exploring the right files, matching the specification, making minimal changes, verifying thoroughly.

3) KwaiClawEnv: a big, diverse tool‑use playground

  • Agents also need to use tools beyond coding (like APIs, data tools, or web utilities). KwaiClawEnv is a structured environment that creates many different “Services” (tools), “Tasks” (problems to solve), and “Eval” (quality checks).
  • It scales up by:
    • Turning human‑written or AI‑generated “skills” into real, callable services,
    • Combining services into more complex workflows,
    • Generating lots of task variations with clear, machine‑checkable success criteria,
    • Filtering and fixing issues so the data stays consistent and high‑quality.
  • The result is long, multi‑step tool‑use trajectories (often 15+ tool calls, and sometimes 100+ steps), which help the model learn to plan and adapt.

4) Stable reinforcement learning (RL) with varied “harnesses”

  • A “harness” is the interface and rules the agent uses to act and read feedback. If the agent only practices with one harness, it overfits to that style.
  • They train with both:
    • White‑box harnesses (simple, transparent, clean signals),
    • Black‑box harnesses (more realistic, compressed, and varied—like what happens in real products).
  • They also:
    • Added a Gateway Server to keep token handling consistent and avoid subtle bugs,
    • Spent a lot of effort making the sandbox reliable (fewer crashes, fewer timeouts, correct environment variables),
    • Used PPO (a popular RL algorithm) with a “hindsight” Critic: during training, the value estimator can see extra info (like final test results), but the acting policy can’t. This makes learning more stable without giving the model unfair peeks at the answers.
    • Designed better rewards:
    • Core reward: pass all the fail‑to‑pass tests (fix the bug) and keep pass‑to‑pass tests green (don’t break other stuff),
    • Behavior rules: reward good habits (accurate tool calls, no garbled text, no messy repeats),
    • Partial credit: if the agent made progress (like some tests passed or it found the right files), it still gets some reward,
    • A model‑judge (GRM) that scores the agent’s overall process with a rubric focused on diagnosis, verification, and strategy.

5) Bringing experts together

  • They train several specialists (software engineering, general tool use, terminal use, web coding, and general knowledge) and then combine them with “multi‑teacher on‑policy distillation,” which merges strengths while the student practices on its own tasks. This avoids the usual back‑and‑forth where boosting one skill hurts another.

What did they find?

  • AutoBuilder greatly improved the ability to set up real projects and run meaningful tests, jumping from 16.5% to 57.2% successful environment builds, and producing 100,000+ verifiable tasks across many languages.
  • The hint‑and‑regenerate loop rescued many “almost there” failures: previously zero‑pass tasks could reach about a 20% pass rate after guided recovery, while keeping the final training data hint‑free.
  • Process‑aware filtering removed passing‑but‑bad habits and turned trajectories into more useful training signals.
  • Harness diversity and sandbox fixes made RL training stable. Environment‑caused errors dropped from about 16% to under 2%, and timeouts fell from around 6–7% to below 1%. Training curves became smoother and more reliable.
  • On benchmarks using a unified harness, KAT-Coder‑V2.5:
    • Achieved the top result on PinchBench,
    • Ranked second on SWE‑Bench Pro and on the internal KAT Code Bench among evaluated models.
  • KwaiClawEnv produced large, diverse tool‑use data with strong quality checks, enabling the agent to handle longer, more complex tool sequences.

Why does this matter?

This work moves AI coding from “typing code” to “being a careful engineer.” By focusing on clean environments, clear tasks, honest processes, and stable training, the agent learns to:

  • Understand big codebases, not just single files,
  • Fix bugs without breaking other parts,
  • Use tools reliably across different interfaces,
  • Keep working through long tasks with partial feedback,
  • Generalize better to new projects and real‑world setups.

In the long run, this approach could make AI helpers more trustworthy for software maintenance, bug fixing, and complex multi‑tool workflows—helping teams move faster while keeping quality high. It also provides reusable infrastructure and benchmarks that others can use to train and evaluate robust AI agents.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a consolidated list of concrete gaps and open questions that the paper leaves unresolved, aimed to guide follow-up research.

  • Public release and reproducibility: Are the AutoBuilder code, KwaiClawEnv generators, harness implementations, curated datasets, build recipes, and internal benchmarks (KAT Code Bench, KAT Claw Bench) publicly available with licenses and documentation to enable full replication?
  • Data contamination controls: What safeguards were used to prevent train–eval leakage (e.g., overlapping repositories, tasks, or regenerated task descriptions) for SWE-Bench/Pro and other reported benchmarks?
  • Model specification transparency: What are the exact model sizes, architectures, tokenizer details, and training compute budgets used for V2.5 and for each specialized expert?
  • Ablation studies: What is the isolated contribution of each component (AutoBuilder verification, hint-based recovery, process scoring, harness randomization, asymmetric PPO, rule-based vs model-based rewards, multi-teacher on-policy distillation) to final performance?
  • Cross-harness generalization evaluation: Despite training with harness randomization, results are reported “under a unified Claude Code harness”; how does performance change across materially different harnesses and unseen tool protocols at test time?
  • External validity of synthetic tool environments: To what extent do skills learned in KwaiClawEnv (LLM-generated or compositional services) transfer to real, messy, rate-limited, or partially documented production tools and APIs?
  • Windows/macOS and non-Linux portability: Can AutoBuilder and the sandbox reliably support non-Linux targets, proprietary build systems, GUI workflows, and OS-specific dependencies?
  • Monorepo and polyglot complexity: How does AutoBuilder handle large monorepos, cross-language build graphs, nonstandard test runners, and deeply nested CI scripts at scale?
  • Environment verification sufficiency: Is the “>90% tests collected + reproducible outcomes” criterion sufficient to ensure test completeness and correct test discovery across frameworks and custom runners?
  • Test flakiness and nondeterminism: How are flaky tests detected, quarantined, or modeled during RL to avoid reward corruption beyond the reported sandbox fixes?
  • Pre-applying reference dependency/config changes: How is the boundary decided between “non-challenge” environment edits vs edits that are part of the true programming task, and what is the impact on realism and learnable setup skills?
  • Reading tests vs true specification understanding: Beyond rule-based filters, how effectively does the process-aware pipeline prevent “test-reading” shortcuts or overfitting to specific assertion patterns?
  • Hint-based recovery leakage: How is it guaranteed that regenerated hint-free trajectories do not implicitly encode hint content, and what quantitative checks validate this independence?
  • Process scoring reliability: What are the inter-rater reliability, drift analyses, and failure modes of the heuristic/rule-based process scores used for filtering and preference signals?
  • GRM (model-based judge) validity: What are the held-out precision/recall, calibration, robustness to adversarial phrasing, and cross-domain transfer of the GRM on trajectory-level rubric judgments?
  • Reward hacking risks: Can the actor exploit rubric blind spots (e.g., superficially “valid” verification steps, style changes to reduce repetition) to accumulate reward without truly improving fixes?
  • Credit assignment granularity: How does asymmetric PPO handle extremely long horizons (>100 turns) without value leakage or bias from privileged hindsight that might distort early-step advantages?
  • Off-policy bias and stability: With hindsight-augmented critic and black-box harnesses, what measures ensure PPO stability and mitigate off-policy drift when sampling student-policy trajectories?
  • Tokenization and deployment drift: The gateway avoids retokenization drift by using /generate, but how does the policy behave under typical chat endpoints used in real deployments?
  • Tool-call parallelism constraints: The chosen thresholds for penalizing parallel calls are unexplained; how do different concurrency policies affect learning and real-world throughput/stability?
  • Coverage-aware rewards: How are code coverage signals computed (tooling, instrumentation overhead, multi-language support), and do they bias edits toward easily coverable code?
  • Long-horizon failure analysis: What residual sandbox error modes remain within the reported ~2% and how do they affect learning on specific task families (e.g., large build/test tasks)?
  • Distributional breadth of SWE tasks: What is the language-wise, domain-wise, and repository-size breakdown of the claimed 100k environments, and where are the major blind spots?
  • Security and safety: What is the threat model for running untrusted code during environment construction and RL, and how are sandbox escapes, data exfiltration, or abuse prevented and audited?
  • Licensing and compliance: How were repository licenses handled for mined code/tests, and can the resulting datasets be redistributed and used for commercial training?
  • Beyond bug fixing: Can the framework handle feature addition, refactoring, performance optimization, and multi-PR change sets with migration steps, and how would verification be defined?
  • Human-in-the-loop validation: What is the minimal human oversight needed to maintain data quality at scale, and which steps benefit most from targeted human curation?
  • Robustness to noisy logs and perturbations: Harness rewriting injects perturbations; what is the measured gain on OOD robustness and failure recovery vs a fixed harness baseline?
  • Multi-teacher distillation dynamics: How are teachers selected and weighted across domains, how is the “see-saw” effect quantitatively monitored, and what safeguards prevent catastrophic forgetting?
  • Efficiency and cost trade-offs: What are the token, compute, and wall-clock costs of end-to-end data generation + RL vs pure SFT, and what is the ROI per additional unit of data or training step?
  • Evaluation transparency: Are prompts, harness configs, seed selection, and scoring scripts for all reported benchmarks released to allow exact reproduction and to test statistical significance?
  • Realistic repo hygiene: The reward includes “debug artifact cleanup”; how is repository pollution robustly detected across heterogeneous projects without false positives that penalize valid artifacts?
  • Generalization to closed-source enterprise code: What adaptations are needed for large proprietary repos with internal tooling, custom CI, or restricted network environments?
  • Memory and planning mechanisms: The paper does not explore explicit memory/plan modules; can integrating structured memory or hierarchical controllers improve long-horizon credit assignment?
  • Test synthesis and augmentation: Can the framework extend to generating or repairing tests to improve specification coverage and reduce reliance on existing tests with gaps?
  • Failure recovery policy: When environment construction fails, which automated repair policies perform best, and can meta-learning learn to predict effective build recipes to reduce iterations?
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following applications can be deployed now by adapting the paper’s training infrastructure, data pipelines, and evaluation methods to production systems.

  • Software: CI/CD hardening with AutoBuilder
    • Use case: Integrate AutoBuilder to automatically reconstruct reproducible, verifiable repo environments in CI, ensure both fail_to_pass (bug-revealing) and pass_to_pass (regression) tests run deterministically across languages.
    • Value: Reduces flaky builds, catches “silent green” pipelines that never executed intended tests, and standardizes cross-repo build scripts.
    • Tools/products/workflows: “AutoBuilder-as-a-Service” for GitHub Actions/GitLab CI/Jenkins; containerized build–verify step; structured test parsing plugin.
    • Assumptions/dependencies: Containerization available; access to repo and dependency mirrors; consistent test frameworks; policy-compliant network access.
  • Software: AI patching bot with process-aware guardrails
    • Use case: Deploy a code-repair agent that proposes PRs while adhering to repository conventions, patch minimality, and verification discipline (guided by process-aware trajectory filters and harness-oriented rewards).
    • Value: Cuts triage time and reviewer load; reduces risky shortcuts (e.g., test tampering, hard-coding).
    • Tools/products/workflows: GitHub/GitLab PR bot; reviewer dashboard showing localization evidence, tests run, verification outcomes.
    • Assumptions/dependencies: High-quality tests; secrets isolation; organization’s coding standards encoded into rules; human-in-the-loop approval.
  • Software QA: Test quality auditing via pass-to-pass/fail-to-pass split
    • Use case: Audit whether new tests genuinely capture the target defect and whether regressions are protected.
    • Value: Prevents reward hacking; improves test suites’ diagnostic power.
    • Tools/products/workflows: Test linter that fails CI if pass_to_pass shrinks or fail_to_pass is not exercised; coverage-linked dashboards.
    • Assumptions/dependencies: Test metadata and structured outcomes available; stable test harness.
  • MLOps: Stable agentic RL training stack
    • Use case: Adopt gateway server to eliminate retokenization drift; reliability-harden sandbox (image GC policies, env-var isolation); multi-harness rollouts to avoid interface overfitting.
    • Value: Fewer training collapses; more reliable learning curves; cross-harness generalization.
    • Tools/products/workflows: Gateway microservice exposing /generate; sandbox resource manager; harness randomizer library.
    • Assumptions/dependencies: Access to inference backends beyond chat endpoints; container registry control; observability (logs/metrics).
  • Software: Cross-harness evaluation and procurement benchmarking
    • Use case: Evaluate internal/external coding agents against KAT Code Bench, KAT Claw Bench, PinchBench-like setups within a unified harness.
    • Value: Apples-to-apples vendor comparisons; procurement transparency; regression testing across agent updates.
    • Tools/products/workflows: Internal benchmark suite runner with white-box and black-box harnesses; scorecards for tool-use, repo-level SWE.
    • Assumptions/dependencies: Licensing/compliance for benchmarks; reproducible sandboxes.
  • Enterprise tools: Synthetic tool-use data generation with KwaiClawEnv
    • Use case: Generate scalable, high-quality tool-use trajectories from internal APIs by turning OpenAPI/Skill specs into executable services and task bundles.
    • Value: Bootstraps enterprise copilots for IT ops, analytics, support; covers long-tail workflows safely.
    • Tools/products/workflows: “ClawEnvKit” to import internal APIs, synthesize tasks, and produce SFT/RL-ready trajectories; closed-loop Eval layer.
    • Assumptions/dependencies: API specs and fixtures; data governance; containerized services; red-team review.
  • Data science/analytics: Multi-service workflow tutors
    • Use case: Train agents to orchestrate data retrieval, transformation, and report generation across BI tools using KwaiClawEnv composite services.
    • Value: Reduces manual glue-work; consistent audit trails.
    • Tools/products/workflows: Skill chains for warehouse + notebook + dashboard; trajectory-transparent grading for compliance.
    • Assumptions/dependencies: Read-only sandboxes; PII-safe fixtures; logging for lineage.
  • Security & compliance: Leakage and shortcut prevention in agent training
    • Use case: Apply repository sanitization (remove git history/metadata), rule-based gates, and process scoring to prevent leakage-based solutions and test tampering.
    • Value: Trustworthy training and evaluation; reduced risk of data leakage in regulated environments.
    • Tools/products/workflows: Sanitization preprocessor; shortcut detectors; process reward checkers.
    • Assumptions/dependencies: Access control audits; reproducibility checks.
  • Education: Autograding of programming assignments across languages
    • Use case: Use AutoBuilder to provision reproducible student environments; define tasks with precise descriptions and verifiers; reward process (exploration/localization/verification) in addition to final output.
    • Value: Fair, scalable grading; feedback beyond pass/fail; multilingual curriculum support.
    • Tools/products/workflows: LMS plugin; assignment packager generating fail_to_pass/pass_to_pass; student-local sandbox images.
    • Assumptions/dependencies: Institution compute; standardized templates; academic integrity policies.
  • Developer productivity: “Run-Ready” local environment assistant
    • Use case: VS Code/JetBrains extension that reconstructs project environments, runs structured tests, and suggests targeted next steps (e.g., where to read, which tests to run).
    • Value: Cuts setup friction; accelerates onboarding and bug reproduction.
    • Tools/products/workflows: Local AutoBuilder client; structured test parsers; hint-only diagnostics panel.
    • Assumptions/dependencies: Container runtime or virtual env manager; project test metadata.
  • RL evaluation: Model-based judge (GRM) for trajectory quality
    • Use case: Add a rubric-trained GRM to score fault reproduction, post-fix validation, and execution strategy for agent trajectories.
    • Value: Dense, process-level feedback; less reliance on brittle heuristics.
    • Tools/products/workflows: GRM inference service; rubric versioning; disagreement resolver with human sampling.
    • Assumptions/dependencies: High-quality labeled trajectories; monitoring for judge drift.
  • Model training: Multi-teacher on-policy distillation
    • Use case: Merge specialized experts (SWE, terminal, tool-use, web coding) using student-policy rollouts to avoid see-saw degradation.
    • Value: Practical path to unified assistants without catastrophic forgetting.
    • Tools/products/workflows: Teacher pool orchestration; on-policy sampling scheduler; capability health checks.
    • Assumptions/dependencies: Access to experts; training budget; continual evaluation.
  • Policy and standards: Verifiable agent evaluation protocol
    • Use case: Adopt fail_to_pass/pass_to_pass-based verification, trajectory transparency, and cross-harness testing in RFPs and audits for AI code assistants.
    • Value: Procurement clarity; mitigates vendor “demo overfitting.”
    • Tools/products/workflows: Evaluation policy templates; audit artifacts; reproducibility attestations.
    • Assumptions/dependencies: Organizational buy-in; reproducibility SLAs from vendors.

Long-Term Applications

The following applications are feasible with further research, scaling, integration, or governance work.

  • Software: Autonomous maintenance bots for large repos
    • Use case: Continuous agents that localize defects, propose minimal patches, run targeted regressions, and open PRs with evidence trails.
    • Sector: Software.
    • Dependencies/assumptions: More robust safety/guardrails, comprehensive test coverage, granular repo permissions, stronger non-regression guarantees.
  • Enterprise automation: Generalized cross-tool orchestrators
    • Use case: Agents executing end-to-end workflows across CRM, ticketing, data warehouses, and observability stacks using KwaiClawEnv-style service chaining.
    • Sector: Enterprise software/IT operations.
    • Dependencies/assumptions: Deep API coverage, identity and access management, auditability, change-management policies.
  • Legacy systems recovery: Automated software archaeology
    • Use case: Use AutoBuilder to reconstruct brittle build systems and environments for legacy codebases to enable modernization, security patching, and compliance testing.
    • Sector: Government/finance/industrial software.
    • Dependencies/assumptions: Heterogeneous OS/toolchain support, offline artifact caches, licensing remediation, human oversight.
  • Standardization: Harness interoperability standards
    • Use case: Define cross-vendor, cross-platform interaction schemas for agent tools (protocols, context layouts, control-flow primitives) inspired by harness scaling axes.
    • Sector: Policy/standards bodies; software.
    • Dependencies/assumptions: Industry consensus; open conformance test suites; backward compatibility.
  • Safety and governance: Compliance-audited agent RL pipelines
    • Use case: Traceable reward shaping (rule- and model-based), sandbox attestations, and trajectory-provenance logs for regulated industries.
    • Sector: Finance, healthcare, public sector.
    • Dependencies/assumptions: Policy-aligned logging; privacy-preserving telemetry; third-party audits; red-team frameworks.
  • Education: Massively scalable, adaptive programming curricula
    • Use case: Generate difficulty-controlled task variants with machine-verifiable objectives and process-aware feedback for personalized learning.
    • Sector: Education/edtech.
    • Dependencies/assumptions: Curriculum alignment; cheating mitigation; on-device or campus compute; accessibility.
  • Robotics and embodied AI: Interface/domain randomization for tool protocols
    • Use case: Transfer harness scaling and domain randomization principles to robotic skill APIs (planners, perception modules) to reduce interface overfitting.
    • Sector: Robotics.
    • Dependencies/assumptions: High-fidelity simulators; standardized skill schemas; safety cases for sim-to-real transfer.
  • Healthcare: Verified agent workflows over EHR and clinical tools
    • Use case: Agents that orchestrate data retrieval, coding, and documentation within validated service compositions; trajectory-transparent evaluation to ensure clinical correctness.
    • Sector: Healthcare/health IT.
    • Dependencies/assumptions: HIPAA/GDPR compliance, robust de-identification, human sign-off, medical device regulations.
  • Finance: Back-office automation with verifiable tool chains
    • Use case: Agents executing reconciliations, regulatory reporting, and risk checks across internal systems using Skill definitions and end-to-end verifiers.
    • Sector: Finance.
    • Dependencies/assumptions: Strong controls (SoD), immutable logs, adversarial evaluation for error containment, model risk management.
  • Energy/industrial: Secure SCADA/OT workflow simulators
    • Use case: ClawEnv-like simulations for SCADA operations to safely train agents on incident response and maintenance planning.
    • Sector: Energy/industrial automation.
    • Dependencies/assumptions: High-fidelity synthetic data; strict network isolation; safety-case validation; human-in-the-loop.
  • Marketplace: Curated Skills and task packs
    • Use case: Ecosystem of verified services and tasks (with fixtures, validators, rubrics) for domains like legal ops, marketing analytics, and customer support.
    • Sector: Software/platforms.
    • Dependencies/assumptions: IP/licensing for fixtures; quality governance; versioned rubrics; community moderation.
  • IDE-native, policy-compliant agent copilots
    • Use case: On-device or VPC-deployed agents with hindsight-trained critics for guidance during development, respecting data sovereignty.
    • Sector: Software/enterprise IT.
    • Dependencies/assumptions: Efficient inference on local/VPC hardware; privacy-by-design telemetry; model update pipelines.
  • Open evaluation commons for agents
    • Use case: Public repositories of trajectory-transparent benchmarks with GRM-like standardized judges and reproducibility kits.
    • Sector: Academia/policy.
    • Dependencies/assumptions: Shared annotation standards; periodic rebaselining; governance to mitigate judge bias and reward hacking.
  • Automated refactoring and upgrade campaigns
    • Use case: Long-horizon agents that plan and execute multi-PR migrations (e.g., framework upgrades) with verification-driven checkpoints.
    • Sector: Software.
    • Dependencies/assumptions: Comprehensive test suites, cross-repo coordination, rollback plans, change-approval workflows.

These applications leverage the paper’s core innovations—verifiable environment construction (AutoBuilder), process-aware trajectory curation, scalable service/task synthesis (KwaiClawEnv), harness randomization, reliability-hardened sandboxing, asymmetric PPO with hindsight critics, harness-oriented rewards, and model-judged process rewards (GRM)—to translate research into operational value across sectors. Assuring feasibility hinges on reproducible environments, trustworthy tests, secure sandboxes, governance for reward shaping and evaluation, and human oversight in high-stakes settings.

View Paper Prompt  View All Prompts 

Glossary

  • Agentic reinforcement learning: A reinforcement learning paradigm where LLMs act as autonomous agents interacting with tools and environments over long horizons. Example: "Long-horizon agentic reinforcement learning suffers from sparse rewards, unstable environment feedback, coarse credit assignment, overfitting to a fixed harness, and difficulty in fusing capabilities across specialized experts."
  • Asymmetric actor-critic: An actor-critic setup where the critic has access to privileged information unavailable to the actor at inference time, improving value estimation. Example: "Following asymmetric actor-critic and centralized-critic paradigms~\cite{liu2025asymmetric,pinto2017asymmetric}, we let the Critic access privileged hindsight information during training, while the Actor only observes the normal harness state available at rollout time."
  • AutoBuilder: An agent-driven pipeline that constructs reproducible, executable software environments and validates them to support verifiable tasks. Example: "We introduce AutoBuilder, an agent-driven pipeline for multilingual execution-environment construction."
  • Black-box harness: An execution interface for agents whose internal mechanics are hidden, emphasizing generalization to varied protocols and control flows. Example: "Black-box harnesses: ClaudeCode, Codex, OpenClaw, OpenHands and etc."
  • Centralized critic: A critic that leverages additional global or hindsight information during training to stabilize learning. Example: "Following asymmetric actor-critic and centralized-critic paradigms~\cite{liu2025asymmetric,pinto2017asymmetric}..."
  • Credit assignment: The problem of determining which actions or tokens contributed to outcomes in long trajectories. Example: "Long-horizon agentic reinforcement learning suffers from sparse rewards, unstable environment feedback, coarse credit assignment..."
  • Domain randomization: Training-time variation of environment/harness factors to promote robustness and generalization. Example: "In essence, this is a form of domain randomization applied at the environment level."
  • Experience Buffer: A storage of completed trajectories from which training batches are sampled for policy updates. Example: "Once a trajectory is completed, the Gateway Server writes it into the Experience Buffer, from which the Train Engine samples batches for policy updates."
  • Fail-to-pass tests: Test cases designed to fail initially that must pass after a correct fix, ensuring the defect is resolved. Example: "fail-to-pass and pass-to-pass tests."
  • F2 score: A metric emphasizing recall more than precision, used here to assess file retrieval quality. Example: "Retrieval quality is evaluated using an F2F_{2} score that jointly considers precision and recall, encouraging accurate file localization while discouraging overly broad searches."
  • Gateway Server: A mediator between rollout and training that standardizes interactions and prevents tokenization inconsistencies. Example: "we introduce an additional Gateway Server module on top of KwaiEnv."
  • Generalized Advantage Estimation (GAE): A method to compute low-variance, bias-controlled advantage estimates in RL. Example: "PPO combined with Generalized Advantage Estimation (GAE) and reward shaping enables turn-level credit assignment..."
  • Golden patch: The reference code change from a real commit/PR that represents the correct solution to a defect. Example: "the merged code change provides a {golden patch} and the accompanying test change provides a {test patch}~\cite{jimenez2024swebench}."
  • GRM: A specialized model-based judge trained to evaluate trajectory processes according to a rubric. Example: "We apply targeted RL to train a specialized judge model GRM for consistent rubric violation detection."
  • GRPO: A family of critic-free RL methods for sequence modeling that can struggle with long-horizon credit assignment. Example: "trajectory-level critic-free methods such as GRPO~\cite{shao2024deepseekmath} and its anchor-state variants~\cite{feng2026group} suffer from coarse credit assignment and high gradient variance."
  • Harness Scaling: Training with diverse harness interfaces and control flows to mitigate overfitting to one protocol. Example: "The key to Harness Scaling lies not in the number of harnesses but in whether the diversity falls along dimensions that are useful for generalization."
  • Harness-oriented reward framework: A structured reward design tied to harness feedback that shapes both task success and process behavior. Example: "combine an asymmetric actor--critic PPO with a harness-oriented reward framework to deliver stable, fine-grained signals for long-horizon tasks."
  • KwaiClawEnv: A scalable, structured environment synthesis framework for tool-use and multi-step agent tasks. Example: "we propose KwaiClawEnv, an environment synthesis framework designed around real business needs and Claw-style Agent tasks~\cite{openclaw2026}."
  • KwaiEnv: The environment manager that hosts execution environments and integrates harnesses for training. Example: "(3) KwaiEnv, which manages environments and integrates harnesses by hosting corresponding execution environments."
  • LLM-as-Judge: Using a LLM to score trajectories along multiple quality dimensions during filtering. Example: "The second layer, LLM-as-Judge evaluation, scores the remaining trajectories along three dimensions: semantic correctness, execution efficiency, and interaction naturalness."
  • Multi-Teacher On-Policy Distillation: A distillation approach that merges expertise from multiple teacher models using trajectories sampled from the student’s own policy. Example: "Multi-Teacher On-Policy Distillation"
  • OpenAPI: A standardized interface specification for describing RESTful APIs used to define executable services. Example: "equipped with OpenAPI specifications, container configurations, and fixture data."
  • Pass-to-pass tests: Regression tests that must continue to pass after a fix to ensure no new breakages. Example: "fail-to-pass and pass-to-pass tests."
  • PinchBench: A benchmark for agentic tool use and reasoning used to evaluate model performance. Example: "achieving the top result on PinchBench and ranking second on both SWE-Bench Pro and KAT Code Bench among the evaluated models."
  • Preference learning: Training that leverages relative judgments between trajectories to guide the model toward preferred behaviors. Example: "These process annotations also provide positive and negative trajectory signals for preference learning, rejection sampling, and process reward modeling."
  • Proximal Policy Optimization (PPO): A policy-gradient RL algorithm with clipped objectives and advantage estimation for stable training. Example: "We therefore adopt Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} as the algorithmic backbone."
  • Process-aware trajectory construction: Building and scoring trajectories with attention to exploration, localization, design, and verification steps, not just final success. Example: "we introduce {process-aware trajectory construction} that evaluates exploration, localization, design, editing, verification, and recovery behavior..."
  • Process reward modeling: Learning reward models that score multi-step agent processes beyond simple pass/fail outcomes. Example: "preference learning, rejection sampling, and process reward modeling."
  • Rejection sampling: Selecting trajectories based on adherence to process or quality criteria to improve training data. Example: "These process annotations also provide positive and negative trajectory signals for preference learning, rejection sampling, and process reward modeling."
  • Reliability-hardened sandbox: An execution environment engineered to minimize errors that corrupt reward signals. Example: "a reliability-hardened sandbox that reduces reward corruption from environment errors"
  • Retokenization drift: Mismatch between rollout and training tokenization due to re-templating at chat interfaces. Example: "This phenomenon, commonly referred to as retokenization drift, has also been documented in NVIDIA \href{https://arxiv.org/abs/([2605.24220](/papers/2605.24220))}{Polar}..."
  • Reward hacking: Exploiting weaknesses in reward signals to score highly without truly solving the task. Example: "This reward prevents reward hacking through ineffective code changes or simplified self-authored tests..."
  • Reward shaping: Adding auxiliary rewards to provide denser, more informative feedback for long-horizon learning. Example: "PPO combined with Generalized Advantage Estimation (GAE) and reward shaping enables turn-level credit assignment..."
  • Rollout Engine: The component that runs policy inference to generate agent actions during environment interaction. Example: "(1) Rollout Engine, which is responsible for policy-model inference and generates model responses during the agent's interaction with the environment;"
  • Service chaining: Composing multiple services/tools into pipelines to solve more complex tasks. Example: "service chaining and orchestrated combination."
  • SFT-ready samples: Data formatted for supervised fine-tuning, with consistent fields and structure. Example: "Raw trajectories are first converted into a unified training format, such as SFT-ready samples, with missing or auxiliary fields populated to ensure consistency across samples."
  • Trajectory compression: Compaction or reorganization of interaction history that alters raw trajectory structure. Example: "they commonly introduce trajectory-compression and context-reorganization mechanisms..."
  • Trajectory-transparent grading: Evaluation that bases judgments on fully visible, auditable interaction traces. Example: "Evaluation follows trustworthy agent evaluation principles including trajectory-transparent grading and multi-dimensional quality assessment~\cite{ye2026claweval,pinchbench}."
  • Verifiable task: A task defined by a precise description, executable environment, and validation tests ensuring objective correctness. Example: "A {verifiable task} is defined as a triplet consisting of a precise task description, an executable repository environment, and a set of validation tests used to determine correctness."
  • White-box harness: A simple, transparent harness exposing clean trajectories that provide low-noise training signals. Example: "White-box harness: mini-swe-agent. Its control flow is simple, it performs no trajectory compression, and its tool-invocation scale is relatively small."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 462 likes about this paper.

Sign Up for Free