InnoGym: Benchmarking the Innovation Potential of AI Agents

Benchmarking the Innovation Potential of AI Agents: InnoGym

Motivation and Conceptual Framework

"InnoGym: Benchmarking the Innovation Potential of AI Agents" (2512.01822) addresses a critical deficiency in agent evaluation: conventional benchmarks predominantly assess correctness and final output quality, neglecting the methodological diversity and originality underlying solutions. To bridge this gap, the authors formulate a principled framework for measuring innovation in AI agents, precisely characterizing each problem by a quadruple $(P, S, V, D)$: problem instance, solution space, performance measure, and solution dissimilarity. Two independent axes, performance gain $G$ and novelty $N$, quantify agent contributions, providing a bidimensional conceptualization where $G$ reflects measurable improvement over known baselines and $N$ captures methodological deviation within the solution space.

This model supports a nuanced taxonomy of tasks—solved, improvable, exploratory—recognizing innovation as context-relative and temporally dynamic. It explicitly models the lifecycle wherein $S_{\text{known}}$ evolves as agents produce novel, effective solutions, and innovation measurements are grounded in the current frontier.

The InnoGym Benchmark and Execution Environment

The InnoGym platform consists of two main subsystems: iBench, a task suite for systematic innovation assessment, and iGym, a unified runtime and agent SDK enabling reproducible, long-horizon problem solving. iBench comprises 18 meticulously curated tasks sourced from diverse engineering and scientific competitions, intentionally selected as improvable problems. Each task is standardized through multi-stage filtering and augmentation, ensuring access to datasets, validators, leaderboards, and validated reference solutions. Evaluators are normalized to guarantee absolute scoring; solution collection is rigorous, covering both classical methods and leaderboard submissions.

iGym abstracts away differences in agent architectures by providing robust, asynchronous tool dispatch and resource management, facilitating equitable comparison and enabling reproducible evaluation even under varying agent paradigms (workflow vs. agent-style).

Innovation Metrics and Evaluation Protocol

The key innovation metrics are as follows:

• Performance Gain ($G$): $G(s) = V(s) - V^*_{\mathrm{known}}$, measuring improvement over best-known scores. Positive $G$ indicates super-human or state-of-the-art performance.
• Novelty ($N$): Minimal method-level distance to $S_{\mathrm{known}}$, for feasible solutions only. Instantiated using Codex-based extraction and GPT-5-based rubric scoring, $N$ operates on a $[0,100]$ scale with multi-dimensional, per-solution granularity.

The evaluation pipeline enforces a structured separation between agent-visible resources and hidden leaderboards. Solutions are first validated for structural correctness via deterministic validators, then scored for performance, and finally compared against known solutions for novelty using the agent-as-judge pipeline. This dual-axis scoring distinctly rewards both effective and creative methodologies.

Experimental Results

InnoGym evaluates three representative agent frameworks—MLAB, CodeAct, and AIDE—using DeepSeek-v3.1, GPT-5, and Gemini-2.5-Pro as backbone LLMs. The experiments span 10 tractable tasks from the benchmark suite.

• Performance Gaps: All agents systematically underperform relative to the best-known human solutions across complex tasks; none surpassed human SOTA, and tasks with intricate formats resulted in unsuccessful submissions.
• Agent Differentiation: MLAB attains the highest macro-average for $G$ and $N$, demonstrating relative strength in both innovation and execution. CodeAct and AIDE lag significantly in both dimensions except for specialized mathematical optimization (CirclePacking), highlighting lack of generalization.
• Novelty vs. Robustness: High $N$ does not translate to high $G$; agents frequently produce novel but non-robust solutions. Explicit innovation prompting further increases $N$ but inflicts a performance penalty.
• Model Dependence: Ablations reveal backbone models are the limiting factor; more powerful LLMs achieve higher scores, indicating that agent frameworks amplify but cannot compensate for model capability bottlenecks.
• Exploration-Exploitation: Controlled temperature sweeps expose trade-offs, with mid-range sampling balancing performance and novelty.

The novelty evaluation pipeline, validated via EquiBench and expert triplet comparisons, demonstrates strong agreement with human assessments both on code-level and high-level methodological differences.

Implications and Future Directions

Practically, InnoGym provides a reproducible, cross-domain platform for benchmarking agents on meaningful innovation axes. The framework is sufficiently general to accommodate a wide spectrum of agent types, evaluation objectives, and methodological diversity scoring functions. Empirically, the persistent gap between creativity and robustness underscores the current fragility of agent-generated innovations for scientific and engineering applications. The findings imply that future progress necessitates more robust execution, effective integration of promising methodologies, and base models with advanced reasoning and tool-use capabilities.

Theoretically, the task formalism and innovation definitions clarify the moving frontier of machine intelligence: innovation is neither static nor absolute but evolves as solution spaces and baselines update. This has direct implications for curriculum learning, automated research, and next-generation agentic search strategies.

Going forward, addressing current limitations is essential: extending to solved and exploratory task classes, integrating efficiency and interpretability dimensions, enriching prior solution sets, and scaling to larger, more resource-intensive domains. Further, the metrics and evaluation protocols could be adapted to foster more nuanced forms of agent creativity, such as conceptual paradigm shifts, generative hypothesis formation, and multi-agent ensemble innovation.

Conclusion

"InnoGym: Benchmarking the Innovation Potential of AI Agents" inaugurates a principled and operational framework for evaluating AI agent innovation, integrating both performance and methodological novelty. The platform advances empirical standards for agent benchmarking and highlights robustness as the primary bottleneck to actionable scientific and engineering innovation. InnoGym is poised to inform and accelerate future agent development toward genuinely creative, effective, and reliable machine intelligence (2512.01822).