Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

Sign up for free to create and run prompts on this paper using GPT-5.

A Simple Explanation of “Ling and Ring 2.6: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale”

1) What is this paper about?

This paper introduces two very large AI models that are built to be practical assistants:

• Ling-2.6: a “fast responder” that answers quickly and packs a lot of useful information into every word it writes.
• Ring-2.6: a “deep thinker” that can plan, search, use tools, and handle long, complex tasks.

The big idea: make AI that is both smart and quick, so it can act like a reliable digital helper in the real world.

2) What questions are the researchers trying to answer?

In simple terms, they focus on three goals:

• How can an AI read and use very long inputs (like whole books or codebases) without slowing down?
• How can it explain well using fewer words (high “token efficiency”)?
• How can it be trained to act like a real agent—planning steps, using tools, browsing, coding—reliably and safely?

3) How did they do it?

They didn’t throw away their old model and start over. Instead, they upgraded their previous model (Ling‑2.0) step by step so it kept its skills while becoming faster and more capable.

Here are the main ideas, explained with everyday analogies:

• Faster “attention,” the AI’s reading strategy:
  • Usual attention is like comparing every word with every other word—very slow for long texts.
  • They switched to a hybrid of two faster methods (Lightning Attention and MLA) so the model can scan long documents more like a smart “summary tracker,” speeding up both training and answering. They mix them roughly 7:1 to balance speed and quality.
• Careful part-by-part swap (migration):
  • Think of replacing parts in a running car without stopping the engine. They converted the attention system in stages so the model didn’t forget what it already knew.
• Mixture-of-Experts (MoE):
  • Imagine a classroom with many specialists; for each sentence, the model only asks a few relevant experts for help. This saves time while keeping quality high.
• Smarter training data:
  • Not just web text—also “agent” data where the AI uses tools, writes code, searches, and follows real workflows. They also built ultra-long reading sets (up to 256,000 tokens) and cleaned them carefully to remove junk and repetition.
• Teach it to be concise and correct:
  • Evolutionary Chain-of-Thought: the model learns to remove extra, unhelpful steps—like practicing the shortest correct math solution.
  • Shortest-correct-response distillation: it studies examples that answer correctly using the fewest words.
  • Linguistic Unit Policy Optimization: instead of tweaking one token at a time, it learns in meaningful chunks (like phrases), which makes training more stable and reduces repetition.
  • Bidirectional preference alignment: a reward system that both praises good behavior (accurate, on-task, clear) and penalizes bad behavior (errors, fluff, rambling).
• Training for real “agent” behavior:
  • KPop, a new reinforcement learning approach: keeps learning stable while the AI tries things in real environments (coding, searching, using tools). “Asynchronous” means different parts of the training pipeline work in parallel—like multiple teams collecting experiences and updating the model at the same time—so it scales to huge models.

4) What did they find, and why does it matter?

• Much better with long inputs: The models can handle very long contexts more efficiently and answer faster, which is vital for tasks like reading long PDFs, codebases, or reports.
• More idea-per-word: Ling‑2.6 produces answers that are denser with useful information—about 4× better “capability per token” than their earlier generation. That means lower cost, less waiting, and fewer words to read.
• Strong reasoning and coding while keeping speed: They improved knowledge, math, coding, and general reasoning benchmarks compared to their previous base models, without sacrificing speed.
• Real agent skills, not just chat: Ring‑2.6 is trained to use tools, plan steps, and recover from failures in realistic environments. It did well on multiple agent-focused tests (coding, search, workflows), showing it can act—not just talk.
• Open releases: They are open-sourcing key model checkpoints so others can study, improve, and build on this work.

Why it matters: Faster, smarter, more reliable AI assistants cost less to run, can help more people, and can tackle harder, longer tasks.

5) What’s the big impact?

These models point toward practical AI “agents” that:

• Answer quickly when you just need help now (Ling‑2.6),
• Think more deeply when a problem is tricky or long (Ring‑2.6),
• Use tools (like search, spreadsheets, or code runners) reliably,
• And do all of this more efficiently—saving time and computing resources.

Because the team shares their models, researchers and developers can build better assistants for education, coding, research, and business workflows. In short, this work moves AI closer to being a fast, trustworthy teammate that can read long materials, plan multi-step tasks, and get real work done.

Below are the major knowledge gaps, limitations, and open questions left by the paper that future work could address.

• Lack of transparent, quantitative latency and throughput reporting: no end-to-end p50/p95/p99 latency, tokens/sec, or system-level throughput numbers across context lengths, model sizes, and hardware.
• Incomplete KV-cache and memory footprint accounting: no concrete measurements of KV-cache compression ratios, memory savings, and bandwidth contention for hybrid Lightning+MLA at varying sequence lengths (e.g., 32K–256K).
• Hybrid ratio generality: the 7:1 Lightning:MLA choice is supported by scaling-law trends, but its robustness across tasks, data regimes, model sizes, and longer horizons (>256K) remains untested; ablations on interleaving patterns and layer placement are missing.
• Quality–efficiency trade-off quantification: no systematic Pareto analysis showing how accuracy degrades vs efficiency gains as the fraction of linear attention increases.
• Migration sensitivity: the QK Norm absorption relies on a calibration dataset; sensitivity to calibration set size, domain mismatch, and statistical estimation error is not analyzed.
• Theoretical guarantees for QK Norm removal: there is no theoretical justification of stability or error bounds for the RMSNorm-based absorption procedure; only brief empirical remarks at “mini” scale are provided.
• Partial RoPE to TransMLA conversion risks: effects of maintaining Partial RoPE (vs full) on extrapolation beyond trained windows, positional generalization, and long-range order fidelity are not quantified.
• Long-context robustness under noise: no stress tests for robustness to injected repetitions, off-topic spans, shuffling, or corrupted sections in 100K–256K contexts.
• Numerical stability with FP8: claims of improved FP8 compatibility are not accompanied by failure-mode analysis (overflow/underflow incidents, gradient spikes) or mitigation strategies.
• MoE routing reliability: no analysis of expert load-balancing stability over long contexts, risks of expert collapse, token-level vs group routing trade-offs, or latency variance induced by MoE sparsity.
• Router precision and cost: router operates in FP32, but there is no exploration of mixed-precision/memory-optimized routing and its impact on accuracy and latency.
• Speculative decoding with continued MTP: acceptance rates, accuracy drift, and hallucination rates under speculation are not reported; Appendix with details is referenced but not provided.
• Post-training “capability per token” metric opacity: the Artificial Analysis Intelligence Index and “4× token efficiency” claim lack formal definitions, normalization details, task mix, and measurement protocol for reproducibility.
• Risk of under-answering: optimizing for shortest-correct responses may bias the model toward terse but incomplete outputs; no user-centric or task-adequacy evaluation of informativeness vs brevity.
• Evo-CoT and LPO side effects: potential loss of reasoning transparency or reduced performance on tasks requiring explicit, stepwise derivations is not evaluated.
• Bidirectional preference alignment (BPA) reliability: details on reward scale calibration, reward hacking safeguards, and generalization of penalties (verbosity/hallucinations) across domains are missing.
• LLM-judge dependence: the semantic redundancy penalty relies on an LLM judge; inter-judge reliability, bias, calibration, and cost–benefit (compute overhead) are not examined.
• Compression-based repetition penalty brittleness: using zlib compression may be gamed by superficial paraphrases; effectiveness across languages and code is untested.
• DPR curriculum risks: prioritizing “hard/unstable” samples could cause underfitting on common/easy distributions; no longitudinal retention tests to detect catastrophic forgetting.
• KPop RL with binary KL: convergence behavior, stability under off-policy/asynchronous updates, sensitivity to KL targets, and comparison to standard PPO/IMPALA-style baselines are absent.
• Asynchronous RL staleness: the impact of policy lag on learning stability and sample efficiency is not quantified; no ablations on queue sizes, update cadence, or rollout–learner skew.
• Tool-use generalization: robustness to unseen tools, API version changes, tool latency/timeouts, and adversarial tool outputs (e.g., injection) is not evaluated.
• Realistic environment robustness: no evaluations in adversarial web/mobile environments (prompt injections, content drift, CAPTCHAs) or recovery strategies from tool-chain failures.
• Safety and alignment coverage: no red-teaming metrics, jailbreak resistance, toxicity/fairness analyses, privacy leakage tests, or safety guardrail efficacy for agentic actions.
• Data governance and licensing: training corpus sources (especially the agentic and web/code data) lack licensing details, PII handling, deduplication, and contamination audits.
• Benchmark contamination checks: no evidence that LiveCodeBench (time-sliced), GAIA-2 Search, PinchBench, ClawEval, or long-context datasets are uncontaminated by pre/post-training data.
• Multilingual limitations: only ~4% multilingual data and limited evaluation (e.g., Belebele, KorBench); no long-context multilingual tests, cross-lingual tool-use, or script-diverse (e.g., Arabic/Hindi) long-horizon evaluations.
• Long-context evaluations are narrow: beyond LEval and LongBenchv2, more task-realistic, verifiable long-document tasks (e.g., multi-document financial analysis with ground-truth) should be included.
• Instant vs CoT evaluation mismatch: some benchmarks are run with CoT prompting while the model is optimized for instant outputs; direct evaluation of instant-mode accuracy vs CoT-mode trade-offs is missing.
• Specialist distillation fidelity: the procedure to consolidate specialists into a unified model lacks ablations on teacher–student gaps, catastrophic interference across domains, and capacity/temperature effects.
• Adaptive thinking mode selection: how to automatically choose between high/xhigh modes, budget tokens, and respect latency/SLOs in production is not specified.
• Serving portability and cost: the linghe fused-kernel library, context-parallel comms, and speculative decoding are not benchmarked across vendors (NVIDIA/AMD/TPU), nor is energy/carbon cost reported.
• Quantization and edge deployment: no results for 4/8-bit quantization with hybrid attention/MLA, nor accuracy/latency trade-offs for CPU/edge accelerators.
• Continual and on-the-fly tool adaptation: no method for rapid post-deployment adaptation to new tools/workflows without full retraining (e.g., lightweight adapters or online RL).
• Failure analysis: no taxonomy of common failure modes (reasoning, tool-use, long-context retrieval errors), their frequencies, or targeted mitigations.
• Interpretability and diagnostics: no attention/latent-space interpretability or probing of MLA/Lightning layers to understand long-range dependency modeling.
• Hardware and compute disclosure: training FLOPs, GPU hours, cluster specs, and cost/carbon accounting for 9.6T-token pre-training and RL are not disclosed.
• Reproducibility barriers: evaluations are run in an internal framework; decoding parameters, seeds, prompts, and harness are not released to ensure external replication.
• Open-source scope ambiguity: while checkpoints are “open-sourced,” clarity on licenses, commercial use, model cards, evaluation scripts, and (even partial) datasets is missing.

Overview

The paper introduces Ling-2.6 (instant, token-efficient) and Ring-2.6 (deep reasoning, agentic) trillion-parameter model families with:

• A hybrid linear attention design (Lightning Attention + MLA, 7:1) for long-context efficiency (up to 256K tokens), lower KV-cache, and faster decoding.
• Post-training methods that increase “capability per output token” (Evolutionary CoT, Linguistic Unit Policy Optimization, shortest-correct-response distillation, bidirectional preference alignment).
• A scalable agentic RL stack (KPop with asynchronous rollouts and binary-KL stabilization) for reliable tool use, coding, search, and workflow execution.
• Open-source checkpoints for both “instant” and “thinking” modes.

Below are concrete, real-world applications mapped to industry, academia, policy, and daily life, with immediate vs. long-term horizons. Each item lists sector fit, potential tools/workflows/products, and key assumptions or dependencies.

Immediate Applications

These can be deployed now using the open-source Ling-2.6/Ring-2.6 checkpoints and described serving/training techniques.

Industry

• Token-efficient customer support and IT helpdesk agents
  • Sectors: software, retail, telecom, SaaS
  • What: Low-latency assistants that resolve tickets, invoke MCP tools (knowledge base, CRM, RMM), and produce concise, accurate resolutions by default (Ling-2.6 “instant”).
  • Tools/workflows: “Instant Agent Layer” with tool-call traces, shortest-correct-response prompting; “Think switch” to Ring-2.6 ($high$/$xhigh$) on complex cases.
  • Assumptions/dependencies: Stable tool APIs; guardrails and preference models; on-prem or VPC deployment for data privacy.
• Long-context enterprise RAG and document intelligence (256K)
  • Sectors: legal, finance, insurance, pharma, manufacturing
  • What: Contract comparison, policy harmonization, due diligence, portfolio/company report analysis (multi-year, multi-entity), with MLA-based KV compression and linear attention for throughput.
  • Tools/workflows: “KV-compressed long-context RAG” pipelines; context-parallelized retrieval; report-logic evaluation for long-form outputs.
  • Assumptions/dependencies: High-quality retrieval & chunking; redaction/compliance pipelines; GPUs with adequate memory and fused-kernel support.
• Codebase assistants for repository-level development
  • Sectors: software, embedded systems, fintech, e-commerce
  • What: Repo search, bug triage, refactors, test-gen, SQL/BI support (BIRD-SQL gains), paired with CI/CD gates.
  • Tools/workflows: PR copilot with “instant” drafting + “thinking” verification; unit-test synthesis; LiveCodeBench-style harnesses.
  • Assumptions/dependencies: Access to repos/tests; policy for auto-commit; human-in-the-loop review.
• Analytics and BI copilots (SQL + narrative insight)
  • Sectors: finance, ops, sales, product analytics
  • What: Query generation, data validation, and concise insight narratives with token-efficiency rewards to reduce verbosity.
  • Tools/workflows: Warehouse connectors, schema-aware prompting; “LPO-based” unit-level instruction adherence for structured outputs.
  • Assumptions/dependencies: Data governance; schema/catalog availability; monitoring for hallucinated fields.
• Cost-optimized LLM serving in production
  • Sectors: platform/infrastructure providers, LLMops
  • What: Adopt hybrid attention, MLA, speculative decoding (MTP), fused kernels, and context-parallelism to cut latency and GPU costs.
  • Tools/workflows: Drop-in inference server plugins; “linghe” fused kernels; MTP-continued post-training for speculative decoding.
  • Assumptions/dependencies: Kernel compatibility with hardware stack; load testing and A/B validation.
• Telecom operations assistants
  • Sectors: telecom, network operations centers
  • What: Triage, troubleshooting, and knowledge retrieval (demonstrated performance on $\tau^2$-Bench Telecom), with tool-call orchestration for diagnostics.
  • Tools/workflows: Incident runbooks, search tools, CLI integrations, controlled Ring-2.6 “think” escalation.
  • Assumptions/dependencies: Secure access to NOC tools; runbook encoding and verification.

Academia and Research

• Literature review and synthesis over ultra-long contexts
  • Sectors: academia, R&D labs
  • What: Summarize 100s of pages (papers/books) with structured logic (ReportLogic), cross-document QA, trivia/fact integration via atomic fact pipelines.
  • Tools/workflows: 256K-context “survey generators”; citation/quote checkers; bidirectional preference alignment for factual focus.
  • Assumptions/dependencies: Access to full texts; evaluation rubrics for hallucination control.
• Agentic experimentation platforms
  • Sectors: AI/ML labs, tool-use research
  • What: Train task-specific agents with KPop (asynchronous RL, binary-KL), verifiable environments (coding, search, MCP tools).
  • Tools/workflows: “Asynchronous Agent Trainer (KPop)”; execution sandboxes and reward models (process + formatting + length).
  • Assumptions/dependencies: Instrumented environments; compute for rollouts; reward design expertise.

Policy and Government

• Rapid analysis of legislation and regulatory documents
  • Sectors: public sector, compliance, regulatory affairs
  • What: Cross-reference spanning long bills, guidance documents, and prior rulings; concise change-impact summaries.
  • Tools/workflows: KV-efficient legislative RAG; rule extraction and verification agents with itemized compliance checks.
  • Assumptions/dependencies: Access to updated statutes; audit trails; on-prem deployment for sensitive data.
• Procurement and oversight document triage
  • Sectors: government, audit, NGOs
  • What: Prioritize large FOIA responses, budget justifications, and oversight reports with long-context clustering and targeted QA.
  • Tools/workflows: “Focus reward–tuned” summarizers to avoid verbosity; multi-criteria scoring dashboards.
  • Assumptions/dependencies: Role-based access; bias/fairness review.

Healthcare and Life Sciences

• Long-document clinical/biomedical summarization and assistance
  • Sectors: providers, payers, pharma R&D
  • What: Summarize clinical notes, prior-auth packets, or research dossiers; compute with tool calculators (dosage, eGFR) via tool calls.
  • Tools/workflows: HIPAA-compliant on-prem instances; “instant” summaries with think-mode escalation for complex cases.
  • Assumptions/dependencies: Strict privacy/compliance; domain evaluation and medical oversight.

Education and Daily Life

• STEM tutors and multilingual learning aids
  • Sectors: education technology, tutoring services
  • What: Concise, step-appropriate reasoning on math/STEM with length-aware rewards; multilingual coverage improvements.
  • Tools/workflows: Difficulty-adaptive reasoning (dynamic length penalties); curriculum-aligned question generation.
  • Assumptions/dependencies: Age-appropriate filters; educator oversight; classroom integration.
• Personal assistants with tool use across web/mobile
  • Sectors: consumer apps, productivity
  • What: Plan-and-execute daily workflows (scheduling, shopping) via MCP tools; default to concise outputs.
  • Tools/workflows: “Instant-first” UX with opt-in “think” on hard tasks; traceable tool calls.
  • Assumptions/dependencies: User consent and API access; safety policies for external actions.

Long-Term Applications

These require additional research, scaling, domain data, or safety work before broad deployment.

Industry

• General-purpose digital workers for cross-system enterprise workflows
  • Sectors: finance, insurance, logistics, manufacturing
  • What: End-to-end planning, multi-tool orchestration, recovery from failures; human-on-the-loop governance.
  • Tools/workflows: Workflow graphs with verifiable tool traces; adaptive $high$/$xhigh$ “thinking” modes for uncertain steps.
  • Assumptions/dependencies: Robust verifiability, audit logging, and escalation; integration across legacy systems.
• Autonomous repository-level coding agents
  • Sectors: software, platform engineering
  • What: From issue intake to PRs with tests, benchmarked via LiveCodeBench-like harnesses at scale.
  • Tools/workflows: Full CI/CD loops; reward models for process alignment and minimal tool calls.
  • Assumptions/dependencies: Policy for automated contributions; robust test coverage; security scanning.
• Enterprise knowledge transformation pipelines
  • Sectors: legal, finance, pharma
  • What: Convert vast corpora into atomic fact graphs with automated consistency checks; feed downstream RAG/analytics.
  • Tools/workflows: Fact extraction + verification agents; continuous updates and provenance tracking.
  • Assumptions/dependencies: Ground-truth sources; disinformation detection; schema governance.

Science, Healthcare, and Safety-Critical Domains

• Clinical copilots with tool-grounded, auditable reasoning
  • Sectors: healthcare (clinical decision support), payers
  • What: Tool-integrated CDS with long-context patient histories, strict audit trails, and fail-safe escalation to clinicians.
  • Tools/workflows: Certified medical calculators; structured checklists (bidirectional preference + focus rewards).
  • Assumptions/dependencies: Regulatory approvals, clinical validation, liability frameworks.
• Autonomous research agents
  • Sectors: academia, biotech, materials science
  • What: Plan experiments, search literature, write code to run analyses, and iteratively refine hypotheses with verifiable logs.
  • Tools/workflows: Wet-lab/compute lab connectors; KPop-style asynchronous RL over long-horizon tasks.
  • Assumptions/dependencies: High-fidelity simulators/lab APIs; safety and reproducibility standards.

Robotics and Real-World Interaction

• Tool-use agents orchestrating robot fleets or IoT systems
  • Sectors: warehousing, manufacturing, agriculture, energy
  • What: Long-horizon task planning, failure recovery, and multi-robot coordination with “think” modes for hard planning.
  • Tools/workflows: Translating tool-call traces to robotic APIs; safety layers for real-world actuation.
  • Assumptions/dependencies: Reliable perception-planning stacks; safety certification; latency constraints.

Public Sector and Infrastructure

• Advanced compliance and regulatory auditors
  • Sectors: government, finance, energy
  • What: Detect inconsistencies, measure rule adherence, and synthesize remediation plans over millions of tokens of regulations, filings, and logs.
  • Tools/workflows: Rule decomposition/verifiers; cross-document linkage with provenance tracking.
  • Assumptions/dependencies: Transparent scoring/explanations; legal acceptance of AI-assisted audits.
• Network operations autonomy
  • Sectors: telecom, cloud infrastructure
  • What: Proactive detection, triage, and remediation with tool-runbooks and closed-loop verification.
  • Tools/workflows: Multi-agent coordination trained with KPop; rollback/guardrail policies.
  • Assumptions/dependencies: Strong simulators; strict safety policies and rollbacks.

Platforms and Ecosystems

• Edge/embedded deployment of long-context agents
  • Sectors: mobile, automotive, AR/VR
  • What: Leverage linear attention/MLA for memory-efficient, on-device summarization and planning.
  • Tools/workflows: Distilled variants of Ling/Ring; adaptive think toggling based on resource budgets.
  • Assumptions/dependencies: Further compression/distillation; hardware accelerators.
• Multi-agent systems with asynchronous coordination
  • Sectors: logistics, finance, operations research
  • What: Agents specialize (search, plan, execute) and coordinate via KPop-trained policies for dynamic tasks.
  • Tools/workflows: Orchestrators with binary-KL stabilization and curriculum scheduling (DPR).
  • Assumptions/dependencies: Reliable inter-agent protocols; conflict resolution and auditing.

Education and Workforce Development

• Personalized curriculum planners and assessment designers
  • Sectors: K-12, higher ed, corporate L&D
  • What: Long-horizon learning paths with concise feedback, adaptive depth (instant vs. think), and verifiable mastery checks.
  • Tools/workflows: Itemized rubric verifiers; difficulty-aware length control (dynamic penalties).
  • Assumptions/dependencies: Alignment with standards; bias/fairness oversight; privacy.

Cross-Cutting Notes on Feasibility

• Compute and infra: Hybrid attention + MLA lower costs, but trillion-parameter MoE models still require substantial GPU memory; fused kernels and speculative decoding need engineering integration.
• Data and evaluation: Success depends on high-quality, verifiable tool traces and long-context corpora; robust reward models (bidirectional/focus) reduce verbosity and reward hacking.
• Safety and compliance: Tool-use agents must include audit trails, permissioning, and rollback; regulated sectors need domain validation and certifications.
• Human-in-the-loop: For high-stakes applications, maintain review checkpoints and escalation (instant by default, “think” when needed).
• Licensing and governance: Confirm open-source license terms for commercial use; establish monitoring for hallucinations, bias, and drift.

• Agentic intelligence: A paradigm where models act as autonomous agents that plan, use tools, and execute workflows in real environments. "Efficient and scalable agentic intelligence requires models that can deliver both low-latency responses and strong reasoning capabilities"
• Agentic reinforcement learning: RL tailored to train agents to act over long horizons with tools and environments, optimizing behavior beyond static chat. "to stabilize agentic reinforcement learning"
• Asynchronous RL: A reinforcement learning setup that decouples experience collection from parameter updates to improve throughput and stability. "combined with asynchronous RL that decouples rollout collection from parameter updates"
• Auxiliary-loss-free load balancing: An MoE routing strategy that balances expert utilization without adding an auxiliary loss term. "We employ the auxiliary-loss-free load balancing strategy with expert bias enabled"
• Bidirectional preference alignment: A training method that uses both rewards and penalties within one reward model to better align outputs with human preferences. "bidirectional preference alignment rewards informative, constraint-satisfying outputs and penalizes logical errors, hallucinations, and formulaic verbosity."
• Binary KL divergence: A divergence measure used here as a stabilization mechanism in RL by constraining policy updates with a binary-structured KL. "replaces the uniform fixed-ratio constraint in IcePop with binary KL divergence"
• Context-parallel communication: A distributed systems optimization that parallelizes across context segments to speed up training/serving of long sequences. "optimized context-parallel communication"
• Context window: The maximum number of tokens the model can attend to in a single input. "extend the context window to 256K"
• Continued pre-training: Further large-scale pre-training starting from a prior checkpoint to adapt to new architectures or data. "continued pre-training"
• Dynamic Pass Rating (DPR): An adaptive curriculum that prioritizes samples based on evolving pass rates and stability during training. "a novel sample selection strategy called Dynamic Pass Rating (DPR)."
• Evolutionary Chain of Thought (Evo-CoT): A method that iteratively refines and prunes reasoning chains to remove redundant steps while preserving correctness. "we integrate Evolutionary Chain of Thought (Evo-CoT)"
• Fine-grained MoE: A Mixture-of-Experts design with many small experts and precise routing to improve capacity and efficiency. "employs a fine-grained MoE architecture"
• FP8 training: Training with 8-bit floating point to reduce memory and improve throughput while maintaining stability. "improve FP8 training compatibility"
• Group Query Attention (GQA): An attention variant that groups queries to reduce computation and memory cost in standard attention mechanisms. "In our earlier Group Query Attention (GQA)-based architecture"
• Group Sequence Policy Optimization (GSPO): An RL algorithm that optimizes grouped sequence-level decisions, here tailored for tool-use efficiency. "we employ Group Sequence Policy Optimization (GSPO)"
• Grouped routing: An MoE routing scheme that selects experts in groups to balance load and improve efficiency. "We adopt a grouped routing strategy with $n_\text{group} = 8$ groups and top-4 group selection"
• Hybrid linear attention: An architecture mixing linear-time attention layers with full/latent attention layers to balance quality and efficiency. "adopt a unified hybrid linear attention architecture"
• IcePop: A prior RL approach whose fixed-ratio constraint is replaced here by binary KL for stability. "replaces the uniform fixed-ratio constraint in IcePop with binary KL divergence"
• KPop: A reinforcement learning framework designed for stable, large-scale agent training with asynchronous scheduling. "we propose KPop, a reinforcement learning framework"
• KV cache: The stored keys and values from previous tokens used to speed up autoregressive decoding. "MLA compresses the KV cache into a low-rank latent space."
• Lightning Attention: A linear-time attention mechanism used to reduce computational cost on long contexts. "integrates Lightning Attention with MLA"
• LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that injects low-rank updates into weight matrices. "KV LoRA Rank"
• Mid-Training: A stage after continued pre-training that focuses on high-quality data and long-context activation. "In the mid-training stage, we train on high-quality data"
• Mixture-of-Experts (MoE): A model architecture with many expert subnetworks, where a router chooses a subset to process each token. "employs a fine-grained MoE architecture"
• MTP (Multi-Token Prediction): An objective that trains the model to predict multiple future tokens, aiding speculative decoding and efficiency. "The MTP loss weight is set to 0.1."
• Multi-Head Attention (MHA): The standard attention mechanism with multiple parallel heads to capture diverse dependencies. "expanded to standard Multi-Head Attention (MHA)"
• Partial RoPE: Applying rotary positional encoding to only a subset of head dimensions rather than all. "Partial RoPE is applied to a subset of head dimensions."
• QK Norm: A normalization applied to queries and keys in attention that can affect stability and compatibility with certain conversions. "QK Norm is a nonlinear operation that prevents the KV weight matrix absorption required for efficient MLA inference."
• ReportLogic: A framework for evaluating long-form writing that emphasizes logical structure and discourse organization. "we propose ReportLogic framework"
• Retrieval-Augmented Generation (RAG): Techniques that combine retrieval from external sources with generation to improve factuality and reasoning. "retrieval-augmented generation (RAG) fusion"
• RMSNorm: Root Mean Square Layer Normalization, a normalization variant used in transformer layers. "We employ SiLU as the activation function and RMSNorm with $\epsilon = 10^{-6}$ for layer normalization."
• RoPE (Rotary Positional Encoding): A method for encoding position by rotating query/key vectors in attention. "via RoPE positional encoding"
• Scaling law: Empirical relationships describing how performance scales with compute, data, or model size. "We conduct scaling law experiments"
• Shortest-correct-response distillation: Training that distills the shortest accurate answers to improve information density per token. "We further apply shortest-correct-response distillation"
• SiLU: The Sigmoid Linear Unit activation function used in transformer layers. "We employ SiLU as the activation function"
• Speculative decoding: A decoding technique that uses a draft model to propose tokens and verifies them with the target model for speedups. "speculative decoding with continued MTP training"
• TransMLA: A method/tooling for converting to or operating with Multi-Head Latent Attention architectures. "TransMLA natively supports Full RoPE"

Sign up for free to add this paper to one or more collections.