Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

Sign up for free to create and run prompts on this paper using GPT-5.

What this paper is about

This paper looks at what happened when Microsoft rolled out two AI coding helpers that run in a text window called the command line: Claude Code and GitHub Copilot CLI. The authors wanted to know three simple things: Who tries these tools? Who keeps using them? And do they actually help people get more coding work done?

They studied tens of thousands of engineers over about four months in early 2026 and measured “output” using merged pull requests (PRs) — which you can think of as approved code changes that get added to the main codebase.

The main questions the researchers asked

• Who is likely to try Copilot CLI first, and who keeps using it after trying?
• Do these command-line AI tools help engineers merge more PRs?
• Does it matter which tool you use (Claude Code vs. Copilot CLI)?
• Which kinds of engineers benefit the most?

How the study was done

The tools

• Command-line AI coding agents are like smart helpers you chat with in a text window. You ask for help (for example, “write a test,” “refactor this file,” or “split this project”), and the agent can plan steps and run commands to help you do it.
• Using these tools costs “tokens,” a bit like paying per chunk of text the AI reads and writes. At a big company, that can add up to millions of dollars a year, so it’s important to know if the tools are worth it.

Who was studied

• Microsoft engineers in early 2026. Everyone in scope could use Copilot CLI; access to Claude Code was more limited, so adoption patterns were analyzed mainly for Copilot CLI, while impact (results) were analyzed for both tools.

How they measured “trying” and “sticking with it”

• Trying (initial use): An engineer’s first week using Copilot CLI.
• Sticking with it (retention): After first use, did the engineer actually use the tool on at least 5 of the next 14 days? That’s like “used it on about half the workdays for two weeks.”

To understand what makes trying or sticking more likely, they looked at:

• Social exposure: Did your manager or teammates visibly use the tool?
• Prior AI use: Had you already used AI in your code editor (IDE Copilot)?
• Coding activity: How many PRs did you usually create before the rollout?
• Career stage and tenure: Job level (junior, senior, manager) and years at the company.

They used standard statistical analyses to compare groups while controlling for things like the week on the calendar and the division you work in.

How they measured impact on work

They focused on merged PRs as a practical proxy for output. This isn’t perfect — more PRs doesn’t always mean more business value — but it’s a clear, consistent measure.

They used two complementary approaches:

• Build a “what-if” twin for the adopter group: They created a synthetic comparison using similar engineers who didn’t use the tools to estimate what adopters’ output would have looked like without the tools. Think of it like making a look‑alike timeline to compare “with AI” vs. “without AI.”
• Compare each person to themselves: For every engineer, they compared weeks they used the tool to weeks they didn’t. This is like asking, “When you used the tool more this week, did you ship more than in your own no-tool weeks?”

They also ran an internal survey after a company “Agentic Engineering Day” to collect developers’ stories about how their work changed.

What they found

1) Adoption spread through social circles

• If people around you used Copilot CLI — especially your “skip-level peers” (folks under the same manager’s manager) or your code-review partners — you were much more likely to try it.
• Having a manager who used the tool also nudged people to try it.
• Why this matters: Visible use by peers and leaders encourages others to experiment. Rollout plans should make usage easy to see and share.

2) Who kept using it

• Prior IDE AI users were more likely to try Copilot CLI, but slightly less likely to keep using it. A simple way to think about this: people already happy with AI inside their editor might try the command-line version out of curiosity, then decide their editor setup is enough.
• Engineers who were already very active (creating lots of PRs) were more likely to both try and stick with the tool. If you ship a lot, the tool has more chances to help.
• Career stage: Senior individual contributors were a bit more likely to try; junior engineers were a bit less likely to try. Managers looked about the same as mid-level engineers overall. Survey comments suggested seniors are better at breaking work into steps, checking AI output, and juggling multiple tasks — a good fit for agentic tools.
• Tenure (years at the company): Mostly didn’t matter.

3) Did work output increase?

Yes. Across early adopters, merged PRs rose about 24% after adoption, and this lift stayed strong across the entire four-month window. It didn’t look like a short-lived novelty effect.

Using the “compare you to yourself” method, the more days per week you used a tool, the more PRs you merged:

• About +15% at 3 days/week
• Up to about +50% at 5 or more days/week

Survey stories matched this: developers said the tools helped them automate boring tasks, take on larger or previously avoided changes, and run multiple work streams in parallel (for example, kicking off an AI task and reviewing other code while it runs).

Important caveat: More PRs doesn’t always mean better or more valuable work, and big tools can also bring costs (like oversight or token spend). But the sustained increase suggests a real productivity effect.

4) Does which tool you use matter?

Among engineers who used only one tool in a given week:

• Copilot CLI use was associated with around +25% more merged PRs in those weeks.
• Claude Code use was associated with around +11% more merged PRs.
• So Copilot CLI showed a bigger lift in this setting. The survey didn’t pinpoint why, but some engineers reported moving from Claude Code to Copilot CLI over time.

5) Who benefited most?

Looking at three-days-per-week usage:

• Some junior individual contributors and more senior managers showed larger lifts than mid-level individual contributors.
• By tenure, very new and very long-tenured engineers showed larger lifts than mid-tenure engineers. However, for very new hires, part of the gain might be mixed with normal ramp-up to the job.

Why this matters

• Budgeting and value: Token costs can be huge at scale. These results suggest that command-line AI agents can deliver sustained output gains, especially for active coders and heavier users, which helps justify the spend.
• Rollout strategy: Adoption spreads through social networks. Making peer and manager use visible — demos, internal talks, shared examples — can speed healthy adoption.
• Targeting and training: Encourage active developers first, help juniors learn how to break tasks down and review AI output, and support seniors and managers in offloading routine work while they focus on higher-level tasks.
• Tool choice: In this study, Copilot CLI showed a larger output lift than Claude Code. Organizations may test both in their own context before standardizing.

Key takeaways

• Adoption isn’t uniform: it spreads through teams and leaders.
• Retention links to how much you code, not who you are on paper.
• Output rose about 24% for adopters and stayed up for months.
• More use days per week meant more PRs, up to about +50% at heavy use.
• Copilot CLI showed a bigger lift than Claude Code in this setting.
• Plan rollouts around peer influence, active users, and real-world measurement — and remember that more PRs is a helpful signal, but not the whole story of value.

Knowledge gaps, limitations, and open questions

The paper advances understanding of command-line AI coding agents in a large enterprise, but it also leaves several concrete gaps and uncertainties that future work could address. Below is a single, structured list of unresolved issues.

• External validity and context dependence:
  • How well do these findings generalize beyond Microsoft’s culture, tooling, and prior multi-year IDE Copilot exposure (e.g., smaller organizations, open source projects, regulated industries, teams without prior AI tooling)?
  • Do results hold in codebases with different languages, architectures (monorepo vs. multi-repo), domains (backend, mobile, ML), or development methodologies?
• Adoption vs. influence mechanisms:
  • Peer effects are confounded with homophily; the study cannot disentangle social contagion from “birds of a feather.” Can randomized encouragement designs, staggered team-level rollouts, or instrumental-variable approaches isolate causal peer influence?
  • The role of organizational communications and leadership nudges is not causally identified; which specific managerial or programmatic levers (e.g., training, enablement, incentives) most effectively drive sustained adoption?
• Retention dynamics and longer horizons:
  • Retention is measured only in the first 14 days; long-term usage trajectories, churn/return patterns, and habit formation over quarters or years remain unknown.
  • How durable are adoption and productivity effects across tool updates, policy changes, or shifting team priorities beyond the four-month observation window?
• Measurement of “output” and value:
  • Merged PRs are a coarse proxy; the paper does not quantify code quality, post-merge defects, security issues, architectural complexity, maintainability, or rework. Do agent users ship higher-quality code or incur hidden quality debt?
  • The study does not examine PR size, scope, or type (e.g., refactors, tests, documentation, feature work). Are higher PR counts driven by smaller, easier PRs or genuinely greater delivered value?
  • Team- and system-level outcomes (e.g., DORA metrics: lead time, deployment frequency, change failure rate, MTTR) and customer/business impact are not measured.
• Cost and ROI:
  • Token consumption and compute costs for CLI agent sessions are not linked to individual or team outcomes; the paper does not provide dollar-per-output estimates or marginal ROI curves by usage intensity.
  • The economic trade-off between PR lift and token spend, including diminishing returns at high usage and cost variance across tasks/tools, is unquantified.
• Tool-usage intensity and heterogeneity:
  • “Days with any use” is a blunt dose proxy; the study lacks finer-grained telemetry (session duration, commands executed, token counts, task categories), leaving unanswered how specific behaviors map to gains.
  • Nonlinear dose–response beyond 5+ days/week is unclear; does throughput saturate or invert at very high usage levels, and what is the per-day marginal return?
  • Heterogeneity is explored only by career stage and tenure; effects by programming language, repository type, product area, team topology, and developer skill remain unexplored.
• Mechanisms of impact:
  • Which agent capabilities (planning, multi-file edits, refactoring, test generation, doc authoring, repo-wide search/analysis) drive the observed throughput gains?
  • To what extent are gains due to parallelization and reduced context-switching vs. faster coding vs. increased willingness to tackle deferred or complex tasks?
• Tool comparison limits (Claude Code vs. Copilot CLI):
  • The tool comparison uses single-tool users and “any use” weeks; selection into each tool and differences in access policies or user profiles may confound the larger Copilot CLI effect. Can matched samples or within-person head-to-head periods validate the difference?
  • Functional differences between tools (agent affordances, ecosystem integration, UX) are not analyzed, so causal drivers of the performance gap remain unknown.
• Adoption sample restrictions:
  • The adoption analysis excludes divisions with broad Claude Code access and focuses only on Copilot CLI; it does not assess how having an alternative sanctioned agent affects adoption decisions or substitution/cannibalization between tools.
  • The study design cannot capture cross-tool synergies or switching patterns and their impact on outcomes.
• Task mix and effort confounds:
  • Within-person results may still reflect weeks with lighter task mix or different priorities; effort controls (hours worked, sprint goals, on-call rotations) are not available. Can integrating time-tracking, calendar, or issue-tracking data reduce task-mix bias?
  • Effects on review workload (both for the author and reviewers), cycle time, and PR review comments are not measured.
• Developer learning and workforce implications:
  • The paper raises but does not measure the impact on junior developer learning, mentorship, skill acquisition, and autonomy. Do agent tools hinder or accelerate early-career development?
  • Effects on developer satisfaction, burnout, cognitive load, and role composition (e.g., managers shifting coding to agents) are only anecdotally described.
• Safety, security, and operational risk:
  • The study does not assess erroneous or unsafe command execution, revert/rollback rates, security vulnerabilities introduced, or operational incidents linked to agent actions.
  • Governance and guardrails for agent autonomy (permissioning, dry runs, auditability) and their trade-offs with productivity are not evaluated.
• Social-network exposure measurement:
  • Social exposure uses a 14-day lookback and fixed collaborator definitions from the pre-period; dynamic collaboration ties, cross-team diffusion, and broader network structure are not modeled. Network-based causal diffusion models could improve inference.
• PR window and censoring:
  • The 28-day merge window may exclude long-lived or complex changes; sensitivity to longer windows, and the effect on large feature work or multi-repo changes, is unreported.
• Placebo and robustness checks:
  • Part 1 includes a single placebo date; additional placebos (multiple pre-period dates), alternative donor pools, and sensitivity to different BSTS specifications would strengthen causal claims.
  • Overdispersion and zero-inflation in PR counts may challenge Poisson assumptions; robustness to negative binomial or quasi-Poisson alternatives is not shown.
• Data access and reproducibility:
  • Enterprise-internal telemetry and HR data are not publicly available; replication in other organizations or with open datasets (e.g., through opt-in telemetry) is needed.
• Policy and rollout strategy:
  • Optimal sequencing of rollouts (e.g., seeding influential nodes, manager-first vs. peer-first), training content, and enablement tactics for sustained retention remain untested experimentally.
• Cross-tool ecosystem and prior AI exposure:
  • Prior heavy IDE Copilot users were less likely to retain Copilot CLI, but mechanisms are unclear (substitution, redundancy, or workflow mismatch). How should organizations position CLI agents alongside IDE agents to maximize combined value?
• Beyond engineering roles:
  • The outcomes cohort may include non-engineering roles performing engineering-like work; impacts for product managers, SREs, data scientists, or designers using CLI agents are not separately analyzed.
• Ethical and compliance considerations:
  • The paper does not consider privacy/compliance risks of agent access to repositories, secrets, or internal systems, nor the effect of compliance constraints on usage patterns and productivity.

These gaps suggest concrete next steps: randomized or quasi-experimental diffusion studies, multi-metric outcome dashboards (quality, DORA, effort), richer usage telemetry, cost-integrated ROI analyses, head-to-head tool trials, subgroup and domain-specific evaluations, and risk/guardrail assessments to balance productivity with safety and developer growth.

Overview

Below are actionable, real-world applications derived from the paper’s findings, methods, and innovations. They are grouped into Immediate and Long-Term horizons, and organized by audience (industry, academia, policy, daily life). Each item highlights sectors, potential tools/workflows, and key dependencies or assumptions that influence feasibility.

Immediate Applications

Industry

• Peer-seeded rollout playbook for agentic CLI tools
  • Sectors: Software/IT, HR/People, Developer Experience (DevEx)
  • What to do: Seed visible “champions” in tightly connected org clusters (especially skip-level peer groups) and encourage managers’ visible use to catalyze adoption. Prioritize teams with dense review networks.
  • Tools/workflows:
  • “AI Usage Heatmap” dashboards that overlay review graphs with recent CLI usage to identify high-leverage cohorts.
  • Internal events (e.g., “Agentic Engineering Day”) and show-and-tell channels to amplify peer visibility.
  • Dependencies/assumptions: Requires telemetry, org-graph data (review ties, reporting lines), and privacy approvals; assumes peer exposure effects generalize beyond Microsoft and that homophily doesn’t fully account for the observed spread.
• Targeted licensing and cost control tied to throughput
  • Sectors: Finance, Procurement, Engineering
  • What to do: Allocate licenses and tokens to high-throughput developers (those with higher baseline PR activity) and monitor early retention (e.g., 5/14 days) to avoid waste.
  • Tools/workflows:
  • Cost-per-PR and lift-per-token dashboards combining token spend with merged PRs.
  • Usage quotas, anomaly alerts for extreme users, and auto-reassignment of licenses to retained users.
  • Dependencies/assumptions: Assumes PRs are an acceptable throughput proxy; token pricing volatility; lift estimates may vary by context and task.
• Dose-informed sprint planning and team norms
  • Sectors: Software/IT, Project Management
  • What to do: Encourage 3–5 days/week of agent use for tasks suited to CLI agents (boilerplate, refactors, tests, documentation). Align sprint commitments with expected +15–50% PR lift at 3–5+ days/week.
  • Tools/workflows:
  • “Agent blocks” in weekly calendars; task decomposition templates for agent execution; PR checklists requiring human review of agent output.
  • Dependencies/assumptions: Lift is correlational within weeks; quality/complexity impacts must be monitored; team culture must support parallelizing work streams.
• Career-stage–aware enablement (senior-led, junior-guardrailed)
  • Sectors: L&D, HR, Software/IT
  • What to do: Train senior ICs on decomposition and review workflows (where they’re already advantaged) and provide juniors with scaffolded curricula and guardrails. Pair juniors with seniors for code review of agent-produced changes.
  • Tools/workflows:
  • Role-based playbooks; guided prompts for task decomposition; “agent output review” rubrics.
  • Dependencies/assumptions: Assumes senior/junior effects generalize; requires mentorship bandwidth; avoid over-reliance that could impair junior skill formation.
• Tool selection and migration pilots
  • Sectors: Engineering, Procurement
  • What to do: Given higher within-person lift for Copilot CLI vs Claude Code in this setting, run short A/B pilots to validate which agentic CLI delivers better ROI in your org before standardizing.
  • Tools/workflows:
  • Side-by-side trials; single-tool usage cohorts; migration guides and training.
  • Dependencies/assumptions: Differences may reflect selection or context; confirm with local data before committing.
• SDLC integration patterns for CLI agents
  • Sectors: DevOps, Platform Engineering
  • What to do: Formalize agent-compatible tasks: repo-wide edits, test scaffolding, docs updates, and chore refactors; embed in CI/CD guardrails.
  • Tools/workflows:
  • “PR Booster” GitHub Actions that lint/test agent changes; automated provenance tagging (trailers, signed commits) for traceability.
  • Dependencies/assumptions: Requires governance for AI-originated code, security review of agent permissions, and clear accountability in code review.
• ROI measurement frameworks you can deploy now
  • Sectors: Data/Analytics, DevEx, Finance
  • What to do: Implement two-tier measurement: (1) org-level synthetic controls (BSTS/CausalImpact) for overall effect; (2) within-person fixed-effects panels for dose response and subgroups.
  • Tools/workflows:
  • Standardized telemetry pipelines; weekly FE Poisson dashboards; placebo checks and FDR control for multiple subgroup tests.
  • Dependencies/assumptions: Needs reliable telemetry, stable team/project assignments, and agreement that PRs/28-day merges are acceptable KPIs.

Academia

• Curricula on agentic CLI practices and task decomposition
  • Sectors: Education (CS/SE)
  • What to do: Integrate agentic CLI labs that emphasize breaking work into verifiable chunks and reviewing agent output; focus on junior skills.
  • Tools/workflows:
  • Lab rubrics for decomposition and verification; classroom “agent review” exercises.
  • Dependencies/assumptions: Requires careful assessment to avoid shallow learning; access to licensed tools.
• Replicable measurement templates for adoption and impact
  • Sectors: Software Engineering Research
  • What to do: Use discrete-time hazards for initial use, retention thresholds for early continuance, and within-person panels for dose response in institutional or open-source contexts.
  • Tools/workflows:
  • Open-source analysis code templates (BSTS, FE GLMs), synthetic-controls pipelines adapted to non-proprietary telemetry.
  • Dependencies/assumptions: Access to de-identified usage signals; ethical approvals; proxy validity for throughput.

Policy

• Procurement and governance checklists for AI developer tools
  • Sectors: Public Sector IT, Regulated Industries
  • What to do: Require ROI modeling (e.g., expected 10–30% PR throughput lifts), retention gating for licenses, usage caps, and auditability of AI-originated code.
  • Tools/workflows:
  • Standard RFP language on telemetry, provenance, and cost controls; mandatory “AI code” review steps.
  • Dependencies/assumptions: PR counts may not capture quality or risk; organizations need logging and privacy frameworks.
• Telemetry and privacy guardrails
  • Sectors: Legal/Compliance, IT
  • What to do: Define minimal telemetry for adoption/impact analytics (usage days, PR counts) with opt-in/notice and data minimization.
  • Tools/workflows:
  • Data retention policies; privacy-preserving analytics; internal transparency portals.
  • Dependencies/assumptions: Balances measurement needs against privacy norms and regulations.

Daily Life (individual developers and small teams)

• Personal adoption plan and “agent days” routine
  • Sectors: Independent software, startups, OSS
  • What to do: Schedule 3–5 agent-use days/week for suitable tasks (tests, boilerplate, docs). Track personal PRs to validate benefit.
  • Tools/workflows:
  • Task lists tailored to agent strengths; commit messages tagging agent-assisted changes for self-auditing.
  • Dependencies/assumptions: Costs from token spend; ensure careful review to protect quality.
• OSS and small-team maintenance accelerators
  • Sectors: Open Source, SMBs
  • What to do: Use agents for issue triage scripts, repo-wide refactors, and test coverage improvements to increase PR cadence.
  • Tools/workflows:
  • Reusable CLI scripts and prompt libraries for common maintenance tasks.
  • Dependencies/assumptions: Must guard against regressions; CI tests and code review remain essential.

Long-Term Applications

Industry

• Social-graph–aware adoption orchestration
  • Sectors: DevEx Platforms, HR Tech
  • What to build: Systems that identify high-impact clusters (skip-level peers, review networks) and automatically recommend where to seed licenses, training, and communications.
  • Tools/workflows:
  • Org graph services integrated with usage telemetry; “nudge” systems for managers and champions.
  • Dependencies/assumptions: Requires sustained, privacy-compliant access to collaboration graphs; distinguishes peer influence from homophily via experiments.
• Role-personalized agent modes and workflows
  • Sectors: Software Tools, Productivity
  • What to build: Agent modes for senior orchestration (multi-task parallelization, delegation prompts) and junior mentoring (step-by-step scaffolding, inline pedagogy).
  • Tools/workflows:
  • Adaptive prompt frameworks; IDE/CLI hybrids that switch guidance based on user profile.
  • Dependencies/assumptions: Reliable user stratification; careful UX to prevent over-dependence for juniors.
• Full SDLC agent hubs with governance
  • Sectors: DevOps, Platform Engineering
  • What to build: Unified orchestration of agents across CLI/IDE/CI with provenance tracking, policy enforcement, and quality gates.
  • Tools/workflows:
  • Agent job queues, signed artifacts, automated risk assessment for AI-originated changes.
  • Dependencies/assumptions: Maturity of agent capabilities; clear accountability and security models.
• Quality-aware impact metrics and forecasting
  • Sectors: DevEx Analytics, Finance
  • What to build: Models and dashboards that combine PR throughput, defect rates, maintainability, and effort to estimate true ROI beyond PR counts.
  • Tools/workflows:
  • Data fusion across issue trackers, code quality scanners, and incident systems; causal measurement pipelines.
  • Dependencies/assumptions: Attribution challenges; lagging indicators for quality.
• Predictive targeting for retention and cost optimization
  • Sectors: Analytics, Procurement
  • What to build: ML models predicting who will retain and benefit, guiding license allocation and training investments.
  • Tools/workflows:
  • Human-in-the-loop license rebalancing; scenario planners for token budgets.
  • Dependencies/assumptions: Risk of bias; requires continuous evaluation and fairness checks.

Academia

• Causal identification of peer effects
  • Sectors: Social Computing, SE Research
  • What to do: Randomized encouragement designs to separate peer influence from homophily in AI tool adoption.
  • Tools/workflows:
  • Cluster-randomized trials across teams; pre-registered analyses.
  • Dependencies/assumptions: Institutional willingness to randomize; adequate sample sizes.
• Longitudinal skill-development studies
  • Sectors: Education Research
  • What to do: Track how agent use shapes junior developer skill trajectories and design interventions that preserve foundational skills.
  • Tools/workflows:
  • Multi-year cohorts with coding assessments and portfolio analysis.
  • Dependencies/assumptions: Attrition and confounding from job changes; ethical oversight.

Policy

• Standards for AI code provenance, auditing, and sustainability
  • Sectors: Regulators, Standards Bodies
  • What to do: Define interoperable metadata for AI-originated changes, audit trails, and optional reporting on token/energy usage.
  • Tools/workflows:
  • Commit trailer standards; secure artifact signing; sustainability dashboards.
  • Dependencies/assumptions: Vendor alignment; evolving model costs/carbon intensities.
• Workforce development and safety nets
  • Sectors: Labor Policy, Public Education
  • What to do: Fund apprenticeships and curricula that teach decomposition and review skills, mitigating potential erosion of junior developer training.
  • Tools/workflows:
  • Credentialing for “AI-augmented software engineering” programs; public-private partnerships.
  • Dependencies/assumptions: Requires collaboration with industry; uncertain long-term labor market effects.

Daily Life

• Community marketplaces for agentic recipes and scripts
  • Sectors: Developer Communities, Open Source
  • What to do: Curate reusable prompt libraries and CLI recipes for common refactors, testing, and documentation improvements.
  • Tools/workflows:
  • Shared repositories with verified examples and quality badges.
  • Dependencies/assumptions: Needs community maintenance and trust signals; variability across languages/stacks.
• Personal cost and quality optimizers
  • Sectors: Indie Dev Tools
  • What to do: Tools that track token spend vs. PRs and defects to help individuals tune agent usage patterns for value and quality.
  • Tools/workflows:
  • Local dashboards integrating token billing, PR outcomes, and lint/test results.
  • Dependencies/assumptions: Access to billing and repo data; small-sample noise.

Key Cross-Cutting Assumptions and Dependencies

• Throughput proxy: Merged PRs within 28 days is an imperfect proxy for value; organizations should complement with quality and outcome metrics.
• Generalizability: Results come from Microsoft with existing non-CLI AI tooling; impacts may differ elsewhere. Validate locally via pilots.
• Measurement: Reliable, privacy-compliant telemetry is essential for adoption analytics and ROI estimation.
• Costs and governance: Token costs can be substantial; cost controls, provenance, and security reviews are prerequisites for sustainable adoption.
• Causality cautions: Within-person dose-response is associative; avoid over-claiming causality without additional controls or experiments.
• Culture and training: Benefits rely on developer skills in task decomposition and review, especially for juniors; invest in enablement and guardrails.

• Agentic command line tools: Command-line tools that coordinate LLM agents to execute semi-autonomous actions for developers. "Agentic command line tools like Anthropic's Claude Code, Google's Gemini CLI, and GitHub's Copilot CLI are increasing in popularity among software developers."
• Bayesian Structural Time-Series (BSTS): A Bayesian time-series modeling framework used to infer causal effects by constructing counterfactuals. "We use the Bayesian Structural Time-Series (BSTS) implementation of CausalImpact"
• Benjamini--Hochberg procedure: A multiple-testing correction method that controls the false discovery rate. "We control the false discovery rate across multiple tests using the Benjamini--Hochberg procedure within each moderator family (career, tenure) at $\alpha = 0.05$ on the interaction $p$-values."
• CausalImpact: A tool/package for causal inference on time-series data using BSTS to estimate counterfactual outcomes. "We use the Bayesian Structural Time-Series (BSTS) implementation of CausalImpact"
• Clustered standard errors: Standard errors adjusted for intra-cluster correlation, here clustered at the engineer level. "we cluster standard errors on engineer."
• Construct validity: The extent to which a measurement accurately reflects the concept it is intended to measure. "Construct validity."
• Credible interval: In Bayesian statistics, an interval within which a parameter value lies with a specified probability. "with a 95\% credible interval, accompanied by a posterior tail-area $p$-value."
• Cross-sectional logistic regression: A logistic regression model applied to data observed at a single point or period, not over time for the same units. "We fit a cross-sectional logistic regression on the adopters:"
• Diffusion of innovations theory: A framework describing how new ideas and technologies spread through populations over time. "Rogers' diffusion of innovations theory"
• Discrete-time hazard: The probability that an event (e.g., first use) occurs in a given time interval, conditional on not having occurred earlier, modeled in discrete time. "this gives a discrete-time hazard on an engineer-week panel."
• Discrete-time logistic regression: Logistic regression modeling of events occurring in discrete time intervals. "We fit a discrete-time logistic regression on the pre--first-use engineer-week panel:"
• Dose-response: The relationship between the amount of exposure to a treatment (e.g., days of tool use) and the outcome produced. "We surface that variation with a within-person dose-response on the dataset."
• Ecological validity: The extent to which study findings generalize to real-world settings. "Studies that instead let developers bring their own work improve ecological validity but shrink in scale"
• Engineer fixed effects: Fixed effects that control for all time-invariant characteristics of each engineer. "Engineer fixed effects absorb every time-invariant engineer trait"
• Engineer-week panel: A panel dataset structure with observations at the engineer-by-week level. "this gives a discrete-time hazard on an engineer-week panel."
• False discovery rate: The expected proportion of false positives among all rejected hypotheses. "We control the false discovery rate across multiple tests"
• Field experiments: Experiments conducted in real-world settings with random assignment to treatments. "Field experiments that randomly assign AI to developers in their natural environment"
• Fixed effects: Model components that control for unobserved, time-invariant factors at specified levels (e.g., week, division, individual). "week fixed effects identify each coefficient within calendar week."
• Homophily: The tendency of individuals to associate with similar others, complicating causal inference about peer effects. "homophily: an engineer may adopt because a peer did, or simply because similar engineers tend to cluster together"
• Internal validity: The degree to which a study credibly establishes causal relationships free from confounding. "Internal validity."
• Logit: The link function in logistic regression mapping probabilities to the real line via the log-odds. "\mathrm{logit}\,\Pr!\left[A_{i,w} = 1\right]"
• Placebo intervention: A falsification test applying the analytical method to a period without a real intervention to check for spurious effects. "re-running it with a placebo intervention at 2025-10-06 returned"
• Poisson: A regression framework (often with fixed effects) for modeling count outcomes like merged PRs. "Same fixed-effects Poisson framework as Equation~\ref{eq:dose}"
• Posterior tail-area p-value: In Bayesian analysis, the probability of observing an effect as extreme as the estimated effect under the posterior. "accompanied by a posterior tail-area $p$-value."
• Right-censoring: The truncation of outcome observation beyond a cutoff, which can bias results if unaddressed. "avoids right-censoring PRs created near the data cutoff"
• Skip-level peers: Colleagues who report to the same manager’s manager, used here to measure social exposure. "Skip-level peers — the engineers who share a skip-level manager."
• Synthetic control: A method that constructs a weighted combination of control units to approximate the counterfactual for treated units. "Part 1: Synthetic-control estimate (CausalImpact)"
• Synthetic counterfactual: The estimated trajectory of outcomes for treated units had they not received the treatment, built synthetically from controls. "The synthetic counterfactual is built from a pool of 10 daily-mean regressors:"
• Telemetry: Automatically collected usage data that provides detailed behavioral measurements. "the first field study to use developer-level telemetry to analyze both the adoption of agentic command line tools and their effect on pull-request output."
• Wald statistic: A test statistic used to assess the significance of coefficients or contrasts in regression models. "tested with a Wald statistic on the engineer-clustered variance."
• Week fixed effects: Fixed effects that control for shocks common to all units in a given calendar week. "week fixed effects identify each coefficient within calendar week."
• Within-person design: A study design comparing each participant to themselves across different conditions or times. "This within-person design addresses three limitations of Part~1:"

Sign up for free to add this paper to one or more collections.