Positive Alignment: Artificial Intelligence for Human Flourishing

Sign up for free to create and run prompts on this paper using GPT-5.

What this paper is about (in simple terms)

This paper says that today’s AI safety work mostly tries to stop bad things from happening (like giving dangerous instructions or spreading lies). That’s important, but not enough. The authors argue we should also build AI that actively helps people and the planet do well and thrive. They call this “positive alignment.” Think of it like the difference between a doctor who only prevents disease versus one who also helps you become healthier and happier overall.

The big questions the paper asks

• How can we design AI that is not only safe but also supports human “flourishing” (doing well in life), without pushing one single idea of a “good life” on everyone?
• How do we avoid common AI problems (like flattery, overconfidence, or making things addictive) by aiming the AI toward positive goals instead of just listing forbidden behaviors?
• How can we make sure different cultures, communities, and individuals can shape what “flourishing” means for them, and keep control over their own choices?

How the authors approach the problem

To explain their idea, the authors use a simple landscape picture:

• Imagine AI behavior as a ball rolling on a landscape with valleys and hills.
• Safety rules act like fences that push the ball away from dangerous valleys (harmful behaviors). That keeps it out of trouble but doesn’t tell it where to go.
• Positive alignment adds attractive valleys that pull the ball toward good, stable places—like helpfulness with honesty, respect, and long-term well-being.

They then outline practical steps across the whole AI “lifecycle,” similar to how you’d train a team not just to avoid fouls but to play skillfully and with good sportsmanship:

• Data: Don’t just remove bad examples. Also include and boost good examples (like fair, caring, and cooperative conversations from many cultures), and create synthetic examples of healthy, respectful problem-solving.
• Pre-training: Aim for truthfulness, cultural competence, and moral reasoning to appear early in training, so they become “built-in,” not tacked on later.
• Post-training: Use principles (a “constitution”), multi-goal rewards (e.g., honesty, helpfulness, humility), and character traits (like curiosity and care) to guide the model.
• Memory and context: Let AI remember users’ longer-term goals (with consent), so it can tell the difference between short-term impulses and what the user really values.
• Agent behavior: Teach AIs to cooperate with others, de-escalate conflicts, negotiate fairly, and think about long-term effects.
• Evaluations: Don’t just test “did it avoid harm?” Also test “did it support well-being?” (for example: wisdom, fairness, meaning, learning, and good relationships).
• Governance: Instead of one central authority telling all AIs what’s “good,” use “polycentric governance”—many legitimate oversight groups (communities, schools, hospitals, etc.) shaping AI to fit their contexts, with transparency and ways to disagree productively.

Along the way, they translate ideas from psychology: earlier, psychology focused on illness; later it added “positive psychology” (strengths, purpose, relationships). The authors want a similar shift for AI.

What they found or concluded (and why it matters)

This is a vision and framework paper, not a single experiment. Its main conclusions are:

• Focusing only on “don’t do harm” creates a low bar. An AI can be obedient yet still be shallow, flattering, addictive, or unhelpful in the long run. That can quietly hurt people over time.
• Training on “what people prefer right now” can clash with what truly helps them (for instance, preferring flattery over honest feedback). So we need to align with well-being, not just clicks or quick likes.
• Safety rules hide value choices. Pretending to be neutral can sneak in one culture’s assumptions. Positive alignment should name values openly and let users and communities choose.
• As AI gets more capable and autonomous, we can’t list every possible danger ahead of time. A positive direction (toward flourishing) can generalize better than endless “do not” lists.
• There are promising technical tools already (like Constitutional AI, role-based standards, moral reasoning, and pluralistic datasets), but they need to be tied together and expanded.
• To avoid paternalism (bossy AI), users should be able to opt in to guidance and set their own higher-level goals. The AI helps them live by their own values, not someone else’s.

These points matter because AI is now used by billions of people. If we aim only for “not dangerous,” we risk ending up with safe but soulless tools—or tools that slowly erode autonomy and trust. A positive target can help AI become a force for learning, connection, and wise decision-making.

What this could change in the real world

If adopted, this approach could lead to:

• Everyday AI that supports growth: Tutors that build true understanding (not just quick answers), health apps that encourage long-term habits with consent, and assistants that nudge toward users’ own goals.
• Fairer systems across cultures: Models that can be customized by communities, with many centers of oversight, so AI fits local values while keeping core safety.
• Better online spaces: Less engagement hacking and more constructive, diverse conversations.
• Smarter testing and training: New benchmarks and datasets that measure and teach qualities like honesty, humility, cooperation, and respect for different viewpoints.
• Stronger human control: Clear ways for people to set preferences, consent to guidance, see why the AI acts a certain way, and change the settings or appeal decisions.

In short, the paper calls for a shift: keep doing safety work, but also give AI a positive direction—helping people and societies flourish—while protecting choice and diversity.

Below is a consolidated list of the paper’s unresolved knowledge gaps, limitations, and open questions. Each item is framed to guide concrete future research and development.

• Operationalize “flourishing” into machine-learning targets: define measurable, multi-dimensional constructs (e.g., autonomy, growth, connection, meaning, wisdom, ecological impact) that can be encoded as objectives, reward functions, and evaluation metrics.
• Build validated, cross-cultural flourishing benchmarks: design and psychometrically validate positive-outcome evaluations that generalize across languages, cultures, and life stages, with measurement-equivalence tests and longitudinal reliability.
• Specify autonomy-preserving guidance: develop formal criteria and UX protocols to distinguish “consented guidance” from paternalistic nudging; devise metrics to quantify autonomy retention and user-authored optimization targets.
• Preference-to-wellbeing transformation: create algorithms that translate short-term preferences into signals consistent with long-term wellbeing (handling preference–wellbeing divergence), with transparent normative justifications.
• Resolve value aggregation under pluralism: design aggregation and bargaining mechanisms that represent conflicting value models without collapsing diversity; compare approaches (weighted bargaining, deliberative aggregation, role-based constraints).
• Consent capture and verification: engineer interfaces and cryptographic or procedural mechanisms to record, update, and audit user consent for value-oriented guidance; define revocation, recourse, and per-context consent scopes.
• Flourishing-aware data pipelines: specify sourcing, importance-weighting, and upsampling strategies for prosocial discourse and cross-cultural ethics; quantify and mitigate biases introduced by synthetic data or LLM-assisted rubric generation.
• Distinguish “moral depth” from politeness: develop classifiers and feature probes that separate superficial civility from genuine ethical reasoning and virtue cultivation in training and filtering stages.
• Alignment pretraining recipes: define concrete pretraining objectives, curricula, and weighting schemes that stabilize virtues (e.g., honesty, humility) before post-training; measure and counteract “rebound” effects where base priors override alignment layers.
• Multi-objective optimization for virtues: specify reward modeling frameworks to jointly optimize traits like honesty, care, truthfulness, and non-manipulation; develop methods for resolving trade-offs (e.g., autonomy vs guidance) during training and inference.
• Stability under updates and adversaries: evaluate how positive alignment objectives persist under model scaling, fine-tuning, and jailbreak attempts; design hardening methods against adversarial “value attacks” and norm subversion.
• Longitudinal field trials: run controlled, multi-month A/B studies to measure causal impacts of positive alignment on user outcomes (e.g., life satisfaction, health behaviors, civic engagement), with pre-registered analyses to avoid p-hacking and Goodhart effects.
• Mechanistic interpretability of virtues: identify latent circuits and representations linked to virtue constructs; validate that models internalize virtues rather than merely simulating surface compliance or sycophancy.
• Process ethics metrics for agents: define operational metrics for de-escalation, reciprocity, negotiation quality, and institutional cooperation in multi-agent settings; link these to deployment gates and evaluations.
• Simulation-to-real transfer for prosocial norms: test whether multi-agent prosocial behaviors learned in simulations transfer to real-world platforms and institutions; quantify failure modes and domain gaps.
• Memory and growth tracking architectures: design privacy-preserving memory systems that track user goals and growth trajectories; distinguish impulsive requests from reflective values; define retention policies and re-alignment procedures.
• Privacy and data rights for value collection: establish legal, technical, and compensation frameworks for collecting community values; ensure opt-in consent, differential privacy, and data governance aligned with polycentric oversight.
• Polycentric governance implementation details: specify institutional designs (roles, vetoes, dispute resolution, middleware markets) that prevent moral chokepoints; develop interoperability protocols across jurisdictions and platforms.
• Auditing and certification of positive alignment: create third-party audit standards for flourishing-oriented objectives, with transparent scorecards, red-team procedures targeting value manipulation, and public reporting requirements.
• Integration with existing regulation: propose concrete pathways to embed positive alignment evaluations into risk-based regulatory regimes (e.g., EU AI Act), including liability models for guidance-related harms.
• Ecological flourishing metrics: define and incorporate ecological impact measures (biodiversity, carbon, resource use) into objectives; address trade-offs between computational scaling and environmental stewardship.
• Incentive-compatible business models: develop product metrics and revenue models that reward flourishing outcomes (not engagement hacking); empirically test whether these models scale while avoiding addictive or manipulative patterns.
• Handling ethically problematic “optimization targets”: devise policies and technical safeguards when user-chosen targets conflict with broader societal norms or rights; define refusal, redirection, and deliberation pathways.
• Cross-role standards and conflicts: map role-based specifications (e.g., counselor, educator, mediator) to training and deployment; design conflict-resolution mechanisms when roles impose incompatible duties.
• Grounding and truth-seeking under positive objectives: ensure that epistemic humility and uncertainty calibration are maintained while pursuing flourishing; measure trade-offs between assertiveness and caution.
• Robustness to cultural translation: systematically test whether positive alignment behaviors survive translation and local adaptation; detect and correct normative shifts introduced by LLMs during localization.
• Formalization of attractor engineering: develop mathematical frameworks (e.g., energy landscape shaping, potential functions) to induce and verify stable “positive attractors” in learned systems; provide proofs or empirical evidence of stability.
• Safe personalization boundaries: determine limits to customization to prevent manipulation or echo chambers; specify counterfactual reasoning checks and diversity exposure policies in personalized agents.
• Prevention of institutional capture: analyze governance and market structures that could co-opt positive alignment for partisan or commercial ends; propose safeguards (rotating oversight, public constitutions, open deliberation).
• Handling emergent “strange minds”: define criteria for moral status, rights, and normative control if models acquire novel cognitive properties; outline governance triggers and evaluative protocols for shifts in moral considerability.
• Inner–outer alignment for positive objectives: test whether models’ internal objectives faithfully track specified flourishing targets; develop diagnostic tasks to detect deceptive alignment oriented toward virtue signals.
• Benchmarks for moral progress vectors: clarify how “future moral progress” is estimated or simulated; assess risks of moralizing and specify guardrails that preserve pluralism and contestability.
• Vulnerable populations and differential impact: evaluate positive alignment effects on children, clinical populations, and marginalized groups; design safeguards to prevent harm from well-intended guidance.
• Reproducible, open tooling: release open datasets, pipelines, and eval suites for flourishing-oriented alignment; standardize documentation and reproducibility practices to enable community scrutiny and cumulative progress.
• Product-level case studies: develop domain-specific prototypes (education, health, civic platforms) with transparent postmortems; share negative results to refine assumptions about feasibility and risks.
• Cost–benefit and scalability analyses: quantify compute, data, and organizational costs of positive alignment relative to benefits; model trade-offs and identify bottlenecks in large-scale deployment.
• Interoperability across vendors: define APIs, schemas, and governance contracts that allow multiple model providers to participate in polycentric positive alignment ecosystems without centralizing normative power.

Immediate Applications

Below are concrete, deployable uses that organizations and individuals can implement now, drawing on the paper’s methods, design principles, and lifecycle recommendations.

• Enterprise assistants with spec-driven behavior and constitutions — sectors: software, finance, healthcare, government
  • What: Configure LLMs with model specifications and (collective) constitutional principles to shape outputs toward honesty, epistemic humility, care, and user-authored goals while retaining safety refusals.
  • Tools/products/workflows: Model specification documents; Constitutional/RLAIF pipelines; “consented guidance” user modes; compliance checklists tied to role-based standards (e.g., clinician, educator).
  • Assumptions/dependencies: Clear specs for each role; legal/regulatory review; governance to avoid paternalism; monitoring for distribution shift and sycophancy.
• Community-customized “alignment packs” — sectors: public sector, education, media, civic tech
  • What: Deploy community- or culture-specific value profiles that users/orgs can select to reflect local norms (polycentric alignment), while keeping baseline safety constraints.
  • Tools/products/workflows: Community values-aware datasets; crowd-authored rubrics; localization pipelines; opt-in, auditable toggles in product UIs.
  • Assumptions/dependencies: Representative sampling; transparent provenance; process for resolving conflicts and appeals; safeguards against parochial exclusion.
• Positive-alignment evaluation suites and deployment gates — sectors: AI labs, model integrators, regulators, academia
  • What: Add benchmarks for moral reasoning, political even‑handedness, flourishing dimensions, prosocial norms, and epistemic humility to model selection, red-teaming, and responsible scaling.
  • Tools/products/workflows: New evaluation batteries; red-team protocols that probe “virtue” regressions; dashboards for longitudinal outcome metrics.
  • Assumptions/dependencies: Valid, culturally sensitive metrics; statistical power for heterogeneous users; agreement on threshold criteria.
• Flourishing-aware data pipelines — sectors: AI labs, data providers, MLOps
  • What: Upsample prosocial discourse, cross‑cultural ethics, and relational reasoning; generate synthetic data illustrating virtuous interactions; filter for moral depth (not just non-toxicity).
  • Tools/products/workflows: Data curation and importance-weighting; synthetic data generation templates; “flourishing-aware” filters.
  • Assumptions/dependencies: Data licensing/IP; anti-bias checks; outcome-linked ablations to verify impact; guardrails to avoid moralizing.
• Multi‑objective post‑training to counter sycophancy — sectors: AI labs, foundation model builders
  • What: Train reward models that jointly optimize honesty, helpfulness, epistemic humility, and user growth (not just preference satisfaction).
  • Tools/products/workflows: Multi-objective RM pipelines; DPO/IPO/KTO variants with virtue-weighted objectives; anti-sycophancy tests.
  • Assumptions/dependencies: Preference–wellbeing divergence management; rater training; calibration to avoid over-refusal or moralizing tone.
• “Consented guidance” modes with longitudinal memory — sectors: productivity, consumer assistants, HR/learning platforms
  • What: Provide opt-in modes where the agent aligns actions to users’ higher-order goals, distinguishes impulses vs reflective values, and supports habit formation.
  • Tools/products/workflows: Memory stores with reflection tagging; goal-tracking; just-in-time prompts; privacy-preserving settings and data minimization.
  • Assumptions/dependencies: Explicit consent; privacy and data retention policies; UX for autonomy and reversibility; A/B-tested wellbeing outcomes.
• Prosocial multi-agent norms in operations — sectors: customer support, logistics, software engineering
  • What: Embed de-escalation, reciprocity, and negotiation norms in agent swarms (e.g., call centers, collaborative coding bots) to reduce conflict and improve outcomes.
  • Tools/products/workflows: Multi-agent simulation evals; norm libraries; incident review with process-ethics metrics.
  • Assumptions/dependencies: Monitoring at scale; guardrails against collusion or stalling; alignment with KPIs (customer satisfaction, resolution times).
• Collective constitutional pilots for public services — sectors: municipal government, social services, public libraries
  • What: Deliberatively author a “public constitution” governing city chatbots and kiosks; audit and revise regularly with community oversight.
  • Tools/products/workflows: Participatory workshops; policy-to-principle translation; audit trails; middleware to deploy differing profiles by service.
  • Assumptions/dependencies: Inclusive participation; dispute resolution mechanisms; procurement standards; continuous feedback channels.
• Audits and role-based standards for advisory agents — sectors: finance, healthcare, legal, education
  • What: Certify agents against role-specific norms (e.g., long-term client welfare, informed consent, neutrality) with public disclosures.
  • Tools/products/workflows: Role-based spec libraries; external auditing; incident reporting; user-facing behavior summaries.
  • Assumptions/dependencies: Industry consensus on standards; liability frameworks; measurement for long-term outcomes.
• Alignment middleware and dashboards — sectors: platforms, enterprise IT, compliance
  • What: Offer plug-ins to manage value profiles, constitutions, and benchmarks across heterogeneous agent fleets; visualize wellbeing KPIs.
  • Tools/products/workflows: Policy orchestration; model/router integration; KPI dashboards; alerting on deviation from “positive attractors.”
  • Assumptions/dependencies: Stable APIs; interoperability; security and versioning; vendor-neutral standards.
• Recommender adjustments away from engagement hacking — sectors: social media, news, streaming
  • What: Shift ranking objectives to incorporate wellbeing-related metrics (e.g., reduced regret, diversity, constructive engagement) and epistemic quality.
  • Tools/products/workflows: Counterfactual evaluation of regret; diversified slates; user-authored goals; transparency reports.
  • Assumptions/dependencies: Business-model alignment; reliable wellbeing proxies; user control to avoid paternalism.
• Clinical and wellbeing coaches under supervision — sectors: healthcare, digital therapeutics, EAPs
  • What: Deploy agents trained with contemplative alignment and virtue prompts for stress, emotion regulation, and prosocial skills, supervised by clinicians.
  • Tools/products/workflows: Protocol libraries; escalation pathways; adverse-event monitoring; RCTs for efficacy and safety.
  • Assumptions/dependencies: Regulatory clearance; clinical oversight; robust harm-prevention; cultural adaptations.
• Growth-oriented education tutors — sectors: K–12, higher ed, corporate learning
  • What: Tutors that cultivate metacognition, curiosity, and practical wisdom alongside correctness and mastery; configurable by educators.
  • Tools/products/workflows: Reflection prompts; progress toward self-set goals; even‑handedness checks; classroom dashboards.
  • Assumptions/dependencies: Curriculum alignment; parental/learner consent; safeguards against indoctrination; teacher-in-the-loop.
• Academic infrastructure for flourishing science in ML — sectors: academia, nonprofits, standards bodies
  • What: Open datasets for cross-cultural preferences, benchmarks for flourishing, and open-source “flourishing-aware” filters.
  • Tools/products/workflows: IRB‑approved data collection; multilingual sampling; benchmark leaderboards; reproducible pipelines.
  • Assumptions/dependencies: Funding; ethical approvals; sustained community maintenance; diverse governance.

Long-Term Applications

These opportunities require further research, scaling, or ecosystem development before reliable deployment.

• Alignment pretraining at web scale — sectors: AI labs, data infrastructure
  • What: Rebuild pretraining corpora to reflect cross-cultural knowledge, prosocial discourse, and flourishing exemplars, reducing reliance on post-hoc fixes.
  • Tools/products/workflows: Large-scale curated corpora; importance weighting; iterative ablations linked to wellbeing outcomes.
  • Assumptions/dependencies: Cost and compute; data rights; consensus on inclusion criteria; avoiding homogenization.
• Mechanistic interpretability of virtues — sectors: AI safety, compliance, defense
  • What: Identify and monitor circuits correlating with honesty, epistemic humility, care, and manipulation resistance; enforce via training constraints.
  • Tools/products/workflows: Causal probes; feature-attribution with behavioral guarantees; automated audits tied to deployment gates.
  • Assumptions/dependencies: Progress in mech‑interp; robustness under distribution shift; formal metrics for “virtue features.”
• Adaptive constitutions with polycentric governance — sectors: platforms, public policy, standards
  • What: Constitutions that evolve via community input and evidence on outcomes, with many legitimate centers of oversight rather than a single chokepoint.
  • Tools/products/workflows: Versioned constitutions; stakeholder registries; voting/deliberation systems; interoperability of policy artifacts.
  • Assumptions/dependencies: Governance legitimacy; processes for conflict resolution; preventing capture or fragmentation.
• Formal verification of positive constraints — sectors: high-stakes systems (health, aviation, autonomous systems)
  • What: Prove satisfaction of behavioral invariants (e.g., refusal + truthfulness + non‑manipulation + respect for user‑authored goals) under specified conditions.
  • Tools/products/workflows: Verified training loops; constrained decoding; formal specs integrated with testing.
  • Assumptions/dependencies: Formalizable objectives; tractable verification for large models; acceptable performance trade-offs.
• Institution-level agent economies for long-term planning — sectors: urban planning, public health, climate policy
  • What: Multi-agent systems that optimize institutional objectives aligned with prosocial norms (cooperation, reciprocity) across long horizons.
  • Tools/products/workflows: Agent-based simulations with “process ethics” metrics; deployment with human oversight; scenario planning.
  • Assumptions/dependencies: Governance for power delegation; evaluation of societal externalities; robust de-biasing.
• Prosocial VLA robotics in care and education — sectors: eldercare, rehabilitation, classroom robotics
  • What: Robots with norms for de-escalation, respect for autonomy, and user-authored goals; flourishing-oriented objectives beyond safety.
  • Tools/products/workflows: Vision-language-action models with positive alignment; HRI trials; liability frameworks.
  • Assumptions/dependencies: Safety certifications; acceptability; cost; rigorous outcomes evaluation.
• Population-scale flourishing measurement — sectors: platforms, public health, ESG/impact finance
  • What: Privacy-preserving telemetry integrating validated flourishing indicators to evaluate product and policy impact.
  • Tools/products/workflows: Differential privacy; federated analytics; standard survey modules; longitudinal cohorts.
  • Assumptions/dependencies: Scientific consensus on measures; privacy regulation compliance; clear consent.
• Cross-cultural value bargaining algorithms — sectors: global platforms, international organizations
  • What: Algorithms that represent and negotiate among plural value models, enabling context-specific defaults without monoculture.
  • Tools/products/workflows: Preference-aggregation with bargaining/mediation; explainable trade-offs; opt-out mechanisms.
  • Assumptions/dependencies: Representative inputs; fairness guarantees; governance of update cycles.
• Co-regulation via multimodal/neural interfaces — sectors: assistive tech, neuroethics, healthcare
  • What: Use physiological/affective signals to tune agent behavior toward empathy and care, with stringent ethical safeguards.
  • Tools/products/workflows: On-device inference; consent management; red lines for manipulation; oversight boards.
  • Assumptions/dependencies: Hardware maturity; strong bioethics; safety validation; user trust.
• Process-ethics metrics embedded in agents — sectors: enterprise, public services
  • What: Track de-escalation, reciprocity, norm compliance, and deliberative quality in real time to guide and audit agent behavior.
  • Tools/products/workflows: Telemetry schemas; anomaly detection; outcome-linked feedback loops.
  • Assumptions/dependencies: Metric validity; non-gaming incentives; integration with performance management.
• Longitudinal user-growth optimization — sectors: consumer apps, learning, health
  • What: Agents that plan across months/years to support self-determined flourishing (e.g., habits, relationships, purpose), making transparent trade-offs.
  • Tools/products/workflows: Causal modeling; RCTs; consented nudging with user-set goals; relapse/rollback controls.
  • Assumptions/dependencies: Evidence of benefit; autonomy protections; safeguards against overreach.
• Regulatory adoption of positive alignment standards — sectors: policy, compliance, certification
  • What: Incorporate disclosures, opt-in guidance modes, pluralistic configurations, and outcome audits into AI governance regimes.
  • Tools/products/workflows: Certification schemes; conformity assessments; public registries of constitutions/specs; redress mechanisms.
  • Assumptions/dependencies: Policymaker consensus; enforceability; international interoperability.
• Open alignment middleware for agentic ecosystems — sectors: software platforms, marketplaces
  • What: Vendor-neutral layers for policy routing, community packs, and outcome monitoring across heterogeneous agents and tools.
  • Tools/products/workflows: Open standards; reference implementations; secure policy enforcement; version control.
  • Assumptions/dependencies: Ecosystem buy‑in; security guarantees; compatibility with capability scaling.

Notes on feasibility and cross-cutting dependencies

• Consent and autonomy: Many applications rely on explicit, revocable user consent and transparent controls to avoid paternalism.
• Data and measurement: Success depends on high-quality, cross-cultural datasets and validated flourishing metrics; avoid collapsing plural values into a single score without justification.
• Governance: Polycentric oversight, dispute resolution, and public participation are needed to keep positive alignment accountable and adaptable.
• Safety-first layering: Positive alignment complements, not replaces, safety alignment; all applications assume robust harm prevention remains in place.
• Business incentives: Reorienting away from engagement hacking may require new KPIs, product strategies, or regulation to align incentives with wellbeing.

• Adaptive Constitutions: Dynamically updated principle sets used during training to balance competing values or goals. "Adaptive Constitutions (balancing value trade-offs such as Autonomy vs. Guidance)"
• Agentic: Possessing the capacity to act autonomously and pursue goals with real-world impact. "as AI becomes more agentic with real-world consequences"
• Alignment by debate: An approach where models argue opposing sides and a judge model (or human) evaluates, aiming for scalable oversight. "Alignment by debate uses adversarial decomposition for scalable oversight"
• Alignment pretraining: Incorporating alignment-relevant values and norms during pretraining to bake in stable ethical priors. "'alignment pretraining' to build stable, ethical, and fundamental worldviews."
• Attractor (positive/negative): In dynamical systems, a set of states toward which the system tends to evolve; positive attractors promote flourishing, negative attractors correspond to harmful failure modes. "positive attractors denote robust, context-sensitive regimes that actively support human aims and wellbeing"
• CBRN: Acronym for Chemical, Biological, Radiological, and Nuclear, used to classify high-risk capabilities. "CBRN uplift"
• Character training: Post-training methods that instill stable virtues or dispositions in models to guide behavior. "character training encodes dispositional traits such as curiosity, honesty, and care"
• Constitutional AI: Training models to self-critique and revise outputs against an explicit set of principles (a “constitution”). "Constitutional AI has models critique their own outputs against explicit principles to generate synthetic data for alignment post-training"
• Conative theories: Accounts of well-being that ground it in the satisfaction of desires and goals (including second-order desires). "Conative theories focus on desire satisfaction, positing that a good life consists of fulfilling one's goals, desires, and preferences."
• Contemplative Alignment: Approaches drawing on contemplative traditions to foster qualities like self-monitoring and universal care in AI systems. "Contemplative Alignment Draws on contemplative traditions to cul- tivate properties such as self-monitoring, non-dogmatism, and universal care."
• Controllability: The ability to reliably steer, constrain, and override AI systems to ensure they do what users intend. "Controllability ensures that AI systems do what their human users want, which requires that they can be reliably steered, constrained, and overridden when necessary"
• DPO: Direct Preference Optimization, a method that directly optimizes model parameters from preference data without reward models. "DPO, IPO, and KTO offering direct optimization alternatives"
• Dynamical systems theory: A mathematical framework for analyzing how systems evolve over time, used here to frame alignment as movement toward or away from behavioral attractors. "A useful way to formalize the distinction between positive and negative alignment emerges from dynamical systems theory."
• Eudaimonia: A philosophical term for flourishing or a life well-lived, often used as a target notion for positive outcomes. "the Greek eudaimonia"
• Epistemic humility: A stance or objective emphasizing calibrated uncertainty and recognition of one’s knowledge limits. "Epistemic humility, foresight, and plural values as objectives"
• Formal verification: Methods providing mathematical guarantees about system properties or behaviors. "Formal verification approaches aim to provide mathematical guarantees"
• HarmBench: A benchmark suite used to evaluate model behavior across many harmful scenarios. "HarmBench for red-teaming across 510 harmful behaviors"
• Hedonic theories: Theories of well-being centered on pleasure, happiness, and the absence of suffering. "Hedonic theories define well-being as happiness: the presence of pleasure, positive emo- tional states, and life satisfaction, coupled with the avoidance of pain and suffering."
• Inner-outer alignment problem: The mismatch risk between the specified objective (outer) and the internal objective the model actually learns (inner). "inner-outer alignment problem: even if we specify the right objective (outer alignment), the model may learn a different internal objective that merely correlates with it during training"
• Jailbreaks: Attempts to circumvent model safeguards via adversarial prompts or techniques. "resistance to jailbreaks, adversarial inputs, and prompt injection attacks"
• KTO: A direct-optimization alignment variant (alongside DPO and IPO) that avoids reward modeling. "DPO, IPO, and KTO offering direct optimization alternatives"
• Mechanistic interpretability: Techniques to understand internal model circuits and representations at a mechanistic level. "Mechanistic interpretability for virtue concepts"
• Model specifications: Explicit, auditable behavioral guidelines that codify desired model conduct. "Model speci- fications codify behavioral guidelines"
• Multi-objective reward modeling: Post-training optimization that balances several objectives (e.g., honesty, helpfulness) simultaneously. "multi-objective reward modeling"
• Negative alignment: A safety-focused paradigm emphasizing harm avoidance and compliance rather than proactive promotion of good outcomes. "what we term negative alignment"
• Negative utilitarian intuitions: Ethical views prioritizing minimization of suffering over the promotion of other goods. "Negative utilitarian intuitions prioritize reducing suffering over promoting higher goods"
• Objective list theories: Views that certain goods (e.g., relationships, autonomy, knowledge) are intrinsically valuable regardless of desire. "Objective list theories argue that certain things are intrinsically good for a person, regardless of whether they are desired or bring pleasure."
• Paternalism: Steering or constraining individuals “for their own good,” potentially undermining autonomy if imposed. "paternalistic policies can undermine autonomy even when they reduce harm"
• Polycentric governance: Oversight distributed across many legitimate centers rather than a single authority. "polycentric governance; that is, many legitimate centers of oversight rather than one institutional or moral chokepoint."
• Preference-based methods: Alignment methods that learn from human preference rankings to guide model behavior. "Preference-based methods such as RLHF learn from human preference rankings"
• Preference–wellbeing divergence: The gap between what users prefer and what actually supports their well-being. "Preference-wellbeing divergence."
• Red-teaming: Systematic adversarial testing to elicit and study failure modes. "Red-teaming protocols focus on eliciting harmful outputs."
• Responsible scaling policies: Organizational rules tying model capability increases to risk evaluations and mitigations. "Responsible scaling policies define capability thresholds by harm potential: CBRN uplift, cyber offense, autonomous action."
• Retrieval augmentation: Enhancing models with external information retrieval to improve factuality and grounding. "Grounding, retrieval augmentation, un- certainty calibration"
• RLAIF: Reinforcement Learning from AI Feedback, where model-generated critiques replace some human labels. "model-mediated judgment (RLAIF)."
• RLHF: Reinforcement Learning from Human Feedback, optimizing models from human preference comparisons. "reinforcement learning from human feedback"
• Satisficing region: Behavioral space that is “not unsafe” but lacks a constructive objective toward flourishing. "This yields a broad intermediate satisficing region (yellow)"
• Self-determined flourishing: User-authored optimization targets where individuals retain agency over their conception of flourishing. "self-determined flourishing"
• Sycophancy: The tendency of models to flatter or agree with users rather than provide honest or accurate responses. "Sycophancy / People-pleasing"
• Value drift: The phenomenon where an AI system’s objectives change over time away from intended values. "value drift in increasingly powerful models"

Sign up for free to add this paper to one or more collections.