Elements of Conformal Prediction for Statisticians

Sign up for free to create and run prompts on this paper using GPT-5.

Overview: What this paper is about

This paper explains a modern way to tell “how confident” we should be about a prediction made by any machine-learning model. The method is called conformal prediction. It wraps around a model like a safety belt, adding reliable uncertainty information (for example, a range of likely values or a small set of likely labels) without needing to know how the model works inside. Even better, the method gives exact guarantees that hold for any dataset that looks like a random shuffle of similar cases (a condition called exchangeability).

The key questions the paper asks

• How can we measure confidence around predictions from complex models without making strong assumptions about the data or the model?
• What kind of guarantees can we give that are valid even for small datasets?
• How do we build practical versions that work for both numbers (regression) and categories (classification)?
• What are the strengths and limits of these guarantees, and how should we interpret them?

How the methods work (in everyday language)

Think of a bouncer at a club. They look at the crowd inside (your past data) and ask, “Does this new person (your new case) fit in?” If the new person looks unusual compared to the crowd, the bouncer is less confident they belong. Conformal prediction turns this idea into a careful, fair procedure with math behind it.

Here are the core ideas, explained simply:

• Exchangeability: Imagine your dataset as a shuffled deck of cards from the same box. If shuffling the order doesn’t change what you expect to see, the data are exchangeable. This is a mild assumption and weaker than “identical and independent” data.
• Prediction sets: Instead of giving just one prediction, conformal prediction gives a set that should contain the correct answer with high probability. For numbers, that might be an interval like [7.2, 9.1]. For categories, it might be a small list like {flu, cold}.
• “How unusual” scores: For each example, we compute a score that measures how weird it is compared to the others. Higher score = more unusual.
• Ranking to get a p-value: Put the new case together with the old ones and compute everyone’s scores. See where the new case ranks. If it ranks among the most ordinary, we keep that candidate value or label in the prediction set; if it looks too unusual, we leave it out. This “rank-based” trick is what gives exact, small-sample guarantees.

Two practical workflows:

• Full (transductive) conformal: Refit the model for each possible answer of the new case. Very accurate but can be slow or impossible for big models.
• Split (inductive) conformal: Train your model once on part of the data. Then use a separate “calibration” set to adjust your predictions so they have the right coverage. Much faster and what people often use in practice.

Examples the paper walks through:

• Predicting a number (like a lab value) without features: You can make a one-sided or two-sided interval by using sample quantiles (think: ordered list and pick a cutoff) with a small correction that guarantees coverage.
• Predicting a category without useful features: Use the observed label frequencies to prefer common labels and form a small set likely to contain the truth.
• Regression with features: Build intervals using model residuals (differences between observed and predicted) or, better, quantile regression, which adapts the interval width when variability changes with the features.
• Classification with features: Use estimated class probabilities to include the most likely labels until you pass a calibrated threshold; this tends to give small, informative sets.

Main findings and why they matter

• Strong, exact guarantees with minimal assumptions:
  • Conformal prediction guarantees “marginal coverage.” If you choose 95%, then across many future cases, about 95% of the time the true answer will fall inside the set. This holds exactly for any dataset size, as long as the data are exchangeable.
  • The method is distribution-free (no special shape assumptions about the data) and model-agnostic (works with any algorithm).
• Sharp, small-sample bounds:
  • Coverage is at least your target (like 95%) and at most that plus a tiny 1/(n+1) term. As you get more calibration data n, this extra bit quickly vanishes.
• Clear interpretation and limits:
  • The guarantee is marginal, not feature-by-feature. That means it’s averaged over who walks in the door next, not tailored to each exact type of person. Getting exact per-feature guarantees with small sets is usually impossible in high dimensions.
• Simple examples show the mechanics:
  • For one-sided numeric prediction, the conformal cutoff is essentially a slightly “inflated” sample quantile. The paper shows this delivers exact coverage now, not just eventually with lots of data.
  • For categories, sorting labels by frequency approximates the ideal (“oracle”) method as data grow.
  • In real data (NHANES):
  • Serum creatinine (a blood test): Both mean-based and quantile-based conformal intervals achieve about 95% coverage, but quantile-based intervals adapt better when variability changes with age (heteroscedasticity).
  • Diabetes classification: Most people get a single, confident label, while uncertain cases get a two-label set; overall coverage hits the target.
• Key practical trade-off:
  • Full conformal can be more accurate but expensive; split conformal is fast and usually good enough if the model is reasonably trained. You don’t need a huge calibration set; a few hundred cases often suffice.

Why this research matters

Conformal prediction gives a practical, trustworthy way to know when to trust a model’s output. It turns “bare predictions” into informed decisions:

• Safer decisions: In medicine, finance, or any high-stakes area, knowing the uncertainty helps avoid overconfidence and supports better choices (like follow-up tests for a patient whose lab result falls outside a conformal interval).
• Works with any model: You don’t have to redesign your favorite machine-learning method. Just “wrap” it with conformal calibration.
• Honest guarantees now, not just in theory: The math ensures coverage even for small datasets, and without needing perfect model assumptions.
• Realistic expectations: The method’s guarantee is averaged over future cases. For very specific subgroups, perfect guarantees with tiny sets are often impossible; the paper explains why and shows methods that still adapt to features as much as possible.

Overall, this paper gives a clear, step-by-step entry point to conformal prediction: what it is, how to do it, what it guarantees, and how to use it well. It shows that adding reliable uncertainty to modern machine learning is not only possible, but also practical.

Knowledge gaps, limitations, and open questions

Below is a concise list of concrete gaps and open problems left unresolved by the paper that future research could address.

• Conditional validity beyond marginal coverage: Develop practical procedures with provable approximate or local conditional coverage guarantees (e.g., PAC-type, group-conditional, or neighborhood-conditional) and quantify conditional miscoverage risk under heteroscedasticity and high-dimensional $X$.
• Robustness to non-exchangeability: Provide diagnostics for exchangeability violations and finite-sample guarantees under covariate shift, selection bias, temporal dependence, and concept drift; design conformal methods with controlled degradation when exchangeability fails.
• Principled design/selection of nonconformity scores: Create systematic, data-driven criteria (or meta-learning schemes) to choose scores that optimize expected set size and conditional coverage; establish theory linking score “quality” to efficiency and miscoverage.
• Full vs split conformal trade-offs: Derive quantitative guidance for optimal data-splitting (training vs calibration) and compare efficiency analytically across model classes; develop cross-conformal/jackknife+ variants with finite-sample guarantees and reduced variance, especially for small $n$.
• Computational scalability of full conformal: Devise general-purpose accelerations for continuous-label problems and deep models (e.g., exploiting monotonicity/convexity/sufficient statistics) with provable exactness or tight approximation-error control.
• Categorical prediction sets: Supply formal proofs and finite-sample excess-risk bounds relative to the oracle; analyze behavior for large $K$, extreme class imbalance, rare/unseen classes (open-set), and the role of synthetic $U$-based tie-breaking versus deterministic alternatives.
• Quantile-based regression intervals (CQR): Establish finite-sample conditional risk guarantees when quantile models are only approximately correct; address quantile crossing and misspecification; develop adaptive conformalization that avoids a global $\hat{\tau}$ and better adapts to local noise scales.
• Choosing between mean-, quantile-, and distributional-score CP: Provide oracle inequalities or criteria that predict when each approach yields shorter sets or better conditional validity; deliver standardized comparisons across realistic regimes.
• Minimax efficiency in regression: Characterize lower bounds on average interval length under marginal coverage and construct procedures that achieve (or adapt to) these bounds across noise/feature distributions.
• Missing data, censoring, and measurement error: Extend conformal workflows to handle MAR/MNAR missingness, right/interval censoring, and noisy labels/features, with finite-sample validity and efficiency guarantees.
• Multiple/online inference and data reuse: Analyze the impact of reusing a fixed calibration set for many test points, adaptive pipelines, and feedback loops; develop time-uniform/anytime conformal guarantees and safe updating strategies in deployment.
• Fairness and subgroup guarantees: Design methods that ensure group-conditional coverage (or bounded disparity) across sensitive attributes while maintaining efficiency; study trade-offs and impossibility frontiers under realistic constraints.
• Privacy-preserving conformal prediction: Integrate differential privacy into training and calibration with coverage guarantees; quantify trade-offs between privacy budget and set size/accuracy.
• Hyperparameter tuning and selection leakage: Provide validated workflows that allow model selection/tuning without invalidating coverage (e.g., selective-inference corrections, data-reuse schemes) and quantify the resulting efficiency.
• Randomization and reproducibility: Measure the impact of tie-breaking and other randomization on coverage variability and interval size; propose deterministic or aggregation strategies to stabilize outputs without sacrificing validity.
• Decision-theoretic integration: Translate prediction sets into actions under cost-sensitive utilities; optimize risk–coverage and set-size–utility trade-offs; provide guidance for clinical and policy decision thresholds.
• Evaluation beyond coverage: Establish standardized benchmarks and metrics (e.g., conditional coverage curves, risk–coverage profiles, set size vs. error-frontiers) for regression, classification, and multivariate tasks.
• Structured and complex outputs: Develop scalable conformal methods for sequences, graphs, images, and other structured $Y$ with interpretable sets and guaranteed coverage.
• Semi-/weak supervision: Leverage unlabeled or weakly labeled data to improve efficiency while retaining distribution-free coverage guarantees.
• Distribution shift at deployment: Create adaptive conformal mechanisms that monitor and adjust to label shift and covariate drift post-deployment, with explicit guarantees on out-of-sample coverage and false-alarm rates.

Immediate Applications

These applications can be deployed now using split conformal prediction, basic full conformal methods where tractable, and off-the-shelf ML models wrapped with conformal calibration to deliver distribution-free finite-sample marginal coverage.

• Clinical reference ranges and decision support (Healthcare; prediction intervals; split/full conformal)
  • Use case: Patient-specific lab test reference bounds (e.g., one-sided troponin upper limits; two-sided creatinine ranges) and small differential-diagnosis sets for triage.
  • Tools/products/workflows: EHR-integrated “Conformal Reference Range” widget; clinical decision support that returns a set of plausible diagnoses and flags when a result falls outside the calibrated range.
  • Assumptions/dependencies: Exchangeable/representative calibration cohort; careful handling of covariates (age, sex, device/site effects); communicate marginal (not feature-conditional) coverage.
• Risk triage with set-valued outputs (Healthcare; classification sets)
  • Use case: Flag patients as {low-risk}, {uncertain}, {high-risk} for conditions like diabetes based on calibrated probability thresholds; escalate uncertain cases for follow-up tests.
  • Tools/products/workflows: Calibrated classifier microservice returning prediction sets rather than single labels; integration with care pathways for escalation.
  • Assumptions/dependencies: Reasonable probabilistic classifier; coverage is marginal—expect variability across subgroups unless explicitly mitigated.
• Outlier/novelty detection and quality monitoring (Cross-sector: healthcare, manufacturing, cybersecurity; conformal p-values)
  • Use case: Detect anomalous samples (instrument faults, sensor failures, fraud attempts) via exchangeability-based conformal p-values.
  • Tools/products/workflows: Real-time “conformal p-value monitor” for lab analyzers, IoT devices, network traffic; alerts when p-values fall below thresholds.
  • Assumptions/dependencies: Calibration data reflect normal operating regime; concept drift will require recalibration or online methods.
• Predictive maintenance and quality control (Manufacturing/Industrial IoT; regression intervals)
  • Use case: Remaining useful life and defect rate prediction intervals; tolerance regions for process outputs without parametric assumptions.
  • Tools/products/workflows: On-line dashboards that display conformalized prediction intervals for key KPIs; automated stop-the-line triggers when realized values fall outside intervals.
  • Assumptions/dependencies: Exchangeability within stable operating windows; adequate calibration sample size for each product line/shift.
• Demand, load, and generation forecasts with uncertainty (Energy, Retail/Logistics; regression/quantile-based conformal)
  • Use case: Day-ahead load and PV/wind generation intervals; SKU-level demand intervals for inventory and staffing buffers.
  • Tools/products/workflows: Forecasting APIs that expose conformalized intervals; planning tools that convert interval width to buffer stock/safety margins.
  • Assumptions/dependencies: Stationary residuals or regime-aware calibration; time-series dependencies handled by appropriate blocking or rolling calibration.
• ETA and travel-time intervals (Transportation; regression intervals)
  • Use case: Maps and delivery apps present ETA intervals rather than point estimates to align user expectations and routing choices.
  • Tools/products/workflows: Conformal wrapper around existing ETA models; UI that adapts routing aggressiveness based on interval width.
  • Assumptions/dependencies: Exchangeability across similar contexts (time-of-day/route segments); periodic recalibration for seasonality and events.
• Credit and insurance risk bounds (Finance/Insurance; classification/regression intervals)
  • Use case: Credit default prediction sets (approve, manual review, decline); loss ratio and claim amount intervals for pricing buffers.
  • Tools/products/workflows: Risk engine returning set-valued decisions; pricing margin calculators tied to interval widths.
  • Assumptions/dependencies: Representative calibration across applicant cohorts; governance to interpret marginal coverage and fairness impacts.
• Fraud and intrusion detection (Finance/Cybersecurity; outlier detection)
  • Use case: Flag transactions/sessions with small conformal p-values as potential fraud or intrusions.
  • Tools/products/workflows: Stream processors computing nonconformity scores and p-values; SOC dashboards prioritizing low-p cases.
  • Assumptions/dependencies: Calibrated on clean traffic/legitimate transactions; drift and adversarial behavior may erode guarantees.
• Safer model deployment and monitoring (Software/MLOps; split conformal)
  • Use case: Wrap any predictive API to return a prediction set/interval with guaranteed marginal coverage; monitor coverage in production.
  • Tools/products/workflows: Conformal calibration SDKs (Python/R) for scikit-learn, XGBoost, PyTorch; microservice that manages calibration scores and thresholds; CI/CD checks for coverage regressions.
  • Assumptions/dependencies: Consistent data pipeline; stable distribution between calibration and inference; coverage logging and recalibration triggers.
• Education analytics with uncertainty (Education; classification/regression)
  • Use case: Early-warning systems return set-valued risk categories or intervals for expected grades; adaptive testing with mastery sets.
  • Tools/products/workflows: LMS plugins that display confidence-aware alerts; item-selection strategies using conformalized mastery probabilities.
  • Assumptions/dependencies: Cohort-representative calibration; transparency around marginal vs subgroup-specific coverage.
• Scientific data analysis and reporting (Academia/Science; prediction intervals/sets)
  • Use case: Publish prediction bands for new observations in experiments/surveys without parametric assumptions; small-sample guarantees for pilot studies.
  • Tools/products/workflows: Analysis templates in R/Python for conformalized quantile regression; lab SOPs for reporting set-valued predictions alongside models.
  • Assumptions/dependencies: Exchangeability within study design; pre-registered calibration split to avoid leakage.
• Content moderation and open-set classification (Software/CV/NLP; classification with reject option)
  • Use case: Serve set-valued labels or abstain when uncertainty is high; detect unseen categories in user-generated content.
  • Tools/products/workflows: Moderation pipelines using adaptive cumulative-probability scores; human-in-the-loop review for non-singleton sets.
  • Assumptions/dependencies: Calibrated class-probability estimators; novelty appears as low p-values relative to calibration distribution.
• Personal decision support with intervals (Daily life; apps)
  • Use case: Budget, calorie, or sleep apps show conformal intervals for weekly spend, weight change, or sleep duration; communicate uncertainty clearly.
  • Tools/products/workflows: App-side conformal wrappers around simple predictive models; user education on interval interpretation.
  • Assumptions/dependencies: Personal history forms the calibration set; behavior changes break exchangeability and require re-calibration.

Long-Term Applications

These applications are promising but depend on further research, scaling, or engineering to address conditional coverage, distribution shift, sequential dependence, fairness, or computational constraints.

• Fairness-aware and subgroup-conditional guarantees (Policy/Industry; conformal with group adaptivity)
  • Use case: Enforce approximate conditional or group-conditional coverage (e.g., across demographics) in lending, hiring, and healthcare triage.
  • Tools/products/workflows: Adaptive calibration schemes that stratify or reweight by subgroup; fairness dashboards for coverage parity.
  • Assumptions/dependencies: Adequate data per subgroup; trade-offs between set size and conditional guarantees; regulatory alignment.
• Time-series, online, and non-exchangeable settings (Energy/Finance/Operations; sequential conformal)
  • Use case: Streaming calibration under drift; rolling prediction intervals for volatile markets or grids with formal guarantees.
  • Tools/products/workflows: Online conformal algorithms with sliding windows, covariate-shift corrections, or blocked residuals; drift detectors tied to recalibration.
  • Assumptions/dependencies: Weakened exchangeability assumptions (e.g., mixing); careful windowing; stability of nonconformity score distributions.
• Multimodal and multivariate prediction sets at scale (Healthcare/Autonomy/Remote sensing)
  • Use case: Joint prediction regions for high-dimensional outputs (e.g., multi-analyte panels, trajectories, images); uncertainty-aware planning.
  • Tools/products/workflows: Normalizing-flow or diffusion-based conditional density models with conformal wrappers; interpretable set visualizations.
  • Assumptions/dependencies: Scalable modeling of complex distributions; computational efficiency for full conformal or tight split approximations.
• Safe autonomy and control with set-based planning (Robotics/Transportation; conformal for control)
  • Use case: Use prediction sets for perception and dynamics to compute robust, safe control actions; abstain or slow when uncertainty is large.
  • Tools/products/workflows: Planning stacks that consume conformal sets/intervals and enforce safety margins; fallback policies keyed to set size.
  • Assumptions/dependencies: Real-time computation; coupling with robust MPC; handling temporal dependence and feedback loops.
• Treatment effect prediction and individualized medicine (Healthcare; conformal for uplift/causal)
  • Use case: Set-valued recommendations when individualized treatment effects are uncertain; communicate bounds on expected benefit.
  • Tools/products/workflows: Conformalized causal forests/metalearners producing prediction sets for outcomes under each treatment; decision policies using overlap of sets.
  • Assumptions/dependencies: Unconfoundedness or valid instruments; careful design of nonconformity scores for counterfactuals; sample size per stratum.
• Human-AI collaboration with abstention and deferral (Software/Operations/Customer support)
  • Use case: Systems that defer to humans when prediction sets are not singletons; allocate expert time where uncertainty is highest.
  • Tools/products/workflows: Triage routers using set size to prioritize cases; learning-to-defer frameworks calibrated by conformal prediction.
  • Assumptions/dependencies: Calibrated uncertainty translates to utility; cost-sensitive optimization over set sizes vs throughput.
• Regulatory-grade uncertainty reporting (Policy/Compliance; standardization)
  • Use case: Standardize uncertainty communication (e.g., in clinical AI or credit scoring) using distribution-free coverage guarantees.
  • Tools/products/workflows: Documentation templates and audit tools that verify coverage on held-out regulatory datasets; certification programs.
  • Assumptions/dependencies: Agreement on metrics (marginal vs conditional coverage); processes for recalibration and post-deployment monitoring.
• Privacy-preserving and federated conformal prediction (Healthcare/Finance; federated/DP)
  • Use case: Conformal calibration across institutions without sharing raw data; differentially private calibration thresholds.
  • Tools/products/workflows: Secure aggregation of calibration residual quantiles; DP mechanisms for quantile release.
  • Assumptions/dependencies: Communication-efficient protocols; utility-privacy trade-offs (DP noise vs interval width).
• Robustness to covariate shift and domain adaptation (All sectors; shift-aware conformal)
  • Use case: Maintain coverage when test covariates differ from training (e.g., new geographies, devices).
  • Tools/products/workflows: Importance-weighted conformal scores; conditional calibration via domain-invariant representations.
  • Assumptions/dependencies: Estimable density ratios or invariant features; risk of coverage degradation when shift is severe.
• LLM/NLP systems with calibrated set outputs and retrieval (Software; multi-label/open-set)
  • Use case: Return sets of plausible labels/explanations or abstain; select knowledge snippets only when within calibrated uncertainty bounds.
  • Tools/products/workflows: Conformal wrappers around classifier heads or retrieval modules; user interfaces that expand/collapse set size on demand.
  • Assumptions/dependencies: Reliable probability proxies from LLM components; managing dependencies across multi-step pipelines.
• Batch and hierarchical decision-making (Supply chain/Taxonomy-rich domains)
  • Use case: Batch-level coverage guarantees (e.g., for a day’s orders) or hierarchical prediction sets consistent with taxonomies.
  • Tools/products/workflows: Batch conformal methods that target aggregate coverage; hierarchy-aware scoring and set pruning.
  • Assumptions/dependencies: Clear batching semantics; taxonomy-consistent learning and calibration.
• Cost-sensitive and utility-optimized conformal sets (Finance/Healthcare/Operations)
  • Use case: Optimize thresholds for asymmetric costs (false positives vs negatives) while preserving coverage.
  • Tools/products/workflows: Post-calibration threshold tuning with cost constraints; decision analytics that map set sizes to actions.
  • Assumptions/dependencies: Well-specified cost models; careful validation to avoid invalidating guarantees.
• End-to-end full conformal for deep models (CV/NLP; fast refitting/inference)
  • Use case: Near-oracle set sizes via re-fitting for each hypothesized label at scale.
  • Tools/products/workflows: Efficient influence-function approximations, weight-sharing, or low-rank updates to enable transductive conformal in deep nets.
  • Assumptions/dependencies: Significant engineering for runtime; numerical stability; verifying that approximations preserve validity.

Notes on feasibility across all applications:

• Core dependency: exchangeability (or a justified relaxation) between calibration and deployment data. Violations (drift, covariate shift, feedback loops) require recalibration or shift-aware methods.
• Guarantees are marginal unless methods specifically target conditional or group-conditional coverage; communicate this to users and stakeholders.
• Informativeness depends on score quality: better predictive models yield tighter sets; calibration sample size affects only the calibration quantile (O(1/n) rate).
• Random tie-breaking and stochastic elements should be controlled/logged for reproducibility in regulated settings.

• Acceptance region: The set of values for which a hypothesis test does not reject the null hypothesis. Example: "In other words, $C_\alpha(X_{n+1}; \mathbf{Z}_{1:n})$ is the acceptance region for this test of $\mathcal{H}_{n+1}$."
• Asymptotically consistent: A property where an estimator or procedure approaches the target (e.g., oracle performance) as sample size grows. Example: "these conformal prediction sets are asymptotically consistent with the oracle from the previous section."
• Calibration subset: A held-out set used to compute nonconformity scores and thresholds for conformal prediction. Example: "and a calibration subset of size $n$, used for conformal prediction;"
• Central limit theorem: A result that the sum (or average) of many independent random variables tends toward a normal distribution, governing typical estimation rates. Example: "for which typically the error converges no faster than $\mathcal{O}(1/\sqrt{n})$ due to the central limit theorem."
• Conditional coverage: A coverage guarantee that holds for each feature value (or subset), not just on average. Example: "achieving exact conditional coverage with reasonably-sized prediction sets is often impossible"
• Conditional pivot: A statistic whose distribution, conditional on a sufficient summary (e.g., a multiset), does not depend on unknown parameters and is known. Example: "In conformal prediction, the vector $\mathbf{Z}_{1:(n+1)}$ is a conditional pivot,"
• Conformal adjustment: The calibrated offset added to model-based bounds to ensure marginal coverage in split conformal methods. Example: "and marginal coverage by the conformal adjustment $\hat{\tau}$."
• Conformal p-value: A data-dependent p-value constructed from nonconformity score ranks under exchangeability; it is super-uniform under the null. Example: "gives a conformal $p$-value $p(Y_{n+1}; \mathbf{Z}_{1:n}, X_{n+1})$"
• Conformal prediction: A distribution-free, model-agnostic framework that quantifies predictive uncertainty with finite-sample guarantees under exchangeability. Example: "conformal prediction has emerged as a rapidly growing alternative framework"
• Conformal prediction set: The set of labels or values not rejected by a conformal test at level α; guaranteed marginal coverage. Example: "The $\alpha$-level conformal prediction set for $Y_{n+1}$ is the set of labels $y \in \mathcal{Y}$ ..."
• Cumulative class probabilities: Sums of predicted class probabilities used to build adaptive-sized classification sets. Example: "uses adaptive scores based on cumulative class probabilities"
• Cumulative distribution function (CDF): The function giving the probability a random variable is less than or equal to a value. Example: "Cumulative distribution function (CDF) defined as $P(y) := P\{Y \le y\}$, where $Y$ is a random sample from the distribution $P$."
• Distribution-free: Procedures that provide guarantees without specifying the form of the data-generating distribution. Example: "distribution-free—relying mainly on symmetry assumptions such as exchangeability"
• Empirical distribution: The discrete distribution placing equal mass on observed samples. Example: "$$\hat{P}(\mathbf{Y}_{1:n}) = \frac{1}{n} \sum_{i=1}^{n} \delta_{Y_i}$$ is the empirical distribution of $\mathbf{Y}_{1:n} = (Y_1,\ldots,Y_n)$"
• Exchangeability: A symmetry condition where the joint distribution is invariant to permutations of indices. Example: "Random variables $Z_1, \ldots, Z_{n+1}$ are exchangeable if their joint distribution does not change when the indices are permuted."
• Feature-conditional coverage: Coverage that holds for every fixed feature value, tailoring uncertainty to X. Example: "feature-conditional coverage,"
• Finite-sample guarantee: A performance assurance (e.g., coverage) that holds for any sample size, not only asymptotically. Example: "conformal prediction provides exact finite-sample guarantees"
• Full (transductive) conformal prediction: A conformal approach that refits the predictive model for each hypothesized label and test point. Example: "Full (or transductive) conformal prediction"
• Generalized additive model: A flexible regression model with additive, possibly nonlinear (smooth) effects of covariates. Example: "we fit a generalized additive model using gam in \textsf{R}, with a smooth age effect and a sex main effect."
• Glivenko–Cantelli theorem: A theorem stating uniform convergence of the empirical CDF to the population CDF. Example: "by the Glivenko-Cantelli theorem"
• Good–Turing estimator: A method for estimating the probability mass of unseen events (missing mass) from sample frequencies. Example: "This reveals a connection between conformal prediction and the classical Good-Turing estimator of the missing mass"
• Heteroscedasticity: Variation in the conditional spread of outcomes across feature values. Example: "adaptivity to heteroscedasticity."
• Homoscedasticity: Constant conditional spread of outcomes across feature values. Example: "nice properties in homoscedastic settings"
• i.i.d. assumption: Independent and identically distributed data assumption often used in statistics and ML. Example: "under the general i.i.d.~assumption"
• Split (inductive) conformal prediction: A conformal approach that trains a model once and calibrates on a held-out set, avoiding re-training per hypothesis. Example: "Split (or inductive) conformal prediction"
• Marginal coverage: An average (unconditional) coverage guarantee over the joint distribution of features and outcomes. Example: "Marginal coverage is a reasonable objective because it is easy to achieve in finite samples under limited assumptions."
• Maximum-likelihood estimator: An estimator that maximizes the likelihood under a specified model. Example: "maximum-likelihood (or, plug-in) estimate"
• Missing mass: The total probability of outcomes not observed in the sample. Example: "the classical Good-Turing estimator of the missing mass"
• Model-agnostic: Methods that treat the predictive model as a black box, making minimal assumptions about its internals. Example: "model-agnostic, treating the learning algorithm as a black box."
• Multinomial model: A probability model for categorical outcomes with multiple classes. Example: "under the general multinomial model"
• Nonconformity score: A function measuring how atypical an example is relative to a reference dataset; used to rank candidates. Example: "non-conformity score function $$s: \mathcal{Z} \times \mathcal{Z}^{n+1} \mapsto \mathbb{R}$$"
• Open-set classification: Classification where test instances may belong to unseen classes, requiring detection of novelty. Example: "These include open-set classification, where test cases may belong to unseen classes"
• Oracle: A hypothetical ideal procedure that has access to the true data-generating distribution. Example: "an oracle could construct the most informative prediction set $C^*_\alpha$ with marginal coverage"
• Outlier detection: Identifying observations that do not conform to the patterns of the data. Example: "outlier detection problems, discussed in Section~\ref{sec:outlier-detection}"
• Permutation test: A nonparametric test that assesses significance by comparing to distributions induced by permuting labels. Example: "rank and permutation tests"
• Pivot: A function of data and parameters with a known distribution independent of unknowns, used to construct intervals. Example: "Exchangeability connects conformal prediction to the classical notion of pivots;"
• Plug-in estimator: An estimator obtained by replacing unknown population quantities with empirical estimates in an oracle formula. Example: "empirical plug-in analogue"
• Quantile: The value below which a given proportion of observations falls. Example: "the $\tau$-quantile of the empirical distribution of $\mathbf{Y}_{1:n}$."
• Quantile regression: Regression modeling of conditional quantiles of the response given features. Example: "A pair of quantile regression models $\hat{q}_\ell(x; D)$ and $\hat{q}_u(x; D)$ is trained"
• Super-uniform: A p-value property where its distribution is stochastically larger than uniform under the null, ensuring validity. Example: "is marginally super-uniform under $\mathcal{H}_{n+1}$"
• Tolerance region: A set that contains a specified proportion of the population with a given confidence level. Example: "coverage requirements for tolerance regions"

Sign up for free to add this paper to one or more collections.