Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Expert Summary: "Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling" (2607.01612)

Motivation and Problem Definition

LLMs trained with reinforcement learning (RL) have achieved strong results in reasoning and question-answering. However, a critical limitation persists: prevailing RL reward schemas tend to focus narrowly on output correctness, without incentivizing LLMs to express or calibrate uncertainty. Consequently, LLMs frequently exhibit poor calibration—manifesting as overconfident hallucinations, especially under uncertainty. This miscalibration is highly detrimental, particularly for safety-critical AI deployment, and directly undermines reliability in downstream decision-making and resource allocation. Existing calibration methods either handle only binary confidence (certain/uncertain), degrade accuracy-calibration trade-offs, or do not leverage calibrated confidence for downstream adaptivity.

Methods: C3RL and CAS

Training: Correctness and Confidence Calibration RL (C3RL)

The C3RL algorithm introduces a composite RL reward that simultaneously optimizes three objectives:

1. Correctness Reward: Standard RLVR-based reward ensuring updates favor correct predictions.
2. Calibration Reward: Confidence thresholding (e.g., $t=5$ on a 1–10 scale); correct high-confidence and incorrect low-confidence responses are rewarded, while incorrect high-confidence and correct low-confidence are penalized, using a designated margin (empirically set).
3. Reference Accuracy Reward: Incorporates sample-specific accuracy information derived from the model’s performance over multiple generations on the training set. Correct responses for previously "all incorrect" examples are strongly rewarded, while hallucinations on "all correct" samples are penalized. This term explicitly discourages "calibrated but wrong" degenerate solutions.

This composite reward structure is designed to induce fine-grained, dataset-informed calibration and to avoid reward hacking via universally low or high confidence.

Inference: Confidence-based Adaptive Test Time Scaling (CAS)

CAS leverages the model’s verbalized (i.e., outputted) confidence to optimize test-time resource allocation. Sampling proceeds batch-wise: after each batch, the confidences associated with unique answers are aggregated. Stopping decisions are made based on the dominance of the highest-confidence answer, as measured by a Beta-distributed probability on the summed confidence scores of top answers. This mechanism preferentially invests further sampling budget on low-confidence examples, while early-stopping on high-confidence, consensus cases.

CAS is model-agnostic (reliant on output, not internal logits), enabling broad applicability.

Experimental Results

Calibration-Accuracy Trade-off

C3RL demonstrates robust improvements in calibration metrics without sacrificing accuracy across a diverse set of benchmarks, including AGIEval, MMLU, and several multimodal tasks (MMMU, Math Vista, Logic Vista). Key observations include:

• Calibration Performance: On OOD datasets, C3RL outperforms the prior SOTA RLCR, with lower Expected Calibration Error (ECE) and higher AUROC, demonstrating both reliability and discrimination between correct/incorrect with confidence signals.
• Accuracy Preservation: C3RL avoids significant performance drops typical of calibration-oriented or self-abstaining methods (e.g., SaySelf), matching or surpassing RLVR/RLCR in accuracy.
• Generalization: Superior robustness is observed across LLM architectures (e.g., Qwen2.5VL-7B-Instruct, Llama-3.2-3B-Instruct), indicating transferable calibration benefits.

An ablation study confirms the necessity of the reference accuracy reward for optimizing calibration without accuracy degradation.

Inference Efficiency and Scaling

CAS achieves substantial reduction in inference sampling budget—up to 12.33$\times$—while matching baseline majority voting accuracy, particularly highlighting its efficacy on OOD data. Notably, CAS outperforms frequency/entropy-based adaptivity (e.g., Adaptive-Consistency) due to its direct exploitation of well-calibrated confidence. Stratified analysis verifies that most efficiency gains stem from shifting budget toward hard, low-confidence samples, while allowing for rapid early-stopping in easy, high-confidence cases.

Critically, CAS’s benefits are contingent on high-quality calibration: models trained without C3RL-derived calibration exhibit minimal gains from CAS, underscoring the synergy between training-time calibration and inference-time adaptivity.

Robustness and Practical Scenarios

C3RL maintains its advantage on tasks emphasizing selective answering (e.g., ALCUNA) and on canonical QA datasets (CommonSenseQA, StrategyQA). Verbalized confidence outperforms probability-based measures for uncertainty identification and hallucination mitigation, especially in CHOKE-type settings (hallucinations with high certainty).

Adding C3RL’s reference-based reward to other RL-based calibrators (e.g., RLCR) is insufficient—threshold-guided calibration is essential for superior performance.

Theoretical and Practical Implications

The integration of dataset reference-aware rewards into RL enables rich, context-sensitive calibration. The threshold-guided calibration furnishes a mechanism for models to learn an actionable uncertainty boundary, rather than degenerate to binary confidence extremes. By explicitly incentivizing correct uncertainty estimation, C3RL cultivates fine-grained confidence signals.

The synergy with CAS demonstrates the practical utility of calibrated LLMs: substantial reductions in inference cost, combined with enhanced reliability and trustworthiness, have direct implications for scalable and safety-critical deployments. The framework is compatible with both text and multimodal, multi-disciplinary tasks.

Limitations and Future Directions

Residual limitations include insufficient high-quality science training data and remaining trade-offs between peak calibration (best in SaySelf) and accuracy. For small-sample inference (CAS with $N<8$), further improvement of stopping criteria and calibration fidelity is needed. The methodology relies on consistent confidence verbalization, which remains an open research area for deployment across more diverse LLM architectures or in generative settings demanding soft calibration.

Future research directions include:

• Automated construction of balanced, difficult calibration-critical datasets
• Novel reward designs for hierarchical or continuous calibration scenarios
• Extending reference-aware calibration to few-shot or continual learning settings
• Further integration of calibrated uncertainty with decision-theoretic or abstention/deferral frameworks in interactive AI systems

Conclusion

This work establishes C3RL as an effective method for simultaneously optimizing correctness and calibration in LLMs, thereby enabling Confidence-based Adaptive Test Time Scaling (CAS). The resulting models achieve optimal calibration-accuracy trade-offs, minimize overconfident hallucinations, and allow inference-time computational allocation to be dynamically adjusted based on epistemic uncertainty. This unified framework sets a robust foundation for developing trustworthy, resource-efficient, and application-ready LLMs.