InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

InfoLaw: A Data- and Repetition-Aware Scaling Law for LLMs

Introduction

The rapid scaling of LLMs has exposed fundamental challenges in selecting optimal data mixtures under rigorous compute and data constraints. While upweighting high-quality data generally improves downstream performance, data scarcity and consequent repetition often lead to quality-repetition trade-offs and diminishing returns, particularly in overtraining regimes. Conventional scaling laws are agnostic to these factors, systematically misestimating model performance when repetition is prominent and when training data mixes of disparate quality. The work "InfoLaw: Information Scaling Laws for LLMs with Quality-Weighted Mixture Data and Repetition" (2605.02364) systematically addresses these issues by introducing a scaling law framework that unifies model size, training token count, data mixture quality, and repetition into a predictive theory for LLM loss.

Motivations and Background

Prevailing scaling laws, such as the Chinchilla law, link error or loss to training and model size via power-law relationships but fundamentally assume essentially infinite, uniform-quality data. In practice, high-quality data is both finite and expensive, forcing practitioners to incorporate lower-quality data or upweight and repeat high-quality samples, often inducing substantial repetition. Prior literature reports that excessive upsampling or repetition can degrade final model performance—a phenomenon not captured by standard scaling laws. Furthermore, downstream performance exhibits nontrivial dependence on the data mixture, and optimal mixture weights are highly regime-dependent (model scale, compute, tokens seen).

Traditional methods require extensive grid search over mixture weights, which becomes computationally prohibitive at scale. InfoLaw aims to replace this by providing a theory-based, data-aware, and repetition-sensitive predictive framework for pretraining loss as a function of measurable quantities.

Formulation of InfoLaw

LayerMix and Dataset Construction

The authors introduce LayerMix, a systematic data generation protocol, partitioning the source corpus (Common Crawl with aggressive deduplication) into quality buckets based on classifier-derived quality scores. LayerMix sampling then constructs batch training datasets via configurable mixture weights. Repetition arises naturally when the number of selected tokens from a quality bucket exceeds its true unique volume, a practical reality in quality-upweighted mixtures and under data scarcity.

Modeling Information Accumulation

The theoretical core of InfoLaw is the treatment of LLM pretraining as an information accumulation process, where:

• Information Density: Each quality bucket’s data contributes according to an estimated, monotonic quality density function.
• Diminishing Returns: The gain from repeated exposures to the same data follows an exponential decay law parameterized by model FLOPs per token and a normalization dependent on log-total consumed tokens.
• Monotonicity: The total learned information is modeled as the sum over buckets of (unique tokens in bucket) × (quality density) × (exponential diminishing term).

Mathematically, the marginal information gain from $t$-th exposure to document $i$ is

$I_i \cdot \lambda(N) e^{-\lambda(N)t} / \log K$

where $\lambda(N)$ is a fitted function of model non-embedding FLOPs per token $N$, and $K$ is the total training token count. The total information from $T$ exposures asymptotes (for large $T$) to the information content $I_i$ of the document, with diminishing marginal returns.

Unified Power-Law for Information-Loss

Once total learned information ("info") is computed for all data mixture / repetition configurations, the final model’s validation loss $L$ is found to obey a power-law in info:

$i$0

This law holds across a range of model sizes, token budgets, and, crucially, mixture/repetition regimes. This enables accurately mapping measured small- and mid-scale pretraining runs to optimal configurations for much larger models, unseen data mixtures, and budgets, simply by maximizing info.

Empirical Validation

Training Runs and Fitting

LayerMix datasets were used to train 9 models ($i$1M to $i$2B) with overtraining ratio $i$3, sampling multiple mixture weights (focusing on high-quality, medium, and low-quality regimes). Validation loss on downstream tasks was measured and converted to perplexity for calibration.

The InfoLaw parameters were fit by minimizing the Spearman correlation between info and measured loss over all data points. The best-fit quality density function and repetition decay function ($i$4) are monotonic and exhibit robust extrapolation properties in both mixture and model size.

Extrapolation Performance

The framework’s predictive accuracy is evaluated by holding out data mixtures, model sizes, and overtraining degrees absent in training. InfoLaw achieves exceptional predictive accuracy: absolute validation loss errors of just $i$5 (mean) and $i$6 (max) on unseen recipes, model sizes (up to $i$7B parameters), and higher overtraining ratios ($i$8). In contrast, conventional compute-based scaling systematically underestimates loss in high-repetition regimes.

Predictions generalize to out-of-distribution data mixtures, enabling direct identification and search for optimal data mixtures at large compute without training auxiliary models. On a $i$9B model run, the recipe minimizing InfoLaw-predicted loss outperformed all four strong, hand-selected baselines.

Trends Identified

Analysis of the optimal mixture weights as a function of model and data scale reveals key regimes:

• Smaller models and lower token budgets: Strongly prioritize highest-quality data (despite higher induced repetition).
• Larger models and bigger budgets: Favor greater diversity, upweighting lower quality buckets to avoid severe overfitting by repetition.

These trends form a principled heuristic for data allocation, reducing expensive empirical tuning.

Robustness

Application to alternative datasets (e.g., RefinedWeb) confirmed that the qualitative and quantitative relationships captured by InfoLaw generalize across different web-scale corpora and quality buckets, provided consistent bucketing and quality scoring procedures.

Implications

Practical Implications

• Data Recipe Optimization: By accurately predicting downstream loss for arbitrary mix/repeat schedules, InfoLaw enables direct optimization of mixture parameters, removing virtually all grid search and large-scale experimentation.
• Overtraining Regime Design: InfoLaw supports robust extrapolation into high-overtraining settings, making it possible to accurately predict trade-offs between model size, data mixture, and compute for tasks where inference efficiency dominates cost.
• Scaling LLMs in Data-Limited Regimes: InfoLaw provides the first systematic framework for scaling LLMs under severe constraints on high-quality data, a regime increasingly relevant as web-scale deduplication and data quality filtering deplete the pool of unique, high-quality corpora.

Theoretical Implications

• Beyond Compute-Only Scaling: InfoLaw demonstrates that true scaling laws are fundamentally data-aware, with explicit functions for quality density and repetition diminishing returns as first-class citizens.
• Predictive Unification: The collapse of loss curves across widely varying mixtures/recipes into a single info axis suggests that "effective information" is a sufficient statistic for scaling LLMs even outside classical regimes.
• Decoupling Quality and Repetition: The InfoLaw framework enables separation of the effects of data diversity (uniqueness) and information density, which opens avenues for further theoretical studies on sample efficiency, catastrophic forgetting, and overfitting in transformers.

Future Directions

• Extension to finer data quality granularity, domain adaptation, or alternative signals for information density.
• Automated and theory-driven design of bucketing/mixture schemes, potentially integrating neural or Bayesian inference for density estimation in continuous quality spaces.
• Exploiting InfoLaw for curriculum learning, lifelong learning, and efficient transfer in multitask LLM pretraining.
• Analytical theory linking overtraining ratio shifts to scaling law intercepts and information saturation regimes.

Conclusion

InfoLaw offers a robust, unified, and empirically validated scaling law for LLM pretraining that incorporates data mixture quality, repetition, model scale, and training budget. Its predictive fidelity enables principled, efficient selection of optimal data recipes and guides the design of LLM systems under practical constraints. This provides a substantial step toward data-efficient, scalable, and cost-effective foundation model pretraining (2605.02364).