Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

Sign up for free to create and run prompts on this paper using GPT-5.

Overview

This paper is about a new way to “shrink” big AI models without losing what they’ve learned. The authors call it “requential coding.” The big idea is simple: instead of saving all the details of a model’s weights or all the exact training data it saw, they save only the tiny bits of information the model actually needed to improve—told to it by a smarter “teacher.” This lets them describe a trained model using far fewer bits than previous methods, sometimes by a huge margin.

What questions does the paper ask?

The paper focuses on a few clear questions:

• Can we compress large neural networks in a way that reflects what they truly learned, not just how many parameters they have?
• Can we avoid paying for the “randomness” in data that the model can’t really learn from?
• Do bigger models actually become easier to compress when you measure what they learn, not how many numbers they store?
• Can better compression give us better guarantees that a model will generalize (perform well on new, unseen data)?
• Can compression help us separate the useful patterns in a dataset from its random noise?

How does it work? (Methods explained simply)

Think of a “student” model and a “teacher” model:

• The student is the model we want to compress.
• The teacher is a stronger model that knows more.

Here’s the process in everyday terms:

1. The student suggests options: At each training step, the student uses its own knowledge to generate several candidate training examples (like suggesting answers it might study).
2. The teacher picks the best one: The teacher looks at those student-suggested candidates and selects one that’s most helpful.
3. Only the choice is sent: Because both sides can recreate the student’s suggestions (they share the same “random seed,” like the same shuffle order), the encoder only needs to send “which index” the teacher picked (e.g., “the 5th one”), not the full example. That tiny message is much shorter than sending the actual sample.
4. The student learns and improves: The student trains on that chosen example. Then the cycle repeats.

Why this is efficient:

• When the teacher and student largely agree, the “index message” is very short, because there’s not much new information to send.
• The number of bits sent roughly matches how much the teacher disagrees with the student—think of it as “how surprising the teacher’s choice is to the student.” Less surprise = fewer bits.

Key terms in simple language:

• Compression: describing something with fewer bits.
• Bits: the basic units of information (like characters in a very tiny alphabet of just 0 and 1).
• Entropy (data unpredictability): how random the data is. Very random data is expensive to describe exactly.
• Relative Entropy Coding (REC): a smart way to only send the “index” of a chosen sample among many that both sides can regenerate, so you pay only for the teacher–student disagreement instead of the whole example.
• KL divergence (don’t worry about the formula): a measure of how much two probability models disagree. Here, it measures how often the teacher says “this is important” when the student wouldn’t have expected it.

What’s new compared to older methods:

• Parameter compression (like 4-bit quantization) shrinks weights but still scales with the number of parameters, even if the model hasn’t really learned that much.
• Prequential coding compresses the exact training data sequence, which still costs a lot if the data is very random, even when the model already understands most of it.
• Requential coding pays only for the teacher–student disagreement and is independent of both parameter count and the randomness in the raw data.

Main findings and why they matter

Here are the highlights, followed by why they’re important:

• Much smaller codes than before: Requential coding often uses orders of magnitude fewer bits than prequential coding. It can even beat strong weight-compression baselines (like idealized 4-bit quantization), especially as models get larger.
• Bigger models can be more compressible: Holding performance (loss) the same, larger models and even ensembles (multiple models averaged together) can be compressed into fewer bits than smaller models. Why it matters: it supports the idea that bigger models learn simpler, more general patterns—even if they have more parameters.
• Stronger generalization guarantees: By plugging the requential code length into a PAC-Bayes bound (a standard way to turn compression into a generalization guarantee), the authors get state-of-the-art theoretical guarantees—even for billion-parameter LLMs. Why it matters: It’s hard to prove big models will generalize well; this gives rigorous, improving guarantees as models scale.
• Bounds get better with scale in the “compute-optimal” regime: When you scale data and model size together in a balanced way, the bound tightens as models grow. Why it matters: It suggests that as we build larger models the “extra gap” between training and test performance can vanish.
• Predicts overfitting when you repeat data too much: Training over the same dataset many times can slowly make the model memorize, and the requential code’s generalization bound captures this trend. Why it matters: It offers a measurable way to detect when extra training may begin to hurt generalization.
• Measures “learnable” information in datasets:
  • Random strings: almost nothing to learn (small compressed model).
  • Images: some structure to learn (more bits).
  • Text: lots of learnable structure (even more bits).
  • Why it matters: This helps explain why text is such a powerful training signal for general-purpose AI and supports smarter data selection.

What could this change? (Implications and impact)

• Better theory and practice for model scaling: The work supports the view that bigger models don’t just memorize—they become more compressible in terms of the information they truly learn, which explains why they can generalize well.
• Stronger, more realistic generalization guarantees: We can now certify that certain large models will perform well on new data, not just hope they do.
• Smarter data pipelines: By measuring how much “learnable” information a dataset contains, we can prioritize the most useful data and avoid spending compute on noise.
• Guidance for training strategies: The method reveals when adding more training passes might start to cause overfitting, informing decisions about training duration.
• Open challenges and future work:
  • Encoding speed: actually transmitting a model with these techniques can be slow; the paper mainly focuses on measuring code length, not fast shipping.
  • Compressing the “teacher” itself: the current scheme gives a lossless code for the student; compressing or replacing the teacher remains an opportunity.
  • Forgetting and code reduction: if a model “forgets” some early-learned details later, a future method might reclaim those bits and compress even more.

In short, requential coding is a new lens on model compression that focuses on what a model truly learns, not how big it is or how random its data was. It compresses better than previous methods, explains why larger models can generalize well, and opens up practical tools for choosing data and training smarter.

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper.

• Encoding practicality: REC encoding time grows roughly exponentially with the teacher–student KL; there is no demonstrated encoder achieving near-KL rate with bounded proposals for large, high-dimensional batches. Develop REC variants with controlled proposal budgets or amortized encoders tailored to sequence models.
• Deterministic reproducibility: The method assumes perfectly reproducible training (fixed PRNG, deterministic kernels, identical numerics). How to make requential decoding robust to real-world nondeterminism (e.g., CUDA/cuDNN nondeterminism, mixed precision, hardware variation) or design protocols that tolerate small divergences?
• Teacher design and optimization: Teacher smoothing and iso-loss projection are heuristic. There is no principled procedure to construct teacher trajectories that minimize cumulative KL subject to target performance. Formalize and solve the optimal teacher-scheduling problem (e.g., as a constrained control/optimization problem).
• Early-training KL and support constraints: REC requires $Q_t \ll P_t$ and suffers when KL is large (e.g., distilling a strong pretrained teacher into a randomly initialized student). Devise initialization, annealing, or curriculum strategies that guarantee support overlap and control early KL without exploding encode time.
• Applicability beyond likelihood-based generative models: The approach requires sampling and tractable likelihood evaluation for both teacher and student. Extend to:
  • implicit/score-based models (e.g., diffusion) without tractable likelihoods,
  • non-autoregressive sequence models,
  • discriminative settings where labels are part of the data.
• Leveraging soft teacher signals: The code currently transmits only hard samples. Explore transmitting limited soft information (e.g., compressed logits, temperature-adjusted distributions, low-rank logit projections) to further reduce effective KL and code length.
• Practical KL estimation: Computing batch-wise KL between two large LMs exactly is expensive and may be noisy with dropout/regularization. Provide scalable, low-variance estimators with quantified error for use during training and code evaluation.
• Information forgetting: Code length is monotone non-decreasing; it does not “garbage-collect” information the student later forgets. Design mechanisms to reclaim bits for forgotten information (e.g., backward passes that certify forgetfulness or reversible update encodings).
• Optimality and lower bounds: There is no information-theoretic lower bound for the minimal code needed to produce a model under specified training constraints. Establish lower bounds and characterize when cumulative KL is tight or loose.
• Sensitivity to batch size and blocking: Claims that logarithmic overhead is negligible rely on large batches; behavior with smaller batches or aggressive block splitting (to tame encode time) is unquantified. Map out the trade-off surface among block size, encode time, and code length.
• Generalization bounds beyond LMs: The PAC-Bayes bound used relies on prediction smoothing for language modeling. Extend and validate analogous bounds for vision, multimodal, and reinforcement learning settings, including the role of constants (e.g., $\Sigma$) and vocabulary analogs.
• Dataset epiplexity comparisons: Compressed size depends on model class and training recipe. Establish standardized, model-controlled protocols to compare “learnable information” across modalities without conflating model inductive biases with dataset structure.
• Practical deployability: Decoding requires retraining the model, incurring compute near training cost. Investigate hybrid schemes (e.g., transmit sparse periodic checkpoints plus requential deltas) or amortized decoding to make distribution practical.
• Privacy and leakage: Although the teacher is not transmitted, the messages and resulting student may encode private training data. Analyze privacy guarantees (e.g., differential privacy) and whether requential coding amplifies or mitigates memorization.
• Seed commitment and gaming: Compression depends on the PRNG seed; an encoder could search over seeds to reduce code length. Specify protocols that force seed commitment (or include seed in the code) and quantify the impact of seed selection.
• Ensemble decoding cost: Larger ensembles are more compressible but require training multiple students at decode time. Analyze the net utility and explore compressing ensemble predictions into a single student within the requential framework.
• Non-i.i.d. and shifted data: The method and bounds assume i.i.d. tokens. Examine behavior under curriculum learning, temporal correlations, domain shift, or adversarial filtering, and adapt bounds accordingly.
• Continuous variables and multimodality: While REC can extend to continuous variables in principle, there is no implementation or evaluation for high-dimensional continuous inputs (e.g., images as continuous values) or mixed-modality sequences. Address numerical stability and discretization.
• Optimizer/state handling: Decoding assumes a known update rule G. Specify how to encode optimizer states (e.g., Adam moments), hyperparameter schedules, and non-smooth operations, and quantify overhead when optimizer state resets or changes.
• Sampling schemes with implicit normalization: REC requires evaluating $Q_t(x)$; common sampling tricks (e.g., top-k/nucleus sampling) alter distributions with implicit normalization. Provide methods to handle or approximate such $Q_t$ reliably.
• Baselines breadth: Comparisons focus on 4-bit PTQ; stronger baselines (structured pruning, low-rank + quantization, bits-back parameter codes, learned parameter codecs) are not evaluated. Expand baselines to situate requential gains more rigorously.
• True LLM-scale validation: Experiments are at ~100M parameters; code-length evaluation at billion+ scale (where claims are most impactful) is not demonstrated. Provide LLM-scale experiments or strong extrapolations with error bars.
• Ablations and guidance: The impact of EMA window, projection cadence, block size, teacher–student lag, and optimizer choices is not systematically studied. Deliver ablations and best-practice recommendations.
• Fault tolerance and resynchronization: Prefix-free index streams are brittle—single-bit errors can desynchronize training reconstruction. Develop error-correcting schemes or periodic resynchronization checkpoints.
• Post-training stages: Integration with instruction tuning, RLHF, PEFT/LoRA, and mixed objectives is not specified. Extend requential coding to these stages and quantify their incremental code contributions.
• Multi-epoch scaling: Overfitting predictions are shown on 100M tokens; it is unclear how predictions scale to very large corpora or under regularization/augmentation. Validate multi-epoch behavior at scale and under modern training protocols.

Immediate Applications

The following items describe practical ways to use the paper’s methods and findings today, with sector tags and concrete workflows or tools that could be implemented immediately.

• Compression-aware generalization auditing for large models (software/AI infrastructure; policy)
  • Use requential code length (cumulative teacher–student KL) to produce PAC-Bayes generalization guarantees for LLMs and vision models, improving over parameter-based baselines (e.g., 4-bit PTQ or FP32 counts).
  • Workflow: “Requential Auditor” library that runs teacher–student training on a held-out real dataset, computes code length and bound terms (per-token complexity C), and exports a “Generalization Certificate” artifact with failure probability and conditions.
  • Assumptions/Dependencies: Access to teacher and student likelihoods, shared PRNG seed, ability to run teacher–student training on real data; bound requires token-level cross-entropy measurements; compute overhead ≈2–2.33× normal training.
• Overfitting and early-stopping diagnostics via bound monitoring (software/AI infrastructure; academia)
  • Monitor bound terms across epochs to detect gradual memorization when data is repeated (multi-epoch training). Stop around the point where complexity penalty starts to dominate training loss reduction.
  • Workflow: “Overfitting Monitor” dashboard that tracks loss gap, cumulative KL, and bound terms, flagging rising per-token complexity C.
  • Assumptions/Dependencies: Repeated access to training and validation streams; compatible loss function; requires running teacher–student steps (or evaluating KL via teacher sampling).
• Dataset epiplexity measurement for data selection and curation (software/AI infrastructure; academia; policy)
  • Rank candidate datasets by learnable structure rather than raw entropy (e.g., text > images > random strings), using compressed model size under requential coding after one epoch to avoid memorizing random content.
  • Workflow: “Epiplexity Meter” tool that trains a fixed student/teacher for one pass per dataset and outputs a compressed-size score for data selection, procurement, and documentation (e.g., governance under AI regulations).
  • Assumptions/Dependencies: Comparable tokenization and batch sizes across datasets; same student/teacher hyperparameters; sufficient compute to run one epoch per dataset.
• Teacher smoothing and iso-loss projection in distillation pipelines (software/AI infrastructure; industry)
  • Reduce code length (and therefore bound terms) by using EMA of teacher checkpoints and iso-loss projection (reset teacher to student, retrain to original loss), improving compression without loss of performance.
  • Workflow: Add “Teacher Smoothing” and “Iso-Loss Projection” steps to existing distillation pipelines (e.g., LLM pretraining/post-training).
  • Assumptions/Dependencies: Teacher accessible for training on real data; student initialized and synchronized; practical EMA and reset intervals tuned to task.
• Model selection under compute budgets using compression curves (academia; industry)
  • Compare architectures and ensemble sizes by plotting loss vs requential code length to select models that are simultaneously more sample-efficient and more compressible (larger models/ensembles often win despite higher parameter counts).
  • Workflow: “Compression–Performance Frontier” evaluation suite that reports loss vs code length and code-per-parameter.
  • Assumptions/Dependencies: Training multiple models/ensembles to comparable loss; consistent teacher setup; compute to produce curves.
• Compliance and documentation artifacts for foundation models (policy; healthcare; finance)
  • Provide compression-based generalization certificates and dataset epiplexity reports as part of model cards or regulatory submissions (e.g., demonstrate non-memorizing behavior and data quality).
  • Workflow: Integrate requential bound outputs into standard model governance documentation.
  • Assumptions/Dependencies: Acceptance of compression-based generalization metrics by stakeholders; clear reporting of bound conditions (δ, dataset size D, vocabulary V).
• Robust benchmark evaluations that decouple parameter count and data entropy (academia)
  • Use requential code length to compare training regimes and datasets more fairly than parameter-size or prequential-loss baselines.
  • Workflow: Benchmark reports that include requential code alongside accuracy/loss.
  • Assumptions/Dependencies: Access to teacher likelihoods; reproducible seeds; enough compute to evaluate code length.

Long-Term Applications

These items require further engineering, research, scaling, or ecosystem development before broad deployment.

• Practical model distribution via requential transmission (software/AI infrastructure; mobile/edge)
  • Transmit models as compact streams of accepted indices/messages instead of full parameter dumps, then decode by training the student locally from seed and messages. Could dramatically reduce distribution size (toward ~1 bit/parameter).
  • Product: “Requential Packager” for model updates/patches; “Requential Decoder” on client side.
  • Assumptions/Dependencies: Efficient REC encoders (today’s encoders can be prohibitively slow, scaling with KL), client training compute availability, secure seed and message handling, robust block-wise encoding strategies.
• Federated/edge requential distillation with low-bandwidth teacher guidance (software/AI infrastructure; robotics; IoT)
  • Server acts as teacher to choose accepted indices; devices generate candidate samples and update locally from minimal messages m_t, reducing data transfer of raw examples/logits.
  • Product: “Requential Federated Distiller” protocol for on-device training.
  • Assumptions/Dependencies: Secure shared randomness; device training capability; privacy and regulatory compliance; efficient REC at scale; robust synchronization.
• Data marketplace valuation using learnable-information scores (policy; finance; healthcare; education)
  • Price and prioritize datasets by epiplexity (learnable structure) rather than volume alone, aligning procurement and licensing with actual utility for model building.
  • Product: “Learnability Scorecard” with standardized scoring and audits.
  • Assumptions/Dependencies: Community standards for scoring, comparable training setups, regulatory acceptance for procurement and reporting.
• AutoML for compute-optimal scaling guided by compressibility (software/AI infrastructure; academia)
  • Optimize tokens-to-parameters ratios and curriculum schedules to minimize per-token complexity C and code-per-parameter, targeting bounds that tighten with scale.
  • Product: “Compression-Aware Scheduler” that tunes batch sizes, model sizes, and training durations.
  • Assumptions/Dependencies: Reliable estimation of scaling laws in target domain; integrated teacher–student evaluation; stable training at large scales.
• Privacy-preserving model updates and audits (healthcare; finance; policy)
  • Provide model improvements via requential messages without sharing raw data or teacher parameters/logits, reducing exposure of sensitive information while enabling verifiable audits.
  • Product: “Privacy-Aware Requential Update” service.
  • Assumptions/Dependencies: Legal acceptance of message-only updates; proof that messages do not leak sensitive content; efficient encoding; governance frameworks.
• Energy-efficient AI operations (energy; software/AI infrastructure)
  • Lower storage and transmission energy by distributing compressed messages; use bound-informed early stopping to avoid unnecessary training epochs and reduce memorization-related waste.
  • Product: “Green Requential Ops” tooling that reports energy savings and bound-informed stop criteria.
  • Assumptions/Dependencies: Viable message-based distribution and decoding; integration with training ops; credible energy accounting.
• Curriculum and sampling strategies derived from teacher–student KL (academia; industry)
  • Design curricula that track and minimize loss gaps, steer teacher updates via iso-loss projection, and select synthetic batches adaptively by KL to accelerate learning.
  • Product: “KL-Guided Curriculum Designer.”
  • Assumptions/Dependencies: Reliable KL estimation; stable teacher–student dynamics; domain-specific tuning.
• Regulatory standards for compression-based generalization certification (policy)
  • Establish guidelines where compressibility and PAC-Bayes bounds are part of model approval for high-stakes domains (e.g., clinical decision support, financial risk models).
  • Product: Standards documents and third-party certification programs.
  • Assumptions/Dependencies: Multi-stakeholder consensus; reproducible evaluation protocols; clarity on failure probability δ and dataset definitions.
• Cross-domain foundation model planning informed by epiplexity (academia; industry)
  • Decide strategic investments (text vs images vs multimodal) based on learnable information yield at scale, predicting data ROI and generalization behavior under compute-optimal regimes.
  • Product: “Epiplexity-Informed Roadmapping” analyses.
  • Assumptions/Dependencies: Stable epiplexity measurements across domains; scalable teacher–student setups; harmonized tokenization and loss metrics.

• arithmetic coding: A lossless entropy coding algorithm used to compress sequences by encoding them into intervals on the real line. "using a streaming entropy code (e.g., arithmetic coding)"
• bits-back coding: A compression technique that uses shared randomness to “get bits back,” allowing code lengths close to certain information-theoretic limits. "analogous to bits-back coding \citep{hinton1993autoencoders}"
• compute-optimal scaling regime: A training regime where the number of training tokens grows in fixed proportion to model size, optimizing performance for given compute. "the compute-optimal scaling regime where the number of tokens $D$ and parameters $N$ grow in fixed proportion."
• counter-based PRNG: A pseudorandom number generator where outputs are indexed by a counter, enabling direct regeneration of specific samples without replaying earlier ones. "With a counter-based PRNG, $Y_{i^\star}$ can be regenerated directly from its index without generating earlier proposals."
• distillation (hard distillation): Training a student model using samples (and not teacher logits) produced by a more capable teacher model. "Specifically, hard distillation where only the samples are used, not the logits."
• epiplexity: A measure of the learnable information in a dataset, as opposed to total entropy. "through epiplexity \citep{finzi2026entropy}"
• exponential moving average (EMA): A smoothing technique that maintains a weighted average of recent model checkpoints to reduce variance. "Teacher smoothing: generating synthetic data from an exponential moving average (EMA) of the raw teacher checkpoints"
• Hessian-based adaptive strategies: Quantization approaches that use second-order (Hessian) information to adapt quantization scales and minimize loss. "with the best methods reaching $\leq 4$ bits per parameter at minor loss in performance via Hessian-based adaptive strategies"
• iso-loss projection: Resetting and retraining the teacher to move it onto the same performance (loss) level while reducing divergence from the student. "Iso-loss projection: periodically resetting the teacher to the current student and briefly training the teacher on real data with the student paused until the pre-reset loss is recovered, which approximately moves the teacher to the projection of the student onto its iso-loss surface"
• Kullback–Leibler divergence (KL): A measure of how one probability distribution diverges from another; here measured in bits. "KL divergence is measured in bits."
• minimum description length principle: The idea that the model with the shortest description that explains the data is preferred, linking compression to generalization. "such as the minimum description length principle"
• ordered random coding (ORC): A practical REC method that selects among ordered proposals to approximate target sampling with near-KL code length. "Ordered random coding (ORC) \citep{theis2022algorithms} implements approximate sampling from $Q$ with the same code length bound while drawing on the order of $2^{\mathrm{KL}(Q\|P)}$ proposals."
• PAC-Bayes bound: A probabilistic bound that relates model complexity (e.g., code length or divergence) and empirical risk to expected generalization error. "A certified compressed model size translates directly into a PAC-Bayes generalization guarantee."
• Poisson functional representation (PFR): A construction used to implement channel simulation/sampling that achieves expected code lengths near the KL divergence. "using the Poisson functional representation (PFR)"
• post-training quantization (PTQ): Compressing a trained model by reducing parameter precision after training. "post-training quantization (PTQ) reduces the precision of the trained parameters far below the 32-bit or 16-bit floating point formats used during training"
• prefix-free code: A code in which no codeword is a prefix of another, enabling instantaneous decoding. "transmits a prefix-free code $m$ for that index."
• prequential coding: Compressing data (and implicitly the model) by sequentially encoding training batches using a model updated after each batch. "prequential coding \citep{dawid1984present, rissanen2003universal} compresses a dataset presented as a sequence of batches $X_0,\dots,X_{T-1},$ using a generative model trained sequentially on that dataset."
• prequential heuristic: A non-rigorous estimate of model complexity obtained by subtracting the data code given the final model from the prequential code. "The prequential heuristic (dashed) subtracts the data code given the model from the prequential code, but does not form a valid model code."
• prediction-smoothed per-token risk: A modified risk for autoregressive models that smooths predictions to enable tighter PAC-Bayes-style bounds. "For autoregressive LLMs, the prediction-smoothed per-token risk admits an analogous bound"
• relative entropy coding (REC): A compression method that transmits a sample from a target distribution using proposals from a reference distribution, with expected code length near KL. "Relative entropy coding (REC) \citep{flamich2020compressing,theis2022algorithms} provides a compression primitive to transmit a random sample from a target distribution $Q$ using fewer bits than its entropy"
• rejection sampling: A sampling technique that draws proposals from a reference distribution and accepts them with probability proportional to the target/reference density ratio. "we show how REC can be implemented with rejection sampling."
• sparsification: The process of making model parameters sparse (many zeros) to reduce complexity and storage while preserving performance. "Pruning and sparsification remove parameters that contribute little to predictive performance,"
• streaming entropy code: An entropy coding scheme that processes data in a stream for lossless compression. "using a streaming entropy code (e.g., arithmetic coding)"
• Zipf code: An integer coding scheme tailored to Zipf-like distributions to reduce overhead relative to generic universal codes. "rather than a Zipf code tuned to $\mathrm{KL}(Q_t\|P_t),$ which the decoder cannot access."

Sign up for free to add this paper to one or more collections.