Create a Video View Paper

Requential Coding: Compressing Models by Teaching Themselves

This presentation introduces requential coding, a breakthrough approach to neural network compression that fundamentally reimagines how we measure model complexity. Unlike traditional methods that scale with parameter count or dataset size, requential coding compresses models by having students learn from samples drawn from their own distribution, selected by a teacher. The result: compression that improves with scale, tighter generalization bounds for large language models, and a new lens for understanding what deep learning actually learns.

Script

Every compression method for neural networks has scaled with something: parameter count, dataset size, or both. Requential coding breaks that ceiling by asking a simpler question: how many bits does it take for a teacher to guide a student who is already guessing from its own learned distribution?

The student generates candidate batches using a shared random seed. The teacher selects one via relative entropy coding, transmitting only an index that costs roughly the KL divergence between their distributions. No parameters are sent, and no raw data is coded.

Across language and image datasets, requential coding reaches the same loss at code lengths orders of magnitude smaller than prequential coding or even idealized 4-bit quantization. The advantage grows with model scale and dataset entropy.

Here is the counterintuitive result: holding loss fixed, larger models compress to smaller codes despite having 100 times more parameters. Requential coding decouples model size from compressibility, revealing that scale buys efficiency.

Because requential codes are valid prefix-free codes, they certify PAC-Bayes generalization bounds. The bounds improve as models scale, and in compute-optimal regimes the gap between bound and test loss vanishes. Compression and generalization are two sides of the same coin.

Requential coding tells us that the true complexity of a model is not in its parameters but in the divergence it must close to match a teacher. To dive deeper into this work and create your own video explainers, visit EmergentMind.com.