Discrete Representations

Codebook Collapse Leaves Vector Quantizers with Unused Entries

A vector quantizer can expose thousands of codebook entries while repeatedly selecting only a small subset. The configured vocabulary then overstates the discrete capacity that is actually used. This state is often called codebook collapse: some entries receive frequent assignments, while others become inactive or nearly inactive. The failure is not a shortage of parameters by itself. It comes from the interaction between encoder outputs, the assignment rule, and the procedure used to update code vectors. Once assignments concentrate around a subset of entries, unused entries may receive little or no signal that would move them toward regions occupied by encoder outputs.