Model Compression

Compress Neural Network Layers with Low-Rank Factorization

Large neural networks spend much of their memory and computation multiplying activations by weight matrices. Some of those matrices contain more independent structure than the model actually needs for a particular deployment. If so, we can approximate one large matrix with two smaller matrices and reduce the number of stored parameters and multiply-add operations. This technique is called low-rank factorization. The central idea is simple, but using it well requires more than choosing a smaller number. Compression changes the weights, approximation error can accumulate through a network, and fewer arithmetic operations do not guarantee lower wall-clock latency on every device.