Representation Learning

Understand Negative Sampling for Representation Learning

Some representation-learning problems have an awkward shape: each training example has one observed target, but the model could choose from thousands or millions of alternatives. Computing a score and normalization term for every alternative on every update can become a major training cost. Negative sampling changes the training problem. Instead of comparing the observed target with every possible alternative, the model learns from the observed positive pair and a small set of deliberately sampled negative pairs. The update becomes much cheaper, but it also optimizes a sampled discrimination objective rather than the exact full-class objective.