Active Learning

Select Active Learning Examples with BALD

When labels are expensive, training on every available example may be impractical. An active learning system tries to spend its labeling budget selectively: train a model on the labels already available, score unlabeled examples, request labels for useful examples, then retrain. A common first idea is to label the examples with the highest predictive entropy. That can help, but entropy mixes together two different reasons for uncertainty. The model may be uncertain because it does not yet know enough, or because the input itself is genuinely ambiguous. More labels are most valuable for the first case.