Consistency Regularization

Learn from Unlabeled Data with Mean Teacher

Many machine learning projects have far more raw examples than labeled ones. A team may have millions of images, audio clips, or sensor readings, but only a small subset has been reviewed by people. Standard supervised training ignores the unlabeled remainder because it has no target labels to compare with the model’s predictions. Mean Teacher provides a way to use those unlabeled examples without pretending that their unknown labels are known. It trains a student model to make predictions that stay consistent with a more slowly changing teacher model. The teacher is not a separately trained expert: its parameters are an exponential moving average of the student’s parameters.