Data Quality

Prevent Target Leakage in Machine Learning Evaluation

A machine learning model can score extremely well in offline evaluation and fail as soon as it reaches production. Sometimes the model is not the main problem. The evaluation accidentally gave it information that would not exist when a real prediction is made. This failure is called target leakage: information related to the outcome enters the model inputs in a way that makes the target easier to predict than it will be at inference time. Leakage can produce impressive metrics because the model is solving an easier, unrealistic problem.