Model Monitoring

Detect Distribution Shift Before Model Quality Fails

A model can pass offline evaluation and still become less useful after deployment. The model may not have changed at all. Instead, the data reaching it may have changed. A fraud classifier trained on last year’s transactions may encounter a new payment pattern. A support-ticket model may see terminology introduced by a new product. An image model deployed to different hardware may receive images with different lighting or compression. These are forms of distribution shift: the statistical conditions seen in production differ from those represented by the data used to develop or evaluate the model.