Alliance AI

Respond to a drift alert

What to do when a production model drifts past the thresholds set at validation, using ETB Conversion Propensity as the example.

For: MLOps Engineer, Builder / Publisher, Model Risk & Governance · 1 min read

Go to the screen:Model operations Platform health

The alert

The Drift Watch Agent measures population stability (PSI), feature stability (CSI) and performance weekly. When a threshold is breached it opens a case, alerts the model owner and Model Risk, and starts the drift response workflow.

Steps

  1. 1Open Model operations. The drift advisory names the model, the metric and the threshold.
  2. 2Read the case: which features moved, since when, and the effect on performance.
  3. 3Tell the consuming business units. Scores may be less reliable until the model is retrained.
  4. 4The business owner accepts the retrain cost at the workflow checkpoint. No job starts before that.
  5. 5Retrain in the sandbox on the refreshed data and compare the candidate with the current champion.
  6. 6Submit the new version. Model Risk decides whether it is a material change needing revalidation.
  7. 7Deploy the approved version. The drift indicators return to green once the new version is serving.

Model operations Retrain Workflow studio

Read next

Was this helpful?