Amazon SageMaker AI has introduced a model monitoring solution that integrates MLflow to detect changes in data and model performance. This approach helps organizations maintain the accuracy of their machine learning models by identifying issues such as data drift and model drift before they impact business outcomes. The solution allows for the integration of monitoring results into dashboards, alert systems, or automated retraining pipelines.
The monitoring process involves calculating baseline statistics for training data and comparing them with statistics from production data to detect data drift. Model drift is identified by comparing the model's performance metrics against those from the training phase. The solution uses the open-source Evidently library to calculate these metrics and can be extended with custom code to detect model drift. The results are stored in MLflow, enabling users to track model performance over time and compare results between different model runs.
This solution is part of a broader MLOps workflow, where model monitoring is integrated into the end-to-end machine learning lifecycle. It supports both batch inference and real-time endpoint use cases, with the latter requiring data capture to log inputs and outputs for monitoring. The monitoring architecture can be implemented using either processing jobs or AWS Lambda functions, depending on the user's preference.
Source: awsml
