Monitoring Machine Learning in Python
Build a model monitoring workflow in Python with NannyML: estimate performance without labels, measure business value and trace drift to its cause.
A machine learning model that scored well at launch can quietly lose accuracy once production data shifts, and the true labels needed to measure it often arrive late or never. Working in Python with the NannyML library, you estimate how a model performs before ground truth exists, check those estimates against realized results later, and trace a drop back to drifting features or data quality problems so you can decide what, if anything, needs fixing.
What You Will Achieve
- Split raw production data into reference and analysis sets that NannyML can monitor
- Estimate a deployed model’s performance without labels using the CBPE and DLE workflow
- Compare estimated and realized results and express a performance drop as lost business value
- Rank drifting features with univariate and multivariate methods to find the root cause of model issues
Program Overview
Data Preparation and Performance Estimation with NannyML
Introduces NannyML, then readies the New York taxi tip data and estimates how the tip prediction model performs in production.
- Key features of NannyML
- Reference and analysis periods
- Specifying the algorithm and problem type
- CBPE and DLE workflow
Realized Performance and Business Value
Covers realized performance calculators for when ground truth arrives, ways of handling results, and the business value of a hotel booking model.
- Estimated versus realized performance
- Filtering, plotting and chunking results
- Custom thresholds
- Business value and drops in monetary value
Drift Detection, Data Quality and Issue Resolution
Traces the hotel booking model’s degradation to its underlying cause and closes by putting a full monitoring workflow together.
- Multivariate and univariate drift detection
- Ranking and visualizing drifting features
- Data quality and summary statistics checks
- Choosing a resolution, including doing nothing
Certificate Awarded (Paid)
Job Outlook
These skills are used wherever machine learning models run in production and need to stay accurate over time, such as on MLOps, data science and machine learning engineering teams.
Good for: Python practitioners who already deploy models and want a hands-on way to watch them in production when labels arrive late or not at all.
Keep in mind: The monitoring system is built with one package, NannyML, so the workflow is learned through its functions on two practice datasets.
Bottom line: A focused NannyML course that follows two datasets through estimating performance without labels, valuing the loss and tracing it to drifting features.
Frequently Asked Questions
Which models does the course monitor?
The first chapter works with New York taxi rides and a model that predicts tip amounts. The later chapters switch to a hotel booking dataset for realized performance, business value and root cause analysis.
How does the course handle missing ground truth labels?
The opening chapter estimates model performance with NannyML before labels exist, using the CBPE and DLE workflow. The second chapter turns to realized performance calculators once ground truth is available and compares the two, including cases where the estimate is off.
Does the course end with a complete monitoring workflow?
Yes. The closing lessons cover issue resolution, including whether doing nothing is the right call, and then implementing a monitoring workflow that brings the course's steps together.


There are no reviews yet.