Model Validation in Python
| Course Details | ||||||||
|---|---|---|---|---|---|---|---|---|
|
||||||||
| Languages | ||||||||
|
||||||||
| Outcomes | ||||||||
|
||||||||
| Price & Terms | ||||||||
|
||||||||
| Technical | ||||||||
|
||||||||
| Specification | ||||||||
|
||||||||
Learn to test scikit-learn models with holdout sets, error metrics and cross-validation, then tune hyperparameters to pick the best model.
A machine learning model that scores well on the data it was trained on can still disappoint once fresh data arrives. Model validation gives a measured answer to how well a model will perform on data it has never seen. You build regression and classification models in scikit-learn, set aside data for testing, read error metrics and confusion matrices, and then rely on repeated validation to compare settings and pick the strongest model.
What You Will Achieve
- Split a dataset into training, validation and testing sets before fitting a model
- Judge regression models with mean absolute error and mean squared error, and classifiers with precision and recall
- Spot underfitting and overfitting by comparing a model’s error on training and testing data
- Tune hyperparameters with RandomizedSearchCV and cross-validation to select a final model
Program Overview
Regression and Classification Modeling in scikit-learn
Builds the model-fitting groundwork the later chapters rely on, with regression and classification models and the difference between seen and unseen data.
- Seen versus unseen data
- Setting parameters and fitting regression models
- Classification predictions with a random forest
- Feature importances and reused model parameters
Holdout Sets, Accuracy Metrics, and the Bias-Variance Tradeoff
Covers splitting data for honest testing, scoring regression and classification models, and recognizing the errors that come with underfitting and overfitting.
- One and two holdout sets
- Error measures for regression and performance on data subsets
- Confusion matrices for classifiers
- The bias-variance tradeoff
Cross-Validation and Hyperparameter Tuning
Moves past a single train and test split to repeated validation, then applies cross-validation to search hyperparameters and choose a final model.
- KFold indices and cross_val_score
- Leave-one-out cross-validation and when to use it
- Hyperparameter ranges with randomized search
- Choosing the best accuracy or precision model
Certificate Awarded (Paid)
Job Outlook
Checking how well a model generalizes is everyday work in data science and machine learning teams, where models are tested on held-out data and tuned before their predictions on new data are trusted.
Good for: Anyone fitting predictive models in Python who wants to check how they will hold up on new data and choose model settings with evidence.
Keep in mind: The first chapter revisits model building in scikit-learn before validation begins, and every technique is taught through that library’s tools.
Bottom line: A practical path through holdout sets, cross-validation and randomized search that ends with choosing a final model by accuracy or precision.
Frequently Asked Questions
What data are the models validated on?
Classification models are validated on the complete set of tic-tac-toe endgame scenarios, and regression models on FiveThirtyEight's Halloween candy power ranking dataset, so each kind of model has its own data.
How does cross-validation improve on a single holdout set?
The cross-validation chapter opens with the problems of relying on one train and test split, then repeats the split with KFold and cross_val_score, and closes with leave-one-out cross-validation and the cases where it fits.
Which hyperparameter search method is taught?
The last chapter uses RandomizedSearchCV. You define hyperparameter ranges, prepare and run the randomized search with cross-validation, and then select a final model by its classification accuracy or by its precision.
What does the bias-variance material cover?
It sits at the end of the validation basics chapter and looks at the errors caused by underfitting and overfitting, with an exercise on checking whether a model is underfitting its data.
User Reviews
Write your review
You must be logged in to post a review.
Reviews can only be posted by registered learners. Log in or create a free account, then you will return here to post.
There are no reviews yet.