Machine Learning with PySpark
Learn to train, evaluate and tune classification and regression models on Spark from Python, with pipelines, cross-validation and ensemble methods.
Training models on Big Data means splitting the computation across many machines, and managing that split by hand pulls attention away from the analysis itself. Spark handles the distribution across a cluster, and PySpark lets you drive it from Python. You read CSV files into a Spark session, shape raw columns into model features, and train classifiers and regression models on flight records and text messages. You then organize the work into reusable pipelines and test it with cross-validation.
What You Will Achieve
- Connect to Spark from Python and load CSV data ready for modeling
- Build classifiers that predict a category, such as flagging spam text messages, and measure how well they perform
- Fit a linear regression model, read its coefficients, and use regularization to keep only the most relevant predictors
- Chain preparation and modeling steps together and pick model parameters with grid search
Program Overview
Spark Basics and Loading Data
Background on Spark and machine learning, how a Spark cluster is put together, and the first steps of connecting and reading data.
- Characteristics of Spark and cluster components
- Creating a SparkSession and locating the Spark master
- Loading flights and SMS spam CSV files
Classification and Regression Models
Preparing columns for modeling, then building classification and linear regression models and improving them with engineered predictors.
- Removing, manipulating and assembling columns, with categorical data
- Decision trees and logistic regression with a train/test split
- Tokens, stopwords and hashing for a spam classifier
- One-hot encoding, bucketing and regularization for a flight duration model
Pipelines, Cross-Validation and Ensembles
Making model code clearer and easier to maintain, testing it more thoroughly, and a short look at two ensemble methods.
- Pipeline stages and fitted pipeline models
- Cross-validation and grid search for parameter selection
- Gradient-boosted trees and random forests for delayed flights
Certificate Awarded (Paid)
Job Outlook
These skills are used where models are trained on Big Data held on Spark clusters, in data engineering and data science teams that build their machine learning workflows in Python.
Good for: Data practitioners who want to move their model building onto Spark and work through classification, regression and parameter tuning in Python.
Keep in mind: The exercises return again and again to two datasets, flight records and SMS messages, and the two ensemble methods get only a brief look in the final chapter.
Bottom line: A hands-on route into Spark’s machine learning tools, where one flight duration model grows step by step through encoding, regularization and a full pipeline.
Frequently Asked Questions
Which datasets are used in the exercises?
Most exercises work with two datasets: flight records, used for flight duration and delayed flight models, and a collection of SMS messages, used to build a spam classifier. A small shirt sizes example also appears in the lessons on encoding categorical values.
How are text messages turned into model input?
A dedicated part of the classification chapter shows how to convert text into tables a model can use. It deals with punctuation, numbers and tokens, removes stopwords, applies hashing, and then trains a classifier that separates spam from ordinary messages.
Which ensemble models appear in the final chapter?
The closing chapter introduces two ensemble methods, gradient-boosted trees and random forests, both applied to predicting delayed flights. The random forest is then evaluated. These lessons are brief, so treat them as a starting point rather than a full study of ensembles.
How are the best model parameters chosen?
The course uses cross-validation to test models more reliably, first on a simple flight duration model and then on a full pipeline. Grid search then compares parameter settings, and one lesson dissects the best flight duration model it finds.


There are no reviews yet.