Foundations of PySpark
Use PySpark to query and join flight data in Spark, build a machine learning pipeline and tune a model that predicts flight delays.
Spark runs parallel computation over large datasets, and PySpark is the package that lets Python code drive it. Working with flight records from Portland and Seattle, you open a Spark session, read its tables, reshape and summarize the data with SQL-style queries, and then prepare it for a model that predicts whether a flight will be delayed.
What You Will Achieve
- Start a SparkSession in Python, list the tables it holds and pull a Spark DataFrame into pandas
- Summarize and combine flight tables in Spark with SQL statements and DataFrame queries
- Prepare raw flight columns as numeric features that a Spark model can use
- Fit and tune a model that predicts flight delays, then score it with a binary classification evaluator
Program Overview
Spark Sessions and DataFrames in Python
How Spark manages data, and how a Python session reads and writes its tables.
- SparkContext and SparkSession setup
- Viewing and querying tables
- Moving data between Spark and pandas
- Reading and writing tables from Python
Manipulating Data with pyspark.sql
The pyspark.sql module and its optimized queries for shaping DataFrames inside a Spark session.
- Creating columns
- SQL queries in a session
- Filtering, selecting and aggregating
- Grouping and joining
Machine Learning Pipelines, Tuning and Selection
Building a full pipeline for the flight data, then tuning and comparing models that predict which flights will be delayed.
- Casting data types and encoding carrier and destination
- Assembling features into a pipeline
- Splitting train and test data
- Logistic regression, grids and cross validation
Certificate Awarded (Paid)
Complete the course to earn it.
Job Outlook
PySpark skills are used wherever teams process large datasets with Python, such as data engineering work that prepares and joins tables and analytics projects that train predictive models at scale.
Good for: Python users who want a first working grasp of Spark, carried through a single flight dataset all the way to a delay prediction model.
Keep in mind: Despite its intermediate label, the course is aimed at people new to Spark, and its modeling work stays with one logistic regression on the flights data.
Bottom line: A compact path through Spark DataFrames, SQL queries and a tuned logistic regression, all built on the same Portland and Seattle flight records.
Frequently Asked Questions
What dataset do the exercises use?
Every chapter works with records of flights from Portland and Seattle. The same data carries you through the first table queries, the joins and feature preparation, and the final model that predicts delays.
Which machine learning model does the course build?
The last chapter builds a logistic regression model that predicts whether a flight will be delayed. You set up a grid of settings, run cross validation over it, and check the result with an evaluator for binary classifiers.
How does PySpark work with pandas here?
An early lesson turns a Spark DataFrame into a pandas DataFrame, within a first chapter on how Spark manages data and how you read and write its tables from Python.
Can I run SQL queries inside Spark?
Yes. The second chapter uses the pyspark.sql module to run SQL against tables in your session, alongside DataFrame methods for filtering, selecting, aggregating, grouping and joining the flight data.


There are no reviews yet.