Big Data Fundamentals with PySpark
| Course Details | ||||||||
|---|---|---|---|---|---|---|---|---|
|
||||||||
| Languages | ||||||||
|
||||||||
| Outcomes | ||||||||
|
||||||||
| Price & Terms | ||||||||
|
||||||||
| Technical | ||||||||
|
||||||||
| Specification | ||||||||
|
||||||||
Learn to process Big Data with PySpark: build RDDs, query DataFrames with Spark SQL, and train MLlib models for recommendations and clustering.
Big Data, and how Apache Spark processes it from Python, is the subject here. Working in PySpark, you start with what Big Data is and how SparkContext and the PySpark shell fit in. You then program resilient distributed datasets, query structured data with Spark SQL, and train machine learning models with MLlib for movie recommendations, spam filtering, and k-means clustering.
What You Will Achieve
- Write RDD code that maps, filters, and reduces a text dataset to its word frequencies.
- Load a CSV file into a DataFrame, clean it, and answer questions with SQL queries.
- Train a collaborative filtering recommender on movie ratings and check its error with MSE.
- Classify spam with logistic regression and group data points with k-means.
Program Overview
Big Data Analysis and RDD Programming
Introduces Big Data concepts and Spark’s core data type, with functional Python and the transformations and actions that run on distributed data.
- The 3 V’s of Big Data
- SparkContext and the PySpark shell
- lambda() with map() and filter()
- Partitions, pair RDDs, and RDD actions
PySpark SQL and DataFrames
Moves to structured data processing, where DataFrames and Spark SQL handle loading, inspecting, querying, and charting tabular data.
- RDD to DataFrame conversion and CSV loading
- Inspecting, subsetting, and filtering DataFrames
- Programmatic SQL queries on tables
- DataFrame visualization
Machine Learning with PySpark MLlib
Applies Spark’s scalable machine learning library to three common tasks: recommendation, classification, and clustering.
- PySpark ML libraries and algorithms
- Collaborative filtering with MovieLens
- Feature hashing and logistic regression
- K-means training and cluster plots
Certificate Awarded (Paid)
Job Outlook
PySpark skills apply wherever datasets are too large to fit in memory, including data engineering pipelines, large-scale analysis, and machine learning on distributed data.
Good for: Python users who want hands-on practice with Spark’s three main layers: RDDs, DataFrames with Spark SQL, and MLlib models.
Keep in mind: Despite its Advanced label, it introduces Big Data and Spark from the ground up, and the machine learning chapter covers basic tasks with three algorithms.
Bottom line: A compact PySpark course that has you write low-level RDD code before moving to DataFrames, SQL queries, and three MLlib models.
Frequently Asked Questions
What is a resilient distributed dataset?
An RDD is the main abstraction Spark provides and the backbone data type of its engine. The course shows how to create RDDs from parallelized collections and external datasets, how partitions split the data, and how transformations and actions run on them.
How do Spark SQL and DataFrames relate?
Spark SQL is the Spark module for structured data processing. It offers DataFrames as its programming abstraction and can also act as a distributed SQL query engine, so the lessons cover working with DataFrames in Python and running SQL queries against them programmatically.
Which datasets does the Spark code process?
The exercises draw on the works of William Shakespeare, FIFA 2018 data, and genomic data. The machine learning chapter loads the MovieLens dataset, a set of spam and non-spam messages, and a dataset of 5000 points for k-means clustering.
Are there longer multi-step exercises?
Yes. The RDD chapter ends with a sequence that builds a base RDD, removes stop words, and prints word frequencies. The Spark SQL chapter closes with a three-part exercise that creates a DataFrame from a CSV file, queries it with SQL, and visualizes the results.
User Reviews
Write your review
You must be logged in to post a review.
Reviews can only be posted by registered learners. Log in or create a free account, then you will return here to post.
There are no reviews yet.