Building Recommendation Engines with PySpark
Learn to build, tune and evaluate ALS recommendation engines in PySpark on explicit movie ratings and ratings inferred from customer behavior data.
Recommendation engines here are built with Alternating Least Squares (ALS) in PySpark, a matrix factorization method that predicts the ratings missing from a large user-item matrix by uncovering latent features. You work through the intuition and the math behind ALS, then write the code to train, test and tune models on movie ratings. Because real customer data seldom holds clean ratings, you also infer ratings from customer behavior and build implicit-feedback models on them.
What You Will Achieve
- Tell whether a recommendation problem suits collaborative filtering and whether its data gives explicit or implicit ratings
- Shape a Spark DataFrame for ALS, with distinct users and integer IDs assigned to each item
- Fit an ALS model on a train/test split, tune its hyperparameters with cross-validation and read the resulting RMSE
- Turn customer behavior data into inferred ratings and build ALS recommenders from them
Program Overview
Recommendation Engine Types and Latent Features
Shows what recommendation engines can do and the distinctions that decide how one is built.
- Collaborative vs content-based filtering
- Implicit and explicit ratings data
- Other uses for recommendation engines
- Hidden latent features in customer datasets
How ALS Works and Data Preparation for Spark
Reviews the matrix math behind ALS, then readies data and builds a first model with an evaluator.
- Matrix multiplication and matrix factorization
- Non-negative matrix factorization
- How ALS alternates to estimate predictions
- ALS parameters, hyperparameters and an RMSE evaluator
Building ALS Models on Movies and Songs
Applies ALS to the MovieLens data, then handles the case with no customer ratings using the Million Songs Dataset.
- MovieLens sparsity and summary statistics
- Cross-validation, best model parameters and sanity checks on recommendations
- Implicit ratings, added zeros and their evaluation
- Binary implicit ratings models
Certificate Awarded (Paid)
Job Outlook
These skills apply wherever customer data feeds product suggestions, such as the movie and music catalogs used in the exercises, and in data science and data engineering work that runs on Spark.
Good for: Learners already working with Spark who want to build collaborative-filtering recommenders and judge them with RMSE and cross-validation.
Keep in mind: Every model you build is an ALS collaborative-filtering model; content-based engines appear only in the opening comparison of engine types.
Bottom line: A focused run through ALS in PySpark that pairs the matrix math with cross-validated models on explicit movie ratings and implicit song data.
Frequently Asked Questions
What is matrix factorization and how does ALS use it?
Matrix factorization splits a large user-item matrix into smaller matrices whose product approximates it, exposing latent features that help predict missing ratings. ALS builds on it, and the lessons show how it alternates to generate predictions and how RMSE changes as it does.
Which datasets do the exercises use?
The MovieLens dataset carries the explicit-ratings work, including a full cross-validated ALS model. The Million Songs Dataset and a second version of MovieLens are used for the chapter on implicit and binary ratings.
How are ratings inferred when customers never rate anything?
You start from customer behavior data, compute summary and grouped statistics, add zeros to the data, and then fit implicit ALS models with their own hyperparameters. A binary version of the ratings is also modeled and its recommendations reviewed.
How does the course judge whether recommendations are good?
Explicit models are scored with an RMSE evaluator, and a lesson covers what the RMSE value means. You also check whether the recommendations make sense, and implicit models get their own evaluation approach.


There are no reviews yet.