Machine Learning with PySpark

Start Free Preview

Buying via our links supports this free directory · Affiliate disclosure.
Course facts
  • Payment: Free Preview (Paid Upgrade)
  • Learning style: Self-paced, start anytime
  • Format: Text & Interactive
  • Video access: Streaming Only
  • Before you start: Supervised Learning with scikit-learn; Introduction to PySpark
DataCamp details
  • Chapters: 4
Created byDataCamp
Available onDataCamp

Course Details
Learning Style

Start Anytime

Level

Advanced

Study Mode

100% Online

Duration

Short (< 10 Hours)

Languages
Audio Language

English

Outcomes
Certificate

Certificate of Completion

Price & Terms
Pricing Model

Free Preview (Paid Upgrade)

Technical
Content Medium

Text & Interactive

Video Access

Streaming Only

Specification
Access

Subscription

SKU: machine-learning-pyspark Category: Tags: , , , Brand:

Learn to train, evaluate and tune classification and regression models on Spark from Python, with pipelines, cross-validation and ensemble methods.

At a glance
Duration4 hrs
LevelAdvanced
CertificateCertificate of Completion
LanguageEnglish
AccessSubscription
Add your review
Add to wishlistAdded to wishlistRemoved from wishlist 0
Add to compare

Training models on Big Data means splitting the computation across many machines, and managing that split by hand pulls attention away from the analysis itself. Spark handles the distribution across a cluster, and PySpark lets you drive it from Python. You read CSV files into a Spark session, shape raw columns into model features, and train classifiers and regression models on flight records and text messages. You then organize the work into reusable pipelines and test it with cross-validation.

What You Will Achieve

  • Connect to Spark from Python and load CSV data ready for modeling
  • Build classifiers that predict a category, such as flagging spam text messages, and measure how well they perform
  • Fit a linear regression model, read its coefficients, and use regularization to keep only the most relevant predictors
  • Chain preparation and modeling steps together and pick model parameters with grid search


Program Overview

Spark Basics and Loading Data

Background on Spark and machine learning, how a Spark cluster is put together, and the first steps of connecting and reading data.

  • Characteristics of Spark and cluster components
  • Creating a SparkSession and locating the Spark master
  • Loading flights and SMS spam CSV files

Classification and Regression Models

Preparing columns for modeling, then building classification and linear regression models and improving them with engineered predictors.

  • Removing, manipulating and assembling columns, with categorical data
  • Decision trees and logistic regression with a train/test split
  • Tokens, stopwords and hashing for a spam classifier
  • One-hot encoding, bucketing and regularization for a flight duration model

Pipelines, Cross-Validation and Ensembles

Making model code clearer and easier to maintain, testing it more thoroughly, and a short look at two ensemble methods.

  • Pipeline stages and fitted pipeline models
  • Cross-validation and grid search for parameter selection
  • Gradient-boosted trees and random forests for delayed flights

Certificate Awarded (Paid)


Job Outlook

These skills are used where models are trained on Big Data held on Spark clusters, in data engineering and data science teams that build their machine learning workflows in Python.


Our take

Good for: Data practitioners who want to move their model building onto Spark and work through classification, regression and parameter tuning in Python.

Keep in mind: The exercises return again and again to two datasets, flight records and SMS messages, and the two ensemble methods get only a brief look in the final chapter.

Bottom line: A hands-on route into Spark’s machine learning tools, where one flight duration model grows step by step through encoding, regularization and a full pipeline.

Frequently Asked Questions

Which datasets are used in the exercises?

Most exercises work with two datasets: flight records, used for flight duration and delayed flight models, and a collection of SMS messages, used to build a spam classifier. A small shirt sizes example also appears in the lessons on encoding categorical values.

How are text messages turned into model input?

A dedicated part of the classification chapter shows how to convert text into tables a model can use. It deals with punctuation, numbers and tokens, removes stopwords, applies hashing, and then trains a classifier that separates spam from ordinary messages.

Which ensemble models appear in the final chapter?

The closing chapter introduces two ensemble methods, gradient-boosted trees and random forests, both applied to predicting delayed flights. The random forest is then evaluated. These lessons are brief, so treat them as a starting point rather than a full study of ensembles.

How are the best model parameters chosen?

The course uses cross-validation to test models more reliably, first on a simple flight duration model and then on a full pipeline. Grid search then compares parameter settings, and one lesson dissects the best flight duration model it finds.

User Reviews

0.0 out of 5
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0

No reviews yet. Be the first to review this course.

There are no reviews yet.

Write your review

Reviews can only be posted by registered learners. Log in, then you will return here to post.

×

    Report this page
    Machine Learning with PySpark
    Machine Learning with PySpark
    LiveTalent
    Logo
    Compare items
    • Total (0)
    Compare
    0