Foundations of PySpark

Start Free Preview

Buying via our links supports this free directory · Affiliate disclosure.
Course facts
  • Payment: Free Preview (Paid Upgrade)
  • Learning style: Self-paced, start anytime
  • Format: Text & Interactive
  • Video access: Streaming Only
  • Before you start: Introduction to Python
DataCamp details
  • Chapters: 4
Created byDataCamp
Available onDataCamp

Course Details
Learning Style

Start Anytime

Level

Intermediate

Study Mode

100% Online

Duration

Short (< 10 Hours)

Languages
Audio Language

English

Outcomes
Certificate

Certificate of Completion

Price & Terms
Pricing Model

Free Preview (Paid Upgrade)

Technical
Content Medium

Text & Interactive

Video Access

Streaming Only

Specification
Access

Subscription

SKU: foundations-pyspark Category: Tags: , , Brand:

Use PySpark to query and join flight data in Spark, build a machine learning pipeline and tune a model that predicts flight delays.

At a glance
Duration4 hrs
LevelIntermediate
CertificateCertificate of Completion
LanguageEnglish
AccessSubscription
Add your review
Add to wishlistAdded to wishlistRemoved from wishlist 0
Add to compare

Spark runs parallel computation over large datasets, and PySpark is the package that lets Python code drive it. Working with flight records from Portland and Seattle, you open a Spark session, read its tables, reshape and summarize the data with SQL-style queries, and then prepare it for a model that predicts whether a flight will be delayed.

What You Will Achieve

  • Start a SparkSession in Python, list the tables it holds and pull a Spark DataFrame into pandas
  • Summarize and combine flight tables in Spark with SQL statements and DataFrame queries
  • Prepare raw flight columns as numeric features that a Spark model can use
  • Fit and tune a model that predicts flight delays, then score it with a binary classification evaluator


Program Overview

Spark Sessions and DataFrames in Python

How Spark manages data, and how a Python session reads and writes its tables.

  • SparkContext and SparkSession setup
  • Viewing and querying tables
  • Moving data between Spark and pandas
  • Reading and writing tables from Python

Manipulating Data with pyspark.sql

The pyspark.sql module and its optimized queries for shaping DataFrames inside a Spark session.

  • Creating columns
  • SQL queries in a session
  • Filtering, selecting and aggregating
  • Grouping and joining

Machine Learning Pipelines, Tuning and Selection

Building a full pipeline for the flight data, then tuning and comparing models that predict which flights will be delayed.

  • Casting data types and encoding carrier and destination
  • Assembling features into a pipeline
  • Splitting train and test data
  • Logistic regression, grids and cross validation

Certificate Awarded (Paid)

Complete the course to earn it.


Job Outlook

PySpark skills are used wherever teams process large datasets with Python, such as data engineering work that prepares and joins tables and analytics projects that train predictive models at scale.


Our take

Good for: Python users who want a first working grasp of Spark, carried through a single flight dataset all the way to a delay prediction model.

Keep in mind: Despite its intermediate label, the course is aimed at people new to Spark, and its modeling work stays with one logistic regression on the flights data.

Bottom line: A compact path through Spark DataFrames, SQL queries and a tuned logistic regression, all built on the same Portland and Seattle flight records.

Frequently Asked Questions

What dataset do the exercises use?

Every chapter works with records of flights from Portland and Seattle. The same data carries you through the first table queries, the joins and feature preparation, and the final model that predicts delays.

Which machine learning model does the course build?

The last chapter builds a logistic regression model that predicts whether a flight will be delayed. You set up a grid of settings, run cross validation over it, and check the result with an evaluator for binary classifiers.

How does PySpark work with pandas here?

An early lesson turns a Spark DataFrame into a pandas DataFrame, within a first chapter on how Spark manages data and how you read and write its tables from Python.

Can I run SQL queries inside Spark?

Yes. The second chapter uses the pyspark.sql module to run SQL against tables in your session, alongside DataFrame methods for filtering, selecting, aggregating, grouping and joining the flight data.

User Reviews

0.0 out of 5
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0

No reviews yet. Be the first to review this course.

There are no reviews yet.

Write your review

Reviews can only be posted by registered learners. Log in, then you will return here to post.

×

    Report this page
    LiveTalent
    Logo
    Compare items
    • Total (0)
    Compare
    0