ETL and ELT in Python

Start Free Preview

Buying via our links supports this free directory · Affiliate disclosure.
Course facts
  • Payment: Free Preview (Paid Upgrade)
  • Learning style: Self-paced, start anytime
  • Format: Text & Interactive
  • Video access: Streaming Only
  • Before you start: Data Warehousing Concepts; Streamlined Data Ingestion with pandas
DataCamp details
  • Chapters: 4
Created byDataCamp
Available onDataCamp

Course Details
Learning Style

Start Anytime

Level

Intermediate

Study Mode

100% Online

Duration

Short (< 10 Hours)

Languages
Audio Language

English

Outcomes
Certificate

Certificate of Completion

Price & Terms
Pricing Model

Free Preview (Paid Upgrade)

Technical
Content Medium

Text & Interactive

Video Access

Streaming Only

Specification
Access

Subscription

Build ETL and ELT pipelines in Python with pandas, load data to CSV files and Postgres, and test them with pytest in an interactive course.

At a glance
Duration4 hrs
LevelIntermediate
CertificateCertificate of Completion
LanguageEnglish
Taught byJake Roach
AccessSubscription
Add your review
Add to wishlistAdded to wishlistRemoved from wishlist 0
Add to compare

Data pipelines collect, process and move data so that it is ready for downstream use, and building them well is core data engineering work. ETL and ELT order those steps differently: one transforms data before loading it, the other loads it first and transforms it inside the warehouse. You write Python that pulls records from Parquet files, SQL databases and JSON, reshapes them with pandas and persists the results, then add logging, exception handling and tests so the pipeline holds up in production.

What You Will Achieve

  • Decide whether a job calls for an ETL design or an ELT design based on where the transform step should run
  • Write reusable Python functions that pull structured and non-tabular data into pandas DataFrames
  • Clean, group and reshape DataFrames, then persist the output so it is ready for downstream use
  • Add logging, exception handling and pytest unit tests that catch broken data before a pipeline reaches production


Program Overview

ETL and ELT Pipeline Foundations

Looks at how pipelines collect, process and move data, and where the transform step sits in each of the two patterns.

  • Structure of ETL and ELT pipelines
  • Running an ETL pipeline and ELT in action
  • Transforming data inside the destination
  • Student scores data exercise

Extracting, Transforming and Loading with pandas

Builds reusable ETL logic in pandas for tabular and JSON sources, with logging, exception handling and alerting added along the way.

  • Extracting from Parquet files and SQL databases
  • Filtering, grouping and filling missing values
  • Ingesting and parsing JSON data and dictionaries
  • Loading to CSV files and a Postgres database

Testing and Deploying a Data Pipeline

Validates and tests the pipeline before it ships, then runs it end to end with visibility into how it performs.

  • Manual checks and validation checkpoints with assert
  • Unit tests and fixtures with pytest
  • End-to-end runs in production
  • Orchestration tools and architecture patterns

Certificate Awarded (Paid)


Job Outlook

Building, testing and running Python pipelines is everyday work on data engineering and analytics teams, where database and file data has to arrive clean, validated and on schedule.


Our take

Good for: Analysts and aspiring data engineers who want to turn one-off data scripts into reusable, tested and logged Python pipelines.

Keep in mind: Despite the title, ELT appears mainly in the opening chapter; the later chapters build ETL pipelines in pandas, loading to CSV files and a Postgres database.

Bottom line: Expect to write a pandas pipeline and then prove it works with pytest fixtures, validation checkpoints and an end-to-end run.

Frequently Asked Questions

How do ETL and ELT pipelines differ?

The difference is where transformation happens. An ETL pipeline reshapes data in a separate processing step before loading it into the warehouse, while an ELT pipeline loads the raw data first and transforms it inside the warehouse. The opening chapter runs both patterns.

What does the pipeline code rely on?

Most of the work is done in pandas, with SQL for pulling and loading data, Parquet and JSON files as sources, and pytest for unit tests and fixtures. Loading targets include CSV files and a Postgres database.

How is a pipeline tested before it goes to production?

The final chapter starts with manual checks, then adds validation checkpoints written with assert, unit tests and fixtures in pytest, and an end-to-end test. It closes with running the pipeline in production, orchestration and ETL tools, and common architecture patterns.

Can the pipelines handle data that is not in tables?

Yes. The third chapter ingests JSON with pandas, reads it into memory, iterates over and parses dictionaries, and turns the parsed records into DataFrames that can be cleaned and then loaded into a SQL database.

User Reviews

0.0 out of 5
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0

No reviews yet. Be the first to review this course.

There are no reviews yet.

Write your review

Reviews can only be posted by registered learners. Log in, then you will return here to post.

×

    Report this page
    LiveTalent
    Logo
    Compare items
    • Total (0)
    Compare
    0