Introduction to Data Engineering

Start Free Preview

Buying via our links supports this free directory · Affiliate disclosure.
Course facts
  • Payment: Free Preview (Paid Upgrade)
  • Learning style: Self-paced, start anytime
  • Format: Text & Interactive
  • Video access: Streaming Only
  • Before you start: Intermediate Python; Intermediate SQL
DataCamp details
  • Chapters: 4
Created byDataCamp
Available onDataCamp

Course Details
Learning Style

Start Anytime

Level

Intermediate

Study Mode

100% Online

Duration

Short (< 10 Hours)

Languages
Audio Language

English

Outcomes
Certificate

Certificate of Completion

Price & Terms
Pricing Model

Free Preview (Paid Upgrade)

Technical
Content Medium

Text & Interactive

Video Access

Streaming Only

Specification
Access

Subscription

SKU: introduction-data-engineering Category: Tags: , Brand:

Learn what data engineers do, compare databases and parallel computing tools, then build an ETL process and schedule it with Airflow.

At a glance
Duration4 hrs
LevelIntermediate
CertificateCertificate of Completion
LanguageEnglish
AccessSubscription
Add your review
Add to wishlistAdded to wishlistRemoved from wishlist 0
Add to compare

Data engineering prepares data for analysis, and this course sets out the role, its tools and its core process. You compare the data engineer’s role with the data scientist’s and look at the databases, parallel computing frameworks and workflow schedulers the job depends on. You also learn how ETL pulls data out of its sources, reshapes it and loads it into a target database. The last stretch has you build and schedule a working ETL process on real course rating data.

What You Will Achieve

  • Describe what a data engineer does and how the role works with data scientists
  • Tell SQL and NoSQL databases apart and read a star schema diagram
  • Break a processing task into parallel subtasks with PySpark
  • Build an ETL process and schedule it to run daily with an Airflow DAG


Program Overview

The Data Engineer’s Role and Cloud Services

Sets out what a data engineer is responsible for, which problems the role solves, and why cloud computing matters to the work.

  • Data engineer and data scientist tasks compared
  • Kinds of databases, processing and scheduling tools
  • Cloud providers and the services they offer

Databases, Parallel Computing and Workflow Scheduling

Looks closely at the databases, parallel computation frameworks and workflow schedulers that make up the data engineer’s everyday kit.

  • SQL vs NoSQL and database schemas
  • Joins on relations and star schemas
  • Spark, Hadoop, Hive and running PySpark files
  • Airflow, Luigi and cron

ETL in Practice and a Course Ratings Case Study

Walks through each ETL stage, then applies it in a case study that filters corrupt records and turns raw course ratings into daily recommendations.

  • Extracting data via an API and a database
  • Transforming, splitting and joining data
  • OLAP vs OLTP and loading into Postgres
  • Defining, enabling and querying a scheduled DAG

Certificate Awarded (Paid)

Complete the course to earn it.


Job Outlook

Data engineering skills are used wherever organizations collect and prepare data for analysis, in teams that build databases, run scheduled processing jobs and supply clean data to data scientists and analysts.


Our take

Good for: Analysts and developers curious about how data engineers move and prepare data, who want a survey of the tools plus one complete project.

Keep in mind: Spark, Hadoop, Hive, Luigi, cron and the cloud providers each get a brief look; the hands-on lessons center on PySpark, Postgres and Airflow DAGs.

Bottom line: Its closing case study, which turns raw course ratings into scheduled recommendations, gives the tool survey a concrete, working use.

Frequently Asked Questions

Which workflow schedulers are compared?

The lessons place Airflow alongside Luigi and cron, then concentrate on Airflow: you define a DAG, set Airflow up, interpret the DAG, and later enable one that runs a job every day.

What data does the closing case study use?

It works with raw course rating data. You explore the table schema, compute average ratings per course, filter out corrupt records, and write the resulting course recommendations to a target table on a daily schedule.

Which kinds of databases are discussed?

The second chapter contrasts SQL and NoSQL databases and explains database schemas, joins on relations and star schema diagrams. The ETL chapter adds the OLAP and OLTP distinction and loads transformed data into Postgres.

How much cloud computing is included?

Cloud computing appears in the first chapter as part of the data engineer's setup. The lessons cover why teams use it, the big players in cloud computing, and the kinds of cloud services on offer.

User Reviews

0.0 out of 5
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0
★★★★★
0

No reviews yet. Be the first to review this course.

There are no reviews yet.

Write your review

Reviews can only be posted by registered learners. Log in, then you will return here to post.

×

    Report this page
    Introduction to Data Engineering
    Introduction to Data Engineering
    LiveTalent
    Logo
    Compare items
    • Total (0)
    Compare
    0