Apache Spark Foundation

Apache Spark Foundation Syllabus

1. Apache Spark Introduction

a) What is Apache Spark?

b) What are Apache Spark Components?

c) What are Spark Opportunities/Job Roles?

1. Apache Spark vs Apache Hadoop

2. Apache Spark Installation on Windows Operating System

3. Apache Spark Programming Introduction

4. Apache Spark with Java Language

5. Apache Spark with Scala Language

6. Apache Spark with Python (PySpark) Language

7. Apache Spark with R Language (SparkR)

Apache Spark Introduction

d) What is Apache Spark?

e) What are Apache Spark Components?

f) What are Spark Opportunities/Job Roles?

What is Apache Spark?

Apache Spark is a Data Processing Framework.

Apache Spark Software implemented using Scala Language.

Apache Spark Applications/Programs we can write using 4 Languages. They are:

1. Apache Spark with Java Language

2. Apache Spark with Scala Language (Spark & Scala)

3. Apache Spark with Python Language (PySpark)

4. Apache Spark with R Language (SparkR)

Apache Spark API Available for Java, Scala, Python and R Language

API – Application Programming Interface (Contains Predefined classes, functions and Variables)

Apache Spark is Pure Data Processing Framework. It does not have Storage. It can process any data.

What are Apache Spark Components?

1. Apache Spark Core

2. Apache Spark SQL

3. Apache Spark Streaming

4. Apache Spark ML/MLib

5. Apache Spark GraphX

Apache Spark Core – RDD Programming (RDD – Resilient Distributed Dataset) (Transformations, Actions using either Java or Scala or Python)

Apache Spark SQL – DataFrames/Tables/Datasets – We write SQL Kind Programming.

Apache Spark Streaming – Streaming + Live Analytics

Apache Spark ML/MLib – Machine Learning

Apache Spark GraphX – Linked Data/Graph Data Processing

Apache Spark Application we can deploy on

Apache Spark Standalone Cluster or

YARN Cluster (Hadoop Cluster)

Mesos Cluster

Kubernetes Cluster

What are Spark Opportunities/Job Roles?

There are Two Kinds of Job Roles/ Opportunities in Apache Spark World. They are:

1. Apache Spark Developer

2. Apache Spark Machine Learning Developer or Apache Spark Data Scientist

Apache Spark Developer

1. Strong Apache Spark Foundation

2. Apache Spark Core Programming

3. Apache Spark SQL

4. Apache Spark Streaming

5. Apache Spark Integration Like RDBSM, NoSQL, Streaming Frameworks and Cloud Computing

6. Programming Language (Java or Scala or Python or R)

7. SQL on Any RDBMS

8. Linux Essentials.

9. Any Cloud Computing like AWS or Azure or GCP

Apache Spark Machine Learning Developer or Apache Spark Data Scientist

1. Apache Spark Developer

2. Apache Spark ML/MLib

3. Apache Spark GraphX

4. ML/DL/Data Science Algorithms

5. Mathematics and Statistics

Who this course is for:

• • Any IT aspirant/professional willing to learn/Become Data Engineering using Apache Spark
• • Python Developers who want to learn Spark to add the key skill to be a Data Engineer
• • Scala based Data Engineers who would like to learn Spark using Python as Programming Language
• • Who are Freshers/Experienced – Who Wants to Become Data Engineers
• • Who are Programmers like Java, Scala, .Net, Python etc.. willing to learn/Become Data Engineering using Apache PySpark
• • Who are Database Developer/DBA willing to learn/Become Data Engineering using Apache PySpark
• • Who are Data Warehouse and Reporting People willing to learn/Become Data Engineering using Apache PySpark
• • Non-Programmers like Test Engineers etc.. willing to learn/Become Data Engineering using Apache PySpark