Big Data using Apache Spark with Scala and AWS

About 
This course offers you hands-on knowledge to create Apache Spark applications using Scala programming language in a completely case study based approach.

Apache Spark is a fast and general-purpose distributed computing system. It provides high-level APIs in Scala, Java, Python and R, and an optimized engine that supports general execution graphs (DAG). It also supports a rich set of higherlevel APIs and tools including DataFrame for Structured data processing using Object Oriented Programming and SQL, Structered Streaming for realtime stream processing, MLlib for machine learning and GraphX for graph processing.

Note: This is not an introductory or theory based course, though we'll be discussing about the basic comparison among Hadoop, Spark and Flink. We'll also be having some POCs around Hive, Pig, Presto, AWS Athena and Redshift Spectrum.

Learning Outcomes. By the end of this course,

1. You will be able to identify the type of data (structured, semi-structured or unstructured) been given to you and choose which of the Spark data abstractions to be used (RDD, DataFrame/Dataset) there.
2. Basis the complexity or the nature of the data pipeline you are working with, you’ll be able to assess which techniques to be used there – DataFrame's DSL or SQL, Streaming, etc.
3. Basis the nature of the data (Confidential/PII or not) you’ll be able to decide whether to go ahead with on-premise or cloud based solution. You can also estimate the computational resources requirement for the given data volume.
4. You would be able to comfortably setup Spark Development Environment in your local system and start working on any given big data application. And later, if you want you can productionalize the same on AWS cloud using AWS EMR cluster and AWS Lambda scripts.
5. You’ll be having enough hands-on experience on Spark that you’ll get a feeling as you’ve more than 3 years experience into “Developing Data Pipelines using Big Data Technologies”.

Recommended background
You should have at least some programming experience in any language, basic level should suffice, i.e., variable declaration, conditional expression - if-else, switch statement, control statements, collections, etc.

PART – 1: Getting Started with Spark – Programming RDD using Databricks Notebook
Setting up your 1st Spark cluster on Databricks Community Edition, it’s a free cloud platform. Introduction to Spark RDD. Transformations and Actions. Distribute key-value pairs (Paired RDD).

1. Writing your 1st Spark Program using Databricks Notebook– Word Count example
2. Creating RDD from different file formats (text file, sequence file, object file, etc.), Scala Collections, etc.
3. Optimization using correct partitioning
4. Transformation and Actions
5. Understanding Cluster Topology
6. Pair RDDs. Transformations and Actions on Pair RDDs – groupByKey(), reduceByKey(), aggregateByKey(), cogroup(), foldByKey(), join(), etc.
7. Narrow transformation vs Wider transformation
8. Spark job execution model, application_id -->> jobs -->> stages -->> tasks 

PART – 2: Setting up your local environment for Spark, Programming Scala and Completing RDD

1. Scala basics – Conditional Statement, iteration, user defined functions and higher order functions. OOPs Concepts – class, object and trait. Scala collections.
2. Introduction to Scala – variable declaration, control statements, loop, pattern matching, higher order functions, function currying and 
3. Object Orientation – class, object and trait. Constructor, Inheritance
4. Companion class/object, case class/object & singleton class
5. Scala Collections – mutable & immutable collections, higer order functions
6. Handling null values in Scala using Option, Some and None, Tuple
7. Exception Handling

PART – 3: Working with Structured Data: DataFrame/Dataset. Productionalizing Spark applications on AWS cloud.

1. DataFrames:
1. Converting RDDs to Dataframes through implicit schema inference or by specifying explicit schema
2. Reading structured data using sparkSession.read.fileFormat() and sparkSession.read.format("fileFormat") family of functions
3. sparkSession.sql("select * from parquet.`path`") and all
2. Applying transformation on dataframes using 
1. Domain Specific Language (DSL), e.g. df.select(), df.groupBy($”col1′′).sum($”col2′′), etc.
2. Spark SQL, e.g. sparkSession.sql("select * from temp_vw")
3. Applying window/analytics function using Dataframe’s DSL and Spark SQL. e.g. rank(), dense_rank(), lead(), lag(), etc.
4. Dataset (Typed dataFrame)
1. Solving compile time safety, domain object information, functional programming issues with DataFrame.
2. typed and untyped transformations
3. Encoders
5. Interoperability – Converting RDD to Dataset and vice versa, Dataframe to Dataset and vice versa
6. Deploying and monitoring spark application using AWS EMR, Lambda, Step Function and CloudWatch

PART – 4: Structured Streaming:

1. Understanding the High Level Streaming API in Spark 2.x
2. Triggers and Output modes
3. Unified APIs for Batch and Streaming
4. Building Advanced Streaming Pipelines Using Structured Streaming
5. Stateful window operations
6. Tumbling and Sliding windows
7. Watermarks and late data
8. Windowed joins
9. Integrating Apache Kafka with Structured Streaming

PART – 5: Real time case study
To make things fun and interesting, we will introduce multiple datasets coming from disparate data sources – SFTP, MS SQLServer, Amazon S3 and Google Analytics. And create an industry standard ETL pipeline to populate a data mart implemented on Amazon Redshift. Spark_Usecase_Architecture

1. Set up an Amazon EMR (Elastic Map Reduce) cluster and start Zeppline notebook
2. Set up Amazon Redshift database and its client
3. Create Spark dataframe out of files from remote SFTP
4. Read data from Amazon S3 in Spark
5. Create Spark dataframe out of data from MS SQL Server tables
6. Read Google Analytics data and process it in Spark
7. Schedule Spark job in Amazon Data Pipeline

Regards,
Data Science Monks
Mob +91 9148426497