Spark Tutorial
Spark Topics
• Difference between Data lake,
• Cluster Manager Types Datawarehouse, Delta lake
• Apache Spark & History • Data lake & delta lake architecture
• Apache Spark Features • Difference between Data frame, RDD,
• Apache Spark Advantages Dataset
• Apache Spark Architecture --------------------------------------------------
• File types • Transformation and actions
• Spark Modules • Data skew and the methods to handle
• Spark Session skew
• Spark Context -------------------------------------------------
--------------------------------------------------------------- • Spark Data Source API
----- • DAG
• Spark Data Frame • Lazy evaluation
• Data Frame creation • Catalyst Optimizer
• Spark Data Frame Advanced concepts • AQE
• Data frame join methods • Cache & Persistence
• Optimization & performance
improvement
• GC ,Join types ,Broadcast variable
Databricks Topics
• Databricks Community edition account creation and UI walk
• Cluster creation
• Cluster types & modes
• Dbutils Command & magic commands
• Autoscaling
• Databricks programming
• Fetching data from ADLS to Databricks and creation of medallion struct (ETL)
• Databricks Transformation
• Workflows - Job creation
• Catalog and data bricks repo
• Migration project
AZURE CLOUD
• Azure Cloud
• Azure services
• Azure data factory
• Azure storages
• Azure key vault
• Azure active directory
• ETL Creation using ADF
• Access azure storage from data bricks
• Blob vs ADF VS ADLS
• Azure Data Factory + Azure Databricks:
• key components in ADF
• AZURE activity
• Realtime scenarios
• Pipeline creation
• Azure end to end CI/CD
Apache Spark & History
Apache Spark is an Open source analytical processing engine for large scale powerful distributed
data processing and machine learning applications. Spark is Originally developed at the University
of California, Berkeley’s, and later donated to Apache Software Foundation. In February 2014,
Spark became a Top-Level Apache Project and has been contributed by thousands of engineers and
made Spark one of the most active open-source projects in Apache.
Apache Spark is a framework that is supported in Scala, Python, R Programming, and Java. Below
are different implementations of Spark.
•Spark – Default interface for Scala and Java
•PySpark – Python interface for Spark
•SparklyR – R interface for Spark.
Apache Spark Features
Apache Spark Advantages
•Spark is a general-purpose, in-memory, fault-tolerant, distributed processing engine that
allows you to process data efficiently in a distributed fashion.
•Applications running on Spark are 100x faster than traditional systems.
•You will get great benefits using Spark for data ingestion pipelines.
•Using Spark we can process data from Hadoop HDFS, AWS S3, Databricks DBFS, Azure
Blob Storage, and many file systems.
•Spark also is used to process real-time data using Streaming and Kafka.
•Using Spark Streaming you can also stream files from the file system and also stream from
the socket.
•Spark natively has machine learning and graph libraries.
•Provides connectors to store the data in NoSQL databases like MongoDB.
Apache Spark disadvantages
• It doesn’t have any automatic code optimization process.
• It doesn’t come with its own file management system. It depends on some other platforms
like Hadoop or other cloud-based platforms.
Spark Modules
Apache Spark Architecture
As you have already seen the basic architectural overview of Apache Spark, now let’s dive deeper into its
working.
In your master node, you have the driver program, which drives your application. The code you are writing
behaves as a driver program or if you are using the interactive shell, the shell acts as the driver [Link]
the driver program, the first thing you do is, you create a Spark Context. Assume that the Spark context is a
gateway to all the Spark functionalities. It is similar to your database connection. Any command you execute in
your database goes through the database connection. Likewise, anything you do on Spark goes through Spark
context.
To know about the workflow of Spark Architecture, you can have a look at the info graphic below:
STEP 1:
The client submits spark user application code. When application code is submitted, the driver implicitly converts user
code that contains transformations and actions into a logically directed acyclic graph called DAG. At this stage, it also
performs optimizations such as pipelining transformations.
STEP 2:
After that, it converts the logical graph called DAG into physical execution plan with many stages. After converting into a
physical execution plan, it creates physical execution units called tasks under each stage. Then the tasks are bundled and
sent to the cluster.
STEP 3:
Now the driver talks to the cluster manager and negotiates the resources. Cluster manager launches executors in worker nodes
on behalf of the driver. At this point, the driver will send the tasks to the executors based on data placement. When executors
start, they register themselves with drivers. So, the driver will have a complete view of executors that are executing the task.
STEP 4:
During the course of the execution of tasks, driver program will monitor the set of executors that runs. Driver node also schedules future
tasks based on data placement.
DAG
Working of DAG
WHY DAG
ON-HEAP MEMORY ARCHITECTURE
On-Heap: The memory that is controlled by JVM & allotted to executor is on-heap a memory.
OFF HEAP MEMORY
Off-Heap: The memory that is controlled or allocated to OS is off-heap memory.
RDD Spark
RDD (Resilient Distributed Dataset) Tutorial
is a fundamental data structure of Spark and it is the primary
data abstraction in Apache Spark and the Spark Core. RDDs are fault-tolerant, immutable
distributed collections of objects, which means once you create an RDD you cannot change it.
Each dataset in RDD is divided into logical partitions, which can be computed on different nodes
of the cluster.
RDD, DATAFRAME, DATASET
An action is one of the ways of sending data from Executer to the driver. Executors are agents
that are responsible for executing a task. While the driver is a JVM process that coordinates
workers and execution of the task
File types
Catalyst Optimizer
Catalyst optimizer will actually check for multiple execution plans to execute one particular
statement or one action on data frame based on which it will analyse the performance and
choose the best execution plan.
Optimization or Performance improvement
[Link] the help of code: like reducing shuffle of data across nodes.
2. With the help of file format: like performance can be reduced when parquet and ORC are
used.
3. with upgraded version: each upgraded version will or may have some performance
increased in it.
To keep in mind about.
[Link] of data
[Link] scan
(INNER) JOIN: Returns records that have matching values in both tables
LEFT (OUTER) JOIN: Returns all records from the left table, and the matched records from
the right table
RIGHT (OUTER) JOIN: Returns all records from the right table, and the matched records from
the left table
FULL (OUTER) JOIN: Returns all records when there is a match in either left or right table
Left semi join: Returns all the common records within two data frames.
Left anti join: Return only the uncommon records from left table.
What is SCD in data warehouse
A Slowly Changing Dimension (SCD) is a dimension that stores and manages both
current and historical data over time in a data warehouse
Type 1 – This model involves overwriting the old current value with the new current
value.
Overwrite the changes
Type 2 – The current and the historical records are kept and maintained in the same
file or table.
History will be added as a new row.
Type 3 – The current data and historical data are kept in the same record.
History will be added as a new column.
Z-ordering
Z –ordering is the extension of optimize and z ordering is used along with optimize
optimize : is used combine many smaller files into larger one at the time it doesn’t care
about data ordering it would randomly combine the file and it will create optimal size of
the file.
and the optimal size is 1GB.
Z-ordering : along with the optimize the does the sorting of the data in order and from
which the delta log statistical table is created which will help in skipping the particular
range of the log metadata so that it will only pick the particular by skipping the
unwanted or unasked records in query.
[Link]
Thank you !
Happy Learning
Its not the end its
beginning
Vacuum Command
Vacuum Command is use to clean obsolete files periodically which are not part of latest version of delta lake
[Link]
What is Change Data
Capture?
Change data capture (CDC) refers to the tracking of all changes in a data source (databases, data
warehouses, etc.) so they can be captured in destination systems.
When data is changed (through INSERT, UPDATE, or DELETE) in the source database—which is
usually a relational database such as MySQL, Microsoft SQL, Oracle, or PostgreSQL—it needs to be
propagated to downstream systems such as a cache, search index, data warehouse, or data lake.
Push vs. Pull
Push: In this approach, the source database does the heavy lifting. It implements logic and
processes to capture changes in the database, then sends those updates to target systems so they
can take appropriate action. The advantage of this method is that target systems will be updated
with the latest data in near real time. The downside is that changed data will be lost if the target
systems are unreachable or offline. To remedy that, a messaging system is typically implemented
between source and target systems so that changes aren’t lost until they are committed to their
final destinations.
Pull: In this method, the source database’s is lighter than in the push method, since it only logs the
data changes in a column on each table. It is the target systems’ responsibility to continuously poll
the source database to retrieve the changes and take the right actions on them. Just like in the
push method, a messaging system is needed between the source and target systems to ensure
that the changed data isn’t lost when the target systems are unavailable. The disadvantage of the
Category ETL ELT
Data is extracted from a source system, Data is extracted from a source system, loaded into a
Definition transformed on a secondary processing server, and destination system, and transformed inside the
loaded into a destination system. destination system.
Transform Raw data is transformed on a processing server. Raw data is transformed inside the target system.
Load Transformed data is loaded into a destination Raw data is loaded directly into the target system.
system.
Speed ETL is a time-intensive process; data is transformed ELT is faster by comparison; data is loaded directly into
before loading into a destination system. a destination system, and transformed in-parallel.
Code-Based Performed on secondary server. Best for compute- Transformations performed in-database; simultaneous
Transformations intensive transformations & pre-cleansing. load & transform; speed & efficiency.
Maintenance Secondary processing server adds to the With fewer systems, the maintenance burden is
maintenance burden. reduced.
Costs Separate servers can create cost issues. Simplified data stack costs less.
Requeries Data is transformed before entering destination Raw data is loaded directly into destination system
system; therefore raw data cannot be requeried. and can be requeried endlessly.
Data Output Structured (typically). Structured, semi-structured, unstructured.
Ideal for small data sets with complicated
Data Volume transformation requirements. Ideal for large datasets that require speed & efficiency.