Ch7:Spark
2022-2023 Ines Slimene
Outline
▪ Introduction
▪ Hadoop ecosystem
▪ HDFS
▪ MapReduce
▪ Hadoop request Language : Hive
▪ Data transfer : flume et sqoop
▪ ElasticSearch
▪ Spark
SPARK 2
History
SPARK 3
Introduction
▪ Provides a comprehensive and unified framework to meet the
needs of Big Data processing for various datasets
– data ype (text, graph, etc.)
– data source type (batch or real-time flow).
▪ Allows Hadoop applications to run up to 100 times faster in
memory and 10 times faster on disk.
▪ Allows to quickly write applications in Java, Scala, Python, or
R
SPARK 4
Hadoop and Spark
▪ Hadoop
– the solution of choice for processing large volumes of data.
▪ MapReduce
– It is necessary to express all use cases in the form of MapReduce
patterns to take advantage of this solution.
– The output data from the execution of each step must be stored on the
distributed file system before the next step begins.
– This approach tends to be slow due to replication and disk storage.
SPARK 5
Hadoop and Spark
▪ Hadoop requires the integration of several tools (such as
Mahout for Machine Learning and Storm for flow processing).
▪ If we want to set up something more complex, we have to
chain a series of MapReduce jobs and execute them
sequentially,
▪ Each of these jobs with high latency and none that can start
until the previous one has completely finished.
SPARK 6
Hadoop and Spark
▪ Spark replaces Mapreduce
▪ Faster than Hadoop: Sorting 100 TB of data
– MR: 72 minutes with 2100 knots (50400 cores)
– Spark: 23 minutes with 206 knots (6592 cores)
▪ Spark can be integrated with:
– Yarn, Zookeeper, Mesos
– HDFS
– Cassandra, Elasticsearch, MongoDB, hive
– Zeppelin
SPARK 7
Hadoop and Spark
SPARK 8
Spark specifications
▪ Spark keeps intermediate results in memory rather than on
disk, which is very useful especially when it is necessary to
work several times on the same dataset.
▪ The execution engine is designed to work both in memory and
on disk.
▪ Spark tries to store as much as possible in memory before
switching to disk.
▪ It is able to work at the same time with data in memory and
data in disk.
SPARK 9
Spark specifications
▪ Spark is written in Scala and runs on the Java virtual machine
(JVM).
▪ Supported languages for application development are:
– Scala
– Java
– Python
–R
SPARK 10
Spark ecosystem
SPARK 11
Spark ecosystem
▪ Spark Streaming: real-time processing of streaming data. It
relies on a "micro batch" processing mode and uses Dstream
real-time data.
▪ Spark SQL: allows to extract, transform and load data in
different formats (JSON, Parquet, database)
▪ Spark MLlib: MLlib is a machine learning library that contains
all the classical learning algorithms and utilities, such as
classification, regression, clustering, etc.
▪ Spark GraphX: API for processing and parallelizing graphs.
GraphX includes a large collection of algorithms and builders
to simplify graph analysis tasks.
SPARK 12
Architecture
SPARK 13
Architecture
SPARK 14
Architecture
▪ Apache Spark has a master/slave architecture with two
daemons and a cluster manager:
– Driver Daemon (Master Process)
– Worker Daemon (Slave Process)
▪ A spark cluster has a single Master and an undetermined
number of slaves/workers.
SPARK 15
Spark Driver
▪ The driver is responsible for running the application main()
function and creating the SparkContext.
▪ Spark driver contains several components ( DAGScheduler,
TaskScheduler, BackendScheduler, BlockManage) that are
responsible for translating the spark code into a job (set of
tasks) that will be executed in the cluster.
▪ The driver schedules job execution and negotiates resources
with the cluster manager.
▪ The driver records the RDD meta-data and its partitions.
SPARK 16
Cluster Manager
▪ An external service responsible for allocating resources to jobs
launched by the drivers.
▪ Three types:
– Standalone,
– Mesos,
– YARN.
▪ These types are different in terms of scheduling, security, and
monitoring
SPARK 17
Executor
▪ The executor is a distributed agent responsible for executing
tasks.
▪ Each spark application has its own executor process.
▪ We can choose static or dynamic allocation (to balance
processing loads)
▪ The executor reads and writes the data from external sources.
▪ The Executor stores the calculations results in memory, cache
or on disk.
SPARK 18
Worker node
▪ The worker, is a node that allows to execute a program in the
cluster.
▪ If a process is launched, the application requires executors in
worker nodes
▪ As soon as SparkContext connects to the cluster manager, it
acquires executors in worker nodes
▪ Executors work independently when performing a task and can
interact together.
SPARK 19
SparkContext
▪ SparkContext allows you to establish a connection with the
cluster manager. Can be used to create RDD and
accumulators, and diffuse variables in the cluster.
▪ It is recommended to have only one SparkContext active per
JVM, so we have to call the stop() method of the SparkContext
active before creating another.
SPARK 20
Job spark execution (1)
▪ When a client validates a spark code, the driver translates the code that
contains actions and transformations into a logical direct acyclic graph
(DAG).
▪ At this stage, the driver optimizes the sequence of transformations and
converts the logical DAG into a physical execution plan with a set of steps
▪ The driver creates small physical units of execution (tasks) for each step.
▪ The driver negotiates resources with the cluster manager
▪ The manager cluster launches executors in worker nodes on behalf of the
driver.
▪ At this point, the driver sends the tasks to the cluster manager based on
the location of the data.
SPARK 21
Job spark execution (2)
▪ Before the tasks are executed, they are registered with the
driver so that it can monitor them.
▪ The Driver also plans future tasks based on their memory
locations.
▪ When the main method of the driver program ends or the
SparkContext stop() method is launched, the driver will
terminate all executors and release the resources of the
cluster manager.
SPARK 22