8/27/2016
age
Apache Spark
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
Spark Overview
Unauthorized copying, distribution and exhibition of this presentation is
TA
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
1
8/27/2016
What is Apache Spark? a
• A fast and general engine for large-scale data processing
• A Open-source cluster computing framework
• End-to-End Analytics platform
• Developed to overcome limitations of Hadoop/Map Reduce
• Runs from a single desktop or a huge cluster
• Iterative, interactive or stream processing
• Supports multiple languages – Scala, Python, R, Java
• Major companies like Amazon, eBay, Yahoo use Spark.
Copyright @2016 V2 Maestros, All rights reserved.
When to use Spark?
• Data Integration and ETL
• Interactive Analytics
• High Performance Batch computation I
• Machine Learning and Advanced Analytics
• Real time stream processing
• Example applications
• Credit Card Fraud Detection
• Network Intrusion Detection
• Advertisement Targeting
Copyright @2016 V2 Maestros, All rights reserved.
2
8/27/2016
Typical Spark workflow
• Load data from source
I
• HDFS, NoSQL,S3, real time sources
• Transform Data
• Filter, Clean, Join, Enhance
• Store processed data
• Memory, HDFS, NoSQL
• Interactive Analytics
• Shells, Spark SQL, third-party tools
• Machine Learning
• Action
Copyright @2016 V2 Maestros, All rights reserved.
Online Reference
• [Link]
Copyright @2016 V2 Maestros, All rights reserved.
3
8/27/2016
Spark Eco-system
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
egg
Spark Framework
Programming Scala Python R Java Tools
Library Spark SQL ML Lib GraphX Streaming
Engine Spark Core
Management YARN Mesos Spark Scheduler
Storage Local HDFS S3 RDBMS NoSQL
Copyright @2016 V2 Maestros, All rights reserved.
4
8/27/2016
one
RDD
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
of
Resilient Distributed Datasets (RDD)
• Spark is built around RDDs. You create, transform, analyze and store
RDDs in a Spark program.
• The Dataset contains a collection of elements of any type.
• Strings, Lines, rows, objects, collections
• The Dataset can be partitioned and distributed across multiple nodes
• RDDs are immutable. They cant be changed.
• They can be cached and persisted
• Transformations act on RDDs to create a new RDD
• Actions analyze RDDs to provide a result
Copyright @2016 V2 Maestros, All rights reserved.
5
8/27/2016
Spark Architecture
e
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
Spark Architecture
Worker Node
Master Node Executor
Task Cache
Driver Program
Spark Cluster
Context Manager
Worker Node
Executor
Task Cache
Copyright @2016 V2 Maestros, All rights reserved.
6
8/27/2016
Driver Program e
• The main executable program from where Spark operations are
performed
• Runs in the master node of a cluster
• Controls and co-ordinates all operations
• The Driver program is the “main” class.
• Executes parallel operations on a cluster
• Defines RDDs
• Each driver program execution is a “Job”
Copyright @2016 V2 Maestros, All rights reserved.
SparkContext
• Driver accesses Spark functionality through a SparkContext object.
• Represents a connection to the computing cluster
• Used to build RDDs. Is
• Partitions RDDs and distributes them on the cluster
• Works with the cluster manager
• Manages executors running on Worker nodes
• Splits jobs as parallel “tasks” and executes them on worker nodes
• Collects results and presents them to the Driver Program
Copyright @2016 V2 Maestros, All rights reserved.
7
8/27/2016
Spark modes
• Batch mode
• A program is scheduled for execution through the scheduler
• Runs fully at periodic intervals and processes data
• Interactive mode
• An interactive shell is used by the user to execute Spark commands one-by-
one.
• Shell acts as the Driver program and provides SparkContext
• Can run tasks on a cluster
• Streaming mode
• An always running program continuously processes data as it arrives
Copyright @2016 V2 Maestros, All rights reserved.
Spark scalability
• Single JVM
Is
• Runs on a single box (Linux or Windows)
• All components (Driver, executors) run within the same JVM
• Managed Cluster
• Can scale from 2 to thousands of nodes
• Can use any cluster manager for managing nodes
• Data is distributed and processed on all nodes
Copyright @2016 V2 Maestros, All rights reserved.
8
8/27/2016
De
Loading and Storing Data
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
tag
Creating RDDs
• RDDs can be created from a number of sources
• Text Files
• JSON
• Parallelize() on local collections
• Java Collections
• Python Lists
• R Data Frames
• Sequence files
• RDBMS – load into location collections first and create RDD
• Very large data – create HDFS files outside of Spark and then create
RDDs from them
Copyright @2016 V2 Maestros, All rights reserved.
9
8/27/2016
Storing RDDs
• Spark provides simple functions to persist RDDs to a variety of data
sinks
• Text Files
• JSON
• Sequence Files
• Collections
• For optimization use language specific libraries for persistence than
using Spark utilities.
• saveAsTextFile()
• RDBMS – move to local collections and then store.
Copyright @2016 V2 Maestros, All rights reserved.
Lazy evaluation
• Lazy evaluation means Spark will not load or transform data unless an
action is performed
• Load file into RDD
• Filter the RDD
• Count no. of elements (only now loading and filtering happens)
• Helps internally optimize operations and resource usage
• Watch out during troubleshooting – errors found while executing
actions might be related to earlier transformations
Copyright @2016 V2 Maestros, All rights reserved.
10
8/27/2016
Transformations
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
to
Copyright @2016 V2 Maestros, All rights reserved.
Overview
• Perform operation on one RDD and create a new RDD
• Operate on one element at a time
• Lazy evaluation
• Can be distributed across multiple nodes based on the partitions they
act upon
Copyright @2016 V2 Maestros, All rights reserved.
11
8/27/2016
Map
newRdd=[Link](function)
• Works similar to the Map Reduce “Map”
• Act upon each element and perform some operation
• Element level computation or transformation
• Result RDD may have the same number of elements as original RDD
• Result can be of different type
• Can pass functions to this operation to perform complex tasks
• Use Cases
• Data Standardization – First Name, Last Name
• Data type conversion
• Element level computations – compute tax
• Add new attributes – Grades based on test scores
Copyright @2016 V2 Maestros, All rights reserved.
flatMap
newRdd=[Link](function)
• Works the same way as map
• Can return more elements than the original map
• Use to break up elements in the original map and create a new map
• Split strings in the original map
• Extract child elements from a nested json string
Copyright @2016 V2 Maestros, All rights reserved.
12
8/27/2016
Filter
newRdd=[Link](function)
I
• Filter a RDD to select elements that match a condition
• Result RDD smaller than the original RDD
• A function can be passed as a condition to perform complex filtering
• Returns a true/false
Copyright @2016 V2 Maestros, All rights reserved.
Set Operations
• Set operations are performed on two RDDs
• Union – Return a new dataset that contains the union of the elements
in the source dataset and the argument.
• unionRDD=[Link](secondRDD)
• Intersection - Return a new RDD that contains the intersection of
elements in the source dataset and the argument.
• intersectionRDD=[Link](secondRDD)
Copyright @2016 V2 Maestros, All rights reserved.
13
8/27/2016
af
Actions
Unauthorized copying, distribution and exhibition of this presentation is
Tf
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
Introduction to actions
• Act on a RDD and product a result (not a RDD)
• Lazy evaluation – Spark does not act until it sees an action
• Simple actions
• collect – return all elements in the RDD as an array. Use to trigger
execution or print values
• count – count the number of elements in the RDD
• first – returns the first element in the RDD
• take(n) – returns the first n elements
Copyright @2016 V2 Maestros, All rights reserved.
14
8/27/2016
reduce
• Perform an operation across all elements of an RDD
• sum, count etc.
• The operation is a function that takes as input two values.
• The function is called for every element in the RDD
inputRDD = [ a, b, c, d, e ]
and the function is func(x,y)
func( func( func( func(a,b), c), d), e)
Copyright @2016 V2 Maestros, All rights reserved.
Reduce Example
vals = [3,5,2,4,1]
sum(x,y) { return x + y }
sum( sum( sum( sum(3,5), 2), 4), 1) = 15
Copyright @2016 V2 Maestros, All rights reserved.
15
8/27/2016
Pair RDD
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
Tty
Pair RDDs
• Pair RDDs are a special type of RDDs that can store key value pairs.
• Can be created through regular map operations
• All transformations for regular RDDs available for Pair RDDs
• Spark supports a set of special functions to handle Pair RDD
operations
• mapValues : transform each value without changing the key
• flatMapValues : generate multiple values with the same key
Copyright @2016 V2 Maestros, All rights reserved.
16
8/27/2016
Pair RDD Actions
• countByKey – produces a count by each key in the RDD
I
• groupByKey – perform aggregation like sum, average by key
• reduceByKey – perform reduce, but by key
• aggregateByKey – perform aggregate by key
• Join - join multiple RDDs with the same key
Copyright @2016 V2 Maestros, All rights reserved.
E
Advanced Spark
Unauthorized copying, distribution and exhibition of this presentation is
F
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
17
8/27/2016
Local Variables in Spark
• Spark makes copies of your code (one per partition) and executes
Ss
them
• Any variable you create in the base programming language are local
to a cluster
• Duplicate copies of local variables for each cluster
• Each variable acted upon independently
Copyright @2016 V2 Maestros, All rights reserved.
Broadcast variables
• A read-only variable that is shared by all nodes
• Used for lookup tables or similar functions
• Spark optimizes distribution and storage for better performance.
Copyright @2016 V2 Maestros, All rights reserved.
18
8/27/2016
Accumulators
• A shared variable across nodes that can be updated by each node
• Helps compute items not done through reduce operations
• Spark optimizes distribution and takes care of race conditions
Copyright @2016 V2 Maestros, All rights reserved.
Partitioning
• By default all RDDs are partitioned
• [Link] parameter
• Default is the total no. of cores available across the entire cluster
• Should configure for large clusters
• Can be specified during RDD creation explicitly
• Derived RDD take the same number as the source.
Copyright @2016 V2 Maestros, All rights reserved.
19
8/27/2016
Persistence
• By default, Spark loads an RDD whenever it required. It drops it once
the action is over
• It will load and re-compute the RDD chain, each time a different operation is
performed
• Persistence allows the intermediate RDD to be persisted so it need
not have to be recomputed.
• persist() can persist the RDD in memory, disk, shared or in other third
party sinks
• cache() provides the default persist() – in memory
Copyright @2016 V2 Maestros, All rights reserved.
Se
Spark SQL
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
20
8/27/2016
Overview
• A library built on Spark Core that supports SQL like data and
operations
• Make it easy for traditional RDBMS developers to transition to big
data
• Works with “structured” data that has a schema
• Seamlessly mix SQL queries with Spark programs.
• Supports JDBC
• Helps “mix” n “match” different RDBMS and NoSQL Data sources
Copyright @2016 V2 Maestros, All rights reserved.
Spark Session
• All functionality for Spark SQL accessed through a Spark Session
• Data Frames are created through Spark Session
• Provides a standard interface to work across different data sources
• Can register Data Frames as temp table and then run SQL queries on
them
Copyright @2016 V2 Maestros, All rights reserved.
21
8/27/2016
DataFrame
• A distributed collection of data organized as rows and columns
• Has a schema – column names, data types
• Built upon RDD, Spark optimizes better since it knows the schema
• Can be created from and persisted to a variety of sources
• CSV
• Database tables
• Hive / NoSQL tables
• JSON
• RDD
Copyright @2016 V2 Maestros, All rights reserved.
Operations supported by Data Frames
• filter – filter data based on a condition
• join – join two Data Frames based on common column
• groupBy – group data frames by specific column values
• agg – compute aggregates like sum, average.
• Operations can be nested.
Copyright @2016 V2 Maestros, All rights reserved.
22
8/27/2016
Try
Temp Tables
Unauthorized copying, distribution and exhibition of this presentation is
punishable under law
Copyright @2016 V2 Maestros, All rights reserved.
Temp tables / Views
• Provide SQL table like operations
• A “wrapper” around a data frame
• Execute ANSI SQL Queries against temp tables.
• Very simple and yet very powerful
• A query on a Temp table generates another data frame
• createOrReplaceTempView – register the Data Frame as a table
within SQL Session
• SparkSession provides the ability to execute SQL
Copyright @2016 V2 Maestros, All rights reserved.
23
teabag
Mi
again
meat
tea
tea
i
ffhm
Bragging
doggy
EEEEdor
trap
TTEff
offer
affiliate