MapReduce
Contents
Scenario
Scenario … Contd.
Scenario … Contd.
Scenario … Contd.
Scenario … Contd.
Scenario … Contd.
MR - Analogy
MR – Analogy … Cont.
What is MapReduce?
What is MapReduce? … Cont.
What is MapReduce? … Cont.
MapReduce – Input data
MapReduce – Mapper
MapReduce – Reducer
MapReduce – Result (Output)
MapReduce – Workflow
• Input
• Mapper
• Combiner
• Partitioner
• Reduced
• Output
MapReduce – Workflow … Contd.
MapReduce – Workflow – Input data
MapReduce – Workflow - Input data
MapReduce – Workflow – Record
Reader
MapReduce – Workflow – Record
Reader
MapReduce – Workflow – Mapper
MapReduce – Workflow – Combiner
MapReduce – Workflow – Partitioner
MapReduce – Workflow – Shuffling &
Sorting
MapReduce – Workflow – Shuffling &
Sorting
MapReduce – Workflow – Reducer
MapReduce – Workflow – Reducer
MapReduce – Workflow – Output
MapReduce – Workflow – Overall
MapReduce – Architecture – Hadoop
1.x
Hadoop 1.x
Datanodes
JobTracker
• Master Daemon
Namenode • Responsible to assign and track
task execution progress
Task Tracker
• Slave daemons
• Run on systems where data nodes
resides
• Responsible to create a child JVM
to execute MR and Intermediate
tasks
Dr Vengadeswaran CSE412 Big Data and Large-Scale Computing 34
MapReduce – Architecture … Contd.
MapReduce – Architecture … Contd.
MapReduce – Architecture … Contd.