Module 2 - MapReduce
MapReduce is a programming model that uses parallel processing to speed large-scale data
processing.
Core data processing framwork of haddop and based on Java
Input Files: The data for a Map Reduce task is stored in input files and these input files are
generally stored in HDFS.
Input Format: Input Format defines how the input files are split and read. I
Input Splitting : Slits the input data so as to processed by an individual Mapper.
The splitting divides the input files into records and each record will be processed by the
mapper.
Map Tasks:
A Map Task is responsible for processing each split block of input data and generating
intermediate key–value pairs.
These <key, value> pairs can be completely different from the input pair.
Input - split block or record
output - collection of <Key , value> pairs
Combiner: A Combiner is an optional component in Hadoop MapReduce that performs local
aggregation of the Mapper’s output before sending it to the Reducer.
It is also called a “mini-reducer” because it performs the same type of operation as the
Reducer, but only on the Mapper’s local output.
Shuffle& Sorting: intermediate key–value pairs from the Map phase are grouped and sorted by
key before passing them to reducers
Reduce Task:
A Reduce Task is responsible for taking all the intermediate key-value pairs (from all Mappers),
grouping them by key, and performing an aggregation or summary operation.
It takes the set of intermediate key-value pairs produced by mappers/combiners as the
input and then runs a reducer function on each of them to generate the output.
The output of the reducer is the final output, which is stored in HDFS.
Reducers run in parallel as they are independent of one another
Function of Map Task:
1. Reads input data from HDFS (usually one block at a time).
2. Converts it into smaller records (lines, words, etc.).
3. Applies the Map Function defined by the user.
4. Produces intermediate (key, value) pairs.
5. Passes the output to the Shuffle and Sort phase.
Function of Reduce Task:
1. Accepts grouped key-value pairs from the Shuffle and Sort phase.
2. Applies the Reduce Function to each key and its list of values.
3. Produces the final output and writes it to HDFS.
Component Role
Job Tracker Responsible for managing all MapReduce jobs. It schedules jobs,
(Master) monitors progress, and assigns tasks to Task Trackers.
Task Tracker Executes Map and Reduce tasks as assigned by the Job Tracker. Sends
(Slave) heartbeat signals and task status updates to the Job Tracker.
Combiner
A Combiner is an optional component in Hadoop MapReduce that performs local aggregation
of the Mapper’s output before sending it to the Reducer.
It is also called a “mini-reducer” because it performs the same type of operation as the
Reducer, but only on the Mapper’s local output.
It performs local aggregation on Mapper output.
It reduces the volume of intermediate data, improving performance.
In case of a huge dataset, when a MapReduce technique is applied, then the mapper phase
generates an enormous amount of key-value pairs and these intermediate results need to be
handed over to the reducer for the reduction process.
But to transfer such a large number of key-value pairs from the mapper to the reducer,
sufficiently large communication network bandwidth is required which generally leads to
network congestion.
So to avoid such congestion we can put some of the work of reducers in to the Map tasks by
using a local aggregator ( Combiner )
Example -
Consider the output when input split is passed to mapper is of following
(cat, 1), (dog, 1), (cat, 1)
IF directly send
Mapper Output from Node A:
(cat, 1), (dog, 1), (cat, 1)
→ Sent directly to Reducer
using Combiner
Combiner Output from Node A:
(cat, 2), (dog, 1)
→ Less data transferred to Reducer