Scribd
Upload
4 views4 pages

Understanding MapReduce Components

MapReduce is a distributed data processing framework in Hadoop that processes large datasets through two main phases: Map and Reduce, with intermediate steps like Shuffling and Sorting. It consists of several components including InputFormat, RecordReader, Mapper, Combiner, Partitioner, Shuffle & Sort, Reducer, OutputFormat, and RecordWriter, each serving specific roles in data processing. This framework enhances scalability and efficiency in handling Big Data applications.

Uploaded by

Dipesh Patel
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views4 pages

Understanding MapReduce Components

MapReduce is a distributed data processing framework in Hadoop that processes large datasets through two main phases: Map and Reduce, with intermediate steps like Shuffling and Sorting. It consists of several components including InputFormat, RecordReader, Mapper, Combiner, Partitioner, Shuffle & Sort, Reducer, OutputFormat, and RecordWriter, each serving specific roles in data processing. This framework enhances scalability and efficiency in handling Big Data applications.

Uploaded by

Dipesh Patel
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

MapReduce is a distributed data processing framework used in Hadoop to process large datasets

across a cluster of machines. It works in two main phases: Map and Reduce, with intermediate steps
like Shuffling and Sorting.

1. Components of MapReduce
MapReduce consists of the following main components:

(A) InputFormat

 Purpose: Defines how input data is split and read.


 Key Components:
o FileInputFormat (default for text files)
o TextInputFormat (reads lines of a file as key-value pairs)
o KeyValueTextInputFormat (splits input data based on a delimiter)
o SequenceFileInputFormat (for binary data)

(B) RecordReader

 Purpose: Converts input data into key-value pairs for the Mapper.
 How it Works: Reads one record at a time and passes it to the Mapper.

Example:
If we have an input file containing:

A RecordReader processes it as:

where 0 and 14 are byte offsets.

(C) Mapper

 Purpose: Processes key-value pairs and produces intermediate key-value pairs.


 Key Function:
o Takes input split → Processes data → Generates intermediate output.

Example (WordCount Mapper):


Input:
Mapper Output:

(D) Combiner (Optional)

 Purpose: Performs local aggregation before sending data to Reducer.


 Benefit: Reduces network traffic by combining local Mapper outputs.

Example:

Without Combiner:

With Combiner:

(E) Partitioner

 Purpose: Distributes intermediate key-value pairs among Reducers.


 Default Behavior: Uses hash-based partitioning
 Example:
o If numReducers = 2, then:
 Keys starting with A-M go to Reducer 1.
 Keys starting with N-Z go to Reducer 2.

(F) Shuffle & Sort


 Purpose: Sorts and groups all values associated with the same key before sending them to
Reducers.
 Process:
1. Shuffle: Transfers output from Mappers to Reducers.
2. Sort: Groups values based on keys.

Example:
Mapper Output:

After Shuffle & Sort:

(G) Reducer

 Purpose: Aggregates values of the same key.


 Function:
o Input: (key, [list of values])
o Output: (key, aggregated result)

Example (WordCount Reducer):


Input:

Reducer Output:
(H) OutputFormat

 Purpose: Defines how the final output is stored.


 Types:
o TextOutputFormat (default) – Saves output as text files.
o SequenceFileOutputFormat – Stores binary output.

(I) RecordWriter

 Purpose: Writes the Reducer's output to the final storage (HDFS).


 Example Output File (part-r-00000 in HDFS):

MapReduce is a powerful framework for parallel processing of large-scale data. It works by breaking
down tasks into multiple Mappers and Reducers, ensuring scalability and efficiency in Big Data
applications.

You might also like