Unit 3: MapReduce Programming

Guide

1. Introduction to the MapReduce Paradigm

MapReduce is a programming model and an associated implementation for processing
and generating large datasets with a parallel, distributed algorithm on a cluster. It is the
core processing framework of Hadoop, designed to handle massive volumes of data
(petabytes) reliably and efficiently.

Analogy: Counting Votes in a National Election

Imagine a national election where millions of ballots need to be counted:

1.​ Input Data (Ballots): Millions of physical ballots arrive at thousands of local
counting centers. This is the raw data.
2.​ The Map Phase (Local Counting): Each local center (Mapper) is responsible for
a small subset of the ballots. They don't count the national total; they only count
the votes for each candidate within their local precinct. The output is a list of
<Candidate Name, 1> for every single vote, or more efficiently, a list like
<Candidate A, 450> and <Candidate B, 320> for their local totals. This
step filters and creates key-value pairs.
3.​ Shuffle & Sort (Transport and Organization): All the local results are gathered
and sent to designated regional centers, grouped by the candidate name. All
votes for Candidate A go to one set of regional centers, all votes for Candidate
B go to another, and so on.
4.​ The Reduce Phase (National Aggregation): The regional centers (Reducer)
receive all the local counts for only one specific candidate. They simply
aggregate (sum) all these local counts to determine the final national total for
that candidate. This step aggregates and summarizes.
5.​ Output: The final, reliable result: the total number of votes for every candidate.

This entire process is MapReduce: Divide and Conquer on a grand scale.

2. Topics to Cover: The Flow of Data

The MapReduce job flow can be broken down into five distinct phases: Input, Map,
Shuffle, Reduce, and Output. The following components are essential to this process:
The Mapper (Filtering and Key-Value Creation)
The Mapper is the first phase of the data processing. It processes an input record and
transforms it into a set of intermediate key-value pairs.

●​ Input: The Mapper receives data as <KeyIn, ValueIn>. In text processing,

KeyIn is often the byte offset of the line in the file, and ValueIn is the actual
line of text.
●​ Action: The user-defined map() function processes the input and emits zero,
one, or multiple <KeyOut, ValueOut> pairs.
●​ Role: The Mapper's primary job is filtering relevant data and preparing it into a
standardized key-value format that the Reducer can understand and aggregate.

The Reducer (Aggregation and Summarizing)

The Reducer aggregates the intermediate values associated with the same key
generated by the Mappers.

●​ Input: The Reducer receives data as <KeyOut, List<ValueOut>>. All values

for the same key are grouped together.
●​ Action: The user-defined reduce() function iterates over the list of values for
a given key, performs an operation (e.g., sum, average, count), and emits the
final result as <KeyFinal, ValueFinal>.
●​ Role: The Reducer's primary job is aggregation and summarization to produce
the final, desired output.

The Driver Code (The Main Configuration)

The Driver code is the main() method that configures and submits the MapReduce job
to the Hadoop cluster.

●​ Configuration: Sets the input/output paths, the Mapper class, the Reducer
class, and the data types for the intermediate and final key-value pairs.
●​ Job Submission: Creates a Job object and calls the submit() or
waitForCompletion() method to execute the job on the cluster.
●​ Hadoop API: Developers must choose between the older
[Link] API or the modern
[Link] API for writing the Driver, Mapper, and
Reducer classes.
 Combiner & Partitioner (Optimization Steps)

Component Role Goal

Combiner An optional, local Reducer that To reduce network traffic by

runs on the Mapper output performing local aggregation. It
before the Shuffle phase. minimizes the data transferred
from Mappers to Reducers.

Partitioner Controls how the intermediate To ensure that all intermediate

keys from the Mapper are values for the same key go to
distributed to the Reducers. the same Reducer. By default,
this is a hash function of the
key modulo the number of
reducers.

Example: Walk-through with the Weather Dataset

The "Weather Dataset" is a classic MapReduce example, often used to find the
maximum or minimum temperature recorded in a year at various weather stations.

Goal: Find the maximum temperature recorded for each year.

Phase Component Key/Value Pairs Conceptual Action

Input Record Reader <LongWritable, Text> Reads raw data (e.g.,

1901-01-01,Station_A,15.
2).

Map Mapper <Text, IntWritable> Key: Year (e.g., 1901). Value:

Temperature (e.g., 15). The
Mapper filters out
non-temperature readings.

Combine (Optional) Combiner <Text, IntWritable> For a given year, finds the local
maximum temperature
recorded by that specific
Mapper.

Shuffle/Sort System <Text, Groups all temperature values

List<IntWritable>> for the same year and sends
them to the correct Reducer.
Phase Component Key/Value Pairs Conceptual Action

Reduce Reducer <Text, IntWritable> Key: Year (1901). List: [10,

15, 8, 12]. The Reducer
iterates through the list to find
the overall maximum value.

Output System <Text, IntWritable> Final result (e.g., 1901, 15).

3. Question Bank

Multiple Choice Questions (MCQs)

1.​ What is the primary role of the Map phase in the MapReduce paradigm?​
a) Aggregation and summarization​
b) Filtering and transforming input data into key-value pairs​
c) Configuring the job and specifying input/output paths​
d) Grouping intermediate values by key
2.​ Which component is responsible for ensuring that all values for a specific key
are sent to the same Reducer?​
a) Combiner​
b) Record Reader​
c) Partitioner​
d) Driver Code
3.​ The Combiner is a local optimization step that acts like a mini-Reducer. Where
does it run?​
a) On the NameNode​
b) On the Reducer task tracker​
c) On the Mapper task tracker​
d) On a dedicated Combiner machine
4.​ In the context of the Hadoop API, what is the data type typically used for the
key in the input of the Mapper when reading plain text files?​
a) Text​
b) LongWritable (representing the byte offset)​
c) IntWritable​
d) NullWritable
5.​ Which component sets the input format, output format, Mapper class, and
Reducer class for a MapReduce job?​
a) The Reducer​
b) The Combiner​
c) The Partitioner​
d) The Driver Code
 6.​ If a MapReduce job produces 10 million intermediate key-value pairs, and the
Combiner reduces this to 2 million pairs, what is the primary benefit?​
a) Increased parallelism on the Mapper side​
b) Reduced disk I/O on the Reducer side​
c) Reduced network bandwidth usage during Shuffle​
d) Simpler Reducer logic
7.​ The final output of a standard MapReduce job is written by which component?​
a) Combiner​
b) Mapper​
c) Reducer​
d) Driver Code
8.​ Which Hadoop API package is considered the modern, preferred one for writing
MapReduce programs?​
a) [Link]​
b) [Link]​
c) [Link]​
d) [Link]
9.​ What is the primary purpose of the RecordReader?​
a) To manage job configuration​
b) To generate the final output file​
c) To convert the raw input bytes into key-value records for the Mapper​
d) To perform the final aggregation
10.​Which statement about the Combiner is FALSE?​
a) It must have the same input/output types as the Reducer.​
b) It is optional and used for optimization.​
c) Its use can significantly speed up job execution.​
d) It runs after the Mapper but before the Shuffle phase.

Question Answer

1 b

2 c

3 c

4 b

5 d

6 c

7 c
Question Answer

8 b

9 c

10 a

Code-Logic Questions
1.​ Code-Logic: Word Count Reducer​
A Mapper emits <"apple", 1> for every occurrence of "apple". A Reducer
receives the following input for the key "apple": <"apple", [1, 1, 1, 1,
1]>. Write the Java-like pseudocode for the reduce() method that calculates
the total count for the word "apple".// Assume KeyIn is Text and ValueIn is
IntWritable

public void reduce(KeyIn key, Iterable<ValueIn> values, Context context) {

// Your Code Logic Here

Solution:public void reduce(Text key, Iterable<IntWritable> values, Context

context) {

int sum = 0;

for (IntWritable value : values) {

sum += [Link]();

[Link](key, new IntWritable(sum));

2.​ Code-Logic: Partitioner Logic​

You are writing a MapReduce job to analyze patient records. You want all
records for male patients to go to Reducer 0 and all records for female patients
to go to Reducer 1. If there are exactly two reducers, write the logic for the
getPartition() method. The key is the patient ID, and the value contains the
 gender [Link] int getPartition(Key key, Value value, int
numReduceTasks) {

// Assume the Gender is available via a hypothetical [Link]() method

// numReduceTasks is guaranteed to be 2

// Your Code Logic Here

Solution:public int getPartition(PatientId key, PatientRecord value, int

numReduceTasks) {

if ([Link]().equals("MALE")) {

return 0; // Send all male records to the first reducer

} else if ([Link]().equals("FEMALE")) {

return 1; // Send all female records to the second reducer

return 0; // Default case for safety

3.​ Code-Logic: Maximum Temperature Mapper​

For the Weather Dataset example, the input line (ValueIn) is a comma-separated
string: YYYY-MM-DD,StationID,Temperature(C). Write the Java-like
pseudocode for the map() method to extract the year and the temperature, and
emit them as the intermediate key-value pair.// Assume KeyIn is LongWritable
and ValueIn is Text

public void map(KeyIn key, ValueIn value, Context context) {

// Input: 1901-01-01,Station_A,15.2

// KeyOut: 1901, ValueOut: 15

// Your Code Logic Here

}
Solution:public void map(LongWritable key, Text value, Context context) {

String line = [Link]();

String[] parts = [Link](",");

if ([Link] >= 3) {

String date = parts[0];

String tempStr = parts[2];

// Extract the year (first 4 characters of the date)

String year = [Link](0, 4);

int temperature = (int) [Link](tempStr); // Assuming temperature is

stored as a String

[Link](new Text(year), new IntWritable(temperature));