UNIT 4

UNDERSTANDIN
G MAP REDUCE
FUNDAMENTALS
Prof Sushma D
Computer Science and
Engineering
MVJ COLLEGE OF
ENGINEERING
1
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 What is Map Reduce ?
• A MapReduce is a data processing tool
which is used to process the data parallelly
in a distributed form.
• It was developed in 2004, based on paper
titled as "MapReduce: Simplified Data
Processing on Large Clusters," published by
Google.
• The MapReduce is a paradigm which has
two phases, the mapper phase, and the
reducer phase. In the Mapper, the input is
given in the form of a key-value pair.
• The output of the Mapper is fed to the
reducer as input.
• The reducer runs only after the
Mapper is over. The reducer too takes
input in key-value format,
An Autonomous and the
Institution, Affiliated to VTU,output of By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
Belagavi, Approved 2 s
 Map Reduce Framework: Exploring the features
of Map Reduce
Scalability
MapReduce can process large amounts of
data by distributing tasks across many
nodes in a cluster.
Fault tolerance
MapReduce can automatically detect and
handle node failures, rerunning tasks on
available nodes.
Data locality
MapReduce processes data on the same
node where it is stored, improving
performance.
Simplicity
MapReduce An Autonomous
abstracts away many 3
Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 Map Reduce Framework: Exploring the features
of Map Reduce Contd….
Cost-effective
MapReduce can store and process large data sets economically.
Parallel processing
MapReduce breaks down large tasks and disperses them among
computers that run simultaneously
Versatile
Businesses can use MapReduce programming to access new data
sources. It makes it possible for companies to work with many
forms of data. Enterprises can access both organized and
unstructured data with this method and acquire valuable insights
from the various data sources.

4
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 Working of Map Reduce

• MapReduce is a programming model that simultaneously processes and

analyzes huge datasets logically into separate clusters.
• While Map sorts the data, Reduce segregates it into logical clusters,
thus removing ’bad’ data and retaining the necessary information.
• Example
Nutri is a well-known courier facility. It transports documents across the
globe. When the staff receives a courier, they color code is based on
the country to which it has to be sent. The dispatch staff then
segregate the courier by the tagged color code. Hence, the reception
functions as “Map”, and the dispatch team as “Reduce.”

5
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 Working of Map Reduce Contd…

• The MapReduce algorithm contains two important tasks, namely Map and
Reduce.
• The Map task takes a set of data and converts it into another set of
data, where individual elements are broken down into tuples (key-
value pairs).
• The Reduce task takes the output from the Map as an input and
combines those data tuples (key-value pairs) into a smaller set of
tuples.
• The reduce task is always performed after the map job.
• Let us now take a close look at each of the phases and try to understand their
significance.

6
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
Working of Map Reduce Contd…

7
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 Working of Map Reduce Contd…
• Input Phase − Here we have a Record Reader that translates each record
in an input file and sends the parsed data to the mapper in the form of
key-value pairs.
• Map − Map is a user-defined function, which takes a series of key-value
pairs and processes each one of them to generate zero or more key-value
pairs.
• Intermediate Keys − They key-value pairs generated by the mapper are
known as intermediate keys.
• Combiner − A combiner is a type of local Reducer that groups similar
data from the map phase into identifiable sets. It takes the intermediate
keys from the mapper as input and applies a user-defined code to
aggregate the values in a small scope of one mapper. It is not a part of the
main MapReduce algorithm; it is optional.
• Shuffle and Sort − The Reducer task starts with the Shuffle and Sort
step. It downloads the grouped key-value pairs onto the local machine,
where the Reducer is running. The individual key-value pairs are sorted by
8
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
 Working of Map Reduce Contd…

• Reducer − The Reducer takes the grouped key-value paired data as

input and runs a Reducer function on each one of them. Here, the data
can be aggregated, filtered, and combined in a number of ways, and it
requires a wide range of processing. Once the execution is over, it gives
zero or more key-value pairs to the final step.
• Output Phase − In the output phase, we have an output formatter
that translates the final key-value pairs from the Reducer function and
writes them onto a file using a record writer.

Let us try to understand the two tasks Map &f Reduce with the help of a
small diagram −

9
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) s
Working of Map Reduce Contd…

10 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
Exploring map and reduce functions
• The MapReduce functions in Hadoop are used to process large
amounts of data. The Map function splits data into smaller blocks, and
the Reduce function combines the data based on keys.
Map function
• Splits input data into smaller blocks
• Assigns each block to a mapper for processing
• Applies the map function to the local data
• Writes the output to temporary storage
Reduce function
• Combines the data tuples based on the key
• Modifies the value of the key accordingly
• Processes each group of <key,value> pairs output data
• Produces <key,value> pairs as output
11 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Exploring map and reduce functions Contd…
MapReduce- Example 1 :
Let us take a real-world example to comprehend the power of MapReduce.
Twitter receives around 500 million tweets per day, which is nearly 3000
tweets per second. The following illustration shows how Tweeter manages its
tweets with the help of MapReduce.

12 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Working of Map Reduce Contd…
As shown in the illustration, the MapReduce algorithm performs the following
actions −
• Tokenize − Tokenizes the tweets into maps of tokens and writes them as
key-value pairs.
• Filter − Filters unwanted words from the maps of tokens and writes the
filtered maps as key-value pairs.
• Count − Generates a token counter per word.
• Aggregate Counters − Prepares an aggregate of similar counter values
into small manageable units.

13 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Working of Map Reduce Contd…

MAPREDUCE EXAMPLE 2:

Welcome to Hadoop Class

Hadoop is good Hadoop is
bad

14 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Techniques to optimize MapReduce jobs
Here are some techniques to optimize MapReduce jobs:
• Data compression: Use LZO compression for transitional data
• Data locality: Store data on the same node or rack as the MapReduce task
• Memory tuning: Use as much memory as possible without triggering swapping
• Combiners: Use a combiner to perform aggregation before the data reaches the
reducer
• Shuffling and sorting: Shuffle data over the network, which can save time
• Optimized input formats: Use the most appropriate and compact writable type
for data
• luster configuration: Properly configure your cluster
• Number of MapReduce tasks: Tune the number of MapReduce tasks
• Skewed joins: Use skewed joins to handle data that is skewed, meaning a large
amount of data goes to a single reducer
• Use Hadoop profilers: Use Hadoop profilers to analyze job performance
15 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Uses of Map Reduce.
By spreading out processing across
numerous nodes and merging or
decreasing the results of those nodes,
MapReduce has the potential to handle
large data volumes. This makes it suitable
for the following use cases:
Entertainment
Hadoop MapReduce assists end users in
finding the most popular movies based on
their preferences and previous viewing
history.
Various OTT services, including Netflix,
regularly release many web series and
movies. It may have happened to you that
you couldn’t pick which movie to watch, so
you looked at Netflix’s recommendations 16 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Uses of Map Reduce Contd….
E-commerce
Several e-commerce companies, including Flipkart, Amazon, and eBay, employ
MapReduce to evaluate consumer buying patterns based on customers’ interests
or historical purchasing patterns. For various e-commerce businesses, it provides
product suggestion methods by analyzing data, purchase history, and user
interaction logs.
Social media
Nearly 500 million tweets, or about 3000 per second, are sent daily on the
microblogging platform Twitter. MapReduce processes Twitter data, performing
operations such as tokenization, filtering, counting, and aggregating counters.
• Tokenization: It creates key-value pairs from the tokenized tweets by mapping
the tweets as maps of tokens.
• Filtering: The terms that are not wanted are removed from the token maps.
• Counting: It creates a token counter for each word in the count.
• Aggregate counters: A grouping of comparable counter values is prepared
into small, manageable pieces using aggregate counters.
17 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Uses of Map Reduce Contd….
Data warehouse
Systems that handle enormous volumes of information are known as
data warehouse systems. The star schema, which consists of a fact table
and several dimension tables, is the most popular data warehouse
model. In a shared-nothing architecture, storing all the necessary data on
a single node is impossible, so retrieving data from other nodes is
essential.
Fraud detection
Conventional methods of preventing fraud are not always very effective.
For instance, data analysts typically manage inaccurate payments by
auditing a tiny sample of claims and requesting medical records from
specific submitters. Hadoop is a system well suited for handling large
volumes of data needed to create fraud detection algorithms

18 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
Controlling MapReduce Execution with InputFormat,Reading Data
with custom RecordReader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases,Developing simple MapReduce
Application
Controlling MapReduce Execution with Input Format :

19 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Controlling MapReduce Execution with Input Format :

• In Hadoop MapReduce, an InputFormat is a crucial component that defines how

input data is read and divided into splits for the Mapper phase. The InputFormat
controls how the data is passed to the Map function, thereby directly influencing the
performance and correctness of a MapReduce job. Understanding and controlling the
InputFormat allows developers to fine-tune how MapReduce processes large datasets.
• Key Components of InputFormat:
[Link] Splits:
1. InputFormat decides how data is divided into splits, and each split is processed by
a single Mapper.
2. The size of splits is important as it determines the parallelism level, as larger
splits might cause fewer mappers while smaller splits might lead to too many
mappers.
[Link]:
1. The RecordReader reads each input split and converts it into key-value pairs that
are passed to the Mapper.
2. Different types of InputFormats use different implementations of RecordReaders,
depending on how data is structured. 20 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Controlling MapReduce Execution with Input
Format :
Commonly Used InputFormats:
[Link] (default):
[Link] plain text files, where each line in the file is treated as a
record.
[Link] key is the byte offset of the line, and the value is the actual line
of text.
[Link]:
[Link] to TextInputFormat, but it expects each line to be in a key-
value pair format, typically separated by a delimiter (such as a tab
or comma).
[Link]:
[Link] for reading sequence files, which store binary key-value pairs.
This format is suitable for handling data that is not text-based.
4. Custom InputFormats:
• Developers can implement their own InputFormat to handle custom
data formats. This provides flexibility in dealing with non-standard or 21 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Controlling MapReduce Execution with Input Format :

How Input Format Controls MapReduce Execution:

• Split Size: The Input Format determines how data is split. Larger splits
may result in fewer mappers, reducing overhead but possibly underutilizing
resources. Smaller splits can increase mapper overhead but might better
utilize the cluster.
• Data Access: The format defines how the Mapper interacts with the data,
whether it's text, binary, or custom. This affects the performance and
scalability of the job.
• Resource Allocation: By influencing how data is split and processed, the
Input Format impacts the number of mappers that are launched. This, in
turn, controls resource utilization and job execution time.

22 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Reading Data with custom Record Reader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases, Developing simple MapReduce
Application
In Hadoop, MapReduce is a powerful paradigm for processing large datasets in a
distributed fashion. To ensure that a MapReduce job runs smoothly and efficiently,
it is essential to understand various components involved in the process. These
include RecordReader, RecordWriter, Partitioner, Combiner, and the
different phases of MapReduce.
Here’s a detailed explanation of each component:
1. RecordReader
The RecordReader is responsible for reading the raw input data and converting it
into key-value pairs (key/value pairs) that can be processed by the mapper. It is the
first step in the MapReduce process.
functionality: The RecordReader reads input data, such as files in HDFS, and splits
it into manageable pieces. It interprets the data (e.g., text, CSV, JSON) and
transforms it into the format that can be passed to the Mapper.

23 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Reading Data with custom Record Reader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases, Developing simple MapReduce
Application Contd….
2. RecordWriter
• The RecordWriter is responsible for writing the output data produced by the
Reducer to the output destination, typically HDFS or another file system. It
writes key-value pairs to an output file.
• Functionality: After the reducer processes data, the RecordWriter takes the
results and writes them to disk in the appropriate format. The key-value pairs
are typically stored in the output file (HDFS, local file system, etc.), but can be
customized to write to other systems, such as databases.
• Customization: You can implement a custom RecordWriter to write output
data in a specific format (e.g., JSON, custom delimited files) or store it in a
different storage system.
3. Partitioner
• A Partitioner controls how the data is distributed among the Reducers. After
the Mapper outputs key-value pairs, the Partitioner's job is to determine which
Reducer should handle which key-value pair.
24 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Reading Data with custom Record Reader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases, Developing simple MapReduce
Application.
functionality: The Partitioner's role is to ensure that the data is distributed evenly
and logically across all the reducers. For instance, it might ensure that all data with
the same key ends up in the same reducer.
Customization: If you have specific logic to distribute the data (for example, based
on some business logic or a custom hash function), you can implement a custom
Partitioner. This is useful when you need to control the flow of data between the
mapper and reducer more precisely.

4. Combiner
The Combiner is an optional component that can optimize performance by
performing partial aggregation of the map outputs before they are sent to the
reducer.
•Functionality: The Combiner is like a "mini-reducer" that runs on the mapper node.
It helps reduce the amount of data transferred from the mappers to the reducers,
improving performance by combining intermediate results locally.
Customization: You can implement a custom Combiner if the default combination
logic does not meet your requirements. For example, if you're aggregating numbers,
the default Combiner might just sum them, but a custom Combiner could apply more 25 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Reading Data with custom Record Reader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases, Developing simple MapReduce
Application.
MapReduce Phases
MapReduce jobs are divided into multiple phases. Here’s a brief overview of each
phase:
a. Map Phase
Input: The input data is read by the Record Reader and passed as key-value pairs to
the Mapper.
Processing: The Mapper processes each key-value pair and produces a set of
intermediate key-value pairs. These intermediate pairs are emitted as output from the
Mapper.
Output: The output of the Mapper is written to local storage before being passed on to
the Partitioner.
b. Shuffle and Sort Phase
Functionality: After the map phase, the framework shuffles and sorts the
intermediate data. It groups all the values associated with the same key and sends
them to the same Reducer.
Customization: You can control this phase to define how keys are sorted or grouped 26 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
Reading Data with custom Record Reader,-Reader, Writer, Combiner,
Partitioners, Map Reduce Phases, Developing simple MapReduce
Application.

c. Reduce Phase
• Input: The sorted key-value pairs from the shuffle and sort phase are
passed to the Reducer. Each Reducer receives a key and a list of values
associated with that key.
• Processing: The Reducer processes the grouped data and performs
the aggregation or transformation logic (e.g., summing values, joining
data).
• Output: After processing, the output is written to disk using the Record
Writer.

27 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 Developing simple MapReduce Application

What is MapReduce?
In MapReduce, we break the job into two main steps:
Map: Split the text into words and associate each word with the number 1
(because each time we see a word; we count it as 1).
Reduce: Group all the words together and add the 1 s for each word to get the
total count.

Simple Steps in the Application:

Mapper: It takes each line of text, splits it into words, and emits the word along
with the number 1
Reducer: It takes the list of words with the number 1, groups them by word,
and adds up the 1s for each word to count the total occurrences.

28 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
Developing a MapReduce application involves several steps:
• 1. Define the problem and identify the data: Identify the problem you
want to solve with MapReduce and determine the input data and its
format.
• 2. Design the MapReduce algorithm: Determine the map function and
reduce function, including the input and output key-value pairs.
• 3. Write the Mapper class: Create a Java class that extends the Mapper
class and implements the map method.
• 4. Write the Reducer class: Create a Java class that extends the
Reducer class and implements the reduce method.
• 5. Write the Driver class: Create a Java class that configures and
submits the MapReduce job to the Hadoop cluster.6. Compile and run
the MapReduce job: Compile the Java code and create a JAR file, then
run the MapReduce job using the Hadoop command-line tool or an IDE.

29 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
• Here's an example of a simple MapReduce application that counts the
frequency of each word in a text file:

30 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 31 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
This example demonstrates the basic structure of a MapReduce application, including the
Mapper, Reducer, and Driver classes.

32 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)
 THANK YOU

33 s
An Autonomous Institution, Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B)