UNIT III

MAP REDUCE APPLICATIONS

MapReduce workflows – unit tests with
MRUnit – test data and local tests – anatomy
of MapReduce job run – classic Map-reduce –
YARN – failures in classic Map-reduce and
YARN – job scheduling – shuffle and sort – task
execution – MapReduce types – input formats
– output formats.
 3.1 MAPREDUCE WORKFLOWS
• MapReduce is a programming model and data processing
technique that was popularized by Google and is widely used
for processing and generating large datasets that can be
parallelized across a distributed cluster of computers.

• MapReduce workflows consist of two main phases: the "Map"

phase and the "Reduce" phase.

• These phases are typically orchestrated by a framework like

Apache Hadoop.
1. Input Data
2. Map Phase
3. Shuffle and Sort Phase
4. Reduce Phase
5. Final Output
 3.1 MAPREDUCE WORKFLOWS
1. Input Data: The first step in a MapReduce workflow is to have a
large dataset that you want to process. This dataset is typically
stored in a distributed file system like Hadoop Distributed File
System (HDFS).

2. Map Phase:
• - Map Function: In this phase, the input dataset is divided into
smaller chunks called "splits," and each split is processed by a
separate Mapper task. A user-defined "Map" function is applied
to each record within the split. The Map function takes the input
data and generates a set of key-value pairs as intermediate
outputs. These key-value pairs are not sorted.
• - Shuffling and Sorting: After the Map phase, all the
intermediate key-value pairs from all Mappers are grouped by
key and sorted. This is necessary because the Reduce phase
 3.1 MAPREDUCE WORKFLOWS
3. Shuffle and Sort Phase:
• - In this phase, the framework sorts and shuffles the intermediate key-
value pairs generated by the Mapper tasks. The purpose is to group all
values associated with the same key together, which allows the Reducer
tasks to process the data efficiently.

4. Reduce Phase:
• - Reduce Function: The Reducer tasks take the sorted and shuffled key-
value pairs from the Map phase and apply a user-defined "Reduce"
function to process and aggregate the data. Each Reducer receives a group
of key-value pairs with the same key. The Reduce function can perform
various operations on the values, such as summarization, aggregation, or
filtering.
• - Output: The output of the Reducer tasks is typically written to an
output location, often in a distributed file system, where it can be used for
further analysis or as the final result of the MapReduce job.
 MAP REDUCE
• What is MapReduce?
• A MapReduce is a data processing tool which is used to
process the data parallelly in a distributed form. It was
developed in 2004, on the basis of paper titled as
"MapReduce: Simplified Data Processing on Large Clusters,"
published by Google.
• The MapReduce is a paradigm which has two phases, the
mapper phase, and the reducer phase. In the Mapper, the
input is given in the form of a key-value pair. The output of the
Mapper is fed to the reducer as input. The reducer runs only
after the Mapper is over. The reducer too takes input in key-
value format, and the output of reducer is the final output.
• MapReduce is a framework using which we can write
applications to process huge amounts of data, in parallel, on
large clusters of commodity hardware in a reliable manner.

• What is MapReduce?
• MapReduce is a processing technique and a program model for
distributed computing based on java.
• The MapReduce algorithm contains two important tasks,
namely Map and Reduce.
• Map takes a set of data and converts it into another set of data,
where individual elements are broken down into tuples
(key/value pairs).
• Secondly, reduce task, which takes the output from a map as an
input and combines those data tuples into a smaller set of tuples.
• As the sequence of the name MapReduce implies, the reduce
task is always performed after the map job.
 • Terminology
• PayLoad − Applications implement the Map and the Reduce functions,
and form the core of the job.
• Mapper − Mapper maps the input key/value pairs to a set of
intermediate key/value pair.
• NamedNode − Node that manages the Hadoop Distributed File System
(HDFS).
• DataNode − Node where data is presented in advance before any
processing takes place.
• MasterNode − Node where JobTracker runs and which accepts job
requests from clients.
• SlaveNode − Node where Map and Reduce program runs.
• JobTracker − Schedules jobs and tracks the assign jobs to Task
tracker.
• Task Tracker − Tracks the task and reports status to JobTracker.
• Job − A program is an execution of a Mapper and Reducer across a
dataset.
• Task − An execution of a Mapper or a Reducer on a slice of data.
• Task Attempt − A particular instance of an attempt to execute a task on
 Steps in Map Reduce

• The map takes data in the form of pairs and returns a list of
<key, value> pairs. The keys will not be unique in this case.
• Using the output of Map, sort and shuffle are applied by the
Hadoop architecture. This sort and shuffle acts on these list of
<key, value> pairs and sends out unique keys and a list of
values associated with this unique key <key, list(values)>.
• An output of sort and shuffle sent to the reducer phase. The
reducer performs a defined function on a list of values for
unique keys, and Final output <key, value> will be
stored/displayed.
 3.2 UNIT TESTS WITH MRUNIT
• MRUnit is a Java library that helps you write unit tests for your
Apache Hadoop MapReduce jobs.
• Unit testing your MapReduce code is crucial to ensure that
your data processing logic works as expected before
deploying it to a Hadoop cluster.
• MRUnit provides a framework for creating and running unit
tests for your Mapper and Reducer classes.
• Here's how you can write unit tests with MRUnit:
1. Set up your project
2. Write your unit test
3. Run the unit test
 ```java
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];

public class MyMapperTest {

private MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;

@Before
public void setUp() {
MyMapper mapper = new MyMapper(); // Replace with your Mapper class
mapDriver = [Link](mapper);
}

@Test
public void testMapper() {
[Link](new LongWritable(1), new Text("Hello, world!"))
.withOutput(new Text("Hello,"), new IntWritable(1))
.withOutput(new Text("world!"), new IntWritable(1))
.runTest();
}
}
 3.3 TEST DATA AND LOCAL TESTS

• MRUnit is a Java library that helps you write unit tests for your
Apache Hadoop MapReduce jobs.
• Unit testing your MapReduce code is crucial to ensure that
your data processing logic works as expected before
deploying it to a Hadoop cluster.
• MRUnit provides a framework for creating and running unit
tests for your Mapper and Reducer classes.
• Here's how you can write unit tests with MRUnit:
1. Set up your project
2. Write your unit test
3. Run the unit test
1. Create Test Data:

1.1. Generate Sample Input Data: You can manually

create small sample input data in a text file or use tools
to generate synthetic data for testing purposes. Ensure
that your test data covers various scenarios, including
edge cases and potential issues that your MapReduce job
might encounter.

1.2. Store Test Data in a Local Directory: Place your

sample input data in a directory on your local machine.
This directory will serve as the input source for your local
tests.
2. Write Unit Tests:

• 2.1. Set Up a Testing Framework: Use a testing framework like JUnit,

TestNG, or a Hadoop-specific testing framework like MRUnit to write
unit tests for your MapReduce job. Ensure that your project is
properly configured to use the testing framework.
• 2.2. Write Test Cases: Create test cases that cover various aspects of
your MapReduce job, including Mapper and Reducer functionality,
handling of different input data scenarios, and edge cases. Your test
cases should include both positive and negative test scenarios.
• 2.3. Configure Test Inputs and Outputs: In your test cases, specify
the test input data, including the directory or file containing your
sample input data. Define the expected output for each test case.
• 2.4. Invoke MapReduce Job: In your test cases, invoke the
MapReduce job by creating an instance of your Mapper and Reducer
classes and running them with the test input data. Capture the
output.
• 2.5. Assertions: Use assertions to compare the actual output of your
MapReduce job with the expected output defined in your test cases.
Ensure that the results match your expectations.
3. Run Local Tests:

• 3.1. Execute Unit Tests: Run your unit tests using the
testing framework you've set up. The framework will
execute your MapReduce job locally using the
sample input data and validate the results.
• 3.2. Interpret Test Results: Review the test results to
identify any failures or issues. The test framework
will report which test cases passed and which ones
failed.
• 3.3. Debug and Refine: If any test cases fail, use the
debugging tools provided by your development
environment to diagnose and fix the issues in your
MapReduce code. Refine your code and tests
iteratively until all test cases pass successfully.
4. Repeat for Different Test Scenarios:
• 4.1. Edge Cases: Make sure to test edge cases, such as
empty input data, data with missing fields, or data that
could potentially cause exceptions in your MapReduce
code.
• 4.2. Performance Testing (Optional): If your MapReduce
job is expected to process large volumes of data, you
may want to test its performance with larger datasets.
You can generate larger test data sets for such scenarios.

• By creating test data and running local tests, you can

catch and fix issues early in the development process,
which helps ensure the correctness and reliability of
your MapReduce jobs before deploying them to a
Hadoop cluster for processing large datasets.
3.4 ANATOMY OF MAPREDUCE JOB RUN

1. Input Data
2. Mapper Phase
3. Shuffling and Sorting
4. Reducer Phase
5. Output Data
6. Job Configuration
7. Job Submission
8. Task Execution
9. Job Monitoring and Logging
10. Job Completion
11. Cleanup
12. Results Analysis
3.5 CLASSIC MAP-REDUCE
• This classic MapReduce workflow is the
foundation for distributed data processing in
frameworks like Hadoop.
• "Classic MapReduce" refers to the original
programming model and framework for
processing and generating large datasets that
can be parallelized across a distributed cluster
of computers.
• Here are the key concepts and components
of the classic MapReduce model:
1. Mapper
2. Reducer
3. Input Data
4. Output Data
5. Shuffling and Sorting
6. Job Configuration
7. Job Submission
8. Task Execution
9. Job Monitoring and Logging
10. Job Completion
3.6 YARN (Yet Another Resource Negotiator)
• YARN is a resource management and job scheduling
component in the Hadoop ecosystem.
• It is designed to separate the resource management
layer from the processing framework, providing a
more flexible and scalable resource management
platform.
• YARN replaced the older MapReduce JobTracker and
TaskTracker architecture, allowing Hadoop to support
various processing frameworks beyond MapReduce.
3.6 YARN

Key components of YARN include:

1. ResourceManager (RM): The ResourceManager is the central resource
allocation and management component in YARN. It manages cluster
resources and schedules applications.

2. NodeManager (NM): NodeManagers run on individual nodes in the

cluster and are responsible for monitoring resource usage and reporting it
back to the ResourceManager.

3. ApplicationMaster (AM): Each application running on the cluster has its

own ApplicationMaster. The ApplicationMaster is responsible for
negotiating resources with the ResourceManager and monitoring the
execution of its application's tasks.

4. Container: A container represents a set of allocated resources (CPU,

memory, etc.) on a cluster node for running a specific task or component of
an application.
3.7 FAILURES IN CLASSIC MAP-REDUCE AND YARN
Failures can occur in both classic MapReduce and YARN due to various reasons.
Handling failures robustly is a critical aspect of distributed computing.

Failures in Classic MapReduce:

1. Task Failures: Individual Mapper or Reducer tasks can fail due to
hardware issues, software errors, or other factors. Task failures are
relatively common.

2. JobTracker Failures: In the classic MapReduce model, the JobTracker

is a single point of failure. If the JobTracker fails, it can result in job
failures or delays.

3. Data Node Failures: If a Data Node hosting HDFS data blocks fails, it
can lead to data unavailability for MapReduce jobs.

4. Network Failures: Network issues can disrupt communication

between nodes and cause job failures or delays.
3.8 YARN – JOB SCHEDULING

• In the context of Hadoop, YARN (Yet Another Resource

Negotiator) is a resource management and job
scheduling component that is responsible for allocating
resources and managing the execution of jobs on a
Hadoop cluster.
• YARN plays a crucial role in the Hadoop ecosystem,
allowing various distributed data processing
frameworks like MapReduce, Apache Spark, and
Apache Flink to efficiently utilize cluster resources.
• Job scheduling in Hadoop typically involves two main
phases:
– Shuffle Phase
– Sort Phase
3.8 YARN – JOB SCHEDULING
Shuffle Phase:
- The shuffle phase occurs after the map phase in a
MapReduce job.
- During the map phase, data is read, processed, and
intermediate key-value pairs are generated.
- - In the shuffle phase, the intermediate data generated
by the map tasks needs to be redistributed and grouped
by keys so that it can be processed by the reduce tasks.
- - The shuffle phase involves data transfer between
nodes in the cluster, which can be a resource-intensive
operation.
- YARN ensures that there are enough resources available
for shuffle operations and that data transfer is efficient.
3.8 YARN – JOB SCHEDULING
2. Sort Phase
- After the shuffle phase, the grouped data (key-value pairs) is sorted by
key to prepare it for processing by the reduce tasks.
- Sorting is necessary because each reduce task receives a sorted subset
of data, making it easier to perform aggregation or computations.
- YARN does not directly control the sorting process, but it does allocate
resources to the reduce tasks that perform the sorting and processing of
the data.

In summary, YARN doesn't handle the shuffle and sort phases directly, but
it plays a critical role in ensuring that there are sufficient resources
available for these phases to run efficiently.
The shuffle and sort operations are primarily managed by the MapReduce
framework itself or other data processing frameworks running on top of
YARN, such as Apache Spark or Apache Flink.
These frameworks implement their own mechanisms for data shuffling
and sorting as part of their job execution process.
[Link]
The MapReduce algorithm contains two important tasks, namely
Map and Reduce.

•The map task is done by means of Mapper Class

•The reduce task is done by means of Reducer Class.
EXAMPLE
Let us assume we have employee data in four different files − A,
B, C, and D.

Let us also assume there are duplicate employee records in

all four files because of importing the employee data from
all database tables repeatedly.
• The Map phase processes each input file and
provides the employee data in key-value pairs (<k,
v> : <emp name, salary>). See the following
illustration.
• The combiner phase (searching technique) will
accept the input from the Map phase as a key-value
pair with employee name and salary.
• Using searching technique, the combiner will check
all the employee salary to find the highest salaried
employee in each file.
<k: employee name, v: salary>
Max= the salary of an first employee. Treated as max salary

if(v(second employee).salary > Max){

Max = v(salary);}
else
{
Continue checking;
}
• The expected result is as follows −
<satish, 26000>
<gopal, 50000>
<kiran, 45000>
<manisha, 45000>

The final output should be as follows −

<gopal, 50000>
Key components of YARN include:

1. ResourceManager (RM): The ResourceManager is the central resource

allocation and management component in YARN. It manages cluster
resources and schedules applications.

2. NodeManager (NM): NodeManagers run on individual nodes in the cluster

and are responsible for monitoring resource usage and reporting it back to
the ResourceManager.

3. ApplicationMaster (AM): Each application running on the cluster has its

own ApplicationMaster. The ApplicationMaster is responsible for negotiating
resources with the ResourceManager and monitoring the execution of its
application's tasks.

4. Container: A container represents a set of allocated resources (CPU,

memory, etc.) on a cluster node for running a specific task or component of
an application.
3.10 TASK EXECUTION
Task execution is a fundamental aspect of the MapReduce framework, where
the actual data processing takes place. In a MapReduce job, tasks are
executed in parallel across a cluster of nodes.

There are two main types of tasks in a MapReduce job:

1. Mapper Tasks: These tasks are responsible for processing a portion of the
input data. Mappers apply a user-defined function to each record within their
assigned data split. The output of the Mapper is a set of intermediate key-
value pairs.

2. Reducer Tasks: Reducer tasks take the intermediate key-value pairs

generated by the Mappers and process them. Each Reducer receives a group
of key-value pairs with the same key, and it applies a user-defined Reduce
function to these pairs. The output of the Reducer is the final result of the
MapReduce job.
Key points about task execution in MapReduce:

- Task execution is parallelized, with multiple Mapper and

Reducer tasks running concurrently on different cluster nodes.
- Tasks are assigned to nodes based on data locality whenever
possible, which reduces data transfer overhead.
- Task progress and status are monitored by the
ResourceManager (in YARN) or the JobTracker (in classic
MapReduce).
- In the event of task failures, the ResourceManager or
JobTracker can reschedule tasks on available nodes.

Efficient task execution is critical for achieving good performance

in MapReduce jobs, as it involves the actual processing of data.
3.11 MAPREDUCE TYPE
• MapReduce jobs can be categorized into several types
based on their use cases and data processing requirements.
• Here are some common types of MapReduce jobs:

1. Batch Processing
2. Iterative Processing
3. Real-time Processing (Streaming)
4. Graph Processing
5. Data Joining
6. Search and Indexing
7. Log Processing
3.12 INPUT FORMATS
Input formats in Hadoop MapReduce specify how input data is read and
processed by the MapReduce job. Hadoop supports various input formats
to handle different types of data sources and formats efficiently. Here are
some common input formats:

1. TextInputFormat (Default): This is the default input format for text-based

data. Each line in the input file is treated as a separate record, and the key
is the byte offset of the line, while the value is the content of the line.

2. KeyValueTextInputFormat: This format is used when input data is in the

form of key-value pairs separated by a delimiter. It allows you to specify a
delimiter character to split key and value pairs.

3. SequenceFileInputFormat: Sequence files are a binary format that stores

key-value pairs. This input format is efficient for storing and processing
large datasets.
3.12 INPUT FORMATS
3.13 OUTPUT FORMATS
• Output formats in Hadoop MapReduce specify how the results of a
MapReduce job are written to the distributed file system or external
storage.

• 1. TextOutputFormat (Default): This is the default output format for

text-based data. Each key-value pair is written as a line in the output file.
• 2. SequenceFileOutputFormat: Sequence files can be used as an output
format, especially when you want to store key-value pairs in a binary
format for efficient storage and subsequent processing.
• 3. Avro, Parquet, and ORC Output Formats: These formats allow you to
write data in Avro, Parquet, or ORC formats, which are optimized for
efficient storage and query performance.
• 4. MultipleOutputFormat: This format allows you to write output to
multiple files or directories based on specific criteria, such as keys or
other attributes. It's useful for scenarios where you need to partition or
distribute the output data.