Big Data Systems

By
[Link]
Course Code: DS307
Module-2
 Module-2 contents

What is Hadoop Cluster?

Pseudo Distributed mode,
Type of clusters,
Hadoop Ecosystem,
Pig, Hive, Oozie, Flume, SQOOP.
Overview of MapReduce Framework,
MapReduce Architecture,
Learn about Jobtracker and Task tracker,
Use cases of MapReduce,
Anatomy of MapReduceProgram.
A case study of tuning
MapReduce for efficient Bioinformatics in the cloud
 Hadoop Cluster
Cluster:
• Cluster is a collection of something, a simple computer cluster is a group of various computers
that are connected with each other through LAN(Local Area Network).
• The nodes in a cluster share the data, work on the same task and this nodes are good enough to
work as a single unit means all of them to work together.

Hadoop cluster:
• Hadoop Cluster is also a collection of various commodity hardware(devices that are inexpensive
and amply available).
• This Hardware components work together as a single unit.
• In the Hadoop cluster, there are lots of nodes (can be computer and servers) contains Master
and Slaves, the Name node and Resource Manager works as Master and data node, and Node
Manager works as a Slave.
• The purpose of Master nodes is to guide the slave nodes in a single Hadoop cluster.
• We design Hadoop clusters for storing, analyzing, understanding, and for finding the facts that
are hidden behind the data or datasets which contain some crucial information.
• The Hadoop cluster stores different types of data and processes them.
Structured-Data: The data which is well structured like Mysql.
Semi-Structured Data: The data which has the structure but not the data type like XML, Json
(Javascript object notation).
Unstructured Data: The data that doesn’t have any structure like audio, video.
 Hadoop Cluster
Hadoop cluster schema:
 Hadoop Cluster
Hadoop cluster properties:
1. Scalability:
• Hadoop clusters are very much capable of scaling-up and scaling-
down the number of nodes i.e. servers or commodity hardware.
Let’s see with an example of what actually this scalable property
means.
• Suppose an organization wants to analyze or maintain around
5PB of data for the upcoming 2 months so he used 10
nodes(servers) in his Hadoop cluster to maintain all of this data.
• But now what happens is, in between this month the organization
has received extra data of 2PB, in that case, the organization has
to set up or upgrade the number of servers in his Hadoop cluster
system from 10 to 12(let’s consider) in order to maintain it.
• The process of scaling up or scaling down the number of servers
in the Hadoop cluster is called scalability.

2. Flexibility:
• This is one of the important properties that a Hadoop cluster
possesses.
• According to this property, the Hadoop cluster is very much
Flexible means they can handle any type of data irrespective of its
type and structure.
• With the help of this property, Hadoop can process any type of
data from online web platforms.
 Types of Hadoop clusters
1. Single Node Hadoop Cluster
• In Single Node Hadoop Cluster as the name suggests the
cluster is of an only single node which means all our
Hadoop Daemons i.e. Name Node, Data Node, Secondary
Name Node, Resource Manager, Node Manager will run on
the same system or on the same machine.
• It also means that all of our processes will be handled by
only single JVM(Java Virtual Machine) Process Instance.

2. Multiple Node Hadoop Cluster:

• In multiple node Hadoop clusters as the name suggests it

contains multiple nodes.
• In this kind of cluster set up all of our Hadoop Daemons,
will store in different-different nodes in the same cluster
setup.
• In general, in multiple node Hadoop cluster setup we try to
utilize our higher processing nodes for Master i.e. Name
node and Resource Manager and we utilize the cheaper
system for the slave Daemon’s [Link] Manager and Data
Node
 Types of Hadoop clusters
3. High Availability cluster:

• A special kind of multi-node cluster known as a high availability cluster offers high availability
for the NameNode.
• There are two NameNodes in a high availability cluster, one of which is active and the other on
standby.
• The backup NameNode takes over automatically if the active NameNode fails.
 Types of Hadoop clusters
4. Hybrid cluster:

• A hybrid cluster combines Hadoop with additional tools like Apache Spark and Apache Storm.
• In a hybrid cluster, Spark and Storm are utilized for real-time processing, while Hadoop is used
to store and analyze massive data volumes.
 Types of Hadoop clusters
5. Cloud-based cluster:

• A Hadoop cluster that is hosted in the cloud is referred to as cloud-based.

• Since they are simple to set up, inexpensive, and scalable, based on demand, cloud-based
clusters popularity is increasing in the Hadoop domain.
 Hadoop – Different Modes of Operation
• Hadoop is an open-source framework which is mainly used for storage purpose and maintaining
and analyzing a large amount of data or datasets on the clusters of commodity hardware, which
means it is actually a data management tool.
• Hadoop also posses a scale-out storage property, which means that we can scale up or scale
down the number of nodes as per are a requirement in the future which is really a cool feature.
• Hadoop Mainly works on 3 different Modes:

• Standalone Mode
• Pseudo-distributed Mode
• Fully-Distributed Mode
 Hadoop – Different Modes of Operation
1. Standalone Mode

• In Standalone Mode none of the Daemon will run i.e. Namenode, Datanode, Secondary Name
node, Job Tracker, and Task Tracker.
• Standalone Mode also means that we are installing Hadoop only in a single system.
• By default, Hadoop is made to run in this Standalone Mode or we can also call it as the Local
mode.
• We mainly use Hadoop in this Mode for the Purpose of Learning, testing, and debugging.
• Hadoop works very much Fastest in this mode among all of these 3 modes.
• As we all know HDFS (Hadoop distributed file system) is one of the major components for
Hadoop which utilized for storage Permission is not utilized in this mode.
• You can think of HDFS as similar to the file system’s available for windows i.e. NTFS (New
Technology File System) and FAT32(File Allocation Table which stores the data in the blocks of
32 bits ).
• when your Hadoop works in this mode there is no need to configure the files – [Link],
[Link], [Link] for Hadoop environment.
• In this Mode, all of your Processes will run on a single JVM(Java Virtual Machine) and this mode
can only be used for small development purposes.
 Hadoop – Different Modes of Operation
1. Standalone Mode

• Standalone mode is usually the fastest Hadoop modes as it uses the local file system for all the
input and output. Here is the summarized view of the standalone mode-

• • Used for debugging purpose

• • HDFS is not being used
• • Uses local file system for input and output
• • No need to change any configuration files
• • Default Hadoop Modes
 Hadoop – Different Modes of Operation
2. Pseudo Distributed Mode (Single Node Cluster)

• In Pseudo-distributed Mode we also use only a single node, but the main thing is that the
cluster is simulated, which means that all the processes inside the cluster will run
independently to each other.
• All the daemons that are Namenode, Datanode, Secondary Name node, Resource Manager, Node
Manager, etc. will be running as a separate process on separate JVM(Java Virtual Machine) or we
can say run on different java processes that is why it is called a Pseudo-distributed.
• One thing we should remember that as we are using only the single node set up so all the
Master and Slave processes are handled by the single system.
• Namenode and Resource Manager are used as Master and Datanode and Node Manager is used as
a slave.
• A secondary name node is also used as a Master. The purpose of the Secondary Name node is to
just keep the hourly based backup of the Name node.
• In this Mode,
Hadoop is used for development and for debugging purposes both.
• Our HDFS utilized for managing the Input and Output processes.
• We need to change the configuration files [Link],
[Link], [Link] for setting up the environment.
 Hadoop – Different Modes of Operation
2. Pseudo Distributed Mode (Single Node Cluster)

• Here is the summarized view of pseudo distributed Mode-

• • Single Node Hadoop deployment running on Hadoop is considered as pseudo distributed mode
• • All the master & slave daemons will be running on the same node
• • Mainly used for testing purpose
• • Replication Factor will be ONE for blocks
• • Changes in configuration files will be required for all the three files- [Link], core-
[Link], [Link]
 Hadoop – Different Modes of Operation
2. Pseudo Distributed Mode (Single Node Cluster)

• Real-Time Example: Sensor Data Analysis in Pseudo-Distributed Hadoop

• Scenario:
• Imagine you are working on a project that involves analyzing data from a network of IoT sensors
deployed in various locations. These sensors collect environmental data such as temperature,
humidity, and air quality.

• Steps:
• Pseudo-Distributed Hadoop Setup:
• Set up Hadoop in pseudo-distributed mode on a single machine. This provides an environment
to develop, test, and analyze large-scale sensor data in a simulated distributed system.

• Generate Simulated Sensor Data:

• Simulate sensor data by creating a file with entries like:

Sensor1,2024-01-23 12:00:00,Temperature,25.5
Sensor2,2024-01-23 12:05:00,Humidity,60.2
Sensor3,2024-01-23 12:10:00,AirQuality,Good
 Hadoop – Different Modes of Operation
2. Pseudo Distributed Mode (Single Node Cluster)

• Ingest Sensor Data into HDFS:

• Copy the sensor data file into HDFS:
• hdfs dfs -copyFromLocal sensor_data.txt /sensor_data

• MapReduce Job - Average Temperature Calculation:

• Write a MapReduce job to calculate the average temperature recorded by all sensors over a
specific time period.

• hadoop jar [Link] AverageTemperature /sensor_data /output/averageTemperature

• MapReduce Job - Air Quality Analysis:

• Create another MapReduce job to analyze air quality data and count the occurrences of
different air quality levels.
• hadoop jar [Link] AirQualityAnalysis /sensor_data /output/airQualityAnalysis
• Review Results:
• Examine the output directories to see the results of your sensor data analysis:
• hdfs dfs -cat /output/averageTemperature/*
• hdfs dfs -cat /output/airQualityAnalysis/*
 Hadoop – Different Modes of Operation
3. Fully Distributed Mode (Multi-Node Cluster)

• This is the most important one in which multiple nodes are used few of them run the Master
Daemon’s that are Namenode and Resource Manager and the rest of them run the Slave
Daemon’s that are DataNode and Node Manager.
• Here Hadoop will run on the clusters of Machine or nodes.
• Here the data that is used is distributed across different nodes.
• This is actually the Production Mode of Hadoop let’s clarify or understand this Mode in a better
way in Physical Terminology.
• Once you download the Hadoop in a tar file format or zip file format then you install it in your
system and you run all the processes in a single system but here in the fully distributed mode
we are extracting this tar or zip file to each of the nodes in the Hadoop cluster and then we are
using a particular node for a particular process.
• Once you distribute the process among the nodes then you’ll define which nodes are working as
a master or which one of them is working as a slave.
 Hadoop – Different Modes of Operation
3. Fully Distributed Mode (Multi-Node Cluster)

• Example 2: Large-Scale Log Analysis

• Scenario:
• Consider a scenario where you have a vast amount of log data generated by an online service, and you want
to extract meaningful insights from this data.
• Steps:
• Data Ingestion:
• Ingest log data from various sources into HDFS.
• MapReduce Job - User Behavior Analysis:
• Write a MapReduce job that analyzes user behavior patterns, such as the most visited pages, average session
duration, or common sequences of actions.
• hadoop jar [Link] UserBehaviorAnalysis /input/logs /output/userBehavior
• MapReduce Job - Error Analysis:
• Implement a MapReduce job to analyze error logs and identify common error patterns or frequently
occurring issues.
• hadoop jar [Link] ErrorAnalysis /input/logs /output/errorAnalysis
• MapReduce Job - Traffic Analysis:
• Write another MapReduce job to analyze traffic patterns, such as peak access times or frequently accessed
resources
• hadoop jar [Link] TrafficAnalysis /input/logs /output/trafficAnalysis
• Review Results:
• Examine the output directories to gather insights into user behavior, errors, and traffic patterns.
 Hadoop EcoSystem
The following are the components of Hadoop ecosystem:
 Hadoop EcoSystem
1. HDFS:
• HDFS is the primary or major component of Hadoop ecosystem and is responsible for storing
large data sets of structured or unstructured data across various nodes and thereby maintaining
the metadata in the form of log files.
• HDFS consists of two core components i.e.
• Name node
• Data Node
• Name Node is the prime node which contains metadata (data about data) requiring
comparatively fewer resources than the data nodes that stores the actual data.
• These data nodes are commodity hardware in the distributed environment. Undoubtedly,
making Hadoop cost effective.
• HDFS maintains all the coordination between the clusters and hardware, thus working at the
heart of the system.
 Hadoop EcoSystem
1. HDFS:
 Hadoop EcoSystem
2. YARN:
• Yet Another Resource Negotiator, as the name
implies, YARN is the one who helps to manage the
resources across the clusters.
• In short, it performs scheduling and resource
allocation for the Hadoop System.
• It Consists of three major components i.e.
• Resource Manager
• Nodes Manager
• Application Manager
• Resource manager has the privilege of allocating
resources for the applications in a system.
• Node managers work on the allocation of resources
such as CPU, memory, bandwidth per machine and
later on acknowledges the resource manager.
• Application manager works as an interface between
the resource manager and node manager and
performs negotiations as per the requirement of
the two.
 Hadoop EcoSystem
2. YARN:
 Hadoop EcoSystem
3. MapReduce:
• By making the use of distributed and parallel algorithms, MapReduce makes it possible to carry
over the processing’s logic and helps to write applications which transform big data sets into a
manageable one.
• MapReduce makes the use of two functions i.e. Map() and Reduce() whose task is:
• Map() performs sorting and filtering of data and thereby organizing them in the form of group.
Map generates a key-value pair based result which is later on processed by the Reduce() method.
• Reduce(), as the name suggests does the summarization by aggregating the mapped data.
• In simple, Reduce() takes the output generated by Map() as input and combines those tuples into
smaller set of tuples.
 Hadoop EcoSystem
4. Sqoop:
Sqoop (SQL-to-Hadoop) is one of the most popular Big Data tools that leverages the competency to
haul out data from a non-Hadoop data store by transforming information into a form that can be
easily accessed and used by Big Data Hadoop, to then upload it into HDFS
 Hadoop EcoSystem
5. Flume:
Apache Flume is an open-source distributed system for transferring streaming data from various
applications to the central repository like HDFS, or HBase.
It is highly reliable, extensible, robust, and fault-tolerant.
 Hadoop EcoSystem
6. PIG:
• Pig was basically developed by Yahoo which works on a pig Latin language, which is Query based
language similar to SQL.
• It is a platform for structuring the data flow, processing and analyzing huge data sets.
• Pig does the work of executing commands and in the background, all the activities of
MapReduce are taken care of. After the processing, pig stores the result in HDFS.
• Pig Latin language is specially designed for this framework which runs on Pig Runtime. Just the
way Java runs on the JVM.
 Hadoop EcoSystem
7. HIVE:
• With the help of SQL methodology and interface, HIVE performs reading and writing of large
data sets. However, its query language is called as HQL (Hive Query Language).
• It is highly scalable as it allows real-time processing and batch processing both. Also, all the SQL
datatypes are supported by Hive thus, making the query processing easier.
• Similar to the Query Processing frameworks, HIVE too comes with two components: JDBC
Drivers and HIVE Command Line.
• JDBC, along with ODBC drivers work on establishing the data storage permissions and
connection whereas HIVE Command line helps in the processing of queries.
 Hadoop EcoSystem
7. HIVE:
 Hadoop EcoSystem
8. Spark:
• Apache Spark is a framework for real time data analytics in a distributed computing
environment.
• The Spark is written in Scala and was originally developed at the University of California,
Berkeley.
• It executes in-memory computations to increase speed of data processing over Map-Reduce.
• It is 100x faster than Hadoop for large scale data processing by exploiting in-memory
computations and other optimizations. Therefore, it requires high processing power than Map-
Reduce.
 Hadoop EcoSystem
9. Mahout:
• Mahout, allows Machine Learnability to a system or application.
• Machine Learning, as the name suggests helps the system to develop itself based on some
patterns, user/environmental interaction or on the basis of algorithms.
• It provides various libraries or functionalities such as collaborative filtering, clustering, and
classification which are nothing but concepts of Machine learning.
• It allows invoking algorithms as per our need with the help of its own libraries.
 Hadoop EcoSystem
10. Oozie:
• Apache Oozie is a tool for Hadoop operations that allows cluster administrators to build complex
data transformations out of multiple component tasks.
• This provides greater control over jobs and also makes it easier to repeat those jobs at
predetermined intervals
 Hadoop EcoSystem
11. Kafka:
• Apache Kafka is an open-source distributed event streaming platform that is widely used for
building real-time data pipelines and streaming applications.
• Originally developed by LinkedIn and later open-sourced as part of the Apache Software
Foundation, Kafka provides a distributed, fault-tolerant, and scalable platform for handling large
volumes of data streams.
 Hadoop EcoSystem
12. Strom:
• Apache Storm is an open-source real-time stream processing system.
• It is designed for processing unbounded streams of data in a fault-tolerant and scalable manner.
• Storm provides a distributed and fault-tolerant runtime for processing data streams, making it
suitable for applications that require low-latency and high-throughput data processing.
 Hadoop EcoSystem
13. Ranger:
• Apache Ranger is an open-source framework for managing security and compliance policies for
Hadoop-based ecosystems.
• It provides centralized security administration and authorization across various Hadoop
components, ensuring consistent access control policies and auditing capabilities.
• Ranger is designed to address security and compliance challenges in big data environments.
 Hadoop EcoSystem
14. Knox:
• Apache Knox is an open-source gateway solution for providing a centralized access point and
security for Apache Hadoop clusters.
• It acts as a reverse proxy that provides a single entry point for all REST and HTTP interactions
with the Hadoop cluster, simplifying access and adding an additional layer of security.
 Hadoop EcoSystem
15. HBase:
• HBase is an open source, non-relational distributed database. In other words, it is a NoSQL
database.
• It supports all types of data and that is why, it’s capable of handling anything and everything
inside a Hadoop ecosystem.
• It is modelled after Google’s BigTable, which is a distributed storage system designed to cope up
with large data sets.
• The HBase was designed to run on top of HDFS and provides BigTable like capabilities.
• HBase is an extremely fault-tolerant way as well as good for storing sparse data
 Hadoop EcoSystem
The following are the components of Hadoop ecosystem:
16. Chukwa:
• Apache Chukwa is an open source large-scale log collection system for monitoring large
distributed systems. It is one of the common big data terms related to Hadoop.
• It is built on the top of Hadoop Distributed File System (HDFS) and Map/Reduce framework.
• It inherits Hadoop’s robustness and scalability.
• Chukwa contains a powerful and flexible toolkit database for monitoring, displaying, and
analyzing results so that collected data can be used in the best possible ways.
 Hadoop EcoSystem
The following are the components of Hadoop ecosystem:
17. Zookeeper:
• Apache Zookeeper is the coordinator of any Hadoop job which includes a combination of various
services in a Hadoop Ecosystem.
• Apache Zookeeper coordinates with various services in a distributed environment.
• Before Zookeeper, it was very difficult and time consuming to coordinate between different
services in Hadoop Ecosystem.
• The services earlier had many problems with interactions like common configuration while
synchronizing data.
 Overview of MapReduce Framework
• MapReduce is a software framework.
• MapReduce programming helps to process massive amounts of data in parallel.
• In MapReduce programming, input data set splits into independent chunks.
• Map tasks process these independent chunks completely in a parallel manner.
• The output produced by the map task serves as intermediate data and is stored on the local disk
of the server.
• The output of the mappers are automatically shuffled and stored by the framework.
• MapReduce framework sorts the output based on keys.
• This sorted output becomes the input to the reduce tasks.
• Reduce task-provides reduced output by combining the output of various mappers.
• Job inputs and outputs are stored in a file system.
• MapReduce framework also takes care of the other tasks such as scheduling, monitoring, re-
executing failed tasks etc.
Overview of MapReduce Framework
 Overview of MapReduce Framework
• HDFS and MapReduce frameworks run on same set of nodes.
• This configuration allows effective scheduling of tasks on the node where data is present( Data
Locality), which results in high throughput.
• There are two daemons associated with MapReduce Programming:
• A single master JobTracker per cluster and one slave TaskTracker per cluster node.
• The JobTracker is responsible for scheduling tasks to the TaskTrackers, monitoring the tasks,
and re-executing the case of failure.
• The TaskTracker executes the tasks.
 MapReduce Daemons
• JobTracker
• JobTracker is a master daemon responsible for executing over MapReduce job.
• It provides connectivity between Hadoop and application.
• Whenever code submitted to a cluster, JobTracker creates the execution plan by deciding which
task to assign to which node.
• It also monitors all the running tasks.
• When task fails it automatically re-schedules the task to a different node after a predefined
number of retires.
• There will be one job Tracker process running on a single Hadoop cluster.
• Job Tracker processes run on their own Java Virtual machine process.
 MapReduce Daemons
• TaskTracker
• This daemon is responsible for executing individual tasks that is assigned by the Job Tracker.
• There is a single TaskTracker per slave and spawns multiple Java Virtual Machines to handle
multiple map or reduce tasks in parallel.
• Task Tracker continuously sends heartbeat message to job tracker.
• When a job tracker fails to receive a heartbeat message from a TaskTracker, the JobTracker
assumes that the TaskTracker has failed and resubmits the task to another available node in the
cluster.
• Once the client submits a job to the JobTracker, it partitions and assigns diverse MapReduce
tasks for each TaskTracker in the cluster
MapReduce Architecture
 MapReduce Architecture
• MapReduce architecture consists of various components.
• A brief description of these components is given below

• Job: This is real work that needs to be done or processed

• Task: This is a piece of real work that needs to be done or processed. The MapReduce task
covers many small tasks that need to be done.
• Job Tracker: This tracker plays a role in organizing tasks and tracking all tasks assigned to a
task tracker.
• Task Tracker: This tracker plays the role of tracking activity and reporting activity status to the
task tracker.
• Input data: This is used for processing in the mapping phase.
• Exit data: This is the result of mapping and mitigation.
• Client: This is a program or Application Programming Interface (API) that sends tasks to
MapReduce. It can accept services from multiple clients.
• Hadoop MapReduce Master: This plays the role of dividing tasks into sections.
• Job Parts: These are small tasks that result in the division of the primary function.
 MapReduce Architecture
• In MapReduce architecture, clients submit tasks to MapReduce Master.
• This manager will then divide the work into smaller equal parts.
• The components of the function will be used for two main tasks in Map Reduce: mapping and
subtraction.
• The developer will write a concept that satisfies the organization’s or company’s needs.
• Input data will be categorized and mapped.
• The central data will then be filtered and merged.
• The slider that will produce the last one stored on HDFS will process the output.
• The following diagram shows a simplified flow diagram of the MapReduce program.
 MapReduce Architecture
• How the JobTracker and TaskTracker works:

• Every task consists of two essential parts:

mapping and reduction functions.
• Map work plays the role of splitting duties
into task segments and central mapping
data, and the reduction function plays the
role of shuffling and reducing the central
data into smaller units.
• The activity tracker works like a master.
• It ensures that we do all the work.
• The activity tracker lists tasks posted by
clients, and it will provide job trackers for
jobs.
• Each task tracker has a map function and
minimizes tasks.
• Activity trackers report the status of each
task assigned to the task tracker.
• The following diagram summarizes how task
trackers and task trackers work.
 Phases of MapReduce
• The MapReduce program comprises three main stages: mapping, navigation, and mitigation.
There is also an optional category known as the merging phase.

• Mapping Phase
• This is the first phase of the program.
• There are two steps in this phase: classification and mapping.
• The database is divided into equal units called units (input divisions) in the division step.
• Hadoop contains a RecordReader that uses TextInputFormat to convert input variables into
keyword pairs.
• Key-value pairs are then used as input on the map step.
• This is the only data format a map editor can read or understand.
• The map step contains the logic of the code used in these data blocks.
• In this step, the map analyzes key pairs and generates an output of the same form (key-value
pairs).

• map(in_key, in_value) -> list(intermediate_key, intermediate_value)

Phases of MapReduce
 Phases of MapReduce
• Shuffling phase
• This is the second phase that occurs after the completion of the Mapping phase.
• It consists of two main steps: filtering and merging.
• In the filter step, keywords are filtered using keys and combining ensures that key-value pairs are
included.
• The shoplifting phase facilitates the removal of duplicate values and the collection of values.
• Different values with the same keys are combined.
• The output of this category will be keys and values, as in the Map section.

• Reducer phase
• In the reduction phase, the output of the push phase is user input.
• The subtractor continuously processes these inputs to reduce the median values into smaller
ones.
• Provides a summary of the entire database.
• Output in this category is stored in HDFS.

• The following diagram illustrates MapReduce with three main categories. Separation is usually
included in the mapping phase.
• reduce(intermediate_key, list(intermediate_value) -> list(out_key, out_value)
 Phases of MapReduce
• The following diagram illustrates MapReduce with three main categories. Separation is usually
included in the mapping phase.
 Phases of MapReduce
• Combiner phase
• This is the optional phase used to improve the MapReduce process.
• It is used to reduce pap output at the node level.
• At this stage, duplicate output from the map output can be merged into a single output.
• The integration phase accelerates the integration phase by improving the performance of tasks.
• The following diagram shows how all four categories of MapReduce are used.
 Benefits of Hadoop MapReduce
• There are numerous benefits of MapReduce; some of them are listed below.

• Speed: It can process large amounts of random data in a short period.

• Fault tolerance: The MapReduce framework can manage failures.
• Less expensive: Hadoop has a rating feature that allows users to process or store data cost-
effectively.
• Scalability: Hadoop provides an excellent framework. MapReduce allows users to run applications
on multiple nodes.
• Data availability: Data matches are sent to various locations within the network. This ensures
that copies of the data are available in case of failure.
• Parallel Processing: On MapReduce, many parts of the same database functions can be processed
similarly. This reduces the time taken to complete the task.
 Applications of Hadoop MapReduce
• The following are some of the most valuable features of the MapReduce program.

• E-commerce

• E-commerce companies like Walmart, eBay, and Amazon use MapReduce to analyze consumer
behavior.
• MapReduce provides sound information that is used as a basis for developing product
recommendations.
• Other information includes site records, e-commerce catalog, purchase history, and contact logs.

• Social media

• The MapReduce editing tool can check certain information on social media platforms such as
Facebook, Twitter, and LinkedIn.
• It can check necessary information, such as who liked your status and viewed your profile.

• Entertainment

• Netflix uses MapReduce to analyze clicks and logs for online clients.
• This information helps the company promote movies based on customer interests and behavior.
 Use cases of MapReduce
• WordCount Program
• WC_Mapper.java
• public class WC_Mapper extends MapReduceBase implements
• Mapper<LongWritable,Text,Text,IntWritable>
• {
• private final static IntWritable one = new IntWritable(1);
• private Text word = new Text();
• public void map(LongWritable key, Text Context context) throws IOException
{
• String line = [Link]();
• StringTokenizer tokenizer = new StringTokenizer(line);
• while ([Link]())
• {
• [Link]([Link]());
• [Link](word, one);
• }
• }
• }
 Use cases of MapReduce
• WC_Reducer.java
• public class WC_Reducer extends MapReduceBase implements
Reducer<Text,IntWritable,Text,IntWritable>
• {
• public void reduce(Text key, Iterator<IntWritable> values, Context context) throws
IOException
• {
• int sum=0;
• while ([Link]())
• {
• sum+=[Link]().get();
• }
• [Link](sum);
• [Link](key, result);
• }
• }
 Use cases of MapReduce
• WordCount Program hadoop jar [Link] WordCount input_directory output_directory
• WC_Driver.java
• public class WordCount {

• public static void main(String[] args) throws Exception {

•
Configuration conf = new Configuration();
• Job job = [Link](conf, "word count");
• [Link]([Link]);
•
• [Link]([Link]);
• [Link]([Link]);
• [Link]([Link]);
•
• [Link]([Link]);
• [Link]([Link]);
•
• [Link](job, new Path(args[0]));
• [Link](job, new Path(args[1]));
• [Link]([Link](true) ? 0 : 1);
• }}
 Use cases of MapReduce
• MaximumAverage
• Mapper
• Task of mapper is to calculate the average of each row.
• Reducer
• Task of reducer is to calculate the maximum of these average values which were provided by the
mapper.
• Use Case
• Calculating max percentage of a student in a class
• where each row will desribe the marks of student in different subject.
 Use cases of MapReduce
• MA_Mapper.java
• public class MaximumAverageMapper extends Mapper<LongWritable, Text, Text, FloatWritable>
• {
• public void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException
• {
• String line = [Link]();
• String [] a = [Link](" ");
• [Link](line);
• int sum=0;
• for(String i:a)
• sum += [Link](i);

• float avg = sum/[Link]; [Link](avg);

• [Link](new Text("maxavg"),new FloatWritable(avg) );
• }
• }
•
 Use cases of MapReduce
• MA_Reducer.java
• public class MaximumAverageReducer extends Reducer<Text, FloatWritable, Text,
FloatWritable>
• {
• public void reduce(Text key, Iterable<FloatWritable> values, Context context)
• throws IOException, InterruptedException {
• float max=0;
• for (FloatWritable val : values)
{
if([Link]()>max)
{
max=[Link]();
}
}
• [Link](key, new FloatWritable(max));
• }
• }
 Use cases of MapReduce
public class MaximumAverageDriver {
public static void main(String[] args) throws Exception
{
Configuration conf = new Configuration();
Job job = [Link](conf, "maxaverage");

[Link]([Link]);
[Link]([Link]);
[Link]([Link]);

[Link]([Link]);
[Link]([Link]);

[Link](job, new Path(args[0]));

[Link](job, new Path(args[1]));

if ()
return;
}
}
 Use cases of MapReduce
• MaxTemp
public class MaxTempMapper extends Mapper<Object, Text, Text, FloatWritable>
{
public void map(Object ikey, Text ivalue, Context context)
throws IOException, InterruptedException
{
String line = [Link]();
if (!([Link]()==0))
{
String year = [Link](6, 10);
float avg = [Link]([Link](54,62).trim());
[Link](new Text(year),new FloatWritable(avg));

}
}
}
 Use cases of MapReduce
• MaxTemp
• public class MaxTempReducer extends Reducer<Text, FloatWritable, Text, FloatWritable> {

• public void reduce(Text key, Iterable<FloatWritable> values, Context context)

• throws IOException, InterruptedException {
• float max = 0;
• for(FloatWritable value:values ){
•
• if(max < [Link]()){
• max=[Link]();
•
• }
•
• }
•
• [Link](key, new FloatWritable(max));
•
• }

• }
 Use cases of MapReduce
• MaxTemp
• public class MaxTempDriver {

• public static void main(String[] args) throws Exception {

• Configuration conf = new Configuration();
• Job job = [Link](conf, "maxtemp");
•
• [Link]([Link]);
• [Link]([Link]);

• [Link]([Link]);
• [Link]([Link]);

• [Link](job, new Path(args[0]));

• [Link](job, new Path(args[1]));

• if ()
• return;
• }

• }
Use cases of MapReduce
Anatomy of MapReduce Program.
 Anatomy of MapReduce Program.
• There are five independent entities:

• The client, which submits the MapReduce job.

• The YARN resource manager, which coordinates the allocation of compute resources on the
cluster.
• The YARN node managers, which launch and monitor the compute containers on machines in the
cluster.
• The MapReduce application master, which coordinates the tasks running the MapReduce job The
application master and the MapReduce tasks run in containers that are scheduled by the
resource manager and managed by the node managers.
• The distributed file system, which is used for sharing job files between the other entities.
 Anatomy of MapReduce Program.
• Job Submission:

• The submit() method on Job creates an internal JobSubmitter instance and calls
submitJobInternal() on it.
• Having submitted the job, waitForCompletion polls the job’s progress once per second and
reports the progress to the console if it has changed since the last report.
• When the job completes successfully, the job counters are displayed Otherwise, the error that
caused the job to fail is logged to the console.
• The job submission process implemented by JobSubmitter does the following:
• Asks the resource manager for a new application ID, used for the MapReduce job ID.
• Checks the output specification of the job For example, if the output directory has not been
specified or it already exists, the job is not submitted and an error is thrown to the MapReduce
program.
• Computes the input splits for the job If the splits cannot be computed (because the input paths
don’t exist, for example), the job is not submitted and an error is thrown to the MapReduce
program.
• Copies the resources needed to run the job, including the job JAR file, the configuration file, and
the computed input splits, to the shared filesystem in a directory named after the job ID.
• Submits the job by calling submitApplication() on the resource manager.
 Anatomy of MapReduce Program.
• Job Initialization:

• When the resource manager receives a call to its submitApplication() method, it hands off the
request to the YARN scheduler.
• The scheduler allocates a container, and the resource manager then launches the application
master’s process there, under the node manager’s management.
• The application master for MapReduce jobs is a Java application whose main class is
MRAppMaster .
• It initializes the job by creating a number of bookkeeping objects to keep track of the job’s
progress, as it will receive progress and completion reports from the tasks.
• It retrieves the input splits computed in the client from the shared file system.
• It then creates a map task object for each split, as well as a number of reduce task objects
determined by the [Link] property (set by the setNumReduceTasks() method on
Job).
 Anatomy of MapReduce Program.
• Task Assignment:

• If the job does not qualify for running as an uber task, then the application master requests
containers for all the map and reduce tasks in the job from the resource manager .
• Requests for map tasks are made first and with a higher priority than those for reduce tasks,
since all the map tasks must complete before the sort phase of the reduce can start.
• Requests for reduce tasks are not made until 5% of map tasks have completed.

• Task Execution:

• Once a task has been assigned resources for a container on a particular node by the resource
manager’s scheduler, the application master starts the container by contacting the node
manager.
• The task is executed by a Java application whose main class is YarnChild. Before it can run the
task, it localizes the resources that the task needs, including the job configuration and JAR file,
and any files from the distributed cache.
• Finally, it runs the map or reduce task.
 Anatomy of MapReduce Program.
• Streaming:

• Streaming runs special map and reduce tasks for the

purpose of launching the user supplied executable and
communicating with it.
• The Streaming task communicates with the process (which
may be written in any language) using standard input and
output streams.
• During execution of the task, the Java process passes input
key value pairs to the external process, which runs it
through the user defined map or reduce function and
passes the output key value pairs back to the Java process.
• From the node manager’s point of view, it is as if the child
process ran the map or reduce code itself.
 Anatomy of MapReduce Program.
• Progress and status updates :
• MapReduce jobs are long running batch jobs, taking anything from tens of seconds to hours to
run.
• A job and each of its tasks have a status, which includes such things as the state of the job or
task (e g running, successfully completed, failed), the progress of maps and reduces, the values of
the job’s counters, and a status message or description (which may be set by user code).
• When a task is running, it keeps track of its progress (i e the proportion of task is completed).
• For map tasks, this is the proportion of the input that has been processed.
• For reduce tasks, it’s a little more complex, but the system can still estimate the proportion of
the reduce input processed.
• It does this by dividing the total progress into three parts, corresponding to the three phases of
the shuffle.
• As the map or reduce task runs, the child process communicates with its parent application
master through the umbilical interface.
• The task reports its progress and status (including counters) back to its application master,
which has an aggregate view of the job, every three seconds over the umbilical interface.
• How status updates are propagated through the MapReduce System The resource manager web UI
displays all the running applications with links to the web UIs of their respective application
masters, each of which displays further details on the MapReduce job, including its progress.
• During the course of the job, the client receives the latest status by polling the application
master every second (the interval is set via [Link]).
 Anatomy of MapReduce Program.
• Progress and status updates :
 Anatomy of MapReduce Program.
• Job Completion:
• When the application master receives a notification that the last task for a job is complete, it
changes the status for the job to Successful.
• Then, when the Job polls for status, it learns that the job has completed successfully, so it prints
a message to tell the user and then returns from the waitForCompletion() .
• Finally, on job completion, the application master and the task containers clean up their working
state and the OutputCommitter’s commitJob () method is called.
• Job information is archived by the job history server to enable later interrogation by users if
desired.
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• Many applications in Bioinformatics revolve around next-generation sequencing
• Process of analyzing sequences such as DNA to understand different characteristics of an
organism.
• Very large DNA strings
• Hundreds of GB of sequence data can be generated from single experiments
• A number of different problems arise: k-mer counting, sequence quality assessment, read
alignment, fast similarity search, etc
• Two sources of knowledge required: domain specific, and big data
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• k-mer Counting
• k-mer counting is a critical first step for many Next Generation Sequencing applications
• A k-mer refers to all the possible subsequences of length k in a DNA/RNA sequence
• k-mer counting returns a count of every k-mer present in a sequence
• "When the k-mer size is large and billions of reads need to be processed, k-mer counting becomes
the most difficult problem in Bioinformatics“

• K-Mer Counting Mapper:

 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• K-Mer Counting Mapper:
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• K-Mer Counting Mapper:
• map(input):
• for i in [0,[Link] - k):
• emit(input[i:i+k], 1)
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• K-Mer Counting Mapper:
• Mapping phase does not require network usage (computation is performed local to the data!)
• Reduce phase requires significant network usage
• Shuffle – can run during map phase, but finishes after map completes
• Sort – runs after shuffle completes
• Reduce – runs after sort completes
• MapReduce allows overlap of shuffle and map phases
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• K-Mer Counting Mapper:
• public class KmerCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
• private final static IntWritable one = new IntWritable(1);
• private Text kmer = new Text();
• private int k; // Set the desired k-mer length here
k=3
• protected void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException {
• String sequence = [Link]().toUpperCase().replaceAll("[^ACGT]", ""); // Remove non-
ACGT characters
• for (int i = 0; i <= length - k; i++) {
• String kmerString = [Link](i, i + k);
• [Link](kmerString);
• [Link](kmer, one);
• }
• }
• }
 A case study of tuning
MapReduce for efficient Bioinformatics in the cloud.
• K-Mer Counting Reducer:
• public class KmerCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
• private IntWritable result = new IntWritable();

• protected void reduce(Text key, Iterable<IntWritable> values, Context context)

• throws IOException, InterruptedException {
• int sum = 0;
• // Sum up the counts for each k-mer
• for (IntWritable value : values) {
• sum += [Link]();
• }

• [Link](sum);
• [Link](key, result);
• }
• }