UNIT III

Map Reduce Applications

MapReduce workflows – unit tests with MRUnit – test data and local tests – anatomy of MapReduce job
run – classic Map-reduce – YARN – failures in classic Map-reduce and YARN – job scheduling – shuffle
and sort – task execution – MapReduce types – input formats – output formats.
[Link] MapReduce.
 MapReduce is a Java - based, distributed execution framework within the Apache Hadoop Ecosystem. It
takes away the complexity of distributed programming by exposing two processing steps that developers
implement: Map and Reduce.
 In the Mapping step, data is split between parallel processing [Link] logic can be applied
to each chunk of data. Once completed, the Reduce phase takes over to handle aggregating data from the
Map set.
 In general, MapReduce uses Hadoop Distributed File System (HDFS) for both input and output. The
MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster -
node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them
and re-executing the failed tasks. The slaves execute the tasks as directed by the master.
 MapReduce is a programming model and software framework first developed by Google. Intended to
facilitate and simplify the processing of vast amounts of data in parallel on large clusters of commodity
hardware in a reliable, fault-tolerant manner.
Q. What are the Characteristics of MapReduce:
1. Very large-scale data: peta, exa bytes.
2. Write once and read many data. It allows for parallelism without mutexes.
3. Map and Reduce are the main operations: Simple code.
4. All the map should be completed before reduce operation starts.
5. Map and reduce operations are typically performed by the same physical processor.
6. Number of map tasks and reduce tasks are configurable.
[Link] Workflows. List the Relational-Algebra operations in mapReduce. Nov/Dec-2023
Q. Discuss workflow and dataflow in MapReduce programming model.
 With HDFS, we are able to distribute the data so that data is stored on hundreds of nodes instead of
a single large machine.
 MapReduce provides the framework for highly parallel processing of data across clusters of commodity
hardware.

1
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 MapReduce data processing

 It removes the complicated programming part from the programmers and moves into the
framework. Programmers can write simple programs to make use of the parallel processing.
 The framework splits the data into smaller chunks that are processed in parallel on cluster of machines
by programs called mappers.
 The output from the mappers is then consolidated by reducers into desired result. The share
nothing architecture of mappers and reducers make them highly parallel.
 Input: This is the input data / file to be processed.
 Split: Hadoop splits the incoming data into smaller pieces called "splits".
 Map: In this step, MapReduce processes each split according to the logic defined in map() function.
Each mapper works on each split at a time. Each mapper is treated as a task and multiple tasks are
executed across different Task Trackers and coordinated by the JobTracker.
 Combine: This is an optional step and is used to improve the performance by reducing the amount of
data transferred across the network. Combiner is the sameas the reduce step and is for aggregating the
output of the map() function before it is passed to the subsequent steps.
 Shuffle and Sort: In this step, outputs from all the mappers are shuffled, sorted to put them in order and
grouped before sending them to the next step.
 Reduce: This step is used to aggregate the outputs of mappers using the reduce()function. Output of
reducer is sent to the next and final step. Each reducer is treated as a task and multiple tasks are executed
across different Task Trackers and coordinated by the JobTracker.
 Output: Finally the output of reduce step is written to a file in HDFS.. The output from the mappers is
then consolidated by reducers into desired result. The share nothing architecture of mappers and
reducers make them highly parallel.

2
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
  Data locality is achieved by mapreduce by working closely with HDFS. When you specify the file
system as HDFS for mapreduce, it automatically schedules them appers on the same node as where the
block of data exists.
 Mapreduce can get the blocks from HDFS and process them. The final output from Mapreduce also can
be stored in HDFS file system. However, the intermediate files between mappers and reducers are not
stored in HDFS and arestored on the local file system of the mappers.
[Link] Flow in the MapReduce Programming Model
 MapReduce: A programming model to facilitate the development and execution of distributed tasks.
 The programmer defines the program logic as two functions:
a. Map transforms the input into key-value pairs to process.
b. Reduce aggregates the list of values for each key.
 The MapReduce environment takes in charge distribution aspects. A complex program can be
decomposed as a succession of Map and Reduce tasks. Higher-level languages (Pig, Hive, etc.) help
with writing distributed applications.
 MapReduce is a parallel programming model especially dedicated for complex and distributed
computations, which has been derived from the functional paradigm.
 In general, MapReduce processing is composed of two consecutive stages, which for most problems are
repeated iteratively: The map and the reduce phase.
 Map processes the data on hosts in parallel, whereas the reduce aggregates the results. At each iteration
independently, the whole data is split into chunks, which, in turn, are used as the input for mappers.

3
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
  Each chunk may be processed by only one mapper. Once the data is processed by mappers, hey can emit
View the MathML source? Key, value? Pairs to the reduce phase.
 Before the reduce phase, the pairs are sorted and collected according to the View the MathML sourcekey
values, therefore each reducer gets the list of values related to a given View the MathML sourcekey. The
consolidated output of the reduce phase is saved into the distributed file system.
Q. Briefly explain about functions of Job Tracker and Task Tracker
Function of job tracker:
 There is a single job tracker that runs on the master node. It is the driver for the map-reduce jobs. Its
functions are:
1. Accepts jobs from client and divides into tasks.
2. Schedules tasks on worker nodes called task trackers.
3. Keeps heartbeat info from task trackers on worker nodes.
4. Reschedules the task on alternate worker if a worker fails.
Function of task tracker:
 Task tracker runs on each worker node and there are as many task trackers as the worker nodes. If HDFS
is also used, then data nodes of HDFS also become worker nodes for task tracker. The functions of a
task tracker are:
a. Takes assignments from job tracker.
b. Executes the tasks locally.
c. Each worker node has specific number of mapper and reducer tasks it can take at one time.
d. The tasks assigned are run in parallel.
e. Normally they can take more map jobs than reduce tasks.
f. Task tracker does a task attempt before executing task.
g. Task tracker may do multiple attempts before declaring a task as failed.
h. Task tracker maintains a connection with the task attempt called umbilical protocol.
i. Task tracker sends a regular heartbeat signal to job tracker indicating its status including available map
and reduce tasks.
j. Task tracker runs each task attempt in a separate JVM. So even if the task has bad code due to which it
fails, it will not cause task tracker to abort.
Q. What are the limitation of MapReduce
1. Cannot control the order in which the maps or reductions are run.
2. For maximum parallelism, we need Maps and Reduces to not depend on data generated in the same
MapReduce job (i.e., stateless).

4
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 3. A database with an index will always be faster than a MapReduce job on unindexed data.
4. Reduce operations do not take place until all Maps are complete.
5. General assumption that the output of Reduce is smaller than the input to Map;large data source used
to generate smaller final values.
[Link] Tests with MRUnit
Q. Explain in details about unit test MRUnit
 MRUnit is a JUnit-based Java library that allows us to unit test HadoopMapReduce programs. There is
specialized test suite for testing mapreduce jobs, known as MRUnit.
 MRUnit removes as much of the Hadoop framework as possible while developing and testing. The
focuses are narrowed to the map and reduce code, their inputs and expected outputs. With MRUnit,
developing and testing MapReduce code can be done entirely in the IDE and these tests take fractions of
a second to run.
 MRUnit is built on top of the popular JUnit testing framework. It uses the object -mocking library,
Mockito, to mock most of the essential Hadoop objects so the user only needs to focus on the map and
reduce logic.
 MRUnit supports testing Mappers and Reducers separately as well as testing MapReduce computations
as a whole.
 To get started, download MRUnit. After we have extracted the tar file, cd into themrunit-0.9.0-
incubating/lib directory. In there we should see the following:
[Link]
[Link]
 The [Link] is for MapReduce version 1 of Hadoopand mrunit-0.9.0-
[Link] is for working the new version ofHadoop's MapReduce.
 Given a MapReduce job that writes to an HBase table called MyTest, which has one column
family called CF, the reducer of such a job could look like the following:
public class MyReducer extends TableReducer<Text, Text, ImmutableBytesWritable> {
public static final byte[] CF = "CF".getBytes();
public static final byte[] QUALIFIER="CQ-1".getBytes();
public void reduce (Text key, Iterable<Text> values, Context context) throws
IOException, InterruptedException {
//bunch of processing to extract data to be inserted, in our case, let’s say we
are simply
//appending all the records we receive from the mapper for this particular

5
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 //key and insert one record into HBase
StringBuffer data = new StringBuffer();
Put put = new Put([Link]([Link]()));
for (Text val values) {
data = [Link](val);
}
[Link](CF, QUALIFIER, [Link]([Link]()));
//write to HBase
[Link](new Immutable Bytes Writable ([Link]([Link]())), put);
}
}
 To test this code, the first step is to add a dependency to MRUnit to MavenPOM file.
<dependency>
<groupId>[Link]</groupId>
<artifactId>mrunit</artifactId>
<version> 1.0.0</version>
<scope>test</scope>
</dependency>

 Next, use the ReducerDriver provided by MRUnit, in Reducer job.

public class MyReducerTest {
ReduceDriver<Text, Text, ImmutableBytesWritable, Writable>reduceDriver;
byte[] CF "CF".getBytes();
byte[] QUALIFIER="CQ-1".gotBytes();
@Before
public void setUp() {
MyReducer reducer= new MyReducer();
reduceDriver = [Link] (reducer);
}
@Test
public void testHBaseInsert() throws IOException {
String strKey = "RowKey-1", strValue = "DATA", strValue1= "DATA1", strValue2=
"DATA2";

6
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 List<Text> list = new ArrayList<Text>();
[Link](new Text(strValue));
[Link](new Text(strValue 1));
[Link](new Text(strValue2));
//since the reducer is doing is appending the records that the mapper
//sends it, we should get the following back
String expected Output = strValue + strValue1 + strValue2;
//Setup Input, mimic what mapper would have passed
//to the reducer and run test
[Link](new Text(strKey), list);
//run the reducer and get its output
List<Pair<ImmutableBytesWritable, Writable>> result = [Link]();
//extract key from result and verify
assertEquals([Link]([Link](0).getFirst().get()), strKey);
//extract value for CF/QUALIFIER and verify
Put a =(Put)[Link](0).getSecond();
String c = [Link]([Link](CF, QUALIFIER).get(0).getValue());
assertEquals(expected Output,c);
}
}
 MRUnit test verifies that the output is as expected, the Put that is inserted intoHBase has the correct
value and the ColumnFamily and ColumnQualifier have thecorrect values. MRUnit includes a
MapperDriver to test mapping jobs and we canuse MRUnit to test other operations, including reading
from HBase, processingdata, or writing to HDFS.
 To Unit test MapReduce jobs:
1. Create a new test class to the existing project
2. Add the mrunit jar file to build path
3. Declare the drivers
4. Write a method for initializations and environment setup
5. Write a method to test mapper
6. Write a method to test reducer
7. Write a method to test the whole MapReduce job
8. Run the test

7
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
[Link] to test Java MapReduce Jobs in Hadoop?
Developer activities:
Step 1: Develop MapReduce Code
Step 2: Unit Testing of Map Reduce code using MRUnit framework
Step 3: Create Jar file for MapReduce code
Testing activities:
Step 1: Create a new directory in HDFS then copy data file from local to HDFSdirectory
Step 2: Run jar file by providing data file as an input
Step 3: Check output file created on HDFS.
[Link] the components involved in the Anatomy of a MapReduce Job Run. Nov/Dec-2023.(13marks)
[Link] explains about anatomy of MapReduce job run with neat diagram.
 We can run a MapReduce job with a single line of code: [Link](conf).
 There are five independent entities:
1. The client, which submits the MapReduce job.
2. The YARN resource manager, which coordinates the allocation of compute resources on the cluster.
3. The YARN node managers, which launch and monitor the compute containers on machines in the
cluster,
4. The MapReduce application master, which coordinates the tasks running theMapReduce job. The
application master and the MapReduce tasks run incontainers that are scheduled by the resource
manager and managed by the node managers.
5. The distributed file system, which is used for sharing job files between theother entities.
Job submission:
1. The submit() method on Job creates an internal Jobsubmitter instance and callssubmitJobInternal() on
it,
2. Having submitted the job, waitForCompletion polls the job's progress once per second and reports the
progress to the console if it has changed since the lastreport,
3. When the job completes successfully, the job counters are displayed otherwise, the error that caused
the job to fail is logged to the console.
The job submission process implemented by JobSubmitter does the following:
1. Asks the resource manager for a new application ID, used for the MapReduce job ID.
2. Checks the output specification of the job for example, If the output directory has not been
specified or it already exists, the job is not submitted and an error is thrown to the MapReduce
program.

8
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 3. Computes the input splits for the job If the splits cannot be computed (because the input paths don't
exist, for example), the job is not submitted and an error is thrown to the MapReduce program.
4. Copies the resources needed to run the job, including the job JAR file, the configuration file and the
computed input splits, to the shared filesystem in a directory named after the job ID.
5. Submits the job by calling submit Application() on the resource manager.
Job Initialization:
1. When the resource manager receives a call to its submit Application() method, it hands off the request
to the YARN scheduler.
2. The scheduler allocates a container and the resource manager then launches the application master's
process there, under the node manager's management.
3. The application masters for MapReduce jobs is a Java application whose main classis MR App Master.
4. It initializes the job by creating a number of bookkeeping objects to keep track of the job's progress,
as it will receive progress and completion reports from the tasks.
5. It retrieves the input splits computed in the client from the shared filesystem.
6. It then creates a map task object for each split, as well as a number of reduce task objects determined
by the map reduce. job. reduces property (set by the set Num Reduce Tasks() method on Job).
Task assignment:
1. If the job does not qualify for running as an uber task, then the application master requests containers
for the entire map and reduce tasks in the job from the resource manager.
2 Requests for map tasks are made first and with a higher priority than those for reduce tasks, since all
the map tasks must complete before the sort phase of the reduce can start.
3. Requests for reduce tasks are not made until 5 % of map tasks have completed.
Task execution:
1. Once a task has been assigned resources for a container on a particular node by the resource
manager's scheduler, the application master starts the container by contacting the node manager.
2. The task is executed by a Java application whose main class is Yarn Child. Before it can run the task,
it localizes the resources that the task needs, including the job configuration and JAR file and any files
from the distributed cache.
3. Finally, it runs the map or reduce task.
Streaming:
 Streaming runs special map and reduce tasks for the purpose of launching the user supplied executable
and communicating with it. The Streaming task communicates with the process using standard input and
output streams.

9
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
  From the node manager's point of view, it is as if the child process ran the map or reduces code itself.
Progress and status updates:
 MapReduce jobs are long running batch jobs, taking anything from tens of seconds to hours to run.
 During execution of the task, the Java process passes input key value pairs to the external process, which
runs it through the user defined map or reduce function and passes the output key value pairs back to the
Java process.
 A job and each of its tasks have a status, which includes such things as the state of the job or task, the
progress of maps and reduces the values of the job's counters and a status message or description.
 When a task is running, it keeps track of its progress. For map tasks, this is the proportion of the input
that has been processed. For reduce tasks, it's a little more complex, but the system can still estimate the
proportion of the reduce input processed.
Job completion:
 When the application master receives a notification that the last task for a job is complete, it changes the
status for the job to Successful. Then, when the Job polls for status, it learns that the job has completed
successfully, so it prints a message to tell the user and then returns from the wait For Completion().
 Finally, on job completion, the application master and the task containers clean up their working state
and the Output Committer's commit Job () method is called.
 Job information is archived by the job history server to enable later interrogation by users if desired.
[Link]
Q. Explain failures in classic MapReduce and YARN.
 YARN stands for yet another Resource Negotiator. It is the next generation computing framework in
Apache Hadoop with support for programming paradigms besides MapReduce. It is a large-scale
distributed operating system for big data applications.
 It has two major responsibilities:
1. Management of cluster resources such as compute, network and memory.
2. Scheduling and monitoring of jobs.
 YARN in Hadoop allows for the execution of various data processing engines such as batch processing,
graph processing, stream processing and interactive processing, as well as the processing of data stored
in HDFS.
[Link] is YARN Used?
a. YARN in Hadoop efficiently and dynamically allocates all cluster resources, resulting in higher
Hadoop utilization compared to previous versions which help in better cluster utilization.

10
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 b. Clusters in YARN in Hadoop can now run streaming data processing and interactive queries in
parallel with MapReduce batch jobs.
c. It can now handle several processing methods and can support wider range of applications.

YARN architecture

 YARN Architecture consists of the following main components:

a. Resource manager: Runs on a master daemon and manages the resource allocation in the cluster.
b. Node manager: They run on the slave daemons and are responsible for the execution of a task on
every single data node.
c. Application master manages the user job lifecycle and resource needs of individual applications.
It works along with the node manager and monitors the execution of tasks.
d. Container: Package of resources including RAM, CPU, Network, HDD etc on a single node.
 The resource manager and the node manager form the data-computation framework. The resource
manager is the ultimate authority that arbitrates resources among all the applications in the system. The
node manager is the per-machine framework agent who is responsible for containers, monitoring their
resource usage (cpu, memory, disk, network) and reporting the same to there source manager/scheduler.
 YARN containers are managed by a container launch context which is Container Life-Cycle (CLC).
This record contains a map of environment variables, dependencies stored in a remotely accessible
storage, security tokens, payload for node manager services and the command necessary to create the
process.
11
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  Application workflow in YARN:
a) Client submits an application
b) Resource manager allocates a container to start application manager
c) Application manager registers with resource manager
d) Application Manager asks containers from resource manager
e) Application Manager notifies node manager to launch containers
f) Application code is executed in the container
g) Client contacts resource manager/application manager to monitor application's status
h) Application manager unregisters with resource manager
Architecture features of YARN:
YARN has become popular for the following reasons -
a. With the scalability of the resource manager of the YARN architecture, Hadoop may
manage thousands of nodes and clusters.
b. YARN compatibility with Hadoop 1.0 is maintained by not affecting map-reduce applications.
c. Dynamic utilization of clusters in Hadoop is facilitated by YARN, which gives better
cluster utilization.
d. Multi-tenancy enables an organization to gain the benefits of multiple engine sat once.
Difference between YARN and MapReduce

YARN MapReduce
Used in Hadoop version 2 Used in Hadoop version 1
The Yarn has a name node, data node, Map Reduce has a name node, data node,
secondary name node, resource manager and secondary name node, job tracker and task
node anager. tracker.

HADOOP2, based on YARN architecture, has Map reduce has a single master and multiple
the concept of multiple masters and slaves slave architecture
Default node size of Data node is 128 MB Default node size of Data node is 64 MB
YARN is more isolated and scalable MapReduce is less scalable than YARN.
YARN can dynamically allocate pool of Provided static allocations of resources for
resources to applications designated work
Supports variety of processing engines and Supported its own batch processing
applications. applications only.

12
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
Merits and Demerits of YARN
1. Merits:
 Scalability: YARN is designed for large number of nodes.
 Utilization: Node manager manages a pool of resources, rather than a fixed number of the designated
slots thus increasing the utilization.
 Multitenancy: Different version of MapReduce can run on YARN, which makes the process of
upgrading MapReduce more manageable.
2. Demerit
Availability - JobTracker is the only point of availability in Hadoop 1.0
[Link] in Classic Map Reduce and YARN
1. Failures in classic MapReduce
MapReduce support three types of failures:
a) Running task failure
b) Task tracker failure
c) Job tracker failure
Task failure
 Child task fail: This happens when user code in the map or reduce task throws a runtime exception. If
this happens, the child JVM reports the error back to its parent task tracker, before it exits. The error
ultimately makes it into the user logs. The task tracker marks the task attempt as failed, freeing up a slot
to run another task.
 Streaming tasks fail: if the streaming process exits with a nonzero exit code, it is marked as failed. This
behavior is governed by the stream. [Link]. failure property.
 The task tracker notices that it hasn't received a progress update for a while and proceeds to mark the
task as failed. The child JVM process will be automatically killed after this period.
Task tracker failure
 If a task tracker fails by crashing, or running very slowly, it will stop sending heartbeats to the job
tracker. The job tracker will notice a task tracker that has stopped sending heartbeats and remove it from
its pool of task trackers to schedule tasks on.
 The job tracker arranges for map tasks that were run and completed successfully on that task tracker to
be rerun if they belong to incomplete jobs, since their intermediate output residing on the failed task
tracker's local filesystem may not be accessible to the reduce task. Any tasks in progress are also
rescheduled.

13
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  A task tracker can also be blacklisted by the job tracker, even if the task tracker has not failed. If more
than four tasks from the same job fail on a particular task tracker, then the job tracker records this as a
fault.
Job tracker failure
 It is most serious failure mode. It is a single point of failure. Hadoop has no mechanism for dealing with
failure of the job tracker.
 YARN is used to overcome this situation.
 After restarting a job tracker, any jobs that were running at the time it was stopped will need to be
re- submitted.
2. Failures in YARN
 Task failure: Failure of the running task is similar to the classic case. Runtime exceptions and sudden
exits of the JVM are propagated back to the application master and the task attempt is marked as failed.
 Node manager failure: If a node manager fails, then it will stop sending heartbeats to the resource
manager and the node manager will be removed from the resource manager's pool of available nodes.
Any task or application master running on the failed node manager will be recovered using the
mechanisms.
 Resource manager failure: Failure of the resource manager is serious, since without it neither jobs nor
task containers can be launched. The resource manager was designed from the outset to be able to
recover from crashes; by using a check pointing mechanism to save its state to persistent storage,
although at the time of writing the latest release did not have a complete implementation.
[Link] Scheduling
Q. Explain in details about different types of schedulers.
 The Hadoop schedulers are designed for better utilization of resources and performance enhancement.
Requirements (issues and challenges) regarding job scheduling in Hadoop are as follows:
 Energy efficiency: To perform operations on large amount of data, a large amount of energy is required
in data centers, which increase the overall cost. The minimization of energy in data centers is a big
challenge in Hadoop.
 Load balancing: Map and Reduce stages are linked with the partition stage. By default, the data is
equally portioned by partition algorithms, which handle the system imbalance in case skewed data is
encountered. As only the key is considered in processing and not the data size, load balancing problem
occurs.
 Mapping scheme: Mapping scheme that could be helpful in the minimization of communication cost is
required.

14
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  Automation and configuration: It help in the deployment of Hadoop cluster by setting all the
parameters. For proper configuration, both the hardware and work amount should be known at the
timing of deployment. During configuration, a small mistake could cause inefficient execution of the
job, leading to performance degradation. To overcome such types of issues, we need to develop
newtechniques and algorithms that perform calculations in such a manner that the setting could be done
efficiently.
 Fairness: Fairness refers to the fairness in scheduling algorithms. It indicates how fairly of scheduling
algorithms divide the resources among users.
 Data locality: The distance between the task node and input node is known as locality. The data transfer
rate depends on the locality. The data transfer time will be short if the computational node is near the
input node.
 Synchronization: The process of transferring the intermediate output data of the mapping process as the
input data to the reduce process is known assynchronization.
FIFO Scheduler
 Default scheduling policy used in Hadoop is First in First Out. More preferences aregiven to the
application coming first than those coming later. It places the applications in a queue and executes them
in the order of their submission (first in, first out).
 Here, irrespective of the size and priority, the requests for the first application in the queue are allocated
first. Once the first application request is satisfied, then only the next application in the queue is served.
Advantage of FIFO scheduler:
1. It is simple to understand and doesn't need any configuration.
2. It does not take into account the balance of resource allocation between the long applications and
short applications,
Disadvantage of FIFO scheduler
1. It is not suitable for shared clusters.
2. Jobs are executed in the order of their submission.
Fair Scheduler
 Fair scheduler is developed Facebook. Fair scheduler aims to give every user a fair share of the cluster
capacity over time. If a single job is running, it gets the entire cluster. As more jobs are submitted, free
task slots are given to the jobs in such away as to give each user a fair share of the cluster.
 The main idea behind fair scheduler is to allocate equal share of resources to each job. It creates groups
of jobs based on configurable attributes like user name, called pools.

15
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  The fair scheduler ensures fairness in sharing of resources between pools. The pools also control job
configurable properties and the configurable properties determine the pool in which a job is placed.
 All the users have their own pools with a minimum share assigned to each. Minimum share means that a
small part of the total number of slots is always achieved by a pool.
 By default, there is a fair allocation of resources among the pools with the MapReduce task slot. If any
pool is free i.e. they are not being used, then their idle slots will be used by the other pools.
 If the same user or same pool sends too many jobs, then the fair scheduler can limit these jobs by
marking the jobs as not runnable. If there is only a single job running at a given time, then it can use the
entire cluster.
Advantages of Fair scheduler
1. This scheduler makes a fair and dynamic resource reallocation.
2. It provides faster response to small jobs than large jobs.
3. It has the ability to fix the number of concurrent running jobs from each user and pool.
Disadvantages Fair scheduler
1. Fair scheduler has more complicated configurations.
2. This scheduler does not consider the weight of each job, which leads to unbalanced performance in
each pool/node.
3. Pools have a limitation on the number of running jobs under fair scheduling.
Capacity Scheduler
 Yahoo developed capacity scheduler. The main objective of this scheduler is to maximize the utilization
of resources and throughput in a cluster environment.
 This scheduling algorithm can ensure fair management of computational resources among a large
number of users. It uses queues and each queue will be assigned to an organization after the resources
have been divided among these queues.
 In order to get control over the queues, a security mechanism is built to ensure that each organization
can access only one of the queues. It can never access the queues of another organization.
 This scheduler guarantees minimum capacity by having limits on the running tasks and jobs from a
single queue.
 When new jobs arrive in a queue, the resources are assigned back to the previous queue after completion
of the currently running jobs. Capacity scheduler allows job scheduling based on priority in an
organization's queue.

16
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
Advantages of capacity scheduler
1. Capacity scheduling policy maximizes utilization of resources and throughput in cluster
environment.
2. This scheduler guarantees the reuse of the unused capacity of the jobs within queues.
3. It also supports the features of hierarchical queues, elasticity and operability.
4. It can allocate and control memory based on the available hardware resources.
Disadvantages of capacity scheduler
1. Capacity scheduler is the most complex among the other schedulers.
2. There is difficulty in choosing proper queues.
3. With regard to pending jobs, it has some limitations in ensuring stability and fairness of the
cluster from a queue and single user.
Difference between Fair and Capacity Scheduler

Fair scheduler Capacity scheduler

Fair scheduler is developed Facebook. Yahoo developed capacity scheduler.
Fair scheduler assigns equal amount of Capacity scheduler assigns resource based on
resource to all running jobs. the capacity required by the organization.
It support hierarchical XML configuration. It cannot support hierarchical XML
configuration.
Fair scheduler is not complex. It is complex amongst the other scheduler.
The main idea behind fair scheduler is to The main objective of this scheduler is to
allocate equal share of resources to each job. maximize the utilization of resources and
throughput in a cluster environment.

[Link] and Sort

Q. Explain details about process of shuffle and sort in MapReduce.
 Map-Reduce gives the guarantee that input to every reducer is sorted by key. Theprocess by which
system performs the sort and transfers the map outputs to thereducers as inputs are called shuffle.
 When we run a MapReduce job and mappers start producing output internallylots of processing is done
by the Hadoop framework before the reducers get theirinput. Hadoop framework also guarantees that the
map output is sorted by [Link] whole internal processing of sorting map output and transferring it
toreducers is known as shuffle phase in Hadoop framework.

17
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
 Shuffle and sort
 It is a phase which happens between each Map and Reduce phase. Just to remind Map and Reduce
handles the data which are organized into key-value pairs. Once the Mappers are done with the
calculations, the results of each Mapper are sorted by the key in so called buffers.
 Once the buffers are filled up, the data spills on the machines' local disks. Then itis straightly pushed to
the Reducers (the shuffle phase), so that records associated with the same key end up in the same
Reducer.
 This shuffling happens as soon as each Mapper finishes in order not to over-flood the network, which
could happen if we waited for all of the Mappers to finish.
 Depending on the setup Reducers may start running before all Mappers have finished their jobs. When
data gets to Reducers it is sorted once again. Results are then written to the HDFS or any other used
filesystem.
 In order to reduce the amount of data shuffled through the network we may define a combiner function,
which does the same calculations as Reducer but on the Mappers' side, so before the data is being
transferred.

18
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  Role of the combiner is to simply pre-calculate the data so that we send less over the network. However
there is no way of controlling how many times and if at all combiner will actually be used - Hadoop
decides on that.
 Hadoop has a default shuffle and sort mechanism which is based on alphabetical sorting and hash
shuffling of the keys. However, there is a way of implementing a custom mechanism by overwriting the
following classes:
1. Partitioner - According to which the data will be shuffled.
2. Raw Comparator - Responsible for data sorting on the Mapper side.
3. Raw Comparator - Which handles the data grouping on the Reducer side.
 The tasks done internally by Hadoop framework within the shuffle phase are as follows -
1. Data from mappers is partitioned as per the number of reducers.
2. Data is also sorted by keys within a partition.
3. Output from Maps is written to disk as many temporary files.
4. Once the map task is finished all the files written to the disk are merged to create a single file.
5. Data from a particular partition (from all mappers) is transferred to a reducer that is supposed to
process that particular partition.
6. If data transferred to a reducer exceeded the memory limit, then it is copied toa disk.
7. Once reducer has got its portion of data from all the mappers, data is again merged while still
maintaining the sort order of keys to create reduces task input.

[Link] Execution
Q. Give a short note on Task execution.
 MapReduce model is to break jobs into tasks and run the tasks in parallel to make the overall job
execution time smaller than it would otherwise be if the tasks ran sequentially.
 In Hadoop, speculative execution is a process that takes place during the slower execution of a task at a
node. In this process, the master node starts executing another instance of that same task on the other
node. And the task which is finished first is accepted and the execution of other is stopped by killing
that.
Speculative execution in Hadoop
 Speculative execution in Hadoop MapReduce is an option to run a duplicate mapor reduce task for the
same input data on an alternative node. This is done so that any slow running task doesn't slow down the
whole job.

19
[Link]., AP/CSE. ., VRSCET. CCS334-Big Data
Analytics
  MapReduce job is dominated by the slowest task. MapReduce attempts to locates low tasks, called
stragglers. If a straggler is discovered, a redundant (speculative) task is run that will optimistically
commit before the corresponding straggler.
 Whichever copy (among the two copies) of a task commits first, it becomes the definitive copy and the
other copy is killed by the Job tracker. This process is known as speculative execution. Only one copy of
a straggler is allowed to be speculated.
 How does Hadoop locate stragglers? Hadoop monitors each task progress using a progress score
between 0 and 1. If a task's progress score is less than average and the task has run for at least 1 minute,
it is marked as a straggler.
 Speculative execution is turned on by default. It can be enabled or disabled independently for map tasks
and reduce tasks, on a cluster-wide basis or on aper-job basis.
 Speculative execution is enabled by default for both map and reduces tasks. Properties for speculative
execution are set in map red - [Link] file.
 Advantages of speculative execution: In many of the production environments, we can have large-
scale clusters with thousands of nodes running MapReduce or the related Hadoop jobs at the same time.
Problems like hardware failure and network clusters are common in large scale clusters. So, it makes
sense to run duplicate tasks in case one server might fail.
[Link] Types
Q. Discuss in details about types of MapReduce
 The map and reduce functions in Hadoop MapReduce have the following general form:
map: (K1, V1) → list (K2, V2)
reduco: (K2, list (V2))→list (K3, V3)
 In general, the map input key and value types (K1 and V1) are different from the map output types (K2
and V2). However, the reduce input must have the same types as the map output, although the reduce
output types may be different again (K3 and V3).
[Link] Formats - Output
1. Input formats
 Hadoop can process many different types of data formats, from flat text files to databases.
 An input split is a chunk of the input that is processed by a single map. Each map processes a single
split. Each split is divided into records and the map processes each record, a key-value pair.
 Splits and records are logical there is nothing that requires them to be tied to files. In a database context,
a split might correspond to a range of rows from a table and a record to a row in that range. Input splits
are represented by the Javainterfacei.e.,InputSplit.

21
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
 An Input Split has a length in bytes and a set of storage locations, which are just hostname strings. An Input
Format is responsible for creating the input splits and dividing them into records.
 File Input Format: File Input Format is the base class for all implementations ofInput Format that use
files as their data source. It provides two things: A place to define which files are included as the input to
a job and an implementation for generating splits for the input files. The job of dividing splits into
records is performed by subclasses.
 Small files and Combine FileInput Format: Hadoop work better with a small number of large files than a
large number of small files. One reason for this is that FileInput Format generates splits in such a way
that each split is all or part of a single file. If the file is very small and there are a lot of them, then each
map task will process very little input and there will be a lot of them (one per file), each of which
imposes extra bookkeeping overhead.
 An InputFormat object creates the input splits that are delegated by the context to get the records. It is
the public and customizable run() method of the Mapper to get the records from context.
 FileInputFormat and DBInput Format classes are derived from InputFormat. The FileInputFormat is
further specialized, with classes that i.e., combine small files or prevent file splitting
2. Text input
 TextInputFormat: This file format is the default Input Format. Each record is a line of input. The key, a
LongWritable, is the byte offset within the file of the beginning of the line. The value is the contents of
the line, excluding any line terminators (newline, carriage return) and is packaged as a Text object.
 TextInput Format is the default Input Format. Each record is a line of input. The key, a Long Writable,
is the byte offset within the file of the beginning of the line. The value is the contents of the line,
excluding any line terminators (e.g., newline, carriage return) and is packaged as a Text object.
 A file is broken into splits at byte, not line, boundaries. Splits are processed independently.
Relationship between input splits and HDFS blocks:
 The logical records that FileInput Formats define usually do not fit neatly into HDFS blocks. For
example, a TextInput Format's logical records are lines, which will cross HDFS boundaries more often
than not.
 This has no bearing on the functioning of our program: Lines are not missed orbroken.
 This means it could be needed to perform some remote reads. The slight overhead these causes are not
normally significant.
 A single file is broken into lines and the line boundaries do not correspond with the HDFS block
boundaries. Splits honor logical record boundaries, in this case lines, so we see that the first split
contains line 5, even though it spans the first and second block. The second split starts at line 6.

22
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
Binary input:
 Hadoop MapReduce is not restricted to processing textual data. It has support forbinary formats, too.

Logical records and HDFSblocks for TextInput Format

 Hadoop's Sequence FileInputFormat stores sequences of binary key-value pairs. Sequence files are
splittable, they support compression as a part of the format and they can store arbitrary types.
 Sequence File As Binary Input Format is a variant of Sequence FileInput Format that retrieves the
sequence file's keys and values as opaque binary objects. They are encapsulated as Bytes Writable
objects and the application is free to interpret the underlying byte array.

Illustrate the application of MapReduce by providing detailed explanations of two instances.

Nov/Dec-2023
 Entertainment: To discover the most popular movies, based on what you like and what you watched in
this case Hadoop MapReduce helps you out. It mainly focuses on their logs and clicks.
 E-commerce: Numerous E-commerce suppliers, like Amazon, Walmart, and eBay, utilize the
MapReduce programming model to distinguish most loved items dependent on clients’ inclinations or
purchasing behavior.
 It incorporates making item proposal Mechanisms for E-commerce inventories, examining website
records, buy history, user interaction logs, etc.
 Data Warehouse: We can utilize MapReduce to analyze large data volumes in data warehouses while
implementing specific business logic for data insights.
 Fraud Detection: Hadoop and MapReduce are utilized in monetary enterprises, including organizations
like banks, insurance providers, and installment areas for misrepresentation recognition, pattern
distinguishing proof, or business metrics through transaction analysis.

List the Relational-Algebra operations. Illustrate the applications of MapReduce by providing detailed
explanations of two instances-Nov/Dec -2023
Before getting a brief overview of relational algebra we need to know what a relation represents. As most of us
are already familiar with SQL there is no point on putting a long description here. A relation represents a

23
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
database table. A major point that distinguishes SQL and relational algebra is that in relational algebra duplicate
rows are implicitly eliminated which is not the case with SQL implementations. Here in this article
implementation of relational algebra operations is discussed, but it’s easily generalizable to the implementations
that don’t eliminate duplicates. If a reader is familiar with relational algebra they can just skim over this section.
Selection: selection(WHERE clause in SQL) lets you apply a condition over the data you have and only get the
rows that satisfy the condition.

Selection operation to select only rows where age is greater than 20

Projection: In order to select some columns only we use the projection operator. It’s analogous to SELECT in
SQL.

Projection operation to select only Name, IsActive column

Union: We concatenate two tables vertically. Similar to UNION in SQL, but the duplicate rows are removed
implicitly. The point to note in the below output table is that (Smith, 16)was a duplicate row so it appears only
once in the output where as (Tom, 17) , (Tom, 19) appears as two, as those are not identical rows.

24
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
Union the two tables
Intersection: Same as INTERSECT in SQL. It intersects two tables and selects only the common rows.

Intersection of two tables

Difference: The rows that are in the first table but not in second are selected for output. Keep in mind that
(Monty, 21) is not considered in the output as its present in the second table but not in first.

Difference between the two tables

25
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
Natural Join: Merge two tables based on some common column. It represents the INNER JOIN in SQL. But the
condition is implicit on the column that is common in both tables. The output will only contain rows for which
the values in the common column match. It will generate one row for every time the column value across two
tables match as is the case below for Tom . If there were multiple Tom values in the table on the top in the
image below, then four rows would have been created in the output table representing all the combinations.

Natural Join With column Name as the common column

Grouping and Aggregation: Group rows based on some set of columns and apply some aggregation (sum, count,
max, min, etc.) on some column of the small groups that are formed. This corresponds to GROUP BY in SQL.

Group By Name and take sum of the Winning

[Link] between big data processing and distributed processing
Big data processing refers to computing against large data sets, in particular using certain mathematical
techniques. Distributed processing refers to computing on devices at various distinct locations, such as a
number of computers on the internet.

26
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
Two Marks Questions with Answers
Q.1 Define MapReduce.
Ans.:MapReduce is a programming model and software framework first developed by Google. Intended to
facilitate and simplify the processing of vast amounts of data in parallel on large clusters of commodity
hardware in a reliable, fault-tolerant manner.
Q.2 List the characteristics of MapReduce?
Ans. Characteristics of MapReduce
1. Very large-scale data: peta, exa bytes.
2. Write once and read many data. It allows for parallelism without mutexes.
3. Map and Reduce are the main operations: Simple code.
4. All the map should be completed before reduce operation starts.
5. Map and reduce operations are typically performed by the same physical
6. Number of map tasks and reduce tasks are configurable.
7. Operations are provisioned near the data.
8. Commodity hardware and storage.
Q3 what are the major responsibilities of YARN?
Ans. It has two major responsibilities:
 Management of cluster resources such as compute, network and memory.
 Scheduling and monitoring of jobs.

Q.4 Why is YARN Used?

Ans.
a. YARN in Hadoop efficiently and dynamically allocates all cluster resources, resulting in higher Hadoop
utilization compared to previous versions which help in better cluster utilization.
b. Clusters in YARN in Hadoop can now run streaming data processing and interactive queries in parallel
with MapReduce batch jobs.
c. It can now handle several processing methods and can support wider range of applications.
Q.5What is fair scheduler?
Ans. Fair scheduler aims to give every user a fair share of the cluster capacity overtime. If a single job is
running, it gets the entire cluster. As more jobs are submitted, free task slots are given to the jobs in such a way
as to give each user a fair share of the cluster.
Q.6 List the failures of MapReduce.
Ans.:MapReduce support three types of failures Running task failure, Tasktracker failure and Job tracker failure.

27
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics
Q.7 Explain First In First Out (FIFO) scheduling.
Ans. FIFO scheduling policy gives more preference to the jobs coming in earlier than those coming in later.
When new jobs arrive, the Job Tracker pulls the earliest job first from the queue.
Q.B Why Hadoop works better with a small number of large files?
Ans. Hadoop works better with a small number of large files than a large number of small files. One reason for
this is that FileInput Format generates splits in such a way that each split is all or part of a single file. If the file
is very small and there are a lot of them, then each map task will process very little input and there will be a lot
of them (one per file), each of which imposes extra bookkeeping overhead.
Q.9 What is TextinputFormat?
Ans. TextInput Format is the default Input Format. Each record is a line of input. The key, a Long Writable, is
the byte offset within the file of the beginning of the line. The value is the contents of the line, excluding any
line terminators (e.g., newline or carriage return) and is packaged as a Text object. A file is broken into splits at
byte, not line, boundaries. Splits are processed independently.
Q.10 What is Node Manager Failure in YARN?
Ans. If a node manager fails, then it will stop sending heartbeats to the resource manager and the node manager
will be removed from the resource manager's pool of available nodes. Any task or application master running
on the failed node manager will be recovered using the mechanisms.

28
[Link]., AP/CSE. ., SRRCET. CCS334-Big Data
Analytics