Module-3:

MapReduce
 Module-3
Map Reduce: Anatomy of a Map Reduce Job Run(1), Failures,(1) Job
Scheduling, Shuffle and Sort, Task Execution,(2) Map Reduce Types and
Formats(3), Map Reduce Features(4).
Hadoop Ecosystem And Yarn: Hadoop ecosystem components-SPARK,
FLUME (5), Hadoop 2.0 New Features- Name Node, High Availability,
HDFS Federation, MRv2, YARN(6)
 Anatomy of a MapReduce Job Run
• You can run a MapReduce job with a single method call: submit() on a Job
object.
• In releases of Hadoop up to and including the 0.20 release series,
[Link] determines the means of execution. If this
configuration property is set to local, the default, then the local job runner
is used. This runner runs the whole job in a single JVM.
• It’s designed for testing and for running MapReduce programs on small
datasets.
• Alternatively, if [Link] is set to a colon-separated host and
port pair, then the property is interpreted as a jobtracker address, and the
runner submits the job to the jobtracker at that address.
• In Hadoop 0.23.0 a new MapReduce implementation was introduced. The
new implementation (called MapReduce 2) is built on a system called
YARN.
• The framework that is used for execution is set by the
[Link] property, which takes the values local (for
the local job runner), classic (for the “classic” MapReduce framework,
also called MapReduce 1, which uses a jobtracker and tasktrackers) and
yarn (for the new framework).
Classic MapReduce (MapReduce 1) :
• A job run in classic MapReduce is illustrated in Figure 6-1. At the highest level, there
are four independent entities:
1. The client, which submits the MapReduce job.
2. The jobtracker, which coordinates the job run. The jobtracker is a Java application
whose main class is JobTracker.
[Link] tasktrackers, which run the tasks that the job has been split into. Tasktrackers are
Java applications whose main class is TaskTracker.
4. The distributed filesystem(normally HDFS), which is used for sharing job files between
the other entities.
[Link] Submission:
Step1:On Job, The submit() method creates an internal JobSummitter
instance and calls submitJobInternal() on [Link] submitted the job,
waitForCompletion() polls the job’s progress once a second and reports the
progress to the console. When the job is complete, if it was successful,the
job counters are displayed. Otherwise, the error that caused the job to fail is
logged to the console.
The job submission process implemented by JobSummitter does the
following:
Step2:Asks the jobtracker for a new job ID (by calling getNewJobId() on JobTracker).
•Computes the input splits for the job. If the splits cannot be computed, because
• the input paths don’t exist, for example, then the job is not submitted and an error is
thrown to the MapReduce program.
•Checks the output specification of the job. For example, if the output directory has not
been specified or it already exists, the job is not submitted and an error is thrown to the
MapReduce program.
Step3:Copies the resources needed to run the job, including the job JAR
file, the configuration file, and the computed input splits, to the jobtracker’s
filesystem in a directory named after the job ID.
•The job JAR is copied with a high replication factor (controlled by the
[Link] property, which defaults to10) so that there are
lots of copies across the cluster for the tasktrackers to access when they
run tasks for the job.

Step4:Tells the jobtracker that the job is ready for execution (by calling
submitJob() onJobTracker).
[Link] Initialization:
Step5: When the JobTracker receives a call to its submitJob() method,
it puts it into an internalqueue from where the job scheduler will pick it
up and initialize it. Initialization involves creating an object to represent the
job being run, which encapsulates its tasks, and bookkeeping information to
keep track of the tasks’ status and progress.
Step6: To create the list of tasks to run, the job scheduler first retrieves the input
splits computed by the client from the shared filesystem . It then creates one map task
for each split. The number of reduce tasks to create is determined by the
[Link] property in the Job, which is set by the
setNumReduceTasks()method, and the scheduler simply creates this number of reduce
tasks to be run. Tasks are given IDs at this point.
• In addition to the map and reduce tasks, two further tasks are created: a job setup
task and a job cleanup task. These are run by tasktrackers and are used to run code
to setup the job before any map tasks run, and to cleanup after all the reduce tasks are
complete.
[Link] Assignment:
Step 7:Tasktrackers run a simple loop that periodically sends heartbeat method
calls to the jobtracker. Heartbeats tell the jobtracker that a tasktracker is alive.
• As a part of the heartbeat, a tasktracker will indicate whether it is ready to run a
new task, and if it is, the jobtracker will allocate it a task, which it communicates
to the tasktracker using the heartbeat return value.
[Link] Execution:
Step8:The tasktracker has been assigned a task, the next step is for it to run the
task.
• First, it localizes the job JAR by copying it from the shared filesystem to the
tasktracker’s filesystem.
• Second, it creates a local working directory for the task, and un-jars the contents
of the JAR into this directory.
• Third, it creates an instance of TaskRunner to run the task.
Step9:TaskRunner launches a new Java Virtual Machine to run each task in (step
10).
Streaming and Pipes: Both Streaming and Pipes run special map and
reduce tasks for the purpose of launching the user-supplied executable and
communicating with it .
• In the case of Streaming, the Streaming task communicates with the
process (which may be written in any language) using standard input and
output streams.
• The Pipes task, on the other hand, listens on a socket and passes the
C++ process a port number in its environment, so that on startup, the C++
process can establish a persistent socket connection back to the
parent Java Pipes task.
• In both cases, during execution of the task, the Java process passes input
key-value pairs to the external process, which runs it through the user-
defined map or reduce function and passes the output key-value pairs
back to the Java process. From the tasktracker’s point of view, it is as if
the tasktracker child process ran the map or reduce code itself.
Progress and Status Updates:
• MapReduce jobs are long-running batch jobs, taking anything from
minutes to hours to run.
• A job and each of its tasks have a status, which includes such things as
the state of the job or task (e.g., running, successfully completed, failed),
the progress of maps and reduces, the values of the job’s counters, and a
status message or description (which may be set by user code). These
statuses change over the course of the job, so how do they get
communicated back to the client.
• When a task is running, it keeps track of its progress, that is, the
proportion of the task completed. For map tasks, this is the proportion of
the input that has been processed. For reduce tasks, it’s a little more
complex, but the system can still estimate the proportion of the reduce
input processed. It does this by dividing the total progress into three parts,
corresponding to the three phases of the shuffle.
• If a task reports progress, it sets a flag to indicate that the status change
should be sent to the tasktracker. The flag is checked in a separate thread
every three seconds, and if set it notifies the tasktracker of the current task
status. Meanwhile, the tasktracker is sending heartbeats to the jobtracker
every five seconds (this is a minimum, as the heartbeat interval is actually
dependent on the size of the cluster: for larger clusters, the interval is
longer), and the status of all the tasks being run by the tasktracker is sent
in the call. Counters are sent less frequently than every five seconds,
because they can be relatively high-bandwidth.
• The jobtracker combines these updates to produce a global view of the
status of all the jobs being run and their constituent tasks. Finally, as
mentioned earlier, the Job receives the latest status by polling the
jobtracker every second. Clients can also use Job’s getStatus() method to
obtain a JobStatus instance, which contains all of the status information for
the job.
YARN (MapReduce 2):
• For very large clusters in the region of 4000 nodes and higher, the
MapReduce system described in the previous section begins to hit
scalability bottlenecks, so in 2010 a group at Yahoo! began to design the
next generation of MapReduce.
• The result was YARN, short for Yet Another Resource Negotiator or YARN
Application Resource Negotiator).
• YARN meets the scalability shortcomings of “classic” MapReduce by
splitting the responsibilities of the jobtracker into separate entities. The
jobtracker takes care of both job scheduling (matching tasks with
tasktrackers) and task progress monitoring (keeping track of tasks and
restarting failed or slow tasks, and doing task bookkeeping such as
maintaining counter totals).
• YARN separates these two roles into two independent daemons: a
resource manager to manage the use of resources across the cluster,
and an application master to manage the lifecycle of applications
running on the cluster.
• In contrast to the jobtracker, each instance of an application—here a
MapReduce job —has a dedicated application master, which runs for the
duration of the application. This model is actually closer to the original
Google MapReduce paper, which describes how a master process is
started to coordinate map and reduce tasks running on a set of workers.
• it is even possible for users to run different versions of MapReduce on the
same YARN cluster, which makes the process of upgrading MapReduce
more managable.
• MapReduce on YARN involves more entities . They are:
1. The client, which submits the MapReduce job.
2. The YARN resource manager, which coordinates the allocation of compute
resources on the cluster.
3. The YARN node managers, which launch and monitor the compute containers on
machines in the cluster.
4. The MapReduce application master, which coordinates the tasks running the
MapReduce job. The application master and the MapReduce tasks run in containers
that are scheduled by the resource manager, and managed by the node managers.
5. The distributed filesystem (normally HDFS), which is used for sharing job files
between the other entities.
[Link] Submission
Step1:Jobs are submitted in MapReduce 2 using the same user API as
MapReduce 1.
MapReduce 2 has an implementation of ClientProtocol that is activated
when [Link] is set to yarn.
Step2:The new job ID is retrieved from the resource manager , although in
the nomenclature of YARN it is an application ID.
Step3:The job client checks the output specification of the job, computes
input splits (although there is an option to generate them on the cluster,
[Link]-splits-in-cluster, which can be
beneficial for jobs with many splits); and copies job resources (including
the job JAR, configuration, and split information) to HDFS.
Step4:. Finally, the job is submitted by calling submitApplication() on
the resource manager.
[Link] Initialization
Step5:When the resource manager receives a call to its submitApplication(),
it hands off the request to the scheduler. The scheduler allocates a
container, and the resource manager then launches the application
master’s process there, under the node manager’s management
(steps 5a and 5b).
• Step6:The application master for MapReduce jobs is a Java application
whose main class is MRAppMaster.
• It initializes the job by creating a number of bookkeeping objects to keep
track of the job’s progress, as it will receive progress and completion
reports from the tasks.
Step7: It retrieves the input splits computed in the client from the
shared filesystem.
•It then creates a map task object for each split, and a number of reduce
task objects determined by the [Link] property.
•The next thing the application master does is decide how to run the
tasks that make up the MapReduce job.
• If the job is small, the application master may choose to run them in the
same JVM as itself, since it judges the overhead of allocating new
containers and running tasks in them as outweighing the gain to be had in
running them in parallel, compared to running them sequentially on one
node. (This is different to MapReduce 1, where small jobs are never run on
a single tasktracker.) Such a job is said to be uberized, or run as an uber
task.
[Link] Assignment
Step8:If the job does not qualify for running as an uber task, then the
application master requests containers for all the map and reduce tasks in
the job from the resource manager.
• Each request, which are piggybacked on heartbeat calls, includes
information about each map task’s data locality, in particular the hosts and
corresponding racks that the input split resides on.
• The scheduler uses this information to make scheduling decisions : it
attempts to place tasks on data-local nodes in the ideal case, but if this is
not possible the scheduler prefers rack-local placement to non-local
placement.
• Requests also specify memory requirements for tasks. By default both
map and reduce tasks are allocated 1024 MB of memory, but this is
configurable by setting [Link] and
[Link].
[Link] Execution:

Step9:Once a task has been assigned a container by the resource manager’s scheduler, the
application master starts the container by contacting the node manager.

Step10:The task is executed by a Java application whose main class is YarnChild. Before it can
run the task it localizes the resources that the task needs, including the job configuration and
JAR file, and any files from the distributed cache.

Step11: Finally, it runs the map or reduce task.

•The YarnChild runs in a dedicated JVM, for the same reason that tasktrackers spawn new JVMs
for tasks in MapReduce 1: to isolate user code from long-running system daemons. Unlike
MapReduce 1, however, YARN does not support JVM reuse so each task runs in a new JVM.
Progress and Status Updates:
• When running under YARN, the task reports its progress and status
(including counters) back to its application master every three seconds
(over the umbilical interface), which has an aggregate view of the job.
 Topic: Failures
• In the real world, user code is buggy, processes crash, and machines fail.
One of the major benefits of using Hadoop is its ability to handle such
failures and allow your job to complete.
Failures in Classic MapReduce
• In the MapReduce 1 runtime there are three failure modes to consider:
[Link] of the running task
[Link] of the tastracker and
[Link] of the jobtracker.
1. Task Failure
a. Child Task Failure: Consider first the case of the child task failing. The most common
way that this happens is when user code in the map or reduce task throws a runtime
exception. If this happens, the child JVM reports the error back to its parent
tasktracker, before it exits. The error ultimately makes it into the user logs. The
tasktracker marks the task attempt as failed, freeing up a slot to run another task.
b. Streaming Task Failure: For Streaming tasks, if the Streaming process exits with
a nonzero exit code, it is marked as failed. This behavior is governed by the
[Link] property (the default is true).
c. Sudden Exit of Child JVM: There is a JVM bug that causes the JVM to exit for a
particular set of circumstances exposed by the MapReduce user code. In this case, the
tasktracker notices that the process has exited and marks the attempt as failed.
d. Hanging Tasks: The tasktracker notices that it hasn’t received a progress update for
a while and proceeds to mark the task as failed. The child JVM process will be
automatically killed after this period. The timeout period after which tasks are considered
failed is normally 10 minutes and can be configured on a per-job basis (or a cluster
basis) by setting the [Link] property to a value in milliseconds.
[Link] in Streaming or Pipes: If a Streaming or Pipes process hangs, the
tasktracker will kill it in one the following circumstances:
• Either [Link]-controller is set to
[Link], or the default task controller is being
used and the setsid command is available on the system. In any other case orphaned
Streaming or Pipes processes will accumulate on the system, which will impact
utilization over time.
f. Setting the timeout to a value of zero disables the timeout, so long-
running tasks are never marked as failed. In this case, a hanging task
will never free up its slot, and over time there may be cluster slowdown as a
result. This approach should therefore be avoided, and making sure that a
task is reporting progress periodically will suffice.
g. Task attempt Failure: When the jobtracker is notified of a task attempt
that has failed (by the tasktracker’s heartbeat call), it will reschedule
execution of the task. The jobtracker will try to avoid rescheduling the task
on a tasktracker where it has previously failed. Furthermore, if a task fails
four times (or more), it will not be retried further. This value is configurable:
the maximum number of attempts to run a task is controlled by the
[Link] property for map tasks and
[Link] for reduced tasks. By default, if any task fails
four times, the whole job fails.
[Link] Failure
• Failure of a tasktracker is another failure mode.
• If a tasktracker fails by crashing, or running very slowly, it will stop sending heartbeats
to the jobtracker (or send them very infrequently). The jobtracker will notice a
tasktracker that has stopped sending heart beats (if it hasn’t received one for 10
minutes, configured via the [Link] [Link] property, in
milliseconds) and remove it from its pool of tasktrackers to schedule tasks on.
• The jobtracker arranges for map tasks that were run and completed successfully on
that tasktracker to be rerun if they belong to incomplete jobs, since their intermediate
output residing on the failed tasktracker’s local filesystem may not be accessible to the
reduce task. Any tasks in progress are also rescheduled.
• A tasktracker can also be blacklisted by the jobtracker, even if the tasktracker has
not failed. If more than four tasks from the same job fail on a particular tasktracker (set
by ([Link]), then the jobtracker records this as a fault. A
tasktracker is blacklisted if the number of faults is over some minimum threshold (four,
set by [Link]) and is significantly higher than the average
number of faults for tasktrackers in the cluster cluster.
• Blacklisted tasktrackers are not assigned tasks, but they continue to
communicate with the jobtracker. Faults expire over time (at the rate of
one per day), so tasktrackers get the chance to run jobs again simply by
leaving them running. Alternatively, if there is an underlying fault that can
be fixed (by replacing hardware, for example), the task tracker will be
removed from the jobtracker’s blacklist after it restarts and rejoins the
cluster.
[Link] Failure
• Failure of the jobtracker is the most serious failure mode. Hadoop has no mechanism
for dealing with failure of the jobtracker—it is a single point of failure—so in this case
the job fails. However, this failure mode has a low chance of occurring, since the
chance of a particular machine failing is low. The good news is that the situation is
improved in YARN, since one of its design goals is to eliminate single points of failure
in Map Reduce. After restarting a jobtracker, any jobs that were running at the time it
was stopped will need to be re-submitted. There is a configuration option that attempts
to recover any running jobs ([Link], turned off by default),
however it is known not to work reliably, so should not be used.
Failures in YARN
• For MapReduce programs running on YARN, we need to consider the
failure of any of the following entities:
[Link] task
[Link] application master
[Link] node manager, and
[Link] resource manager.
1. Task Failure
• Failure of the running task is similar to the classic case. Runtime
exceptions and sudden exits of the JVM are propagated back to the
application master and the task attempt is marked as failed. Likewise,
hanging tasks are noticed by the application master by the absence of a
ping over the umbilical channel (the timeout is set by [Link]
out), and again the task attempt is marked as failed.
• The configuration properties for determining when a task is considered to
be failed are the same as the classic case: a task is marked as failed
after four attempts (set by [Link] for map tasks
and [Link] for re ducer tasks). A job will be
failed if more than [Link] percent of the
map tasks in the job fail, or more than [Link]
cent percent of the reduce tasks fail.
2. Resource Manager Failure:
• Resource Manager is the single point of failure in YARN. To achieve high
availability (HA), it is necessary to run a pair of resource managers in an
active-standby configuration. If the active resource manager fails, then the
standby can take over without a significant interruption to the client.
• Information about all the running applications is stored in a highly available
state store (backed by ZooKeeper or HDFS), so that the standby can
recover the core state of the failed active resource manager. Node
manager information is not stored in the state store since it can be
reconstructed relatively quickly by the new resource manager as the node
managers send their first heartbeats.
• The transition of a resource manager from standby to active is handled by
a failover controller. The default failover controller is an automatic one,
which uses ZooKeeper leader election to ensure that there is only a single
active resource manager at one time.
[Link] failures
• If an Application Master fails in Hadoop, then all the information related to that
particular job execution will be lost. By default the maximum number of attempts to run
an application master is 2, so if an application master fails twice it will not be tried again
and the job will fail. This can be controlled using the property
[Link]-attempts.
• An application master sends periodic heartbeats to the RM, and in the event of application master
failure, the RM will detect the failure and starts a new instance of the master running in a new container,
it will use the job history to recover the state of the tasks that were already run by the (failed) application
so they don’t have to be [Link] is enabled by default, but can be disabled by setting
[Link] to false.
• The MapReduce client polls the application master for progress reports, but if its application master
fails, the client needs to locate the new instance. During job initialization, the client asks the resource
manager for the application master’s address, and then caches it so it doesn’t overload the resource
manager with a request every time it needs to poll the application master. If the application master
fails, however, the client will experience a timeout when it issues a status update, at which point
the client will go back to the resource manager to ask for the new application master’s address.
[Link] Manager Failure
• If a node manager fails, then it will stop sending heartbeats to the resource
manager, and the node manager will be removed from the resource
manager’s pool of available nodes.
• The property [Link]-interval
ms, which defaults to 600000 (10 minutes), determines the minimum time
the resource manager waits before considering a node manager that has
sent no heartbeat in that time as failed.
 Topic: Job Scheduling
• Early versions of Hadoop had a very simple approach to scheduling users’
jobs: they ran in order of submission, using a FIFO scheduler. Typically,
each job would use the whole cluster, so jobs had to wait their turn.
• Later on, the ability to set a job’s priority was added, via the
[Link] property or the setJobPriority() method on JobClient
(both of which take one of the values VERY_HIGH, HIGH, NORMAL,
LOW, VERY_LOW). When the job scheduler is choosing the next job to
run, it selects one with the highest priority.
• However, with the FIFO scheduler, priorities do not support preemption, so
a high-priority job can still be blocked by a long-running low priority job that
started before the high-priority job was scheduled.
• MapReduce in Hadoop comes with a choice of schedulers. The default in
MapReduce 1 is the original FIFO queue-based scheduler, and there
are also multiuser schedulers called the Fair Scheduler and the Capacity
Scheduler.
• MapReduce 2 comes with the Capacity Scheduler (the default), and the
FIFO scheduler.
a. FIFO Scheduler:
• As the name suggests FIFO i.e. First In First Out, therefore the tasks or application that
comes first are going to be served first.
• This is the default Scheduler we use in Hadoop.
• The tasks are placed during a queue and therefore the tasks are performed in their
submission order.
• In this method, once the work is scheduled, no intervention is allowed.
• So sometimes the high priority process has got to wait an extended time since the
priority of the task doesn't matter during this method.
b. The Capacity Scheduler:
• In Capacity Scheduler we've multiple job queues for scheduling our
[Link] Capacity Scheduler allows multiple occupants to share an
outsized size Hadoop
cluster.
• In the Capacity Scheduler corresponding for every job queue, we offer some
slots or cluster resources for performing job operations. Each job queue
has its own slots to perform its task. just in case we've tasks to perform in
just one queue then the tasks of that queue can access the slots of other
queues also as they're liberal to use, and when the new task enters to
another queue then jobs in running in its own slots of the cluster are
replaced with its own job.
• Capacity Scheduler also provides A level of abstraction to understand which
occupant is utilizing the more cluster resource or slots, so that the only user or
application doesn’t take disappropriate or unnecessary slots within the cluster.
• The capacity Scheduler mainly contains 3 sorts of the queue that are root,
parent, and leaf which are wont to represent cluster, organization, or any
subgroup, application submission respectively.
c. Fair Scheduler
• The Fair Scheduler is very much similar to that of the capacity scheduler.
The priority of the job is kept in consideration.
• With the help of Fair Scheduler, the YARN applications can share the
resources in the large Hadoop Cluster and these resources are
maintained dynamically so no need for prior capacity.
• The resources are distributed in such a manner that all applications within
a cluster get an equal amount of time.
• Fair Scheduler takes Scheduling decisions based on memory, we can
configure it to work with CPU also.
• As we told you it is similar to Capacity Scheduler but the major thing to
notice is that in Fair Scheduler whenever any high priority job arises
in the same queue, the task is processed in parallel by replacing
some portion from the already dedicated slots.
 Topic: Shuffle and Sort
• MapReduce makes the guarantee that the input to every reducer is sorted
by key. The process by which the system performs the sort—and transfers
the map outputs to the reducers as inputs—is known as the shuffle.
The Map Side:
• When the map function starts producing output, it is not simply written to disk.
The process is more involved, and takes advantage of buffering writes in
memory and doing some presorting for efficiency reasons.
• Each map task has a circular memory buffer that it writes the output to. The
buffer is 100 MB by default, a size which can be tuned by changing the
[Link] property.
• When the contents of the buffer reaches a certain threshold size ([Link]
cent, default 0.80, or 80%), a background thread will start to spill the
contents to disk. Map outputs will continue to be written to the buffer while the
spill takes place, but if the buffer fills up during this time, the map will block until
the spill is complete. Spills are written in round-robin fashion to the directories
specified by the [Link] property, in a job-specific subdirectory.
• Before it writes to disk, the thread first divides the data into partitions
corresponding to the reducers that they will ultimately be sent to.
• Within each partition, the back ground thread performs an in-memory sort
by key, and if there is a combiner function, it is run on the output of the
sort. Running the combiner function makes for a more compact map
output, so there is less data to write to local disk and to transfer to the
reducer.
• Each time the memory buffer reaches the spill threshold, a new spill file is
created, so after the map task has written its last output record there could
be several spill files. Before the task is finished, the spill files are merged
into a single partitioned and sorted output file.
• If there are at least three spill files (set by the [Link]
property) then the combiner is run again before the output file is written.
• If there are only one or two spills, then the potential reduction in
map output size is not worth the overhead in invoking the combiner.
The Reduce Side :
• Let’s turn now to the reduce part of the process. The map output file is
sitting on the local disk of the machine that ran the map task (note that
although map outputs always get written to local disk, reduce outputs may
not be) but now it is needed by the machine that is about to run the reduce
task for the partition.
• The reduce task needs the map output for its particular partition from
several map tasks across the cluster. The map tasks may finish at
different times, so the reduce task starts copying their outputs as
soon as each completes. This is known as the copy phase of the
reduce task. The reduce task has a small number of copier threads so
that it can fetch map outputs in parallel. The default is five threads, but this
number can be changed by setting the [Link]
property.
• The map outputs are copied to the reduce task JVM’s memory if they are
small enough otherwise, they are copied to disk.
• When the in-memory buffer reaches a threshold size (controlled by
[Link]), or reaches a threshold number of map
outputs ([Link]), it is merged and spilled to disk.
• If a combiner is specified it will be run during the merge to reduce the
amount of data written to disk.
• As the copies accumulate on disk, a background thread merges them into
larger, sorted files.
• When all the map outputs have been copied, the reduce task moves into
the sort phase (which should properly be called the merge phase, as the
sorting was carried out on the map side), which merges the map outputs,
maintaining their sort ordering.
 Topic: Task Execution
• some more controls that MapReduce users have over task execution.
i) The Task Execution Environment
• Hadoop provides information to a map or reduce task about the environment in which it
is running.
• The MapReduce model is to break jobs into tasks and run the tasks in parallel to make
the overall job execution time smaller than it would otherwise be if the tasks ran
sequentially.
• This makes job execution time sensitive to slow-running tasks, as it takes only one
slow task to make the whole job take significantly longer than it would have done
otherwise. When a job consists of hundreds or thousands of tasks, the possibility of a
few straggling tasks is very real.
• Tasks may be slow for various reasons, including hardware degradation or software
mis-configuration, but the causes may be hard to detect since the tasks still complete
successfully, albeit after a longer time than expected.
• Hadoop doesn’t try to diagnose and fix slow-running tasks; instead, it tries
to detect when a task is running slower than expected and launches
another, equivalent, task as a backup. This is termed speculative
execution of tasks.
• A speculative task is launched only after all the tasks for a job have been launched,
and then only for tasks that have been running for some time (at least a minute) and
have failed to make as much progress, on average, as the other tasks from the job.
• When a task completes successfully, any duplicate tasks that are running are killed
since they are no longer needed. So if the original task completes before the
speculative task, then the speculative task is killed; on the other hand, if the
speculative task finishes first, then the original is killed.
• Speculative execution is turned on by default. It can be enabled or disabled
independently for map tasks and reduce tasks, on a cluster-wide basis, or on a per-
job basis.
• The goal of speculative execution is to reduce job execution time, but this comes
at the cost of cluster efficiency.
Output Committers
• Hadoop MapReduce uses a commit protocol to ensure that jobs and
tasks either succeed, or fail cleanly.
• The behavior is implemented by the OutputCommitter in use for the job, and this is
set in the old MapReduce API by calling the setOutputCommitter() on JobConf, or
by setting [Link] in the configuration. In the new MapReduce
API, the OutputCommitter is determined by the OutputFormat, via its
getOutputCommitter() method. The default is FileOutputCommitter, which is
appropriate for file-based MapReduce.
• If the job succeeds then the commitJob() method is called, which in the default file
based implementation deletes the temporary working space, and creates a hidden
empty marker file in the output directory called _SUCCESS to indicate to filesystem
clients that the job completed successfully. If the job did not succeed, then the
abortJob() is called with a state object indicating whether the job failed or was killed
ii) Task JVM Reuse
• Hadoop runs tasks in their own Java Virtual Machine to isolate them
from other run ning tasks. The overhead of starting a new JVM for each
task can take around a second, which for jobs that run for a minute or so
is insignificant.
• jobs that have a large number of very short-lived tasks (these are usually
map tasks), or that have lengthy initialization, can see performance gains
when the JVM is reused for subsequent tasks.
• with task JVM reuse enabled, tasks are not run concurrently in a
single JVM; rather, the JVM runs tasks sequentially. Tasktrackers can,
however, run more than one task at a time, but this is always done in
separate JVMs.
 Topic : MapReduce Types and Formats
• MapReduce has a simple model of data processing: inputs and outputs
for the map and reduce functions are key-value pairs.
• The MapReduce model in particular, how data in various formats, can
be used with this model.
MapReduce Types
• The map and reduce functions in Hadoop MapReduce have the following
general form:
map: (K1, V1) → list(K2, V2)
reduce: (K2, list(V2)) → list(K3, V3)
• In general, the map input key and value types (K1 and V1) are different
from the map output types (K2 and V2). However, the reduce input
must have the same types as the map output, although the reduce
output types may be different again (K3 and V3).
The Java API mirrors this general form:
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
{
public class Context extends MapContext<KEYIN, VALUEIN, KEYOUT,
VALUEOUT>
{
// ...
}
protected void map(KEYIN key, VALUEIN value, Context context) throws
IOException, InterruptedException
{
// ...
}
}
public class Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
{
public class Context extends ReducerContext<KEYIN, VALUEIN,
KEYOUT, VALUEOUT>
{
// ...
}
protected void reduce(KEYIN key, Iterable values, Context context)
throws IOException, InterruptedException
{
// ...
}
}
• The context objects are used for emitting key-value pairs, so they are
parameterized by the output types, so that the signature of the write()
method is:
public void write(KEYOUT key, VALUEOUT value) throws IOException,
InterruptedException.
• If a combine function is used, then it is the same form as the reduce
function (and is an implementation of Reducer), except its output types
are the intermediate key and value types (K2 and V2), so they can feed
the reduce function:
map: (K1, V1) → list(K2, V2)
combine: (K2, list(V2)) → list(K2, V2)
reduce: (K2, list(V2)) → list(K3, V3)
• Often the combine and reduce functions are the same, in which case, K3
is the same as K2, and V3 is the same as V2.
The partition function operates on the intermediate key and value types (K2
and V2), and returns the partition index. In practice, the partition is
determined solely by the key (the value is ignored):
partition: (K2, V2) → integer
• Input types are set by the input format. So, for instance, a
TextInputFormat generates keys of type LongWritable and values of
type Text. The other types are set explicitly by calling the methods on the
Job. If not set explicitly, the intermediate types default to the (final) output
types, which default to LongWritable and Text. So if K2 and K3 are the
same, you don’t need to call setMapOutputKeyClass(), since it falls back
to the type set by calling setOutputKeyClass(). Similarly, if V2 and V3 are
the same, you only need to use setOutputValueClass().
• The default mapper is just the Mapper class, which writes the input key
and value unchanged to the output:
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
{
protected void map(KEYIN key, VALUEIN value, Context context)
throws IOException, InterruptedException
{
[Link]((KEYOUT) key, (VALUEOUT) value);
}
}
 The default reducer is Reducer, again a generic type, which simply writes all its input to
its output:
public class Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT>
{
protected void reduce(KEYIN key, Iterable<VALUEIN> values, Context context) throws
IOException, InterruptedException
{
for (VALUEIN value: values)
{
[Link]((KEYOUT) key, (VALUEOUT) value);
}
}
}
• For this job, the output key is LongWritable, and the output value is Text.
• The default output format is TextOutputFormat, which writes out records, one per line, by converting keys and
values to strings and separating them with a tab character.
Input Formats:
• Hadoop can process many different types of data formats, from flat
text files to data bases.
• An InputFormat is responsible for creating the input splits and
dividing them into records.
FileInputFormat
• FileInputFormat is the base class for all implementations of InputFormat
that use files as their data source. It provides two things: a place to define
which files are included as the input to a job, and an implementation for
generating splits for the input files. The job of dividing splits into records is
performed by subclasses.
1. Text Input
• The different InputFormats that Hadoop provides to process text.
[Link]
• TextInputFormat is the default InputFormat. Each record is a line of
input. The key, a LongWritable, is the byte offset within the file of the
beginning of the line. The value is the contents of the line, excluding any
line terminators, and is packaged as a Text object.
• So a file containing the following text:
• Example:
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.
 On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.
• is divided into one split of four records. The records are interpreted as the
following
key-value pairs:
(0, On the top of the Crumpetty Tree)
(33, The Quangle Wangle sat,)
(57, But his face you could not see,)
(89, On account of his Beaver Hat.)
b. KeyValueTextInputFormat
• TextInputFormat’s keys, being simply the offset within the file, are not
normally very useful.
• It is common for each line in a file to be a key-value pair, separated by a
delimiter such as a tab character.
• You can specify the separator via the
[Link] property (or
[Link] in the old API).
• It is a tab character by default. Consider the following input file, where →
represents a (horizontal) tab character:
line1→On the top of the Crumpetty Tree
line2→The Quangle Wangle sat,
line3→But his face you could not see,
line4→On account of his Beaver Hat.
• Like in the TextInputFormat case, the input is in a single split comprising
four records, although this times the keys are the Text sequences before
the tab in each line:
(line1, On the top of the Crumpetty Tree)
(line2, The Quangle Wangle sat,)
(line3, But his face you could not see,)
(line4, On account of his Beaver Hat.)
c. NLineInputFormat
• With TextInputFormat and KeyValueTextInputFormat, each mapper receives a variable
number of lines of input. The number depends on the size of the split and the length of
the lines. If you want your mappers to receive a fixed number of lines of input,
then NLineInputFormat is the InputFormat to use. Like TextInputFormat, the keys
are the byte offsets within the file and the values are the lines themselves.
• N refers to the number of lines of input that each mapper receives. With N set to one
(the default), each mapper receives exactly one line of input.
 On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.

• If, for example, N is two, then each split contains two lines. One mapper
will receive the first two key-value pairs:
(0, On the top of the Crumpetty Tree)
(33, The Quangle Wangle sat,)
• And another mapper will receive the second two key-value pairs:
(57, But his face you could not see,)
(89, On account of his Beaver Hat.)
[Link]
• Most XML parsers operate on whole XML documents, so if a large XML
document is made up of multiple input splits, then it is a challenge to parse
these individually.
• Large XML documents that are composed of a series of “records” (XML
document fragments) can be broken into these records using simple string
or regular-expression matching to find start and end tags of records.
• set your input format to StreamInputFormat and set the
[Link] property to
[Link] to use xml
as an input format.
• To take an example, Wikipedia provides dumps of its content in XML form,
which are appropriate for processing in parallel using MapReduce using
this approach.
2. Binary Input
• Hadoop MapReduce is not just restricted to processing textual data—it
has support for binary formats, too.
[Link]
• Hadoop’s sequence file format stores sequences of binary key-value pairs.
Sequence files are well suited as a format for MapReduce data since
they are splittable they support compression as a part of the format
[Link]
• SequenceFileAsTextInputFormat is a variant of SequenceFileInputFormat
that converts the sequence file’s keys and values to Text objects.
[Link]
• SequenceFileAsBinaryInputFormat is a variant of
SequenceFileInputFormat that retrieves the sequence file’s keys and
values as opaque binary objects.
3. Multiple Inputs
• The input to a MapReduce job may consist of multiple input files. This
case is handled elegantly by using the MultipleInputs class. For
example, if we had weather data from the UK Met Office6 that we wanted
to combine with the NCDC data for our maximum temperature analysis,
then we might set up the input as follows:
[Link](job,ncdcInputPath,[Link],
[Link]);
[Link](job,metOfficeInputPath,
[Link], [Link]);
4. Database Input
• DBInputFormat is an input format for reading data from a relational
database, using JDBC. It is best used for loading relatively small
datasets, perhaps for joining with larger datasets from HDFS, using
MultipleInputs.
Output Formats
• Hadoop has output data formats that correspond to the input formats.
1. Text Output
• The default output format, TextOutputFormat, writes records as lines of text. Its
keys and values may be of any type, since TextOutputFormat turns them to
strings by calling toString() on them. Each key-value pair is separated by a tab
character. The counterpart to TextOutput Format for reading in this case is
KeyValueTextInputFormat.
2. Binary Output
[Link]
• SequenceFileOutputFormat As the name indicates, SequenceFileOutputFormat writes sequence files
for its output.
[Link]
• SequenceFileAsBinaryOutputFormat is the counterpart to SequenceFileAsBinaryInputFormat, and it
writes keys and values in raw binary format into a SequenceFile container.
[Link]
• MapFileOutputFormat writes MapFiles as output. The keys in a MapFile must be
added in order, so you need to ensure that your reducers emit keys in sorted order.
4. Multiple Outputs
• FileOutputFormat and its subclasses generate a set of files in the output directory.
There is one file per reducer, and files are named by the partition number: part-r-
00000, partr-00001, etc. There is sometimes a need to have more control over the
naming of the files or to produce multiple files per reducer. MapReduce comes
with the MultipleOutputs class to help you do this.
5. Lazy Output
• FileOutputFormat subclasses will create output (part-r-nnnnn) files, even if they are empty.
• Some applications prefer that empty files not be created, which is where LazyOutputFormat helps.
6. Database Output
The output formats for writing to relational databases and to HBase.
DBOutputFormat, which is useful for dumping job outputs (of modest size)
into a database.
 Topic : MapReduce Features
1. Counters
• Counters are a useful channel for gathering statistics about the job: for quality control
or for application level-statistics. They are also useful for problem diagnosis.
a. Built-in Counters
• Hadoop maintains some built-in counters for every job, which report various metrics
for your job.
• Counters are divided into groups, and there are several groups for the built-in counters.
• Each group either contains task counters (which are updated as a task progresses) or
job counters (which are updated as a job progresses).
i)Task counters
• Task counters gather information about tasks over the course of their execution, and
the results are aggregated over all the tasks in a job.
• Task counters are maintained by each task attempt, and periodically sent to the task
tracker and then to the jobtracker, so they can be globally aggregated.
• Task counters are sent in full every time.
ii)Job counters
• Job counters are maintained by the jobtracker (or application master in YARN), so
they don’t need to be sent across the network, unlike all other counters, including user-
defined ones. They measure job-level statistics.
b. User-Defined Java Counters
• MapReduce allows user code to define a set of counters, which are then incremented
as desired in the mapper or reducer. Counters are defined by a Java enum, which
serves to group related counters. A job may define an arbitrary number of enums, each
with an arbitrary number of fields. The name of the enum is the group name, and the
enum’s fields are the counter names.
 enum Temperature
{
MISSING,
MALFORMED
}
[Link]-Defined Streaming Counters
• A Streaming MapReduce program can increment counters by sending a
specially formatted line to the standard error stream.
• The line must have the following format:
reporter:counter:group,counter,amount.
• In a similar way, a status message may be sent with a line formatted like
this:
reporter:status:message
2. Sorting
• The ability to sort data is at the heart of MapReduce.
• Even if your application isn’t concerned with sorting perse, it may be
able to use the sorting stage that MapReduce provides to organize its
data. we will examine different ways of sorting datasets and how you can
control the sort order in MapReduce.
i) Preparation
• We are going to sort the weather dataset by temperature. Storing temperatures as Text
objects doesn’t work for sorting purposes, since signed integers don’t sort
lexicographically. Instead, we are going to store the data using sequence files
whose IntWritable keys represent the temperature (and sort correctly), and whose
Text values are the lines of data.
ii) Partial Sort
• In “The Default MapReduce Job”, we saw that, by default, MapReduce
will sort input records by their keys.
iii) Total Sort
• it is possible to produce a set of sorted files that, if concatenated, would form a globally sorted file. The
secret to doing this is to use a partitioner that respects the total order of the output.
• The objective of Total Order Sorting is to have all outputs sorted across all reducers :
• Reducer 1 output : (a,5) (b,2) (c,5)
• Reducer 2 output : (d,6) (e,7) (w,5)
• This way the outputs can be read/searched/concatenated sequentially as a single ordered output.
iv) Secondary Sort
• A secondary sort problem relates to sorting values associated with a key in the reduce phase.
Sometimes, it is called value-to-key conversion. The secondary sorting technique will enable us to sort
the values (in ascending or descending order) passed to each reducer.
• A dump of the temperature data might look something like the following (columns are year, month, day,
and daily temperature, respectively): 2012, 01, 01, 5
2012, 01, 02, 45
3. Joins
• MapReduce can perform joins between large datasets, but writing the
code to do joins.
• If the join is performed by the mapper, it is called a map-side join,
whereas if it is performed by the reducer it is called a reduce-side join.
i) Map-Side Joins
• A map-side join between large inputs works by performing the join before
the data reaches the map function. For this to work, though, the inputs to
each map must be partitioned and sorted in a particular way. Each input
dataset must be divided into the same number of partitions, and it must be
sorted by the same key (the join key) in each source. All the records for a
particular key must reside in the same partition.
ii) Reduce-Side Joins
• A reduce-side join is more general than a map-side join, in that the input
datasets don’t have to be structured in any particular way, but it is less
efficient as both datasets have to go through the MapReduce shuffle.
• The basic idea is that the mapper tags each record with its source and
uses the join key as the map output key, so that the records with the same
key are brought together in the reducer.
 Topic: Hadoop Ecosystem
• Hadoop is a framework that enables processing of large data sets which
reside in the form of clusters. Being a framework, Hadoop is made up of
several modules that are supported by a large ecosystem of technologies.
• Introduction: Hadoop Ecosystem is a platform or a suite which provides
various services to solve the big data problems. It includes Apache
projects and various commercial tools and solutions.
• There are four major elements of Hadoop i.e. HDFS, MapReduce, YARN,
and
• Hadoop Common. Most of the tools or solutions are used to supplement
or support these major elements. All these tools work collectively to
provide services such as absorption,analysis, storage and maintenance of
data etc.
• Following are the components that collectively form a
Hadoop ecosystem:
HDFS: Hadoop Distributed File System
YARN: Yet Another Resource Negotiator
MapReduce: Programming based Data Processing
Spark: In-Memory data processing
PIG, HIVE: Query based processing of data services
HBase: NoSQL Database
Mahout, Spark MLLib: Machine Learning algorithm libraries
Zookeeper: Managing cluster
Oozie: Job Scheduling
• All these toolkits or components revolve around one term i.e. Data. That’s the beauty of
Hadoop that it revolves around data and hence making its synthesis easier.
[Link]:
• HDFS is the primary or major component of Hadoop ecosystem and is
responsible for storing large data sets of structured or unstructured data
across various nodes and thereby maintaining the metadata in the form of
log files.
• HDFS consists of two core components i.e.
1. Name node
2. Data Node
• Name Node is the prime node which contains metadata (data about data)
requiring comparatively fewer resources than the data nodes that stores
the actual data. These data nodes are commodity hardware in the
distributed environment. Undoubtedly, making Hadoop cost effective.
• HDFS maintains all the coordination between the clusters and hardware,
thus working at the heart of the system.
[Link]:
Yet Another Resource Negotiator, as the name implies, YARN is the one
who helps to manage the resources across the clusters. In short, it performs
scheduling and resource allocation for the Hadoop System.
Consists of three major components i.e.
1. Resource Manager
2. Nodes Manager
3. Application Manager
Resource manager has the privilege of allocating resources for the
applications in a system whereas Node managers work on the allocation of
resources such as CPU, memory, bandwidth per machine and later on
acknowledges the resource manager. Application manager works as an
interface between the resource manager and node manager and performs
negotiations as per the requirement of the two.
3. MapReduce:
• By making the use of distributed and parallel algorithms, MapReduce
makes it possible to carry over the processing’s logic and helps to write
applications which transform big data sets into a manageable one.
MapReduce makes the use of two functions i.e. Map() and Reduce() whose
task is:
1. Map() performs sorting and filtering of data and thereby organizing them
in the form of group. Map generates a key-value pair based result which is
later on processed by the Reduce() method.
2. Reduce(), as the name suggests does the summarization by aggregating
the mapped data. In simple, Reduce() takes the output generated by Map()
as input and combines those tuples into smaller set of tuples.
4. PIG:
• Pig was basically developed by Yahoo which works on a pig Latin
language, which is Query based language similar to SQL.
• It is a platform for structuring the data flow, processing and analyzing huge
data sets.
• Pig does the work of executing commands and in the background, all the
activities of MapReduce are taken care of. After the processing, pig stores
the result in HDFS.
• Pig Latin language is specially designed for this framework which runs on
Pig Runtime. Just the way Java runs on the JVM.
• Pig helps to achieve ease of programming and optimization and hence is a
major segment of the Hadoop Ecosystem.
5. HIVE:
• With the help of SQL methodology and interface, HIVE performs reading
and writing of large data sets. However, its query language is called as
HQL (Hive Query Language).
• It is highly scalable as it allows real-time processing and batch processing
both. Also, all the SQL datatypes are supported by Hive thus, making the
query processing easier.
• Similar to the Query Processing frameworks, HIVE too comes with two
components: JDBC Drivers and HIVE Command Line.
• JDBC, along with ODBC drivers work on establishing the data storage
permissions and connection whereas HIVE Command line helps in the
processing of queries.
6. Mahout:
• Mahout, allows Machine Learnability to a system or application.
• Machine Learning, as the name suggests helps the system to develop itself based on
some patterns, user/environmental interaction or on the basis of algorithms.
• It provides various libraries or functionalities such as collaborative filtering, clustering,
and classification which are nothing but concepts of Machine learning. It allows
invoking algorithms as per our need with the help of its own libraries.
7. Apache-Spark:
• It’s a platform that handles all the process consumptive tasks like batch processing,
interactive or iterative real-time processing, graph conversions, and visualization, etc.
• It consumes in memory resources hence, thus being faster than the prior in terms of
optimization.
• Spark is best suited for real-time data whereas Hadoop is best suited for structured
data or batch processing, hence both are used in most of the companies
interchangeably.
[Link]-HBase:
• It’s a NoSQL database which supports all kinds of data and thus capable
of handling anything of Hadoop Database.
• It provides capabilities of Google’s BigTable, thus able to work on Big
Data sets effectively.
• At times where we need to search or retrieve the occurrences of
something small in a huge database, the request must be processed
within a short quick span of time. At such times,
• HBase comes handy as it gives us a tolerant way of storing limited data.
9. Zookeeper: There was a huge issue of management of coordination and
synchronization among the resources or the components of Hadoop which
resulted in inconsistency, often.
• Zookeeper overcame all the problems by performing synchronization,
inter-component based communication, grouping, and maintenance.
[Link]:
• Oozie simply performs the task of a scheduler, thus scheduling jobs and binding them
together as a single unit.
• Oozie is a workflow management system that allows users to monitor and control
workflows. It can be used to automate tasks for a variety of purposes, including data
processing, system administration, and debugging.
• There is two kinds of jobs .i.e Oozie workflow and Oozie coordinator jobs. Oozie
workflow is the jobs that need to be executed in a sequentially ordered manner
whereas Oozie Coordinator jobs are those that are triggered when some data or
external stimulus is given to it.
11. Avro
• It is an open source project that provides data serialization and data exchange
services for Hadoop. Using serialization, service programs can serialize data into files
or messages.
• It also stores data definition and data together in one message or file. Hence, this
makes it easy for programs to dynamically understand information stored in Avro file or
message
12. Sqoop
• A tool designed to transfer data between Hadoop and relational databases like MySQL
and Oracle.
• It is mainly used for importing and exporting data. So, it imports data from external
sources into related Hadoop components like HDFS, HBase or Hive. It also exports
data from Hadoop to other external sources.
• Sqoop works with relational databases such as Teradata, Netezza, Oracle, MySQL
13. Chukwa – Monitoring
• Chukwa is an open-source distributed monitoring system for high performance
computing clusters.
• The tool collects data from Hadoop Distributed File System (HDFS), MapReduce, and
YARN applications.
• It provides a web interface to view the data collected by Chukwa agents running on
each node in the cluster.
14. Flume – Monitoring
• Flume is an open-source distributed log collection system storing log
events from sources such as web servers or application servers into
HDFS or other systems.
15. Apache Ambari:
• A management platform for provisioning, managing, and monitoring
Hadoop clusters through an easy-to-use web interface.
 Topic: YARN
• Hadoop YARN (Yet Another Resource Negotiator) is a Hadoop
ecosystem component that provides the resource management.
• Yarn is also one the most important component of Hadoop Ecosystem.
• YARN is called as the operating system of Hadoop as it is responsible for
managing and monitoring workloads.
• It allows multiple data processing engines such as real-time streaming and
batch processing to handle data stored on a single platform.
1. Client submits an application
2. The Resource Manager allocates a container to start the Application
Manager
3. The Application Manager registers itself with the Resource Manager
4. The Application Manager negotiates containers from the Resource
Manager
5. The Application Manager notifies the Node Manager to launch
containers
6. Application code is executed in the container
7. Client contacts Resource Manager/Application Manager to monitor
application’s status
8. Once the processing is complete, the Application Manager un-registers
with the Resource Manager
Advantages
[Link]: YARN offers flexibility to run various types of distributed processing
systems such as Apache Spark, Apache Flink, Apache Storm, and others. It allows
multiple processing engines to run simultaneously on a single Hadoop cluster.
[Link] Management: YARN provides an efficient way of managing resources in
the Hadoop cluster. It allows administrators to allocate and monitor the resources
required by each application in a cluster, such as CPU, memory, and disk space.
[Link]: YARN is designed to be highly scalable and can handle thousands of
nodes in a cluster. It can scale up or down based on the requirements of the applications
running on the cluster.
[Link] Performance: YARN offers better performance by providing a centralized
resource management system. It ensures that the resources are optimally utilized, and
applications are efficiently scheduled on the available resources.
[Link]: YARN provides robust security features such as Kerberos authentication,
Secure Shell (SSH) access, and secure data transmission. It ensures that the data
stored and processed on the Hadoop cluster is secure.
 Topic: MRV1 and MRV2
[Link]:
MRv1: Tightly coupled MapReduce and HDFS; a single JobTracker
managed resource allocation and job scheduling.
MRv2 (YARN): Decoupled resource management from MapReduce,
creating a more flexible and scalable platform.
[Link]:
MRv1: Consisted of a JobTracker (resource manager and scheduler) and
TaskTrackers on data nodes.
MRv2: Features a global ResourceManager (replaces JobTracker for
resource management), NodeManagers on each node, and an
ApplicationMaster per application for job execution.
[Link] and Bottlenecks:
• MRv1: The single JobTracker became a bottleneck as the cluster grew, limiting
scalability.
• MRv2: The distributed resource management model with NodeManagers and
ApplicationMasters significantly improves scalability and isolation.
[Link] Management:
• MRv1: Resource slots for map and reduce tasks were fixed and dedicated.
• MRv2: Resources are managed more flexibly in terms of memory and CPU, with no
distinction between map and reduce slots.
[Link] Support:
• MRv1: Primarily supported MapReduce, and other applications could not run on the
Hadoop 1.x cluster.
• MRv2: The YARN execution model is more generic, allowing various data processing
frameworks and applications to run on the cluster, making it more versatile.