MODULE 3

BIG DATA PROCESSING – HADOOP, SPARK, HIVE, PIG

SYLLABUS:
• Hadoop, brief history of Hadoop, comparison with other systems
• MapReduce data flow, weather dataset example
• Hadoop Distributed File System (HDFS) concepts, basic commands, HDFS Java interface
• HBase
• YARN, anatomy of a YARN application, scheduling
• Pig Latin language, running an example, comparison with databases
• Hive data warehousing, shell, running an example, Hive architecture, comparison with databases
• Spark framework, example, anatomy of a SPARK job run

IMPORTANT QUESTIONS:

• Explain the role of MapReduce in Hadoop with a suitable example.

• Describe Spark with an example.
• Explain the architecture of HDFS.
• Discuss on how the MapReduce framework is modified using YARN.
•

SHADHA K|DEPT OF IT|MESCE 1

YARN
Apache YARN (Yet Another Resource Negotiator) is Hadoop’s cluster resource management system.
YARN was introduced in Hadoop 2 to improve the MapReduce implementation, but it is general
enough to support other distributed computing paradigms as well.
YARN provides APIs for requesting and working with cluster resources, but these APIs are not
typically used directly by user code. Instead, users write to higher-level APIs provided by distributed
computing frameworks, which themselves are built on YARN and hide the resource management
details from the user. Figure 8, which shows some distributed computing frameworks (MapReduce,
Spark, and so on) running as YARN applications on the cluster compute layer (YARN) and the cluster
storage layer (HDFS and HBase). Pig, Hive, and Crunch are all examples of processing frameworks
that run on MapRe‐ duce, Spark, or Tez (or on all three), and don’t interact with YARN directly.

Figure 8: YARN Applications

Anatomy of a YARN Application Run

• YARN provides its core services via two types of long-running process:
o Resource manager
o Node managers
• A resource manager (one per cluster) manages the use of resources across the cluster.
• Node managers runs on all the nodes in the cluster to launch and monitor containers.
• A container executes an application-specific process with a constrained set of resources
(memory, CPU, and so on). Figure 9 shows how YARN runs an application.

SHADHA K|DEPT OF IT|MESCE 2

 Figure 9: How YARN runs an application
The different steps in yarn application are:
1. A client contacts the resource manager and asks it to run an application master process.
2. The resource manager then finds a node manager that can launch the application master in a
container (2a ans 2b.
3. The application master depends on the application. It could simply run a computation in the
container it is running in and return the result to the client.
4. It could request more containers from the resource managers and use them to run a distributed
computation (steps 4a and 4b).
Resource Requests

• YARN has a flexible model for making resource requests.

• A request for a set of containers can express the amount of computer resources required for
each container (memory and CPU), as well as locality constraints for the containers in that
request.
• Locality is critical in ensuring that distributed data processing algorithms use the cluster
bandwidth efficiently, so YARN allows an application to specify locality constraints for the
containers it is requesting.
• Locality constraints can be used to request a container on a specific node or rack, or anywhere
on the cluster (off-rack).
• Sometimes the locality constraint cannot be met, in which case either no allocation is made or,
optionally, the constraint can be loosened. For example, if a specific node was requested but it
is not possible to start a container on it (because other containers are running on it), then YARN

SHADHA K|DEPT OF IT|MESCE 3

 will try to start a container on a node in the same rack, or, if that’s not possible, on any node in
the cluster.
• In the common case of launching a container to process an HDFS block (to run a map task in
MapReduce, say), the application will request a container on one of the nodes hosting the
block’s three replicas, or on a node in one of the racks hosting the replicas, or, failing that, on
any node in the cluster.
• A YARN application can make resource requests at any time while it is running. For example,
an application can make all of its requests up front, or it can take a more dynamic approach
whereby it requests more resources dynamically to meet the changing needs of the application.
Application Lifespan

• The lifespan of a YARN application can vary dramatically: from a short-lived application
of a few seconds to a long-running application that runs for days or even months.
• The simplest case is one application per user job, which is the approach that MapReduce takes.
• The second model is to run one application per workflow or user session of (possibly unrelated)
jobs. This approach can be more efficient than the first, since containers can be reused between
jobs, and there is also the potential to cache intermediate data between jobs. Spark is an
example that uses this model.
• The third model is a long-running application that is shared by different users. Such an
application often acts in some kind of coordination role. For example, Apache Slider has a long
running application master for launching other applications on the cluster.
YARN compared MapReduce1
MapReduce in the original version of Hadoop (version 1 and earlier) is sometimes referred to as
“MapReduce 1” to distinguish it from MapReduce 2, the implementation that uses YARN.
MapReduce 1

• There are two types of daemons that control the job execution process: a jobtracker(Datanode)
and one or more tasktrackers(Namenode).
• The jobtracker coordinates all the jobs run on the system by scheduling tasks to run on
tasktrackers.
• Tasktrackers run tasks and send progress reports to the jobtracker, which keeps a record of the
overall progress of each job.
• If a task fails, the jobtracker can reschedule it on a different tasktracker.
• The jobtracker takes care of both job scheduling (matching tasks with tasktrackers) and task
progress monitoring (keeping track of tasks, restarting failed or slow tasks, and doing task
bookkeeping, such as maintaining counter totals).
• The jobtracker is also responsible for storing job history for completed jobs, although it is
possible to run a job history server as a separate daemon to take the load off the jobtracker.
YARN

• These responsibilities are handled by separate entities: the resource manager and an application
master (one for each MapReduce job).

SHADHA K|DEPT OF IT|MESCE 4

 • The timeline server, which stores application history of completed jobs.
Figure 10 shows the comparison of MR1 and yarn components.

Figure 10: A comparison of MapReduce 1 and YARN components

Advantages of YARN

• Scalability
o YARN can run on larger clusters than MapReduce 1. MapReduce 1 hits scalability
bottlenecks in the region of 4,000 nodes and 40,000 tasks, stemming from the fact
that the jobtracker has to manage both jobs and tasks.
o YARN overcomes these limitations by virtue of its split resource manager/application
master architecture: it is designed to scale up to 10,000 nodes and 100,000 tasks.
• Availability
o High availability (HA) is usually achieved by replicating the state needed for another
daemon to take over the work needed to provide the service, in the event of the service
daemon failing. However, the large amount of rapidly changing complex state in the
jobtracker’s memory (each task status is updated every few seconds, for example)
makes it very difficult to retrofit HA into the jobtracker service.
o With the jobtracker’s responsibilities split between the resource manager and
application master in YARN, makes the service highly available as a result of divide
and-conquer.
• Utilization
o In YARN, a node manager manages a pool of resources, rather than a fixed number of
designated slots. MapReduce running on YARN will not hit the situation where a
reduce task has to wait because only map slots are available on the cluster, which can
happen in MapReduce 1. If the resources to run the task are available, then the
application will be eligible for them.
• Multitenancy
o It is even possible for users to run different versions of MapReduce on the same YARN
cluster, which makes the process of upgrading MapReduce more manageable.

SHADHA K|DEPT OF IT|MESCE 5

Scheduling in YARN
It is the job of the YARN scheduler to allocate resources to applications according to some defined
policy. Scheduling in general is a difficult problem and there is no one “best” policy, which is why
YARN provides a choice of schedulers and configurable policies.
Three schedulers are available in YARN:

• FIFO Scheduler
• Capacity Scheduler
• Fair Scheduler

FIFO Scheduler

Figure 11: FIFO Scheduler

As the name suggests FIFO scheduler (Figure 11) i.e. First In First Out, so the tasks or application that
comes first will be served first. This is the default Scheduler we use in Hadoop. The tasks are placed
in a queue and the tasks are performed in their submission order. In this method, once the job is
scheduled, no intervention is allowed. So sometimes the high-priority process has to wait for a long
time since the priority of the task does not matter in this method.
Advantage:

• No need for configuration

• First Come First Serve
• Simple to execute
Disadvantage:

• Priority of task doesn’t matter, so high priority jobs need to wait

• Not suitable for shared cluster

SHADHA K|DEPT OF IT|MESCE 6

Capacity Scheduler

Figure 12: Capacity Scheduler

In Capacity Scheduler (Figure 12) we have multiple job queues for scheduling our tasks. The Capacity
Scheduler allows multiple occupants to share a large size Hadoop cluster. In Capacity Scheduler
corresponding for each job queue, we provide some slots or cluster resources for performing job
operation. Each job queue has its own slots to perform its task. In case we have tasks to perform in
only one queue then the tasks of that queue can access the slots of other queues also as they are free to
use, and when the new task enters to some other queue then jobs in running in its own slots of the
cluster are replaced with its own job.
Capacity Scheduler also provides a level of abstraction to know which occupant is utilizing the more
cluster resource or slots, so that the single user or application doesn’t take disappropriate or
unnecessary slots in the cluster. The capacity Scheduler mainly contains 3 types of the queue that are
root, parent, and leaf which are used to represent cluster, organization, or any subgroup, application
submission respectively.
Advantage:

• Best for working with Multiple clients or priority jobs in a Hadoop cluster
• Maximizes throughput in the Hadoop cluster
Disadvantage:

• More complex
• Not easy to configure for everyone

SHADHA K|DEPT OF IT|MESCE 7

Fair Scheduler

Figure 13: Fair Scheduler

The Fair Scheduler (Figure 13) attempts to allocate resources so that all running applications get the
same share of resources. There is no need to set amount of capacity, since it will dynamically balance
resources between all running jobs. Just after the first (large) job starts, it is the only job running, so it
gets all the resources in the cluster. When the second (small) job starts, it is allocated half of the cluster
resources so that each job is using its fair share of resources.
Note that there is a lag between the time the second job starts and when it receives its fair share, since
it has to wait for resources to free up as containers used by the first job complete. After the small job
completes and no longer requires resources, the large job goes back to using the full cluster capacity
again. The overall effect is both high cluster utilization and timely small job completion.

Figure 14: Fair sharing between user queues

Fair sharing works for applications in the same queue; however, fair sharing actually works between
queues too (Figure 14). Imagine two users A and B, each with their own queue. A starts a job, and it
is allocated all the resources available since there is no demand from B. Then B starts a job while A’s
job is still running, and after a while each job is using half of the resources, in the way we saw earlier.
Now if B starts a second job while the other jobs are still running, it will share its resources with B’s

SHADHA K|DEPT OF IT|MESCE 8

other job, so each of B’s jobs will have one-fourth of the resources, while A’s will continue to have
half. The result is that resources are shared fairly between users.

Advantages:

• High cluster utilization and timely small job completion.

• it can limit the concurrent running task in a particular pool or queue.
Disadvantages:

• The configuration is required.

SHADHA K|DEPT OF IT|MESCE 9