MODULE-2

CHAPTER-1

Topics

Introduction to Hadoop: Introducing hadoop, Why hadoop, Why not RDBMS, RDBMS Vs
Hadoop, History of Hadoop, Hadoop overview, Use case of Hadoop, HDFS (Hadoop
Distributed File System),Processing data with Hadoop, Managing resources and applications
with Hadoop YARN(Yet Another Resource Negotiator).

[Link] HADOOP
Big Data refers to the massive amounts of structured, semi-structured, and unstructured data generated daily
from various sources. Its major characteristics are:
• Volume: The sheer amount of data produced (e.g., NYSE generates 1.5 billion shares and trade data
daily, Facebook stores 2.7 billion comments and likes daily, Google processes about 24 petabytes of
data daily).
• Variety: Data comes in many formats—text, videos, logs, images, etc.—from varied sources.
• Velocity: The speed at which data is generated and needs to be processed (e.g., Twitter users tweet
300,000 times per minute, email users send 200 million messages per minute).
• Value: The potential insights and business advantages that can be extracted from analyzing this data.
Importance of Big Data
Big Data offers several benefits:
• Enables new business opportunities such as product recommendations and trend analysis.
• Provides early indicators that help businesses respond quickly to market changes.
• Improves precision in decision-making through detailed analysis.
1.1 Data: The Treasure Trove
Challenges in Big Data (as illustrated in Figure 5.1)
Figure 5.1 highlights three major challenges:
1. Volume: "How to store terabytes of mounting data?" Organizations are overwhelmed by the increasing
data volume.
2. Variety: "How to work with data that is so very different?" Data comes in various formats and from
different sources, making integration and analysis difficult.

Prof. Babitha P K 1 CIT, Ponnampet

 3. Velocity: "How to access/process information quickly?" Decisions must often be made rapidly, so
processing speed is crucial.
Why Systems like Hadoop Matter
To handle the challenges above, systems such as Hadoop are necessary because they:
• Scale efficiently to accommodate the growth of data.
• Handle data in various formats (structured, semi-structured, unstructured).
• Support fast processing to deliver timely results.
In summary, big data represents both a tremendous opportunity and a set of technical challenges that systems
like Hadoop are designed to address by efficiently storing, processing, and analyzing vast and fast-moving
volumes of diverse data

2. WHY HADOOP

Why is Hadoop Popular?

Key Capability:
• Hadoop can handle huge amounts of different types of data very quickly.
• This is why it is so widely adopted for big data tasks.
Key Considerations of Hadoop (Figure 5.2):
1. Low Cost:
• Hadoop is open-source and uses inexpensive, easily available hardware to store large volumes of
data.
2. Computing Power:
• Hadoop processes very large volumes of data by using many computers (nodes) together. More
nodes = more processing power.

Prof. Babitha P K 2 CIT, Ponnampet

 3. Scalability:
• If you need more processing, you simply add more nodes. Administration is easy as the system
grows.
4. Storage Flexibility:
• You don’t need to pre-process data to store it. Hadoop lets you store any type, size, or format
(structured, unstructured, free-form) of data.
5. Inherent Data Protection:
• Hadoop protects data by creating multiple copies across nodes. If one node fails, work is
redirected to others, so data and processes continue without interruption.

How Does Hadoop Work? (Figure 5.3):

Hadoop uses a distributed file system on low-cost hardware. Here’s how it manages data:
1. Distributes and Duplicates Data:
• Breaks down data files into chunks (for example, “25–30” could be one chunk) and duplicates
each chunk across several nodes. This ensures reliability and quick access.
2. Parallel Processing:
• Each node processes its own chunk of data at the same time (in parallel), speeding up
computation.
3. Automatic Failover:
• If a node fails, Hadoop automatically reassigns its tasks to working nodes, keeping the system
running smoothly.
In Summary:
Hadoop is popular because it's affordable, powerful, flexible, scalable, and fault-tolerant. It is designed to
handle the biggest data problems by distributing both storage and computation across many inexpensive
machines, making processing of vast and varied data not just possible—but efficient and reliable.

Prof. Babitha P K 3 CIT, Ponnampet

 3. WHY NOT RDBMS FOR BIG DATA?

• Poor fit for large files:

RDBMS is not suitable for storing and processing large files, images, and videos. These types of data
are increasingly common in big data scenarios.
• Not ideal for advanced analytics:
RDBMS systems struggle with advanced analytics and machine learning workloads, often required in
modern data applications.
• Cost Issues:
As data grows, the cost per GB of storage in RDBMS rises steeply.
• The image and the explanation (Figure 5.4) illustrate that as you scale up (from GBs to TBs to
PBs), the cost per GB (€ to €€ to €€€) increases significantly.
• Scaling an RDBMS to petabyte levels becomes prohibitively expensive.
• Scaling Model (Scale-Up):
Traditional RDBMS scale by adding more powerful (and expensive) hardware, not by adding more
cheap computers. This is called "scaling up" and is costly.
Summary of Figure 5.4
• As you increase the amount of data (from GBs to TBs to PBs), the cost per GB goes up drastically,
shown by € → €€ → €€€.

Prof. Babitha P K 4 CIT, Ponnampet

 • The arrow labeled "Scale Up" shows that this increasing cost is tied to the way RDBMS systems are
traditionally scaled for big data—by investing in larger, more powerful machines.
Key Takeaways
• RDBMS systems are expensive and inefficient for large, varied, and rapidly-growing datasets (such as
media files or logs).
• They do not scale cost-effectively for modern big data needs, particularly for machine learning and
analytics.
• This is why alternative systems like Hadoop, with their cheap, scalable, distributed storage and
processing, have become popular for big data applications.

4. RDBM Versus HADOOP

PARAMETERS RDBMS HADOOP

Relational Database Management

System System. Node Based Flat Structure.

Prof. Babitha P K 5 CIT, Ponnampet

 PARAMETERS RDBMS HADOOP

Suitable for structured, unstructured data.

Supports variety of data formats in real time
such as XML, JSON, text based flat file
Data Suitable for structured data. formats, etc.

Processing OLTP Analytical, Big Data Processing

When the data needs consistent Big Data processing, which does not require any
Choice relationship. consistent relationships between data.

Needs expensive hardware or high-

end processors to store huge In a Hadoop Cluster, a node requires only a
Processor volumes of data. processor, a network card, and few hard drives.

Cost around $10,000 to $14,000

Cost per terabytes of storage. Cost around $4,000 per terabytes of storage.

5. DISTRIBUTED COMPUTING CHALLENGES

5.1. Hardware Failure

• In distributed systems, multiple servers or hard disks are networked together, increasing the likelihood
of hardware failure.
• For instance, a typical hard disk might fail once in three years, but with 1,000 disks, failures are a daily
possibility.
• Main problem: How to retrieve data if a hardware component fails?
Hadoop’s Solution: Replication Factor
• Replication Factor (RF): This denotes the number of copies of each data block stored across the
network.
• Example: If RF=2, there are two copies of each data block on different servers.
• Why it matters: If one node fails, data can still be accessed from the other copy, ensuring fault-tolerance
and reliability.
• The system constantly maintains the set number of replicas, so if one is lost due to hardware failure, it is
automatically recreated elsewhere.
5.2. How to Process This Gigantic Store of Data?

Prof. Babitha P K 6 CIT, Ponnampet

 • Distributed systems store data across many networked machines. The challenge is integrating all these
pieces for efficient processing.
• Main problem: How to process large volumes of distributed data as a unified dataset?
Hadoop’s Solution: MapReduce Programming
• MapReduce is a programming model to process data across multiple machines.
• It lets developers write programs that handle massive data sets by dividing the tasks among
many nodes and aggregating the results.
• This enables efficient computation and data integration, even when data is distributed across a huge
cluster.
Summary:
Distributed computing faces challenges of hardware failure and massive, distributed data processing. Hadoop
addresses these using Replication Factor for fault-tolerant storage and MapReduce for parallel data processing,
making it highly suitable for big data environments

[Link] OF HADOOP

• Creator:
Hadoop was created by Doug Cutting, who is also known for developing Apache Lucene (a popular text
search library).
• Origins:
Hadoop began as part of the Apache Nutch (Yahoo) project, which was an open-source web search
engine.
Timeline of Key Events (as shown in Figure 5.6)
• 2002:
Doug Cutting and Mike Cafarella started working on Nutch.

Prof. Babitha P K 7 CIT, Ponnampet

 • 2003-2004:
Google published papers on Google File System (GFS) and MapReduce. These concepts influenced
Hadoop's architecture.
• 2005:
Doug Cutting added Distributed File System (DFS) and MapReduce to Nutch.
• 2006:
Yahoo hired Doug Cutting, and Hadoop was spun out from Nutch as a separate project.
• 2008:
Cloudera, a major Hadoop vendor, was founded.
• 2009:
Doug Cutting joined Cloudera.
The Name "Hadoop"
• The name isn't an acronym—it's a made-up name.
• Doug Cutting explained that "Hadoop" was the name of his child’s stuffed yellow elephant. He wanted
something:
• Short
• Easy to spell and pronounce
• Meaningless, and not used elsewhere.
"The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless,
and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid's
term."
Subprojects and Naming
• Many Hadoop modules are named after animals or have whimsical names, like "Pig".
In summary:
Hadoop originated as a solution for large-scale web search and data processing, influenced by Google's
systems. Its development history is closely tied to open-source innovation and collaboration, and its unique
name reflects a playful, personal touch from its creator’s family.

Prof. Babitha P K 8 CIT, Ponnampet

 7. HADOOP OVERVIEW
Hadoop is an open-source software framework designed for storing and processing massive amounts of
data across clusters of inexpensive, commodity hardware. It accomplishes two major goals:
• Massive data storage
• Faster data processing (in parallel)
7.1 Key Aspects of Hadoop
1. Open-Source Software:
• Free to download, use, and contribute to.
2. Framework Approach:
• Provides all necessary tools and programs to develop and execute distributed data processing.
3. Distributed:
• Data is divided and stored across multiple connected computers (nodes).
• Processing tasks are distributed and performed in parallel, improving speed.

4. Massive Storage:
• Stores colossal volumes of data using low-cost hardware.
5. Faster Processing:
• Handles large-scale data processing in parallel for quick results.

Prof. Babitha P K 9 CIT, Ponnampet

7.2 Hadoop Components

Core Components
1. HDFS (Hadoop Distributed File System):
• Main storage system which splits and distributes data across multiple nodes.
• Built-in redundancy ensures data durability.
2. MapReduce:
• Programming model for processing and computing large datasets across distributed nodes in a
parallel fashion.
Hadoop Ecosystem (Supporting Projects)
• HIVE: Data warehousing, SQL-like queries
• PIG: High-level scripting for data analysis
• SQOOP: Data transfer between Hadoop and RDBMS
• HBASE: NoSQL database on Hadoop
• FLUME: Collecting and loading log data
• OOZIE: Workflow scheduling
• MAHOUT: Machine learning
7.3 Conceptual Layers of Hadoop
• Data Storage Layer:
Handles storing huge amounts of data (HDFS).
• Data Processing Layer:
Processes data in parallel to extract insights (MapReduce).
![Diagram of conceptual layers: 'Data Storage' and 'Data Processing'. Figure 5.9]
7.4 High-Level Architecture
Hadoop uses a Master-Slave architecture:
• Master Node (NameNode):
• Manages file system namespace and controls access (where to find what data).

Prof. Babitha P K 10 CIT, Ponnampet

 • Responsible for coordination, storage partitioning.
• Slave Nodes (DataNodes):
• Actually store the data and perform computation (processing tasks).
Each node runs components for:
• Computation (MapReduce)
• Storage (HDFS)
![High-level architecture diagram showing one master node and multiple slave nodes, each handling both
computation and storage. Figure 5.10]
In summary:
• Hadoop provides a scalable, fault-tolerant, and parallel-processing framework for big data.
• Its architecture of distributed storage (HDFS) and processing (MapReduce), plus robust ecosystem
tools, make it ideal for handling huge, diverse, fast-moving datasets efficiently and reliably.

Prof. Babitha P K 11 CIT, Ponnampet

 [Link] CASE OF HADOOP

8.1 ClickStream Data

• ClickStream data refers to the sequence of mouse clicks made by users as they navigate websites.
• By analyzing this data, businesses can understand customer purchasing behavior.
• Marketers use ClickStream analysis to optimize product pages, promotional content, and other aspects
of their websites to improve business outcomes

Why Use Hadoop for ClickStream Data?

Three Key Benefits:
1. Integration with Other Data Sources
• Hadoop enables joining ClickStream data with other business data sources such as:
• CRM (Customer Relationship Management)
• Customer demographics
• Sales data
• Advertising campaign data
• This comprehensive integration delivers deeper insights into customer behavior.
2. Cost-Effective Scalability
• Hadoop’s scalable architecture allows storage of years of clickstream data with minimal
incremental cost.
• This means you can perform historical and year-over-year analysis without the rising storage
costs that traditional systems may incur.
3. Flexible Analysis with Hive and Pig
• Business analysts can use Apache Hive or Apache Pig to:
• Organize clickstream data by user session
• Refine and preprocess it
• Feed it into visualization or advanced analytics tools
• These tools make it much easier to extract valuable patterns from massive clickstream datasets.
Key Summary Table

Prof. Babitha P K 12 CIT, Ponnampet

 Benefit Description

Joins ClickStream with CRM & sales data Enables combined analysis for deeper insights

Facilitates long-term analysis without high incremental

Stores years of data inexpensively cost

Analysis using Hive or Pig Streamlines data organization and analytics

In summary:
Hadoop is ideal for ClickStream data analysis because it enables integration with other datasets, handles huge
volumes of historical data efficiently and affordably, and provides tools (Pig, Hive) for powerful, flexible
analysis at scale

[Link] (HADOOP DISTRIBUTED FILE SYTEM)

What is HDFS?
HDFS (Hadoop Distributed File System) is the primary storage system of Hadoop, designed for storing and
managing large volumes of data reliably and efficiently across multiple nodes.
Key Features of HDFS
1. Storage Component of Hadoop:
HDFS is responsible for storing all the data in a Hadoop cluster.
2. Distributed File System:
It spreads data across multiple computers (nodes) and allows parallel reading and writing, improving
speed and fault tolerance.
3. Modeled after Google File System:
The design draws inspiration from Google’s GFS, using similar ideas like block storage and metadata
management.
4. Optimized for High Throughput:
HDFS uses large block sizes and tries to move computation closer to where data resides (“data locality”)
for efficient processing of big files.
5. Replication for Fault Tolerance:
Each file is split into blocks (default 64MB) and each block is replicated (default 3 copies) across the
cluster. This ensures data is safe even if nodes fail.
6. Automatic Block Re-Replication:
If a node fails, HDFS automatically re-copies (“re-replicates”) its blocks to healthy nodes to maintain
the required replication factor.

Prof. Babitha P K 13 CIT, Ponnampet

 7. Handles Very Large Files:
HDFS excels at storing and processing files that are gigabytes or terabytes in size.
8. Runs on Top of Existing File Systems:
It operates over native OS file systems like ext3/ext4, managing how and where files are stored and
retrieved.

Architectural Highlights
• Block Structured:
Files are broken into blocks. Each block is replicated and stored on different nodes for fault tolerance
and efficient access.
• Default Block Size:
Each block is typically 64MB (can be configured).
• Default Replication Factor:
By default, each block has 3 copies on different nodes.
9.1 HDFS Daemons
• When a file (e.g., “[Link]” of size 192MB) is stored, it is split into three blocks of 64MB each. Each
block is replicated three times (default setting) and distributed across different nodes.
9.1.1 NameNode: Manages all file system metadata (file names, block locations, permissions). It ensures files
are split and replicated as per configuration.
The NameNode is the master server in the HDFS architecture. Its primary role is to manage the file system
namespace. Here are its key responsibilities:

Prof. Babitha P K 14 CIT, Ponnampet

 • Manages Metadata: Keeps records of the directory structure, file names, permissions, and the location
of blocks within the cluster.
• Stores the Namespace: The complete mapping of file names to blocks and blocks to DataNodes is
called the file system namespace. This is stored in a file called the FsImage.
• Tracks Transactions: Every change to the metadata (like creating, deleting, or renaming files) is
logged in an EditLog (transaction log).
• Rack Awareness: Uses rack IDs to identify DataNodes in different racks, optimizing data placement
and network utilization.
• No Data Storage: The NameNode does NOT store the actual data—only the metadata.
• Single Point of Management: Typically, there is one active NameNode per cluster (for high
availability, a standby may exist).

You give a file to Hadoop

Example: You upload [Link] using the Hadoop File System Client.
NameNode stores metadata
The NameNode doesn’t store your file itself.
• It keeps information about the file → file name ([Link]), file size, how many blocks it has, and
where each block is stored.

Prof. Babitha P K 15 CIT, Ponnampet

 • In the diagram, [Link] is divided into Block A, Block B, Block C.
DataNodes store the actual data
• Block A, B, C (the actual content of your file) are stored in DataNode A, DataNode B, DataNode
C.
• Notice: Each block is stored in multiple DataNodes (replication). This keeps data safe even if one
DataNode crashes.
How client reads/writes data
• When you want to read/write [Link], you first talk to the NameNode.
• NameNode tells you: “Block A is on DataNode A, B, and C; Block B is on … etc.”
• Then, the client directly goes to the DataNodes to get the actual content.
FsImage and EditLog (bottom of diagram)
• FsImage → A snapshot file, like a master catalog of the whole file system (all metadata).
• EditLog → A diary/journal that records every change (new file, delete, rename, etc.).
Together, they help the NameNode recover if it restarts or crashes.

9.1.2 DataNodes: Store the actual data blocks on their local disks.
If a DataNode fails, HDFS automatically makes new replicas to maintain the set replication factor.
The DataNodes are the worker nodes in HDFS and are responsible for storing the actual file data in the form of
blocks. Key points about DataNodes include:
• Stores Data Blocks: Each DataNode stores a set of blocks on its local disks. The files are split into
blocks (default: 64MB), and each block is replicated as per the replication factor (default: 3).
• Handles Read/Write Requests: DataNodes serve client read and write requests by accessing the
physical blocks.
• Block Reports: Regularly send block reports and heartbeats to the NameNode to confirm availability
and block status.
• Re-Replication: If a DataNode fails or a block is lost, the HDFS system uses other DataNodes’ replicas
to restore the lost blocks automatically.
• Scalability and Fault Tolerance: Data blocks are spread and replicated across many DataNodes,
ensuring fault tolerance and improved performance.
Together, the NameNode and DataNodes form the backbone of HDFS, allowing for distributed, fault-tolerant,
and highly scalable data storage and retrieval.

Prof. Babitha P K 16 CIT, Ponnampet

1. Heartbeat
• Each DataNode keeps telling the NameNode:
→ “Hello, I’m alive! I’m working fine.”
• This small message is called a heartbeat.
• It’s like a worker checking in with the boss every few seconds.

2. No Heartbeat
• If a DataNode stops sending heartbeat (maybe it crashed, got disconnected, or failed),
the NameNode thinks that DataNode is dead.
• Example: Middle DataNode in the picture → “No heartbeat”.
• The boss (NameNode) decides: “Ok, I cannot trust this worker anymore.”

3. Replication
• To keep data safe, Hadoop always keeps copies (replicas) of every block.
• If one DataNode dies (no heartbeat), the NameNode makes sure the missing block is copied to another
healthy DataNode.
• That’s what the “Replicates” arrow means → data is copied to keep 3 safe copies (by default)

Summary Table

Key Point Description

Storage Component Stores all Hadoop data (block-based)

Distributed Data spread and replicated across many nodes

Prof. Babitha P K 17 CIT, Ponnampet

 Key Point Description

Block Size Default is 64MB

Replication Factor Default is 3 (can be configured)

Fault Tolerance Replicates data, recovers from node failure

NameNode Stores metadata; manages namespace and block maps

DataNode Stores actual data blocks

In summary:
HDFS enables Hadoop to store and process huge, diverse datasets affordably and reliably. Its architecture
ensures data is consistently available and protected from hardware failures, while supporting high-throughput
analysis for big data tasks.

9.1.3 Secondary NameNode:

Secondary NameNode
• Purpose:
The Secondary NameNode periodically takes a snapshot of HDFS metadata (the information that tells
Hadoop which data blocks belong to which file, where they are stored, etc.) at intervals specified in the
Hadoop configuration.
• Functionality:
Unlike its name might suggest, the Secondary NameNode is not a backup NameNode. Instead, it helps
the main NameNode by:
• Merging the FsImage (a persistent snapshot of HDFS metadata) and the EditLog (which records
all recent changes to metadata).
• Producing a new, up-to-date FsImage file, which helps prevent the EditLog from growing too
large and keeps recovery times manageable in case of failure.
• Memory Requirements:
The memory requirements for the Secondary NameNode are similar to those of the main NameNode.
• Recommendation: It is best practice to run the NameNode and Secondary NameNode on
different machines to avoid overloading one machine.
• Disaster Recovery:
• If the main NameNode fails, the Secondary NameNode can be manually used to help restore the
cluster—but not automatically.

Prof. Babitha P K 18 CIT, Ponnampet

 • The Secondary NameNode does not continuously record real-time HDFS metadata changes; it
only has the latest snapshot from its last merge operation.
Summary:
The Secondary NameNode performs routine housekeeping for HDFS metadata, supporting the NameNode by
periodically merging its metadata files. It helps keep the system efficient and assists in recovery, but it does not
serve as a real-time backup or replacement for the NameNode.

9.2 anatomy of file read

Suppose you have a file stored in HDFS (like a document or dataset).

Now, you (the client) want to read that file.
To do this, your program uses the HDFS Client API (a library that lets programs talk
to HDFS).
You call a function named open() from the DistributedFileSystem class.

Steps in HDFS File Read

1. Client Initiates Read (Open):
• The client (using the HDFS Client API) wants to read a file and calls open() on the Distributed
FileSystem.

Prof. Babitha P K 19 CIT, Ponnampet

 2. Find Block Location (NameNode):
• The Distributed FileSystem communicates with the NameNode to request the location of the
file’s data blocks.
• The NameNode responds with the addresses of the DataNodes that store replicas of each block.
3. Input Stream Initialization:
• The DistributedFileSystem returns a FSDataInputStream object to the client.
• This stream enables the client to read from the file. It contains information about which
DataNodes hold the required blocks.
4. Reading Data from DataNodes:
• The client uses the FSDataInputStream to call read().
• For the first file block, the client connects to the nearest (most optimal) DataNode containing
that block.
5. Streaming Blocks:
• The client repeatedly calls read() to stream the data from the connected DataNode.
• When it reaches the end of the current block, the client closes the connection to that DataNode.
• The input stream then moves to the next block, connecting t o the best DataNode for that block,
and the process continues until the whole file is read.
6. Closing Connection:
• After the file is completely read, the client calls close() on the FSDataInputStream to close the
connection and end the operation.
Diagram Summary
• The diagram visually shows the interaction:
• The client (Client JVM) opens a file, gets DataNode locations from the NameNode, and then
directly reads data from one or more DataNodes.
Key Points
• NameNode is only involved at the block-location lookup stage (not during actual data streaming).
• DataNodes serve the file blocks directly to the client.
• Parallel and Sequential Reading: If a file contains multiple blocks, the client retrieves each block from
an optimal DataNode, one at a time, until the full file is read.
• Efficiency: This design maximizes throughput and avoids overloading the NameNode during data-
heavy operations.
This sequence ensures efficient, reliable, and scalable file access across the Hadoop cluster.

9.3 Anatomy of file write

Steps in HDFS File Write:
1. Client Requests File Creation

Prof. Babitha P K 20 CIT, Ponnampet

 o The HDFS Client (running inside Client JVM) asks the DistributedFileSystem to create a new
file.
2. NameNode Involvement
o The DistributedFileSystem contacts the NameNode to create a file entry (metadata only, no
data yet).
o The NameNode checks if the file already exists, user permissions, etc.
o If valid, it creates an empty file entry in its namespace.
3. Write Starts
o The client starts writing data to the file using FSDataOutputStream.
4. Data Written in Packets → Sent to DataNodes
o The data is split into packets.
o These packets are sent to a pipeline of DataNodes (as per replication factor, usually 3).
o Example: Client → DataNode1 → DataNode2 → DataNode3.
5. Acknowledgement Packets (Ack)
o Each DataNode stores the packet and then sends an acknowledgment back through the pipeline:
▪ DataNode3 → DataNode2 → DataNode1 → Client.
o This ensures all replicas have safely stored the data before moving to the next packet.
6. Close Operation
o After finishing all writes, the client calls close() on FSDataOutputStream.
7. Completion Notification
o Finally, the DistributedFileSystem informs the NameNode that the file write is complete.
o The NameNode updates its metadata to mark the file as closed and ready for use.

Prof. Babitha P K 21 CIT, Ponnampet

9.4 Replica Placement Strategy
9.4.1 Hadoop Default Replica Placement Strategy—the method Hadoop uses to decide where to store the
multiple copies (replicas) of each data block for reliability and fault tolerance.
Steps of the Default Replica Placement Strategy
1. First Replica
• Placed on the same node as the client writing the data. This optimizes for write performance
since one copy is immediately local.
2. Second Replica
• Placed on a node that is on a different rack from the first.
• Racks are physical groupings of nodes; having replicas on different racks protects data from
rack-level failures (like a network switch or power outage).
3. Third Replica
• Placed on the same rack as the second replica, but on a different node within that rack. This
balances fault tolerance and network usage.
Key Points of the Strategy
• Pipeline Creation:
Once replica locations are determined, Hadoop sets up a pipeline between these nodes for data transfer
and replication during writes.

Prof. Babitha P K 22 CIT, Ponnampet

 • Reliability:
By placing one replica on the local node, one on a remote rack, and one more on the same remote rack,
Hadoop ensures that data is protected from both node and rack failures, while also reducing cross-rack
network traffic.
In summary, this strategy ensures a balance of reliability (protection from failures) and network efficiency by
distributing replicas intelligently across nodes and racks within the Hadoop cluster

Steps of the Default Replica Placement Strategy

1. First Replica
• Placed on the same node as the client writing the data. This optimizes for write performance
since one copy is immediately local.
2. Second Replica
• Placed on a node that is on a different rack from the first.
• Racks are physical groupings of nodes; having replicas on different racks protects data from rack-
level failures (like a network switch or power outage).
3. Tshird Replica
• Placed on the same rack as the second replica, but on a different node within that rack. This
balances fault tolerance and network usage.
Key Points of the Strategy

Prof. Babitha P K 23 CIT, Ponnampet

 • Pipeline Creation:
Once replica locations are determined, Hadoop sets up a pipeline between these nodes for data transfer
and replication during writes.
• Reliability:
By placing one replica on the local node, one on a remote rack, and one more on the same remote rack,
Hadoop ensures that data is protected from both node and rack failures, while also reducing cross-rack
network traffic.
In summary, this strategy ensures a balance of reliability (protection from failures) and network efficiency by
distributing replicas intelligently across nodes and racks within the Hadoop cluster.

9.5 Working with HDFS Command

Common HDFS (Hadoop Distributed File System) commands and their objectives. Here’s
an explanation of each command along with its use:

1. Listing Directories and Files at the Root of HDFS

• Objective: Get the list of directories and files at the root of HDFS.
• Command:
hadoop fs -ls /

-ls → list files and directories.

/ → root directory in HDFS.

2. Listing All Directories and Files Recursively

• Objective: Get the list of all directories and files in HDFS recursively.
• Command:
hadoop fs -ls -R /

-R → recursive (goes inside subdirectories too).

This shows all files and folders in HDFS.

3. Creating a Directory
• Objective: Create a directory (e.g., sample) in HDFS.
• Command:
hadoop fs -mkdir /sample

Prof. Babitha P K 24 CIT, Ponnampet

-mkdir → make directory.
/sample → directory name in HDFS.
Creates a new directory named /sample.

4. Copying a File from Local to HDFS

• Objective: Copy a file from the local file system to HDFS.
• Command:
hadoop fs -put /root/sample/[Link] /sample/[Link]

put → upload file from local filesystem to HDFS.

First path → local file location.
Second path → destination in HDFS.
Uploads [Link] from the local path to the /sample directory in HDFS.

5. Copying a File from HDFS to Local

• Objective: Copy a file from HDFS to the local file system.
• Command:
hadoop fs -get /sample/[Link] /root/sample/[Link]
-get → download file from HDFS to local filesystem.
First path → HDFS file.
Second path → local destination.
download [Link] from your local machine to HDFS.

6. Copy from Local Using copyFromLocal

• Objective: Copy a file from local file system to HDFS using copyFromLocal.
• Command:
hadoop fs -copyFromLocal /root/sample/[Link] /sample/[Link]
Another way to upload a local file to HDFS.
-copyFromLocal works like -put.

7. Copy to Local Using copyToLocal

• Objective: Copy a file from HDFS to local file system using copyToLocal.
• Command:
hadoop fs -copyToLocal /sample/[Link] /root/sample/[Link]

Prof. Babitha P K 25 CIT, Ponnampet

-copyToLocal works like -get.
Another command to download a file from HDFS to local.

8. Displaying Contents of a File

• Objective: Display the contents of a file stored in HDFS on the console.
• Command:
hadoop fs -cat /sample/[Link]

-cat → shows file content on the terminal.

Outputs the contents of [Link] from HDFS.

9. Copying a File within HDFS

• Objective: Copy a file from one directory to another within HDFS.
• Command:
hadoop fs -cp /sample/[Link] /sample1

-cp → copy within HDFS.

First path → source file.
Second path → destination directory.
Copies [Link] to /sample1.

10. Removing a Directory

• Objective: Remove a directory from HDFS.
• Command:
hadoop fs -rm -r /sample1

-rm → remove file/directory.

-r → recursive (needed for directories).
Deletes the /sample1 directory and all its contents recursively.

These commands are fundamental for managing files and directories in HDFS, enabling users to list, copy,
move, display, and delete data efficiently in Hadoop environments.

9.6 Special Features of HDFS

Prof. Babitha P K 26 CIT, Ponnampet

1. Data Replication
• What it is: HDFS automatically makes multiple copies (replicas) of every data block.
• Why it matters: Clients (applications accessing HDFS) do not need to keep track of where each piece of
data is stored. If you need to read or write data, the system automatically directs the operation to the
nearest available copy (replica) to ensure high performance and reliability.
• Benefit: This ensures that even if some machines fail, the data remains safe and quickly accessible.
2. Data Pipeline
• What it is: When a client writes data to HDFS, it doesn't write to all replicas at once. Instead, it sends
the data block to the first DataNode (storage server) in a list called a "pipeline."
• How it works:
• The first DataNode writes the block, then passes it to the second DataNode, which in turn passes
it to the third, and so on, until all replicas receive the data.
• This process continues for all blocks of the file, making sure every replica is written in order.
• Benefit: This pipeline process allows for efficient and reliable replication of blocks, improves write
performance, and maintains data integrity across the system.
[Link] DATA WITH HADOOP
MapReduce is a software framework that processes massive amounts of data in parallel by splitting the input
data into independent chunks.
10.1Mapreduce Daemons
The two main roles in the MapReduce framework are:
• JobTracker (Master): JobTracker is the boss in Hadoop’s MapReduce.
• When you give Hadoop some code (a job), the JobTracker:
• Makes a plan for how the work will be split.
• Assigns small tasks to different computers (nodes).
• Keeps an eye on all tasks.
• If something fails, it gives the work to another node.
• There is only one JobTracker in the whole cluster.

• TaskTracker (Slave):
• TaskTracker is the worker in Hadoop’s MapReduce.
• Every computer (node) has one TaskTracker.
• The TaskTracker:
• Gets tasks from the JobTracker and runs them.
• Starts multiple JVMs (small programs) to do work in parallel.
• Sends a heartbeat message regularly to the JobTracker to say “I’m alive and working.”

Prof. Babitha P K 27 CIT, Ponnampet

 • If the JobTracker doesn’t hear from it, it thinks the TaskTracker has failed.

Phases in MapReduce:
• Map: Converts inputs into key-value pairs, with each map task operating in parallel.
Example:(map)
Input: A list of words in a file.
Map output:
• (cat, 1)
• (dog, 1)
• (cat, 1)

• Reduce: Combines outputs from the mappers and generates a reduced result, aggregating the map
outputs.
Reduce input:
 (cat, [1,1])
 (dog, [1])

ssReduce output:

• (cat, 2)
• (dog, 1)

Prof. Babitha P K 28 CIT, Ponnampet

10.2 How Does MapReduce Work?
MapReduce is a programming model for processing large amounts of data in parallel across a distributed cluster
(multiple computers). It breaks the data analysis task into two main stages:
• Map: Processes and transforms input data into key/value pairs.
• Reduce: Aggregates and summarizes the output from the map stage.

Components & Architecture

It divides work into two phases:

1. Map Phase → Takes input data, processes it, and converts it into key-value pairs.

Prof. Babitha P K 29 CIT, Ponnampet

2. Reduce Phase → Aggregates (combines) and summarizes the output from the map phase.

The Components in the Diagram

1. Client
The user/application that submits a job (task to be executed).
2. JobTracker (Master Node)
Receives the job from the client.
Divides the job into smaller tasks.
Assigns tasks to TaskTrackers on different worker nodes.
Monitors the progress and handles failures.
3. TaskTracker (Worker Node)
Executes the tasks (Map and Reduce) assigned by the JobTracker.
Reports the task status (success/failure) back to the JobTracker.
4. Map Tasks
Process raw input data.
Transform it into intermediate key-value pairs.
Example: Counting words → ("apple", 1), ("banana", 1).
5. Reduce Tasks
Collect all key-value pairs from the Map tasks.
Aggregate them to give the final result.
Example: Combine counts → ("apple", 5), ("banana", 3).

Flow of Execution

1. Client submits a job → goes to JobTracker.

2. JobTracker splits job into tasks → assigns them to multiple TaskTrackers.
3. Each TaskTracker runs Map tasks first.
These produce intermediate results (key-value pairs).
4. Intermediate results are shuffled and sent to Reduce tasks.
5. TaskTracker executes the Reduce tasks.
6. Final results are sent back to the Client.

Prof. Babitha P K 30 CIT, Ponnampet

 • Figure 5.22, 5.24: Shows JobTracker managing multiple TaskTrackers (each running map or reduce
tasks).
• Figure 5.23: Depicts data flow: data split → map tasks → partition → reduce task → final output.
Key Points
• Parallelism: Map tasks run in parallel on different nodes for speed.
• Fault Tolerance: JobTracker supervises; failed tasks are rescheduled.
• Scalability: Easily handles huge datasets by distributing work.
In essence, MapReduce takes large data, chops it up, processes pieces in parallel (map), then combines results
in an efficient, scalable, and fault-tolerant way (reduce).

Prof. Babitha P K 31 CIT, Ponnampet

10.3 MapReduce Example

Objective
Count the number of occurrences of each word in a large collection of text files using Hadoop MapReduce.
Key Components
1. Driver Class
• Sets up and configures the job (specifies mapper, reducer, input/output paths, etc.)
• Example: WordCounter class
2. Mapper Class

Prof. Babitha P K 32 CIT, Ponnampet

 • Reads lines of text, splits them into words, and outputs each word as a key with value 1.
• Example: WordCounterMap class
3. Reducer Class
• Sums up the counts for each word key to get the total occurrences.
1) This is a Hadoop MapReduce program to count the number of times each word
appears in a text file.
[Link]: DRIVER PROGRAM

What this driver does:

 Creates a Hadoop job.
 Attaches the Mapper (WordCounterMap) and Reducer (WordCounterRed).
 Sets input file and output folder.
 Submits job to Hadoop framework for execution.

Package [Link]; /Defines the Java package.

The program imports Hadoop classes:
Import [Link]; /Needed for input/output error handling.
For example, if Hadoop fails to read/write files, IOException is thrown.
import [Link]; /Hadoop uses its own special class → Path.
In Hadoop, we don’t use normal Java File.
Instead, we use Hadoop’s Path class to point to HDFS files.
import [Link];
import [Link];
Hadoop doesn’t use normal Java types (int, String) because they are not serializable by default.
IntWritable = Hadoop’s version of int
Text = Hadoop’s version of String
import [Link];
import [Link];
Job is used to set everything about the MapReduce program (mapper, reducer, inpoutput, etc).
import [Link];
import [Link];
Base classes for writing Mapper and Reducer.
import [Link];
import [Link];
import [Link];

Prof. Babitha P K 33 CIT, Ponnampet

import [Link];
These are ready-made input/output classes.
 FileInputFormat → Reads files from HDFS as input.
 TextInputFormat → Treats each line of text as one record.
 FileOutputFormat → Writes job’s result to HDFS.
 TextOutputFormat → Writes output as plain text.
public class WordCounter {
Defines the main driver class of the program.
public static void main (String [] args) throws Exception {
• Entry point of the program (when you run it).
• throws Exception means: if something goes wrong (I/O, Hadoop error), program exits.
Job job = new Job();
• Creates a job configuration object.
• Hadoop uses this to know what to run.
[Link]("wordcounter");
Helpful for logs and debugging.
When you run Hadoop jobs, you’ll see the job name in tracking UI.
[Link]([Link]);
Tells Hadoop which JAR file contains your program.
It uses this class (WordCounter) to locate the JAR.
What happens in Hadoop jobs
1. Your MapReduce program (Driver, Mapper, Reducer) is compiled into .class files.
2. Those .class files are packaged into a JAR file (like a zip file for Java programs).
Example: [Link]
3. When you run your program on Hadoop, that JAR file must be sent to all the computers (nodes)
in the Hadoop cluster so they can run the Mapper and Reducer.

.[Link]([Link]);
.[Link]([Link]);
• Tells Hadoop which Mapper and Reducer to use.
• These are user-defined classes (you’ll write them separately).
[Link]([Link]);
[Link]([Link]);
• Defines the output data type from the Reducer.
• Output will be → (word, count)
• word → Text

Prof. Babitha P K 34 CIT, Ponnampet

 • count → IntWritable
[Link](job, new Path("/sample/[Link]"));
[Link](job, new Path("/sample/wordcount"));
 Tells Hadoop where to read input from (/sample/[Link])
 And where to write results (/sample/wordcount)
 Both are HDFS paths (not local file system).
[Link]([Link](true) ? 0 : 1);
}
}
 [Link](true) → Submits job to Hadoop and waits until it finishes.
 If job is successful → exit with code 0.
 If job fails → exit with code 1.

What are .class files?

• In Java, when you write code, you save it in a .java file.
• This file is just plain text (human-readable).
• To run it, Java first compiles it using the Java compiler (javac).
• The compiler converts your .java file into a .class file.
.class files contain Java bytecode (machine-readable instructions that the JVM can run).

What Happens When You Run It

1. Input file ([Link]) contains some text.
2. Mapper splits lines into words → emits (word, 1).
3. Hadoop shuffles/sorts words.
4. Reducer adds up counts → emits (word, total_count).
5. Output is saved in /sample/wordcount/part-r-00000.

Example:
If [Link] contains:
hello world
hello Hadoop
Output will be:
hadoop 1
hello 2

Prof. Babitha P K 35 CIT, Ponnampet

 world 1
[Link] MapReduce program for Word Count.
package [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
public class WordCounterMap extends Mapper<LongWritable, Text, Text, IntWritable>
{
Defines the Mapper class.
 <LongWritable, Text, Text, IntWritable> means:
 Input key → LongWritable (position of line in file).
 Input value → Text (the actual line of text).
 Output key → Text (the word).
 Output value → IntWritable (the count = 1).
@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
• This is the map method that Hadoop calls automatically for each line of the input.
• Parameters:
• key → line offset in file (not usually used in word count).
• value → the actual text line.
• context → used to emit output (key-value pairs).
• throws IOException, InterruptedException → handles errors that may occur during processing.
String[] words = [Link]().split(",");
• [Link]() → Convert the Hadoop Text object into a normal Java String.
• .split(",") → Split the line into words, using comma (,) as a separator.
So if a line is: "apple,banana,apple" → it becomes an array ["apple", "banana", "apple"].
for (String word : words) {
Loop through each word in the array.
Example: For ["apple", "banana", "apple"], it loops 3 times.
[Link](new Text(word), new IntWritable(1));

Prof. Babitha P K 36 CIT, Ponnampet

}
}
}
 Emits the key-value pair (word, 1) for each word found.
 Example:
"apple" → (apple, 1)
"banana" → (banana, 1)
"apple" → (apple, 1)
 These outputs are sent to Hadoop’s Shuffle & Sort phase before going to the Reducer.
3)[Link]:Reduce Class
package [Link];
Defines that this class belongs to the package [Link].
Packages are like folders in Java used to organize classes.

import [Link];
import [Link];
import [Link];
import [Link];

public class WordCounterRed extends Reducer<Text, IntWritable, Text, IntWritable> {

• Defines the Reducer class.
• <Text, IntWritable, Text, IntWritable> means:
o Input key → Text (the word).
o Input values → IntWritable (list of counts, usually many 1s).
o Output key → Text (the word).
o Output value → IntWritable (the final count).

@Override
protected void reduce(Text word, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
• This is the reduce method that Hadoop calls automatically for each unique key (word).
• Parameters:
o word → one unique word (e.g., "apple").
o values → all the counts (e.g., [1, 1, 1, 1]).
o context → used to emit final results (word, total_count).
Integer count = 0;

Prof. Babitha P K 37 CIT, Ponnampet

 • Initialize a local counter variable.
• This will be used to sum up all the counts for the current word.

for (IntWritable val : values) {

count += [Link]();
}
• Loops through the list of counts (values).
• [Link]() converts Hadoop’s IntWritable into a normal int.
• Adds each count to count.
• Example: for "apple" with values [1,1,1], after loop → count = 3.

[Link](word, new IntWritable(count));

}
}
• Emits the final result for that word.
• Example: (apple, 3)
• This gets written to the output file in HDFS.

Summary of Reducer
• Input: (word, [list of counts])
• Process: sum all counts.
• Output: (word, total_count)
Example after Mapper + Shuffle:
(apple, [1,1,1])
(banana, [1])
Reducer output:
apple 3
banana 1

3. Summary Table: SQL vs. MapReduce

After the Reducer code, there's a table comparing SQL and MapReduce. Here are the key differences:

SQL MapReduce

Access Interactive and Batch Batch

Prof. Babitha P K 38 CIT, Ponnampet

 SQL MapReduce

Structure Static Dynamic

Updates Read/write many times Write once, read many times

Integrity High Low

Scalability Nonlinear Linear

• SQL: Used for interactive/batch queries on structured data; supports many reads/writes with high
integrity.
• MapReduce: Used for batch processing on large/possibly unstructured data; typically processes in one
go, is highly scalable and fault-tolerant, but has lower data integrity and is not interactive.
In Short
• The Reducer sums all counts per word sent by the Mappers and emits (word, totalCount).
• Where SQL is for interactive structured data analysis, MapReduce is all about scalable, batch processing
of vast data amounts.

[Link] RESOURCES AND APPLICATIONS WITH HADOOP YARN (YET

ANOTHER RESOURCES NEGOTIATOR)
What is Hadoop YARN?
• YARN is a core component introduced in Hadoop 2.x.
• Stands for Yet Another Resource Negotiator.
• It’s responsible for cluster resource management and enables running multiple types of applications
(not just MapReduce) on Hadoop.
• YARN allows Hadoop to support various processing models: batch, interactive, streaming, graph, and
more.
11.1 Limitations of Hadoop 1.0 Architecture
• Single NameNode Bottleneck: In Hadoop 1.0, only one NameNode manages the entire file system's
metadata for the Hadoop cluster. If it becomes overloaded or fails, it can disrupt the whole cluster.
• Restricted Processing Model: Hadoop 1.0 is built mainly for batch-oriented processing using
MapReduce. It doesn’t support interactive, online, or streaming workloads.
• Not Suitable for Interactive or Advanced Analysis: The MapReduce model is poorly suited for real-
time, interactive analytics, or in-memory machine learning.
• Resource Utilization Issues: Resource slots are rigidly divided into "map" and "reduce" slots.
Sometimes, map slots are full while reduce slots sit idle, and vice versa, resulting in inefficient resource
use.

Prof. Babitha P K 39 CIT, Ponnampet

 • MapReduce Constrained: MapReduce is responsible for both resource management and data
processing, leading to limited flexibility and scalability.
11.2 HDFS Limitation
Metadata Bottleneck: All file system metadata is stored in the RAM of a single NameNode. Even with
increased RAM, there’s a hard limit on scalability. As data and files grow, the NameNode risks running out of
memory and becoming a bottleneck.
11.3 Hadoop 2: HDFS
Components:
• (a) Namespace: Handles files/directories, manages creation, modification, and deletion.
• (b) Blocks Storage Service: Handles data blocks, replication, and storage management.
HDFS 2 Features:
1. Horizontal Scalability: Uses federation of multiple independent NameNodes, allowing more clusters to
scale out and store bigger datasets.
2. High Availability: Passive Standby NameNode enables a backup NameNode that can take over
automatically if the active one fails, preventing single points of failure.
How It Works:
• Multiple NameNodes, each with its own namespace, share the DataNodes for block storage—enabling
parallel management and failover.
• All namespace changes are saved to shared storage. If the Active NameNode fails, the Passive Standby
is updated and quickly takes over.
11.4 Hadoop 2 YARN: Taking Hadoop Beyond Batch
YARN helps us to store all data in one place. WE can interact in multiple ways to get predictable performance
and quality of [Link] was originally architected by Yahoo.

Prof. Babitha P K 40 CIT, Ponnampet

 • Figure 5.26: Shows active and passive NameNode interaction—high availability and failover.
• Figure 5.27: Compares Hadoop 1.x (HDFS + MapReduce) with Hadoop 2.x (HDFS + YARN + diverse
applications).
• Active NameNode: This is the primary NameNode that handles all client requests (read/write
operations) in HDFS.
• Passive NameNode (Standby): This is a backup NameNode that remains in standby mode. It does not
serve requests directly but is kept updated.
• Shared Edit Logs: Both Active and Passive NameNodes share the edit log storage.
• The Active NameNode writes changes (metadata updates) into the shared log.
• The Passive NameNode continuously reads these logs to stay synchronized with the Active NameNode.
• Purpose: This setup ensures high availability (HA). If the Active NameNode fails, the Passive can
quickly take over without major downtime
Figure 5.27: Hadoop 1.x vs Hadoop 2.x
• This figure shows the architectural difference between Hadoop versions.
• Hadoop 1.x
• Consists of two main components:
• 1. MapReduce: Handles both cluster resource management and data processing.
• 2. HDFS (Hadoop Distributed File System): Provides redundant and reliable storage.
• Limitation: MapReduce was tightly coupled with resource management → it could only run
MapReduce jobs, no support for other processing models.

Prof. Babitha P K 41 CIT, Ponnampet

 • Hadoop 2.x
• Enhanced architecture by separating cluster management from data processing:
• 1. YARN (Yet Another Resource Negotiator): A new resource manager that handles cluster
management and job scheduling.
• 2. MapReduce: Still present but now only for data processing.
• 3. Others: Hadoop 2.x supports other data processing engines like Spark, Tez, etc., not just MapReduce.
• 4. HDFS: Continues to provide redundant and reliable storage.
• Advantages of Hadoop 2.x:
• Supports multiple frameworks beyond MapReduce (e.g., Spark, Storm).
• Better resource utilization through YARN.
• Scalability and flexibility improved.
•

• Figure 5.28: Illustrates how YARN manages resources for various processing types above HDFS2
(batch, interactive, streaming, in-memory, etc.

• This diagram is about Hadoop YARN (Yet Another Resource Negotiator).

• It shows how YARN manages different kinds of applications in a Hadoop cluster.
• 1. Applications Supported
• At the top, different kinds of applications are shown that can run on Hadoop with the help of
YARN:
• Batch (MR) → Traditional MapReduce batch jobs.
• Interactive (TEZ) → Framework for fast interactive query processing (used in Hive, Pig).
• Online (HBASE) → Real-time, online database processing with HBase.
• Streaming (Storm) → Stream data processing in real time.
• In-Memory (Spark) → In-memory analytics for very fast data computation.
• Others (Graph, Search) → Specialized applications like graph processing (e.g., Giraph) or search
engines.
[Link] Resource Management
• This is the YARN core layer.

Prof. Babitha P K 42 CIT, Ponnampet

 • YARN allocates cluster resources (CPU, memory, etc.) and manages scheduling among all types of
applications.
• Ensures fairness and efficient utilization of resources.
3. HDFS 2
• At the bottom, HDFS (Hadoop Distributed File System) is shown as the storage layer.
• All applications above use HDFS to store and retrieve data.
• The figure shows that Hadoop YARN sits in the middle layer – it acts as a resource manager for
the entire cluster.
• Different applications (batch, interactive, online, streaming, in-memory, graph/search) run on top
of YARN, while HDFS provides the underlying storage at the bottom.

11.4.1 Fundamental Idea

YARN (Yet Another Resource Negotiator) was introduced to split the JobTracker’s responsibilities in Hadoop:
resource management and job scheduling/monitoring. It divides these tasks among different daemons,
improving scalability and flexibility.
Key Components
1. Client – Submits jobs to YARN.
2. ResourceManager (RM) – The master that manages resources across the cluster.
3. NodeManager (NM) – Runs on each node and manages containers on that node.
4. ApplicationMaster (AppMstr) – Created per application; negotiates resources from RM and
works with NodeManagers.
5. Container – A resource allocation (CPU, memory, etc.) where tasks run.

Flow of Execution (Steps in Diagram):

1. Job Submission –
The client submits the application/job to the ResourceManager.
2. Start ApplicationMaster –
The RM contacts a NodeManager to start an ApplicationMaster for that job in a container.
3. Resource Request –
The ApplicationMaster communicates with the ResourceManager, asking for resources
(containers) to run tasks.
4. Resource Allocation –
The ResourceManager allocates containers across various NodeManagers.
5. Launch Containers –
The ApplicationMaster communicates with the NodeManagers to launch containers.

Prof. Babitha P K 43 CIT, Ponnampet

 6. Task Execution –
Containers run the actual tasks (map, reduce, or other processing logic).
7. Client Communication –
The client communicates with the ResourceManager or directly with the ApplicationMaster for
status updates.
8. Progress & Completion –
The ApplicationMaster monitors tasks, reports progress to the ResourceManager, and releases
resources once the job completes.

CHAPTER-2
Topics

Introduction to Map Reduce Programming: Introduction, Mapper, Reducer, Combiner,

Partitioner,Searching, Sorting, Compression

1. INTRODUCTION
• MapReduce jobs are divided into two types of tasks: map tasks and reduce tasks. These tasks run in a
distributed manner across the nodes (computers) in a Hadoop cluster. Each node handles a small subset
of data, allowing the entire job to be processed efficiently and in parallel.
• The data to be processed is stored in HDFS (Hadoop Distributed File System). Hadoop splits the
processing into the following:

Prof. Babitha P K 44 CIT, Ponnampet

 • Map tasks handle loading, parsing, transforming, and filtering the data.
• Reduce tasks group and aggregate the intermediate data from map tasks to generate the final
output.
Map Task Phases:
1. RecordReader: Reads data from HDFS and prepares it for the mapper.
2. Mapper: Processes input data and produces key-value pairs.
3. Combiner: Optionally aggregates intermediate data to reduce data transfer.
4. Partitioner: Determines how the data is distributed to different reducers.
Reduce Task Phases:
1. Shuffle: Moves intermediate key-value pairs from mappers to reducers.
2. Sort: Sorts the data based on the key.
3. Reducer: Processes the sorted data and generates the final results.
4. Output Format: Writes the final output to HDFS.
Hadoop assigns map tasks directly to the node where the data physically lives (data locality), which means
large volumes of data do not need to be moved across the network—only the computational code is sent to the
data. This approach saves bandwidth and speeds up processing

[Link]
A mapper maps the input key-value pairs into a set of intermediate key-value pairs. Maps are individual tasks
that have the responsibility of transforming input records into intermediate key-value pairs.

Prof. Babitha P K 45 CIT, Ponnampet

 [Link]
The Reducer performs the final processing in a MapReduce job. Its primary task is to take the intermediate
key-value pairs generated by the Mappers (and shuffled and sorted), and combine them into a smaller set of
results. The process happens in three main steps:
1. Shuffle and Sort
• What happens:
All output from the Mappers (after combiners and partitioners) is collected by Reducers. Data is
transferred across the network to the machine(s) running the Reducers.
• Sorting:
The data is sorted by key so that all values for a particular key come together.
• Purpose:
This grouping makes it easy for the Reducer to process all values associated with each key in one go.
2. Reduce
• What happens:
For each unique key, the Reducer receives a list of all its values (for instance, all the counts for the word
"apple").
• Processing:
The Reducer runs its function on these values; common operations are summing, counting, filtering, or
other combinations.
• Result:
The Reducer outputs a single result per key (for example, the total count for "apple").
3. Output Format
• What happens:
The final key-value result for each key is written to storage (e.g., HDFS).
• Format:
By default, the key and value are separated by a tab, and each key-value pair is written as one line by
the output writer.

Prof. Babitha P K 46 CIT, Ponnampet

Visual Example (refer to the figure)
• Mapper Stage:
Each input split is processed by a Mapper, which emits key-value pairs (e.g., X 1, Y 1, etc.).
• Combiner & Partitioner:
The data is optionally combined and partitioned by key for efficient processing.
• Shuffle and Sort:
All key-value pairs for a specific key (e.g., all Xs, all Ys, all Zs) are gathered and sorted.
• Reducer Stage:
Each Reducer gets all values for a key (e.g., X: 1, 6), processes them (e.g., sums to get X: 7), and
outputs a final result.
• Output:
The results (X:7, Y:7, Z:15) are written to the final output.

[Link]
A Combiner is an optimization technique in MapReduce.
It reduces the amount of data that needs to be transferred between the Mapper and the
Reducer.
You can think of it as a “mini reducer” that runs locally on the Mapper’s output before
sending it across the network.
Difference between Combiner and Reducer
1. Combiner Output

Prof. Babitha P K 47 CIT, Ponnampet

Works on intermediate data (output from Mapper).
Sends results to the Reducer (not the final file).
2. Reducer Output
Works on the data received from Mappers (and possibly Combiners).
Produces the final output stored in HDFS.
 Write a MapReduce program to count the occurrence of similar words in a file.
Use a Combiner to optimize performance.(refer previous driver class,mapper and reducer pgm )
Example Input Data
Welcome to Hadoop Session
Introduction to Hadoop
Introducing Hive
Hive Session
Pig Session
How It Works
1. Mapper
Breaks lines into words → emits (word, 1) pairs.
Example: "Welcome to Hadoop Session" →
(Welcome, 1), (to, 1), (Hadoop, 1), (Session, 1)
2. Combiner (mini reducer at Mapper side)
 Locally adds up counts for the same word before sending to Reducer.
 Example: If one mapper sees "Hadoop" twice, instead of sending (Hadoop, 1), (Hadoop,
1), it sends:
 (Hadoop, 2)
3. Reducer
 Receives combined counts from all mappers.
 Aggregates again to produce final word counts.

Example Final Output:

 Hadoop 2
 Session 2
 Hive 2

Prof. Babitha P K 48 CIT, Ponnampet

  Welcome 1
 to 1
 Introduction 1
 Introducing 1
 Pig 1
The Combiner reduces network traffic by summarizing intermediate results locally, but the
Reducer always produces the final result.
Act: In the driver program, set the combiner class as shown below.
[Link]([Link]);
Sets the Combiner class for the job
[Link](job, new Path("/mapreducedemos/[Link]"));
shows the input file ([Link]) being given to the job.
[Link](job, new Path("/mapreducedemos/output/wordcount/"));
output directory where results will be stored
[Link]
 The partitioning phase happen after the map phase and before the reduce phase
 Usually the no. of partitions are equal to the number of reducers
 By default, Hadoop uses a hash partitioner, but you can write your own for custom splitting logic.
Objective in This Example
 Goal: Count the occurrence of similar words in a file.
 Custom Requirement: Use a partitioner that sends all words starting with the same alphabet letter to the
same reducer.
 This means, for example, all words starting with 'A' go to one partition/reducer, all starting with 'B' to
another, and so on.
 'A' → partition 1
 'B' → partition 2
 ... up to
 'Z' → partition 26
 Default is 0 for any non-alphabet keys.
Example:
If the word is "Hadoop", the partitioner will:
 Take 'H'
 Assign it to partition 8 (since 'H' is the 8th letter).
2. Setting Up Reducers and Partitioner in the Driver

Prof. Babitha P K 49 CIT, Ponnampet

  You set the number of reducers to 27 (one for each letter + default) with:
java
[Link](27);
Set the custom partitioner:
java
 [Link]([Link]);
. Output
 After running, you get 27 output files/partitions (one for each letter).
 Example:
part-r-00008 contains all words starting with 'H' (since 'H' is partition 8).
 The provided sample output shows files like part-r-00008, which is associated with the letter 'H', and so
on up to part-r-00026 for 'Z’.
Summary
 Input: A text file with several lines.
 Mapper: Emits (word, 1) pairs.
 Partitioner: Looks at each word’s first letter, assigns a partition/reducer by alphabet.
 Reducer: Tallies counts for words in its partition.
 Output: 27 files, one for each starting letter (plus default).

Prof. Babitha P K 50 CIT, Ponnampet

Prof. Babitha P K 51 CIT, Ponnampet
Prof. Babitha P K 52 CIT, Ponnampet
 [Link]
 Objective
 To write a MapReduce program that searches for a specific keyword (e.g., "Jack") in a file (e.g., a
student record CSV).
Input Data Example
 text
 1001,John,45
 1002,Jack,39
 1003,Alex,44
 1004,Smith,38
 1005,Bob,33
 Each line represents a student record with an ID, Name, and Age.
 Program Overview
Main Job Driver ([Link])
 Sets up the MapReduce job configuration.
 Specifies the input CSV path, output directory, input and output formats, Mapper and Reducer classes,
and the search keyword (“Jack”).
Key Steps:
 Set up the MapReduce job’s classes, input and output types.
 Set the keyword to search: [Link]().set("keyword", "Jack");
 Specify the input file and output file locations.
 Start the job.
Mapper ([Link])
 Reads each line (record) from the file.
 Checks if the keyword exists in that line.
 If found, emits (writes) the relevant output with some metadata (file name, line/position info).
Main Job Driver ([Link])
 Sets up the MapReduce job configuration.
 Specifies the input CSV path, output directory, input and output formats, Mapper and Reducer classes,
and the search keyword (“Jack”).
Key Steps:
 Set up the MapReduce job’s classes, input and output types.
 Set the keyword to search: [Link]().set("keyword", "Jack");
 Specify the input file and output file locations.
 Start the job.

Prof. Babitha P K 53 CIT, Ponnampet

Mapper ([Link])
 Reads each line (record) from the file.
 Checks if the keyword exists in that line.
 If found, emits (writes) the relevant output with some metadata (file name, line/position info).
text
1002,Jack,39 [Link],2,5
Which means:
• The line containing the keyword,
• The input file name,
• Line number,
• Position of the keyword within the line.

Prof. Babitha P K 54 CIT, Ponnampet

Prof. Babitha P K 55 CIT, Ponnampet
Prof. Babitha P K 56 CIT, Ponnampet
 [Link]
Objective
Write a MapReduce program to sort data by student name (the value).
Input Data
Sample input, each line is:
text
ID,Name,Score
Example:
text
1001,John,45
1002,Jack,39
1003,Alex,44
1004,Smith,38
1005,Bob,33
Process Overview
Mapper Phase
• Purpose: Transform the input so that student names become the key. This way, during the shuffle/sort
phase of MapReduce, all records will be sorted by name automatically.
How the Mapper Works:
• Input: Each whole line of the CSV (e.g., 1001,John,45)
• It splits the line by ,, producing an array: [1001, John, 45]
• The key it emits is the student's name (token[1])
• The value is the rest, combining id and marks (token + "-" + token[1]), or possibly more depending on
implementation.
• Output example: Key: "John", Value: "1001-John"
Reducer Phase
• Purpose: Just outputs the values, sorted by keys (which are student names).
• Each reduce call gets a student name and a set of associated details (in this case, just one because names
are unique).
• Writes out the sorted results.

Prof. Babitha P K 57 CIT, Ponnampet

Prof. Babitha P K 58 CIT, Ponnampet
Sample Output
After running, you’ll get the sorted output, for example:
text
1003,Alex,44
1005,Bob,33
1002,Jack,39
1001,John,45
1004,Smith,38
This list is sorted alphabetically by student name.
Summary Table

Step What Happens

Mapper Emits (Name, Details) for each student

Shuffle/Sort (Hadoop Automatically sorts by Name (Key)

Prof. Babitha P K 59 CIT, Ponnampet

 Step What Happens

Reducer Outputs details, resulting in a sorted list

In Short
• Mapper: Makes student names the key, so the framework can sort by them.
• Reducer: Just outputs the records as sorted by the framework.
• End Result: Output file is a CSV sorted by student name.

[Link]
When you run MapReduce jobs in Hadoop, the output files can be large. Hadoop lets you compress these
output files automatically.
Why Compress Output Files?
Hadoop lists two key benefits of output compression:
1. Reduces space: Compressed files are smaller, so they use less disk storage.
2. Speeds up transfer: Smaller files move faster when transferring data over the network (between nodes or
when downloading results).
How to Enable Compression in Code?
In your Driver Program (the main Java program that sets up the job), you add these lines:
java
[Link]("[Link]", true);
[Link]("[Link]", [Link], [Link]);
Explanation:
• First line: Tells Hadoop to compress the output.
• Second line: Specifies which compression method to use.
Here, it's GzipCodec, which uses the popular GZIP format to compress files.
Extra Details:
• The word “codec” means compressor/decompressor — it’s an algorithm for compressing and
decompressing data.
• GzipCodec is just one of several you can use in Hadoop (others include Snappy, LZO, etc.)
Summary Table

Configuration Line Purpose

[Link]("[Link]", true); Enables compression on output

Prof. Babitha P K 60 CIT, Ponnampet

 Configuration Line Purpose

Chooses the compression algorithm

[Link]("[Link]", ...); (here, GZIP)

In Simple Words
By setting these two lines in your Hadoop job, your MapReduce output files will be compressed using GZIP.
This saves space and makes moving data faster.

Prof. Babitha P K 61 CIT, Ponnampet