Department of Computer Science and Engineering

Bigdata and Analytics (BCS714D)

Module-02
Introduction to Hadoop and Map Reduce Programming

.IN
Introduction to Hadoop

• Hadoop is an open-source framework designed to store and process massive volumes of

data efficiently using a distributed file system (HDFS) and a distributed computing model.
C
• It enables organizations to manage and analyze structured, semi-structured, and
N
unstructured data spread across networks.

• Hadoop uses MapReduce programming to process data in parallel across clusters of

SY

computers.

• Understanding Hadoop is crucial due to the exponential growth of data generated daily by
enterprises worldwide.
U

Big Data and Hadoop

VT

• Big Data refers to the huge volumes of diverse data generated every second, minute, and
day.

• Enterprises are increasingly recognizing Big Data as a valuable resource for insights and
innovation.

Data Generation Statistics

• Every Day:

o NYSE generates 1.5 billion shares and trade data.

o Facebook stores 2.7 billion comments and likes.

o Google processes about 24 petabytes of data.

• Every Minute:

o Facebook users share 2.5 million pieces of content.

o Twitter users tweet 300,000 times.

o Instagram users post 220,000 new photos.

Prof. Deepika G, Dept. of CSE, SVIT Page 1

Studied smart, not hard — thanks to [Link]
 o YouTube users upload 72 hours of new videos.

o Apple users download 50,000 apps.

o Email users send 200 million messages.

o Amazon generates over $80,000 in sales.

o Google handles over 4 million search queries.

• Every Second:

o Banking applications process over 10,000 credit card transactions.

.IN
The Need for Hadoop
• To handle, process, and analyze the vast and varied data efficiently, a scalable system like
C
Hadoop is essential.

• Hadoop addresses challenges related to storing and processing colossal volumes of data
N
by distributing the workload.
SY

5.1.1 Data: The Treasure Trove

• Provides business advantages such as generating product recommendations, inventing
new products, and market analysis.
U

• Offers key indicators that can influence business success.

VT

• Allows for more precise analysis — the more data available, the better the accuracy of
insights.

Prof. Deepika G, Dept. of CSE, SVIT Page 2

Studied smart, not hard — thanks to [Link]
 Challenges Hadoop Addresses (Figure 5.1):
• Volume: “How to store terabytes of mounting data?”
• Variety: “How to work with data from varied sources – structured, semi-structured,
unstructured?”
• Velocity: “How to process data quickly for timely decision-making?”

5.2 Why Hadoop?

Hadoop is widely adopted because of its ability to handle massive amounts of data, in
varied formats, and process it quickly. Here's a breakdown of the key points:

.IN
Other Considerations for Using Hadoop (Figure 5.2):

1. Low Cost
C
o Open-source framework
N
o Uses commodity hardware, which is inexpensive and easy to obtain.
2. Inherent Data Protection
SY

o Built-in fault tolerance and data replication across nodes.

3. Storage Flexibility
o Capable of storing structured, semi-structured, and unstructured data.
U

4. Scalability
o Can scale horizontally by adding more machines (nodes).
VT

5. Computing Power
o Distributed processing allows for parallel computation, reducing processing time.
Hadoop provides a cost-effective, scalable, and efficient way to manage the challenges
of Big Data—making it one of the most in-demand technologies in data-driven enterprises.

Prof. Deepika G, Dept. of CSE, SVIT Page 3

Studied smart, not hard — thanks to [Link]
 Key Benefits of Hadoop (Continued from Figure 5.2):

1. Computing Power

o Based on a distributed computing model.

o Can process very large volumes of data quickly.

o More computing nodes = More processing power.

2. Scalability

o Easily scalable by adding more nodes.

.IN
o Requires minimal system administration as the system grows.

3. Storage Flexibility
C
o Unlike traditional RDBMS, no need to pre-process data before storing it.
N
o Can store structured, semi-structured, and unstructured data like images,
videos, text.
SY

o Allows flexibility in deciding how to use stored data later.

4. Inherent Data Protection

U

o Automatically redirects tasks if a node fails.

VT

o Ensures system reliability and high availability.

o Stores replicas of data across different nodes in a cluster to prevent data loss.

Prof. Deepika G, Dept. of CSE, SVIT Page 4

Studied smart, not hard — thanks to [Link]
 � Hadoop Framework Overview (Figure 5.3)

• Cluster-based design: A group of machines (nodes) that work together.

• Data is:

1. Distributed and duplicated across nodes.

2. Processed in parallel using local resources.

3. Managed with automatic failover and fault tolerance.

.IN
5.3 Why Not RDBMS?
• RDBMS is not ideal for:
C
o Handling large files, images, and videos.
N
o Advanced analytics and machine learning workloads.
SY

• Requires:

o High investment for scalability.

More complex architecture and higher storage costs.

U

• Hadoop outperforms RDBMS in terms of cost-efficiency, scalability, and flexibility for

VT

large, complex data sets.

Prof. Deepika G, Dept. of CSE, SVIT Page 5

Studied smart, not hard — thanks to [Link]
Figure 5.4: RDBMS Cost Scaling

• As data size increases (e.g., TB → PB), cost per GB grows rapidly with RDBMS.

• Hadoop provides better scalability at much lower cost.

5.4 RDBMS vs Hadoop

Parameter RDBMS Hadoop

Relational Database

.IN
System Node-Based Flat Structure
Management System

Suitable for structured data Supports structured, semi-structured, and

Data
only
C unstructured data (e.g., XML, JSON, text files)
N
OLTP (Online Transaction
Processing Analytical, Big Data Processing
Processing)
SY

Ideal for consistent

Choice Ideal for Big Data with no need for consistency
relationships between data
U

Requires expensive/high-end Uses commodity hardware (basic processor,

Processor
hardware network card, and a few hard drives)
VT

~$10,000–$14,000 per
Cost ~$4,000 per terabyte
terabyte

5.6 HISTORY OF HADOOP

• Hadoop was created by Doug Cutting, who also developed Apache Lucene, a popular
text search library.

• Hadoop originated as a part of the Apache Nutch project (an open-source web search
engine).

• It is also tied to the Lucene project.

῿ 5.6.1 The Name "Hadoop"

• "Hadoop" is not an acronym.

• It's a made-up name given by Doug Cutting’s son to a stuffed yellow toy elephant.

Prof. Deepika G, Dept. of CSE, SVIT Page 6

Studied smart, not hard — thanks to [Link]
 • Doug Cutting chose the name because it was:

o Short and easy to spell

o Easy to pronounce

o Meaningless (not previously used)

o Memorable and unique

.IN
C
N
SY
U

Year Event
2002 Doug Cutting and Mike Cafarella started working on Nutch.
VT

2003 Continued development of Nutch.

2004 Google published GFS and MapReduce papers.
2005 Doug added DFS and MapReduce concepts to Nutch.
2006 Yahoo hired Doug; Hadoop spins out of Nutch as a separate project.
2008 Cloudera founded.
2009 Doug Cutting joined Cloudera.

What is Hadoop?

• Hadoop is an open-source software framework.

• It is used to store and process massive amounts of data in a distributed manner.

• Operates on large clusters of commodity hardware (low-cost machines).

Prof. Deepika G, Dept. of CSE, SVIT Page 7

Studied smart, not hard — thanks to [Link]
 Core Objectives of Hadoop

1. Massive Data Storage

2. Faster Data Processing

5.7.1 Key Aspects of Hadoop

Aspect Description
Open Source
Free to download, use, and contribute to.

.IN
Software
Includes tools, programs, and infrastructure needed for development and
Framework
execution. C
Distributed Data is divided and stored across multiple machines. Processing is parallelized.

Stores huge volumes of data using low-cost hardware.

N
Massive Storage
SY

Faster Processing Enables quick response times by processing data in parallel across nodes.
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 8

Studied smart, not hard — thanks to [Link]
5.7.2 Hadoop Components
Core Components:

1. HDFS (Hadoop Distributed File System)

o (a) Acts as the storage component

o (b) Distributes data across nodes

o (c) Redundant by design (stores multiple copies)

.IN
2. MapReduce

o (a) A computational framework

(b) Splits tasks across multiple nodes

o
C
o (c) Processes data in parallel
N
Hadoop Ecosystem Tools (Enhance Core Capabilities):
SY

1. HIVE – SQL-like querying of big data

2. PIG – Scripting for data transformation

U

3. SQOOP – Transfers data between RDBMS and Hadoop

4. HBASE – NoSQL database on Hadoop

VT

5. FLUME – Collects and moves large amounts of log data

6. OOZIE – Workflow scheduler

7. MAHOUT – Machine learning algorithms

Prof. Deepika G, Dept. of CSE, SVIT Page 9

Studied smart, not hard — thanks to [Link]
5.7.3 Hadoop Conceptual Layer

• Conceptually divided into two layers:

1. Data Storage Layer – For storing massive data

2. Data Processing Layer – For parallel processing and generating insights

5.7.4 High-Level Architecture of Hadoop

• Distributed Master-Slave Architecture:

.IN
o Master Node = NameNode

o Slave Nodes = DataNodes C

• The NameNode manages metadata and coordination.
N
• DataNodes handle storage and actual data processing.
SY

5.8 Use case of Hadoop

Three Key Benefits:

U

1. Data Integration:
VT

o Combines ClickStream data with:

▪ CRM (Customer Relationship Management) data

▪ Sales data

▪ Advertising campaign info

o Gives deeper customer insights

2. Scalability + Cost Efficiency:

o Stores years of data at low incremental cost

o Enables long-term or year-over-year trend analysis

o Gives competitive edge in understanding customer behavior over time

3. Powerful Analytics with Tools:

o Tools: Apache Pig or Apache Hive

Prof. Deepika G, Dept. of CSE, SVIT Page 10

Studied smart, not hard — thanks to [Link]
 o Allows:

▪ Organizing data by user sessions

▪ Refinement and analysis

▪ Feeding data into visualization or analytics tools.

5.8.1 ClickStream Data

• ClickStream data = User activity data (e.g., mouse clicks on websites)

.IN
• Helps understand customer purchasing behavior

• Used by online marketers to: C

o Optimize product pages
N
o Improve promotional content

o Enhance overall business strategy

SY

Benefits of Using Hadoop for ClickStream Data Analysis

(Refer to Figure 5.11)

U

Three Key Benefits:

VT

1. Data Integration:

o Combines ClickStream data with:

▪ CRM (Customer Relationship Management) data

▪ Sales data

▪ Advertising campaign info

o Gives deeper customer insights

2. Scalability + Cost Efficiency:

o Stores years of data at low incremental cost

o Enables long-term or year-over-year trend analysis

o Gives a competitive edge in understanding customer behavior over time

3. Powerful Analytics with Tools:

Prof. Deepika G, Dept. of CSE, SVIT Page 11

Studied smart, not hard — thanks to [Link]
 o Tools: Apache Pig or Apache Hive

o Allows:

▪ Organizing data by user sessions

▪ Refinement and analysis

▪ Feeding data into visualization or analytics tools

.IN
C
N
SY

5.10 HDFS (Hadoop Distributed File System)

Key Features of HDFS

U

1. Storage Component of the Hadoop framework.

VT

2. A Distributed File System that stores data across multiple nodes.

3. Modeled after Google File System (GFS).

4. High throughput optimization:

o Uses large block sizes

o Moves computation to the data, not vice versa

5. Replication:

o Files can be replicated multiple times (configurable), making the system tolerant
to hardware/software failures

6. Automatic re-replication of data blocks if a node fails

7. Designed for large files (e.g., GBs to TBs) — optimized for reading/writing large
datasets

8. Built on top of native file systems like ext3/ext4

Prof. Deepika G, Dept. of CSE, SVIT Page 12
Studied smart, not hard — thanks to [Link]
 Example:

• A file [Link] of 192 MB

• Default block size = 64 MB

• File will be split into 3 blocks (192 ÷ 64)

• Each block is replicated across nodes in the cluster according to the default
replication factor (usually 3)

5.10.1 HDFS Daemons

.IN
[Link] NameNode

• The NameNode is the master daemon in the HDFS architecture.

C
• It manages the file system namespace, which includes:
N
o Mapping of blocks to files
SY

o Tracking file metadata

o Handling read, write, create, delete operations

U

Key Responsibilities of NameNode:

VT

1. File System Namespace:

o A logical representation of files and directories in HDFS.

o Stored in a file called FsImage.

2. Transaction Logging:

o Every metadata change is recorded in the EditLog.

o On startup, NameNode:

▪ Reads both FsImage and EditLog

▪ Applies the transactions to reconstruct the latest state

▪ Writes a new FsImage and clears the old EditLog

3. Rack Awareness:

o Uses Rack ID to identify the DataNodes in different racks (helps in fault tolerance
and efficient data transfer).
Prof. Deepika G, Dept. of CSE, SVIT Page 13
Studied smart, not hard — thanks to [Link]
 4. Single NameNode per Hadoop cluster.

• Figure 5.13: Shows the layered structure:

Disk Storage → Native OS File System → HDFS
• Figure 5.14: Key defaults in HDFS:
o Block Structure
o Replication Factor = 3
o Default Block Size = 64 MB

.IN
C
N
SY
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 14

Studied smart, not hard — thanks to [Link]
 .IN
C
N
SY
U
VT

[Link] Data Node

1. Multiple DataNodes exist in a Hadoop cluster.

2. DataNodes communicate with each other and the NameNode during read/write
operations.

3. Data Nodes send "heartbeat" messages to the NameNode regularly.

4. The heartbeat confirms that the DataNode is alive and working.

5. If a Data Node fails to send a heartbeat, the NameNode assumes it is down.

6. The NameNode then replicates the data from the failed DataNode to other active
DataNodes.

7. This allows the system to keep functioning smoothly even if a DataNode goes down.

Prof. Deepika G, Dept. of CSE, SVIT Page 15

Studied smart, not hard — thanks to [Link]
Real-Life Example (Analogy):

• Just like employees swipe in at the office to mark attendance,

• DataNodes send heartbeat signals to show they are present.

• The manager (like the NameNode) assigns tasks only to those who have checked in.

• If no swipe (heartbeat), the employee (DataNode) is considered absent, and tasks are
reassigned.

.IN
C
N
SY
U
VT

5.10.2 Anatomy of File Read

1. Client Opens File

o The client uses open() to request a file via the DistributedFileSystem (DFS).

2. Get Block Location from NameNode

o DFS contacts the NameNode to get the locations (addresses) of the DataNodes
storing the file's data blocks.

3. Receive Input Stream

o DFS gives the client an FSDataInputStream to begin reading the file.

4. Read from Closest DataNode

o The client reads the file data block-by-block, starting with the closest DataNode.

5. Read Continues with Next Block

Prof. Deepika G, Dept. of CSE, SVIT Page 16

Studied smart, not hard — thanks to [Link]
 o After reading one block, the client connects to the next best DataNode for the next
block and continues reading.

6. Close the Stream

o When the reading is complete, the client calls close() to close the
FSDataInputStream.

Flow:

.IN
open() → DFS → NameNode → Block locations → FSDataInputStream → Read from
DataNode(s) → close()
C
N
SY
U
VT

5.10.3 Anatomy of File Write

1. Client Requests to Create File

o Client calls create() using DistributedFileSystem to begin the file creation

process.

2. File Creation Request Sent to NameNode

Prof. Deepika G, Dept. of CSE, SVIT Page 17
Studied smart, not hard — thanks to [Link]
 o An RPC (Remote Procedure Call) is made to the NameNode to create a new file.

o NameNode does checks (e.g., if file already exists) and creates a file without data
blocks initially.

3. Client Starts Writing Data

o The client writes data through FSDataOutputStream.

o Data is split into packets, placed into a data queue, and processed by
DataStreamer.

.IN
4. Data Packets Streamed to DataNodes

o Packets are streamed to a pipeline of 3 DataNodes (default replication factor = 3):

C
▪ First DataNode stores and forwards to second.
N
▪ Second DataNode stores and forwards to third.
SY

▪ Third DataNode stores the final copy.

5. Acknowledgment Process

Each DataNode sends an acknowledgment (ack) back up the pipeline.

U

o DFSOutputStream maintains an Ack Queue to track unacknowledged packets.

VT

o A packet is removed from this queue only when all three DataNodes have
acknowledged it.

6. Client Finishes Writing

o Once all data is written, the client calls close() on the stream.

7. Final Acknowledgment to NameNode

o Remaining packets are flushed.

o NameNode is informed that the file write is complete.

Flow:

create() → DFS → NameNode → FSDataOutputStream → DataStreamer → DataNode

Pipeline → ack packets → close() → Notify NameNode

Prof. Deepika G, Dept. of CSE, SVIT Page 18

Studied smart, not hard — thanks to [Link]
 .IN
C
N
SY

5.10.4 Replica Placement Strategy

[Link] Hadoop Default Replica Placement Strategy:

U

1. First Replica

Placed on the same node as the client (local write).

VT

2. Second Replica

o Placed on a different rack, on a different node (for fault tolerance).

3. Third Replica

o Placed on the same rack as the second replica, but on a different node.

5.10.5 Working with HDFS Commands

Objective Command Description

Shows top-level directories
List root directories/files hadoop fs -ls /
and files in HDFS.
List all files and hadoop fs -ls -R /
Shows full directory tree in
subdirectories recursively HDFS.
Create a directory (e.g. hadoop fs -mkdir /sample
Creates a new directory in
/sample) HDFS.

Prof. Deepika G, Dept. of CSE, SVIT Page 19

Studied smart, not hard — thanks to [Link]
 Objective Command Description
hadoop fs -put /root/sample/[Link] Copies file from local
Upload file to HDFS /sample/[Link] filesystem to HDFS.
hadoop fs -get /sample/[Link] Copies file from HDFS to
Download file from HDFS /root/sample/[Link]
local filesystem.
Upload using hadoop fs -copyFromLocal /root/sample/[Link] Similar to put, copies file
copyFromLocal /sample/[Link] from local to HDFS.
Download using hadoop fs -copyToLocal /sample/[Link] Similar to get, copies file
copyToLocal /root/sample/[Link] from HDFS to local.
Prints content of a file in

.IN
Display file content hadoop fs -cat /sample/[Link]
HDFS.
Copies file from one HDFS
Copy file within HDFS hadoop fs -cp /sample/[Link] /sample1
C dir to another.
Remove directory from hadoop fs -rm -r /sample1
Deletes directory and
HDFS contents from HDFS.
N
SY

5.10.6 Special Features of HDFS

Data Replication

• The client doesn’t need to manage block locations.

U

• HDFS automatically directs the client to the nearest replica for better performance.
VT

Data Pipeline

• When writing data, the client sends it to the first DataNode.

• That DataNode forwards it to the second, and then to the third, forming a pipeline.

• All replicas are written to disk through this flow.

5.11 – Processing Data with Hadoop

What is MapReduce?

• MapReduce is a tool in Hadoop that helps process big data in parallel.

• It splits the data into parts and processes each part at the same time.

How It Works

1. Map Task

o Breaks input into key-value pairs.

Prof. Deepika G, Dept. of CSE, SVIT Page 20
Studied smart, not hard — thanks to [Link]
 o Each mapper works independently.

2. Shuffle & Sort

o The outputs from all mappers are shuffled and sorted by keys.

3. Reduce Task

o Combines mapper outputs into a final result.

Key Features

.IN
• Data Locality: Tries to run the task where the data is stored to save time.

• Stored on Disk: Outputs from mappers are stored locally on disk.

C
Important Components
N
Component Role
SY

JobTracker Master – gives tasks to others.

TaskTracker Worker – does the task given to it.

U
VT

How a Job Runs

1. You (Job Client) send the job to JobTracker.

2. JobTracker splits it into tasks.

3. TaskTrackers do the work.

4. JobTracker tracks progress and tells you when done.

Prof. Deepika G, Dept. of CSE, SVIT Page 21

Studied smart, not hard — thanks to [Link]
5.11.1 MapReduce Daemons

1. JobTracker (Master)

• Connects your code to the Hadoop cluster.

• Decides which task runs on which node.

• Monitors running tasks.

• If a task fails, it retries on another node.

.IN
• Only one JobTracker per cluster.

2. TaskTracker (Worker) C
• Executes the actual Map or Reduce tasks given by JobTracker.
N
• One TaskTracker per slave node.

• Uses multiple JVMs to run tasks in parallel.

SY

• Sends regular heartbeat signals to JobTracker.

• If JobTracker doesn't get a heartbeat, it assumes failure and reassigns the task.
U

5.11.2 How does Mapreduce Work

VT

1. Input Data is Split

The big data you want to analyze is split into many small parts. Each part is called a
data segment.

2. JobTracker and TaskTrackers

o There is one JobTracker (the master) that controls the whole job.

o There are many TaskTrackers (workers) that do the actual work on the data.

3. Map Tasks

o Each TaskTracker gets a small part of the data.

o They perform the map function, which processes the data and creates key-value
pairs (like labeling the data).

4. Shuffle and Sort

o The output from the map tasks is shuffled and sorted by keys automatically by
the framework.

Prof. Deepika G, Dept. of CSE, SVIT Page 22

Studied smart, not hard — thanks to [Link]
 5. Reduce Tasks

o Reduce workers get the sorted key-value pairs.

o They combine or summarize the data based on the keys, producing the final
result.

6. Output

o The final combined output is saved or sent back to the user program.

Roles

.IN
• JobTracker: Master controller that assigns tasks to workers.

• TaskTracker: Worker nodes that run map and reduce tasks.

C
• Map Task: Processes data pieces to generate key-value pairs.
N
• Reduce Task: Combines data based on keys to produce final results.
SY
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 23

Studied smart, not hard — thanks to [Link]
 .IN
C
N
SY
U
VT

5.11.3 MapReduce Example

The famous example for MapReduce Programming is Word Count. For example, consider you
need to count the occurrences of similar words across 50 files. You can achieve this using
MapReduce Programming. Refer Figure 5.25.

Word Count MapReduce Programming using Java

The MapReduce Programming requires three things:

1. Driver Class: This class specifies Job Configuration details.

Prof. Deepika G, Dept. of CSE, SVIT Page 24

Studied smart, not hard — thanks to [Link]
 2. Mapper Class: This class overrides the Map Function based on the problem statement.

3. Reducer Class: This class overrides the Reduce Function based on the problem
statement.

.IN
C
N
SY
U

[Link]: Driver Program

VT

package [Link];

import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];

public class WordCounter {

public static void main(String[] args) throws IOException, InterruptedException,

ClassNotFoundException {
Job job = new Job();
[Link]("wordcounter");

Prof. Deepika G, Dept. of CSE, SVIT Page 25

Studied smart, not hard — thanks to [Link]
 [Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);

[Link](job, new Path("/sample/[Link]"));

[Link](job, new Path("/sample/wordcount"));

[Link]([Link](true) ? 0 : 1);
}

.IN
}

[Link]: Map Class

C
package [Link];
N
import [Link];
SY

import [Link];
import [Link];
import [Link];
import [Link];
U

public class WordCounterMap extends Mapper<LongWritable, Text, Text, IntWritable> {

VT

@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String[] words = [Link]().split(",");
for (String word : words) {
[Link](new Text(word), new IntWritable(1));
}
}
}

[Link]: Reduce Class

package [Link];

import [Link];
import [Link];
import [Link];

Prof. Deepika G, Dept. of CSE, SVIT Page 26

Studied smart, not hard — thanks to [Link]
import [Link];

public class WordCounterRed extends Reducer<Text, IntWritable, Text, IntWritable> {

@Override
protected void reduce(Text word, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
Integer count = 0;
for (IntWritable val : values) {
count += [Link]();
}

.IN
[Link](word, new IntWritable(count));
}
}
C
N
5.12 MANAGING RESOURCES AND APPLICATIONS WITH HADOOP YARN (YET ANOTHER
RESOURCE NEGOTIATOR)
SY

Apache Hadoop YARN is a sub-project of Hadoop 2.x. Hadoop 2.x is YARN-based architecture.
It is a general processing platform. YARN is not constrained to MapReduce only. You can run
multiple applications in Hadoop 2.x in which all applications share a common resource
U

management. Now Hadoop can be used for various types of processing such as Batch,
Interactive, Online, Streaming, Graph, and others.
VT

5.12.1 Limitations of Hadoop 1.0 Architecture

In Hadoop 1.0, HDFS and MapReduce are Core Components, while other components are built
around the core.

1. Single NameNode is responsible for managing entire namespace for Hadoop Cluster.

2. It has a restricted processing model which is suitable for batch-oriented MapReduce

jobs.

3. Hadoop MapReduce is not suitable for interactive analysis.

4. Hadoop 1.0 is not suitable for machine learning algorithms, graphs, and other memory
intensive algorithms.

5. MapReduce is responsible for cluster resource management and data processing.

In this Architecture, map slots might be “full”, while the reduce slots are empty and vice versa.
This causes resource utilization issues. This needs to be improved for proper resource
utilization.

Prof. Deepika G, Dept. of CSE, SVIT Page 27

Studied smart, not hard — thanks to [Link]
5.12.2 HDFS Limitation
NameNode saves all its file metadata in main memory. Although the main memory today is not
as small and as expensive as it used to be two decades ago, still there is a limit on the number
of objects that one can have in the memory on a single NameNode. The NameNode can quickly
become overwhelmed with load on the system increasing.

In Hadoop 2.x, this is resolved with the help of HDFS Federation.

5.12.3 Hadoop 2: HDFS

HDFS 2 has two main parts:

.IN
1. Namespace (manages files and directories)

2. Blocks storage service (handles data storage and replication)

C
Key features of HDFS 2:

Horizontal scalability (can easily grow by adding more nodes)

N
•

• High availability (system stays reliable and available)

SY

Uses multiple independent NameNodes that don’t need to coordinate with each other.

DataNodes store blocks and are shared among all NameNodes.

U

High availability is achieved using Passive Standby NameNode:

VT

• Active-Passive NameNode setup where passive node takes over automatically if active
node fails.

• Namespace edits are saved to shared storage.

• Passive NameNode reads from shared storage and keeps metadata updated.

• If active NameNode fails, passive becomes active and continues writing to shared
storage.

5.12.4 Hadoop 2 YARN:

Prof. Deepika G, Dept. of CSE, SVIT Page 28

Studied smart, not hard — thanks to [Link]
 .IN
C
N
SY
U
VT

[Link]. YARN's fundamental idea

• YARN splits the JobTracker's responsibilities into separate daemons for resource
management and job scheduling/monitoring.

Key Components:

1. Global ResourceManager:

o Distributes resources among running applications.

o Has two parts:

▪ Scheduler: Decides resource allocation; does not monitor application
status.

▪ ApplicationManager: Accepts job submissions, negotiates resources,

restarts ApplicationMaster if it fails.

2. NodeManager:

Prof. Deepika G, Dept. of CSE, SVIT Page 29

Studied smart, not hard — thanks to [Link]
 o Runs on each slave machine.

o Launches application containers.

o Monitors resources like memory, CPU, disk, network.

o Reports usage to ResourceManager.

3. Per-application ApplicationMaster:

o Manages resource negotiation with ResourceManager for its application.

o Works with NodeManager to execute and monitor tasks.

.IN
[Link].2 Basic Concepts of YARN

Basic Concepts
C
Application:
N
1. A job submitted to the YARN framework.
SY

2. Example: A MapReduce job.

Container:
U

1. The basic unit of resource allocation.

2. Provides fine-grained resource allocation across multiple resource types such as

VT

memory, CPU, disk, and network.

o Example:

▪ container_0 = 2GB memory, 1 CPU

▪ container_1 = 1GB memory, 6 CPUs

3. Replaces the fixed map/reduce slots in earlier Hadoop versions.

YARN Architecture (Key Steps):

1. A client submits an application specifying how to launch the application-specific

ApplicationMaster.

2. The ResourceManager launches the ApplicationMaster by assigning a container.

3. The ApplicationMaster registers with the ResourceManager during startup, enabling

direct queries about resource status.

Prof. Deepika G, Dept. of CSE, SVIT Page 30

Studied smart, not hard — thanks to [Link]
 4. The ApplicationMaster negotiates resource containers via a resource-request protocol
during the job execution.

5. Launching Containers:

6. The ApplicationMaster launches containers by sending container launch instructions

to the NodeManager after successful resource allocation.

7. Execution by NodeManager:

8. The NodeManager runs the application code and reports progress, status, etc., back to

.IN
the ApplicationMaster using a specific protocol.

9. Client Communication:

10.
C
The client that submitted the job talks directly to the ApplicationMaster to
receive job updates (e.g., status, progress), also via a protocol.
N
11. Completion & Shutdown:
SY

12. After job completion, the ApplicationMaster deregisters from the

ResourceManager and shuts down, freeing up its container for reuse.
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 31

Studied smart, not hard — thanks to [Link]
 Introduction to MapReduce Programming

Introduction
• MapReduce jobs are split into map tasks and reduce tasks, executed on a Hadoop
cluster.
• Each task processes a small portion of the data, allowing Hadoop to distribute the
load.

• Input data is stored in HDFS (Hadoop Distributed File System).

.IN
Map tasks handle:

1. RecordReader – loads data. C

2. Mapper – processes data.

3. Combiner – optional local aggregation.

N
4. Partitioner – splits data for reducers.
SY

Reduce tasks handle:

1. Shuffle – move data to reducers.

U

2. Sort – organize keys.

3. Reducer – combine values.

VT

4. Output Format – write results.

• Output from map tasks = intermediate keys and values, sent to reducers.

• Hadoop runs map tasks where the data is stored (DataNode) to reduce network usage
and save bandwidth.

8.2 Mapper

• Mapper turns input key–value pairs into intermediate key–value pairs.

• The process has four main parts:

1. RecordReader – Converts raw byte data into record-oriented form, giving key–
value pairs to the mapper.
2. Map – Processes the input pairs to produce zero or more intermediate pairs.

3. Combiner (optional) – Aggregates intermediate data locally to save bandwidth and

disk space.

Prof. Deepika G, Dept. of CSE, SVIT Page 32

Studied smart, not hard — thanks to [Link]
 4. Partitioner – Splits intermediate pairs into shards, sending each shard to the
correct reducer (same keys go to the same reducer).

• Partitioned data is stored locally and then fetched by the reducer.

8.3 Reducer
Reducer’s job: Take intermediate values with the same key and reduce them into a smaller
set.

• It works in three phases:

1. Shuffle and Sort – Collects outputs from all partitioners, downloads them to the

.IN
reducer’s local machine, and sorts them by key so similar items are grouped
together.

2. Reduce – Processes each group of values for a key (e.g., sum, filter, aggregate, or
C
combine data), producing zero or more output key–value pairs.
N
3. Output Format – Writes the final key–value pairs to a file (default separator is a
tab).
SY

• Figure 8.1 shows how data flows through Mapper → Combiner → Partitioner → Shuffle
& Sort → Reducer.
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 33

Studied smart, not hard — thanks to [Link]
 8.4 Combiner :
• Purpose: Optimization technique in MapReduce to reduce data transfer between Mapper
and Reducer.

• Difference from Reducer:

1. Combiner output = intermediate data → sent to Reducer.

2. Reducer output = final result → written to disk.

• Use case example: Counting occurrences of similar words in a file before sending to
Reducer.

.IN
• How it works:

o Set the combiner class in the driver program (often same as Reducer class).
C
o Runs locally on Mapper output to aggregate data early.
N
• Sample code:
SY

[Link]([Link]);

[Link](job, new Path("/mapreducedemos/[Link]"));

[Link](job, new Path("/mapreducedemos/output/wordcount/"));

U

• Execution:
VT

Run the job with Hadoop command:

hadoop jar <jar name> <driver class> <input path> <output path>

• Output location: Stored in part-r-00000 file by default in the output directory.

• Example output (word count):

python-repl

CopyEdit

hadoop 2

hive 2

pig 1

session 3
...

Prof. Deepika G, Dept. of CSE, SVIT Page 34

Studied smart, not hard — thanks to [Link]
8.5 Practitioner

Purpose:

• Runs after the Map phase and before the Reduce phase.

• Decides how intermediate key–value pairs are divided among reducers.

• Default method: Hash partitioner.

Number of partitions = number of reducers.

.IN
Example task: Count occurrences of similar words and partition them based on the first
alphabet.

How it works:
C
• Custom WordCountPartitioner checks the first letter of the word.
N
• Assigns a partition number (1–26 for A–Z, plus default).
SY

• Ensures that all words starting with the same letter go to the same reducer.
U
VT

Code: [Link]
import [Link];
import [Link];
import [Link];

public class WordCountPartitioner extends Partitioner<Text, IntWritable> {

@Override
public int getPartition(Text key, IntWritable value, int numPartitions) {
String word = [Link]();
char alphabet = [Link]().charAt(0);
int partitionNumber = 0;

switch (alphabet) {
case 'A': partitionNumber = 1; break;
case 'B': partitionNumber = 2; break;
case 'C': partitionNumber = 3; break;
case 'D': partitionNumber = 4; break;
case 'E': partitionNumber = 5; break;
case 'F': partitionNumber = 6; break;
case 'G': partitionNumber = 7; break;
case 'H': partitionNumber = 8; break;
case 'I': partitionNumber = 9; break;
case 'J': partitionNumber = 10; break;

Prof. Deepika G, Dept. of CSE, SVIT Page 35

Studied smart, not hard — thanks to [Link]
 case 'K': partitionNumber = 11; break;
case 'L': partitionNumber = 12; break;
case 'M': partitionNumber = 13; break;
case 'N': partitionNumber = 14; break;
case 'O': partitionNumber = 15; break;
case 'P': partitionNumber = 16; break;
case 'Q': partitionNumber = 17; break;
case 'R': partitionNumber = 18; break;
case 'S': partitionNumber = 19; break;
case 'T': partitionNumber = 20; break;
case 'U': partitionNumber = 21; break;
case 'V': partitionNumber = 22; break;
case 'W': partitionNumber = 23; break;
case 'X': partitionNumber = 24; break;

.IN
case 'Y': partitionNumber = 25; break;
case 'Z': partitionNumber = 26; break;
default: partitionNumber = 0; break;
}
return partitionNumber;
}
C
}
N
SY

Driver Program (relevant part)

[Link](27);
[Link]([Link]);
U

// Input and Output Path

[Link](job, new Path("/mapreducedemos/[Link]"));
[Link](job, new
VT

Path("/mapreducedemos/output/wordcountpartitioner/"));

Input Data ([Link])

Welcome to Hadoop Session
Introduction to Hadoop
Introducing Hive
Hive Session
Pig Session

Example Output (part-r-00008 for 'H')

Hadoop 2
Hive 2

Prof. Deepika G, Dept. of CSE, SVIT Page 36

Studied smart, not hard — thanks to [Link]
8.6 Searching:

The objective: To write a MapReduce program to search for a specific keyword in a file.

In this case, the program scans an input file (e.g., [Link]) and finds lines containing the
given keyword ("Jack"), then outputs the matching record along with its filename and position
in the file.

1. Driver Program — [Link]

import [Link];

.IN
import [Link];
import [Link];
import [Link];
import [Link];
C
import [Link];
import [Link];
N
import [Link];
SY

import [Link];

public class WordSearcher {

public static void main(String[] args) throws IOException, InterruptedException,
U

ClassNotFoundException {
Configuration conf = new Configuration();
VT

[Link]("keyword", "Jack"); // Keyword to search

Job job = [Link](conf, "Keyword Search");

[Link]([Link]);

[Link]([Link]);
[Link]([Link]);

[Link]([Link]);

[Link]([Link]);

[Link]([Link]);

[Link]([Link]);

Prof. Deepika G, Dept. of CSE, SVIT Page 37

Studied smart, not hard — thanks to [Link]
 [Link](1);

[Link](job, new Path("/mapreduce/[Link]"));

[Link](job, new Path("/mapreduce/output/search"));

[Link]([Link](true) ? 0 : 1);
}
}

.IN
2. Mapper — [Link]

import [Link];
import [Link];
C
import [Link];
N
import [Link];
import [Link];
SY

import [Link];
import [Link];
U

public class WordSearchMapper extends Mapper<LongWritable, Text, Text, Text> {

static String keyword;
VT

static int pos = 0;

@Override
protected void setup(Context context) throws IOException, InterruptedException {
Configuration configuration = [Link]();
keyword = [Link]("keyword");
}

@Override
protected void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException {
FileSplit fileSplit = (FileSplit) [Link]();
String fileName = [Link]().getName();
pos++;

if ([Link]().contains(keyword)) {
[Link](value, new Text(fileName + "," + pos));

Prof. Deepika G, Dept. of CSE, SVIT Page 38

Studied smart, not hard — thanks to [Link]
 }
}
}
4. Reducer — [Link]

import [Link];
import [Link];
import [Link];

public class WordSearchReducer extends Reducer<Text, Text, Text, Text> {

.IN
@Override
protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException,
InterruptedException {
for (Text value : values) {
C
[Link](key, value);
N
}
}
SY

4. Input File ([Link])

U

1001,John,45
1002,Jack,39
VT

1003,Alex,44
1004,Smith,38
1005,Bob,33

5. Output File (/mapreduce/output/search/part-r-00000)

1002,Jack,39 [Link],2

8.7 Sorting

1. Understanding the Problem

You have a CSV file ([Link]) with the following data:
1001,John,45
1002,Jack,39
1003,Alex,44
1004,Smith,38
1005,Bob,33
We need to sort the data by the student name (the second column).
Prof. Deepika G, Dept. of CSE, SVIT Page 39
Studied smart, not hard — thanks to [Link]
2. How the MapReduce Works
1. Mapper — reads each line, splits it into fields, and emits the student name as the key,
the rest of the data as the value.
2. Reducer — receives keys (names) in sorted order automatically from Hadoop, then
writes out the corresponding data.

3. Corrected Code

[Link]
import [Link];

.IN
import [Link];
import [Link];
import [Link];
import [Link];
C
import [Link];
N
import [Link];
import [Link];
SY

import [Link];
import [Link];
import [Link];
U

public class SortStudNames {

VT

public static class SortMapper extends Mapper<LongWritable, Text, Text, Text> {

protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String[] token = [Link]().split(",");
if ([Link] == 3) {
[Link](new Text(token[1]), new Text(token[0] + "," + token[1] + "," + token[2]));
}
}
}

public static class SortReducer extends Reducer<Text, Text, NullWritable, Text> {

protected void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
for (Text details : values) {
[Link]([Link](), details);
}
}

Prof. Deepika G, Dept. of CSE, SVIT Page 40

Studied smart, not hard — thanks to [Link]
 }

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();
Job job = [Link](conf, "Sort by Student Name");

[Link]([Link]);
[Link]([Link]);
[Link]([Link]);

.IN
[Link]([Link]);
[Link]([Link]);
C
[Link](job, new Path(args[0]));
[Link](job, new Path(args[1]));
N
[Link]([Link](true) ? 0 : 1);
SY

}
}
U

4. How to Run
hadoop jar [Link] SortStudNames /mapreduce/[Link]
VT

/mapreduce/output/sorted

5. Expected Output
Hadoop sorts keys (names) alphabetically, so output will be:
1003,Alex,44
1005,Bob,33
1002,Jack,39
1001,John,45
1004,Smith,38

6. Summary
• Input: CSV with IDs, Names, Scores.
• Processing: Sort by name (second field).
• Output: Alphabetically sorted list of students.
• Key Trick: Use the student name as the Mapper output key so Hadoop handles sorting
automaticall

Prof. Deepika G, Dept. of CSE, SVIT Page 41

Studied smart, not hard — thanks to [Link]
8.8 COMPRESSION

• Why compress MapReduce output?

1. Saves storage space.
2. Speeds up data transfer across the network.
• How to enable compression in the Driver program:

[Link]("[Link]", true);
[Link]("[Link]",
[Link],

.IN
[Link]);
o GzipCodec → compression algorithm.
o CompressionCodec → interface for compression/decompression.
• Effect: Output files are compressed in .gz format.
C
N
SY
U
VT

Prof. Deepika G, Dept. of CSE, SVIT Page 42

Studied smart, not hard — thanks to [Link]