Big Data Analytics

MODULE 2

Introduction to Hadoop (T1)

Introduction to Hadoop
Hadoop is a free, open-source software from Apache that helps you store and process a
large amount of data using many computers working together.

 It is written in Java.
 It splits big data into smaller parts and processes them across multiple computers at the
same time.
 It uses simple programming methods, making it easier for developers.
 Hadoop is scalable, meaning:
o You can start with one computer and add many more computers as your data
grows.
o Each computer stores part of the data and does part of the work.
 Hadoop is mainly designed for Big Data, helping organizations handle huge amounts of
data efficiently.

Big Data and Hadoop

Today, Big Data is a popular topic. Companies everywhere are realizing that they have a huge
amount of information around them in many forms:

 Structured data (like tables in databases),

 Semi-structured data (like XML, JSON),
 Unstructured data (like images, videos, social media posts).

All this data is spread across networks and the internet, and it is not fully used yet.

How much data is generated?

📅 Every Day:

 NYSE (New York Stock Exchange) creates data for 1.5 billion share trades.
 Facebook stores 2.7 billion comments and Likes.
 Google processes 24 petabytes of data (1 petabyte = 1000 terabytes).

Every Minute:

 Facebook: 2.5 million posts shared.

 Twitter: 300,000 tweets sent.
 Instagram: 220,000 new photos posted.
AI&ML VVIT Page 1
Big Data Analytics

 YouTube: 72 hours of video uploaded.

 Apple App Store: 50,000 apps downloaded.
 Emails: 200 million emails sent.
 Amazon: $80,000 sales made online.
 Google: 4 million searches made.

Every Second:

 Banks process over 10,000 credit card transactions.

Why is this important?

 A huge amount of data is created continuously.

 This data has valuable information for businesses and society.
 However, handling and processing this large data using normal tools is difficult.

This is why we need Hadoop, a framework that helps in storing and processing Big Data
efficiently across many computers.

5.1.1 Data: The Treasure Trove

Why is data valuable for businesses?

1️⃣ Gives Business Advantages:

 Helps in creating product suggestions,

 Helps in designing new products,
 Helps in analyzing the market,
 Helps in understanding customers.

2️⃣ Gives Early Indicators:

 Data shows early signs that help businesses take correct actions at the right time and
improve profits.

3️⃣ Allows Precise Analysis:

 The more data we have, the more accurate our analysis and predictions will be.

Why do we need a system like Hadoop?

AI&ML VVIT Page 2

Big Data Analytics

✅ Since data is valuable but large, varied, and fast,

✅ We need a system that can:

 Scale up easily (handle large amounts of data),

 Store different types of data, and
 Process data quickly.

👉 This is why Hadoop is needed to handle the Volume, Variety, and Velocity challenges
shown in Figure 5.1 and to extract the full value of data.

The figure shows three people with three challenges related to Big Data:

1️⃣ First person:

“I am inundated with data. How to store terabytes of mounting data?”

✅ This shows the challenge of Volume – too much data is coming in, and we need to store it
safely.

2️⃣ Second person:

“I have data in varied sources. Structured, semi-structured, unstructured. How to work with such
different data?”
✅ This shows the challenge of Variety – data comes in many forms (tables, images, social
media posts), making it difficult to handle.

AI&ML VVIT Page 3

Big Data Analytics

3️⃣ Third person:

“I need this data to be processed quickly. My decision is pending.”

✅ This shows the challenge of Velocity – data needs to be processed quickly for faster
decisions.

Why Hadoop?
Hadoop is popular because:

✅ It can handle a massive amount of data (large size, many types of data)
✅ It can process this data quickly.

Figure 5.2: Key considerations of Hadoop

The diagram shows 5 reasons why Hadoop is used:

1️⃣ Low Cost

 Hadoop is open-source (free to use).

 It uses cheap, easily available hardware (commodity hardware) to store huge data,
reducing costs.

2️⃣ Computing Power

 Hadoop uses distributed computing, meaning it uses many computers working

together.
 More computers = more power to process data quickly.

3️⃣ Scalability

 Hadoop can grow easily by adding more computers (nodes) when needed.
 It requires less management even when it scales up.

4️⃣ Storage Flexibility

 Unlike traditional databases, Hadoop can store data without pre-processing.

 You can store unstructured data like images, videos, text easily.
 You can decide later how to use the stored data.

5️⃣ Inherent Data Protection

 Hadoop protects data even if hardware fails.

 If one computer fails, the work is automatically given to other working computers.

AI&ML VVIT Page 4

Big Data Analytics

 Hadoop stores multiple copies (replicas) of data on different computers for safety.

Summary Table for Quick Revision:

Reason Explanation
Low Cost Free, uses cheap hardware.
Computing Power Uses many computers for fast data processing.
Scalability Easy to add more computers as needed.
Storage Flexibility Can store any type of data, no need for pre-processing.
Data Protection Handles hardware failures, stores multiple data copies.

Hadoop, therefore, is widely used in Big Data because it can store, process, and protect
massive amounts of different types of data efficiently and cheaply.

Hadoop uses:
✅ Commodity Hardware (cheap, easily available computers)
✅ Distributed File System (data stored across many computers)
✅ Distributed Computing (processing done across many computers at once)

Figure 5.3 Explanation:

AI&ML VVIT Page 5

Big Data Analytics

The diagram shows:

 Many disks (computers) working together to store and process data.

 This group of machines working together is called a Cluster.
 Hadoop allows us to add more machines easily when needed (Scale Out) to handle
more data.

How Hadoop manages data:

1️⃣ Distributes and duplicates data:

 Big data is split into smaller parts (chunks), e.g., 25–30, 31–35, etc. shown in the
figure.
 These chunks are stored on multiple computers (nodes) to ensure safety and
availability.

2️⃣ Processes data locally in parallel:

 Each computer processes the data chunk it stores.

 All chunks are processed simultaneously (parallel processing), which saves time.

3️⃣ Handles failures automatically:

 If one computer fails, Hadoop automatically shifts the work to other computers.
 It keeps multiple copies of data chunks on different machines to protect against data
loss.

AI&ML VVIT Page 6

Big Data Analytics

Summary Table for Fast Revision:

Feature Explanation
Cluster Group of machines working together in Hadoop.
Data distribution Splits big data into smaller chunks stored on nodes.
Parallel Processing Each node processes its data chunk simultaneously.
Failover Handling Automatically shifts work if a node fails, protects data.

Key Points to Remember:

✅ Hadoop uses cheap hardware, making it cost-effective.

✅ Hadoop splits, stores, and processes data efficiently across many machines.
✅ Hadoop scales easily by adding more machines when data grows.
✅ Hadoop protects data automatically using multiple copies.

Why Not RDBMS?

📌 RDBMS (Relational Database Management System) is not suitable for Big Data because:

1️⃣ It cannot handle large files like images and videos easily.
2️⃣ It is not good for advanced analytics (like machine learning) on large datasets.
3️⃣ When data increases, the cost of storing and processing data in RDBMS becomes very
high.

Figure 5.4 Explanation:

AI&ML VVIT Page 7

Big Data Analytics

The diagram shows:

 As data volume increases (from terabytes to petabytes),

 The cost per GB of storage increases significantly in RDBMS systems.
 Scaling in RDBMS means “Scale Up” (buying bigger, more expensive servers), which
requires huge investment.

In Simple Points:

✅ RDBMS is costly when data increases.

✅ Cannot handle large files and unstructured data easily.
✅ Not suitable for advanced analytics and machine learning with big data.
✅ Scaling requires expensive hardware upgrades, making it less practical for Big Data.

Summary Table for Revision:

Limitation of RDBMS Explanation

Cannot handle large files Difficult to store images, videos, big logs efficiently.
Not suitable for ML/advanced
Cannot handle large datasets needed for ML effectively.
analytics
High cost per GB as data grows; scaling requires expensive
Costly at scale
servers.
Scale Up only Requires bigger machines, not easy scaling like Hadoop.

AI&ML VVIT Page 8

Big Data Analytics

This is why Hadoop is preferred over RDBMS for Big Data storage and processing, as it
can scale out cheaply and efficiently handle large, varied data.

5.4 RDBMS versus HADOOP

Table: Difference Between RDBMS and Hadoop (Simple Explanation)

PARAMETER RDBMS HADOOP

Relational Database Management
System Node-based flat structure
System
Suitable for structured data only Suitable for structured and unstructured
Data
(tables with rows and columns) data (images, videos, JSON, XML, text)
Used for OLTP (Online Used for Big Data analytics and
Processing
Transaction Processing) processing
Best when data needs consistent
Best for Big Data processing, even when
Choice relationships (e.g., banking
data does not need consistent relationships
transactions)
Needs expensive hardware and Needs only processor, network card, and
Processor high-end processors to store large few hard drives per node in the Hadoop
data cluster
$10,000–$14,000 per terabyte of
Cost $4,000 per terabyte of storage
storage

Key Points in Simple Language:

✅ RDBMS:

 Good for structured data only.

 Needs costly hardware for large data.
 Expensive when data grows.
 Used for transaction processing where relationships in data matter.

✅ HADOOP:

 Can store structured + unstructured data easily.

 Uses cheap hardware (commodity hardware).
 Much cheaper as data grows.
 Best for Big Data analysis and processing.

Why Hadoop is better for Big Data?

AI&ML VVIT Page 9
Big Data Analytics

✅ Can store any type of data (text, images, videos).

✅ Processes large data faster using clusters.
✅ Costs less compared to RDBMS.
✅ Easy to scale by adding more nodes.

5.5 DISTRIBUTED COMPUTING CHALLENGES

What is Distributed Computing?

It means using many computers connected in a network to store and process data together.

While distributed computing is powerful, it has two main challenges:

5.5.1 Hardware Failure

✅ In a distributed system, many servers are connected together.

✅ Hardware (like hard disks) can fail.

 For example, 1 hard disk may fail every 3 years.

 If you have 1000 hard disks, a few will fail daily.

Question: How can we protect data if a hard disk fails?

✅ Hadoop solves this using Replication Factor (RF).

Figure 5.5 Explanation: Replication Factor

 Replication Factor (RF) = The number of copies of data stored across the network to
protect data from hardware failures.
 Example: If RF = 2, it means there are 2 copies of each data block on different
computers.
 If one copy is lost due to hardware failure, the other copy can be used to recover the
data without loss.
AI&ML VVIT Page 10
Big Data Analytics

5.5.2 How to Process Gigantic Data?

✅ In a distributed system, data is spread across many computers.

✅ Before processing, data from different computers needs to be combined, which is
challenging.

How does Hadoop solve this?

✅ Hadoop uses MapReduce Programming,

✅ which processes data in parallel across different computers efficiently.

(You will study MapReduce in detail later.)

Summary Table for Quick Revision:

Challenge Problem Hadoop Solution

Hardware Replication Factor stores multiple
Disks can fail, causing data loss
Failure data copies on different nodes
Processing Data spread across many machines MapReduce Programming processes
Gigantic Data needs integration before processing data in parallel

Key Points to Remember:

✅ Replication Factor ensures data safety in case of hardware failures.

✅ MapReduce helps in efficiently processing large amounts of data spread across many
computers.
✅ These features make Hadoop reliable for Big Data processing.

5.6 HISTORY OF HADООР

✅ Hadoop was created by Doug Cutting, who also created Apache Lucene (a popular text
search library).
✅ Hadoop started as part of the Apache Nutch project, which is an open-source web search
engine project (Yahoo).
✅ Hadoop is also connected with the Lucene project.

AI&ML VVIT Page 11

Big Data Analytics

Figure 5.6 Explanation: Hadoop Timeline

📅 2002 – Doug Cutting and Mike Cafarella started working on Nutch (open-source search
engine).
📅 2003 – Google published papers on GFS (Google File System) and MapReduce, which
inspired Hadoop’s design.
📅 2004 – Doug Cutting added DFS (Distributed File System) and MapReduce to Nutch.
📅 2006 – Yahoo hired Doug Cutting, and Hadoop spun out as a separate project from Nutch.
📅 2008 – Cloudera was founded, a company providing Hadoop-based services.
📅 2009 – Doug Cutting joined Cloudera.

This timeline shows how Hadoop evolved from a small project to a widely used Big Data
framework.

5.6.1 The Name "Hadoop"

✅ The name “Hadoop” is not an acronym.

✅ It was a name given by Doug Cutting’s child to a stuffed yellow elephant toy.
✅ Doug Cutting chose it because:

 It is short and easy to spell and pronounce.

 It is unique and not used anywhere else.

🔸 Subprojects of Hadoop also have fun, animal-related names, e.g.:

 Pig (another Hadoop subproject for data analysis).

Key Points for Revision:

✅ Doug Cutting created Hadoop while working on Nutch at Yahoo.

✅ Inspired by Google’s GFS and MapReduce papers.

AI&ML VVIT Page 12

Big Data Analytics

✅ Became a separate project in 2006 under Yahoo.

✅ Doug Cutting joined Cloudera in 2009.
✅ The name “Hadoop” comes from his child’s toy elephant.

5.7 HADOOP OVERVIEW

✅ Hadoop is an open-source software framework that helps to store and process massive
amounts of data.

✅ It works in a distributed manner:

 Uses many connected computers (clusters) to store and process data together.
 These computers are commodity hardware (cheap and easily available).

✅ Hadoop mainly does two important tasks:

1️⃣ Massive data storage (stores large amounts of data easily).
2️⃣ Faster data processing (processes data quickly using parallel processing).

Figure 5.7: Key Aspects of Hadoop

The figure shows five key aspects of Hadoop:

1️⃣ Open-source software:

AI&ML VVIT Page 13

Big Data Analytics

 Hadoop is free to download, use, and contribute to.

2️⃣ Framework:

 Hadoop provides everything needed to develop and run applications on it:

o Programs
o Tools
o Libraries

3️⃣ Distributed:

 Hadoop divides and stores data across multiple computers.

 Data processing happens in parallel on these computers.

4️⃣ Massive storage:

 Hadoop can store huge amounts of data using low-cost hardware spread across many
computers.

5️⃣ Faster processing:

 Hadoop processes large data in parallel, making it faster and giving quick responses.

Summary Table for Quick Revision:

Aspect Explanation
Open-source Free to use, download, and contribute.
Framework Provides tools and programs to build/run applications.
Distributed Stores and processes data across multiple computers.
Massive storage Can store huge data using cheap hardware.
Faster processing Processes data quickly using parallel computing.

Key Points to Remember:

✅ Hadoop is open-source and cost-effective.

✅ Can store and process big data easily and quickly.
✅ Uses distributed, parallel processing on clusters.

5.7.2 Hadoop Components

AI&ML VVIT Page 14

Big Data Analytics

Figure 5.8 shows two parts:

1️⃣ Hadoop Core Components (main parts of Hadoop)

2️⃣ Hadoop Ecosystem (tools that support and enhance Hadoop’s core)

1️⃣ Hadoop Core Components:

✅ HDFS (Hadoop Distributed File System)

 It is the storage component of Hadoop.

 Stores and distributes data across many computers (nodes).
 Data is natively redundant (multiple copies are stored automatically), ensuring data
safety.

✅ MapReduce

 It is the computational (processing) framework of Hadoop.

 Splits large tasks into smaller parts and distributes them across many computers.
 Processes data in parallel, which makes it fast.

2️⃣ Hadoop Ecosystem:

These are support tools/projects that enhance Hadoop’s core functionalities:

 HIVE – For querying and managing large datasets using SQL-like language.
 PIG – For analyzing large data easily using a simple scripting language.
 SQOOP – For transferring data between Hadoop and relational databases.
 HBASE – A NoSQL database that runs on top of HDFS for real-time read/write access.

AI&ML VVIT Page 15

Big Data Analytics

 FLUME – For collecting and moving large amounts of log data into Hadoop.
 OOZIE – For managing and scheduling Hadoop jobs.
 MAHOUT – For machine learning and data mining on Hadoop.

Summary Table for Quick Revision:

Category Components Purpose

Core HDFS Stores and distributes data across nodes
MapReduce Processes data in parallel across nodes
Hive, Pig, Sqoop, HBase, Flume, Tools to enhance storage, processing, querying,
Ecosystem
Oozie, Mahout and machine learning in Hadoop

Key Points to Remember:

✅ HDFS stores and distributes data safely across nodes.

✅ MapReduce processes large data in parallel quickly.
✅ Hadoop Ecosystem tools add functionalities like querying, data transfer, scheduling,
real-time access, and machine learning to Hadoop.

5.7.3 Hadoop Conceptual Layer

It is conceptually divided into Data Storage Layer which stores huge volumes of data and Data Processing
Layer which processes data in parallel to extract richer and meaningful insights from data (Figure 5.9)

5.7.4 High-Level Architecture of Hadoop

Figure 5.9: Hadoop Conceptual Layer (Simple Explanation)

Hadoop has two main tasks:

🔹 Data Storage (HDFS) – Storing large amounts of data on many computers.

🔹 Data Processing (MapReduce) – Processing the data efficiently across many computers.

These two work together continuously in Hadoop.

Figure 5.10: Hadoop High-Level Architecture

AI&ML VVIT Page 16

Big Data Analytics

✅ Hadoop uses a Master-Slave Architecture:

 Master Node (NameNode):

o Controls and manages the system.
o Assigns tasks to slave nodes.
o Tracks where data is stored.
 Slave Nodes (DataNodes):
o Store actual data.
o Perform computations on the data as assigned by the master.

Key Components:

1️⃣ Master Node Responsibilities:

✅ Master HDFS:

 Manages data storage across slave nodes.

 Keeps track of which data is stored on which slave node (DataNode).

✅ Master MapReduce:

 Schedules and assigns computation tasks to slave nodes for processing data.

AI&ML VVIT Page 17

Big Data Analytics

2️⃣ Slave Node Responsibilities:

Each slave node (DataNode) has:

✅ Storage (HDFS) – Stores the actual data.
✅ Computation (MapReduce) – Processes data as instructed by the master node.

Summary Table for Quick Revision:

Component Role

Master Node Manages system, partitions storage, tracks data, and schedules computation
(NameNode) tasks

Slave Node (DataNode) Stores data and processes data as assigned by the master

HDFS Storage system of Hadoop

MapReduce Processing system of Hadoop

Key Points to Remember:

✅ Hadoop uses Master-Slave architecture for Big Data storage and processing.
✅ Master Node = NameNode (manages storage and processing).
✅ Slave Nodes = DataNodes (store data and process it).
✅ HDFS handles storage, MapReduce handles processing in Hadoop.

5.8 USE CASE OF HADOOP

5.8.1ClickStream Data Analysis using Hadoop — Simple Explanation

📌 What is ClickStream Data?

ClickStream data is information collected from users' clicks on a website. It shows what users
click, when they click, and in what order, helping businesses understand customer behavior —
like what products they look at or how they navigate a site.

🧠 Why Analyze ClickStream Data?

AI&ML VVIT Page 18

Big Data Analytics

ClickStream analysis helps online businesses:

 Understand customer purchasing behavior

 Improve product pages
 Optimize promotions and content for better engagement

📈 Benefits of Using Hadoop for ClickStream Data Analysis

The image (Figure 5.11) highlights three main benefits of using Hadoop for analyzing
ClickStream data:

💡 Key Benefit 📘 Explanation

Hadoop lets you combine click data with customer data (CRM),
🔗 Joins ClickStream data
sales, and marketing info. This helps you understand the full picture
with CRM and sales data
of customer behavior.
Hadoop is highly scalable, meaning you can store large volumes of
Stores years of data
ClickStream data for many years at a low cost. This helps in year-
without much cost
over-year trend analysis.
Tools like Hive and Pig make it easy to process and analyze
Uses Hive or Pig to
ClickStream data. You can organize data by user sessions, clean it
analyze data
up, and send it to dashboards or analytics tools.

Summary of Figure 5.11:

Figure 5.11: ClickStream Data Analysis

 Left box: "Joins ClickStream data with CRM and sales data."
 Middle box: "Stores years of data without much incremental cost."
 Right box: "Hive or Pig Script to analyze data."

This figure summarizes the core advantages of combining Hadoop with ClickStream analytics.

5.10 HDFS (HADOOP DISTRIBUTED FILE SYSTEM)

AI&ML VVIT Page 19

Big Data Analytics

HDFS is the storage system used in Hadoop. It lets you store very large files across many
computers in a reliable and efficient way.

🔑 Key Features of HDFS:

1. ✅ Storage Component of Hadoop — Used to store big data.

2. 🌐 Distributed File System — Files are split and stored across multiple machines.
3. 📄 Based on Google File System (GFS) — Inspired by Google’s method for managing
large files.
4. 🚀 High Throughput — Designed for fast reading/writing of large files by using:
o Large block sizes
o Moving computation to the data
5. 🔁 Replication for Fault Tolerance — Each file is copied (replicated) to multiple
machines to prevent data loss.
6. Self-Healing — If a machine fails, HDFS re-creates lost data blocks on other machines.
7. 📁 Optimized for Large Files — Works best with files that are gigabytes or larger.
8. 📦 Built on Native File System — Runs on top of traditional systems like ext3 or ext4,
as shown in Figure 5.13.

Figure 5.13: Hadoop Distributed File System

The image shows the layers of storage in HDFS:

pgsql
CopyEdit
+-----------------+ ← HDFS (Hadoop file system layer)
| HDFS |
+-----------------+
| Native OS Filesystem | ← Underlying OS file system (e.g., ext3, ext4)

AI&ML VVIT Page 20

Big Data Analytics

+-----------------+
| Disk Storage | ← Physical hard disk
+-----------------+

So, HDFS sits on top of the operating system’s file system, which in turn sits on the actual disk
hardware.

📚 Example:

Let’s say you have a file called [Link] with 192 MB size.

 HDFS splits it into blocks of 64 MB (default block size).

 So, it becomes 3 blocks (64 + 64 + 64 MB).
 If the replication factor is 3, each block is stored on 3 different nodes (for safety and
performance).

🔄 Data Flow in HDFS:

 Client App talks to:

o NameNode → for metadata (like file names, block locations)
o DataNodes → to read/write actual data
 DataNodes communicate with each other to manage block transfers (e.g., during
replication or recovery).

Hadoop Distributed File System (HDFS) – Key Points Explained Simply

Figure 5.14 shows some important settings and structure of HDFS in a very simple form.
Let’s break it down:

🧱 Feature 💬 Meaning
Block Structured Files in HDFS are divided into blocks instead of being stored as a whole.
File Each block is stored separately across machines in the Hadoop cluster.
Default Replication Every block is automatically copied 3 times to different machines. This
Factor: 3 helps prevent data loss if a machine fails.
Default Block Size: Each block is by default 64 MB in size. So a file of 192 MB will be split
64 MB into 3 blocks (64 MB + 64 MB + 64 MB).

Figure 5.14 Summary:

AI&ML VVIT Page 21

Big Data Analytics

This figure highlights:

 HDFS uses block-based storage.

 It has a default replication factor of 3 (for fault tolerance).
 It uses 64 MB block size by default (this can be changed based on needs).

📘 Real-World Example:

If you upload a large file to Hadoop:

 The file is broken into 64 MB blocks.

 Each block is replicated 3 times.
 These blocks are spread across different nodes in the Hadoop cluster.

This design makes HDFS reliable, scalable, and perfect for storing large data.

AI&ML VVIT Page 22

Big Data Analytics

Figure 5.15: Hadoop Distributed File System Architecture

🔧 Main Components:

1. Client Application (Hadoop File System Client):

o The software you use to read or write files into the Hadoop system.
2. NameNode:
o Acts like the "file manager" of Hadoop.
o It does not store the actual data but stores metadata (info about files and where
their parts are stored).
o Example: If you upload a file [Link], the NameNode breaks it into blocks
(A, B, C) and tells where each block is located.
3. DataNodes (A, B, C):
o These are the actual systems where the data blocks are stored.
o Each DataNode can hold copies of blocks (replicated blocks), as shown in the
image:
 DataNode A: Block A, B
 DataNode B: Block A, C
 DataNode C: Block B, C
4. Block Storage and Replication:

AI&ML VVIT Page 23

Big Data Analytics

o The file [Link] is split into 3 blocks: A, B, and C.

o These blocks are replicated across the DataNodes for fault tolerance.

📊 What Happens in This Architecture:

 The client contacts the NameNode to ask:

“Where should I read or write my file?”
 The NameNode replies with the locations of the blocks (which DataNodes hold which
block).
 The client then directly communicates with the DataNodes to read or write the actual
data.

🧠 Figure 5.16: NameNode — Simple Description

The NameNode manages everything related to files and their locations in the cluster.

🔑 NameNode Internals:

🧾 Term 📘 Meaning

A snapshot of the entire file system at a certain point. Think of it as a “master copy” of all file
FsImage
info.

EditLog A logbook that records every single change (like adding or deleting a file).

👉 These help recover the filesystem state in case the system restarts or crashes.

📌 Summary:

 Figure 5.15 shows how data flows in HDFS:

Client → NameNode (metadata) → DataNode (actual data)
 Figure 5.16 shows how the NameNode tracks all file operations using FsImage and
EditLog.

5.10.1 HDFS Daemons

[Link] NameNode

AI&ML VVIT Page 24

Big Data Analytics

 The NameNode is the master server in the Hadoop Distributed File System (HDFS).
 It controls all file operations like read, write, create, delete.
 It does not store actual data, but it keeps track of where each part of the file is stored
in the cluster.

🧱 How Files Are Stored

 HDFS breaks a large file into small blocks (default 128 MB).
 These blocks are stored on different DataNodes in the cluster.
 The NameNode manages all this mapping – which block is stored on which DataNode.

Rack Awareness

 Hadoop clusters are arranged in racks (a rack is a group of DataNodes).

 NameNode knows the rack ID of each DataNode.
 This helps in fault tolerance and network traffic optimization.

📚 File System Namespace

 The file system namespace is like a directory structure of all files and folders in
HDFS.
 The NameNode manages and stores this namespace.
 This includes:
o Which file is made up of which blocks
o Where the blocks are stored
o Properties of each file (size, permissions, timestamps)

📝 Important Files Used by NameNode

File Purpose
FsImage Permanent record of the file system (like a snapshot)
EditLog Keeps a log of every change or update made to the file system

What Happens When NameNode Starts

1. Reads FsImage and EditLog from the disk.

AI&ML VVIT Page 25

Big Data Analytics

2. Applies all updates from EditLog to FsImage.

3. Creates a new updated FsImage and saves it.
4. Deletes the old EditLog, because its changes are now in the new FsImage.

🚩 Key Points to Remember

 📌 Only one NameNode per cluster (single master).

 📌 NameNode does not store actual file data – only the metadata.
 📌 Keeps track of block-to-file mapping, rack locations, and file operations.
 📌 Uses FsImage and EditLog to maintain and update the file system.

 The NameNode is the brain of HDFS.

 It manages all file-related operations like:
o Creating files
o Reading and writing files
o Deleting files

📁 Important Components of the NameNode

Component What It Does

This is a file that stores the entire structure of the file system — like a snapshot of
FsImage
all files, directories, and block mappings.
This keeps a record of all changes (transactions) that happen to the file system. For
EditLog
example: if a file is created, renamed, or deleted, it is written in the EditLog.

🔁 How NameNode Uses These

1. When the NameNode starts up, it reads both:

o FsImage (the last known state)
o EditLog (all new updates)
2. It applies all changes from the EditLog to FsImage.
3. Then it saves a new FsImage (updated snapshot) and clears the old EditLog, because
its changes are already saved.

AI&ML VVIT Page 26

Big Data Analytics

🧾 Summary of Figure 5.16

 🔹 NameNode = Handles all file operations.

 🔸 FsImage = Full file system image (like a backup snapshot).
 🔸 EditLog = Records of every small update to the file system.

[Link] DataNode
 A DataNode is where actual data is stored in the Hadoop system.
 There are many DataNodes in a Hadoop cluster.

🔄 How DataNodes Communicate

 When data is being read or written, the DataNodes talk to each other directly — this is
called pipeline communication.

❤️What is a Heartbeat?

 A heartbeat is a small signal or message that a DataNode sends regularly to the

NameNode.
 It tells the NameNode:
✅ “I’m alive and working fine!”

❌ What Happens If There’s No Heartbeat?

 If the NameNode doesn’t receive a heartbeat, it assumes that the DataNode has failed
or gone offline.
 The NameNode then:
o Creates copies (replicas) of the lost data from that DataNode.
o Stores them on other working DataNodes.
o Keeps the system running smoothly, without any break.

🧠 Summary

AI&ML VVIT Page 27

Big Data Analytics

 DataNodes store the real data.

 They send heartbeats to the NameNode regularly.
 If a DataNode fails (no heartbeat), the NameNode replaces it using replicas — so the
system keeps working without any problem.

1. Heartbeat Signal:
o Each DataNode sends a regular signal called a "heartbeat" to the NameNode.
o This tells the NameNode:
👉 "I'm active and working fine!"
2. No Heartbeat = Problem:
o If the NameNode does not receive a heartbeat, it thinks the DataNode has
failed or is offline.
3. Replication:
o When a DataNode fails, the NameNode creates a copy (replica) of the lost data
and saves it on another working DataNode.
o This keeps the data safe and available.

🔁 Summary with Image:

 ✔️Left DataNode is healthy — sending heartbeat to NameNode.

 ❌ Middle DataNode failed — no heartbeat received.
 🔁 Right DataNode — receives replicated data from NameNode to replace the failed
one.

This system helps Hadoop be fault-tolerant and reliable — your data is always safe, even if a
node goes down!

AI&ML VVIT Page 28

Big Data Analytics

[Link] Secondary NameNode

 The Secondary NameNode is like a helper or assistant to the main NameNode.

 It takes regular snapshots (backups) of important HDFS metadata (information about
files and blocks).

🔄 How it works:

 It does not replace the NameNode.

 It does not keep track of live or real-time changes.
 It simply copies and stores metadata at regular time intervals.
 These intervals are set in the Hadoop configuration settings.

💡 Why run it on a different machine?

 Because it needs almost the same amount of memory as the NameNode.

 Running both on the same machine could slow things down or cause memory issues.

🆘 What happens if the NameNode fails?

 The Secondary NameNode can be used manually to recover the system, but only using
the last snapshot it took.
 It cannot restore the most recent changes that happened after the snapshot.

📝 Summary:

 ✅ Takes periodic snapshots of HDFS metadata.

 ❌ Does not track real-time updates.
 Should run on a different machine from NameNode.
 Can help recover the system if the NameNode fails — but manually, and only from the
last snapshot.

5.10.2 Anatomy of File Read

This diagram shows how a client reads a file from HDFS (Hadoop Distributed File System) in 6 easy
steps.

AI&ML VVIT Page 29

Big Data Analytics

Steps Explained Simply:

Step 1: Open the File

 The HDFS client wants to read a file.

 It calls open() on DistributedFileSystem to begin.

Step 2: Ask NameNode

 The DistributedFileSystem contacts the NameNode.

 It asks: "Where are the blocks of this file stored?"

Step 3: Get Data Location

 The NameNode replies with the locations of blocks (which DataNodes have the file
parts).
 Then, the client gets a special reader called FSDataInputStream to start reading.

Step 4: Start Reading from DataNode

 The client now knows where the first block is.

 It connects to the nearest DataNode that has the block and starts reading.

Step 5: Read Next Blocks

 When one block is done, the client disconnects.

 Then it repeats the process: connects to the next best DataNode (based on location or
availability) to read the next block.

Step 6: Close Connection

 After all blocks are read and the file is complete, the client calls close() to end the
reading session.

🧠 Summary with Roles:

 Client: Requests and reads the file.

 NameNode: Tells where file blocks are located.
 DataNodes: Actually store and send file blocks to the client.

AI&ML VVIT Page 30

Big Data Analytics


 Steps 1–3 happen between the client and the NameNode.
 Steps 4–6 happen between the client and the DataNodes.

5.10.3 Anatomy of File Write

his diagram and steps show how a file is saved (written) into HDFS in 7 easy steps.

Steps Explained Simply:

Step 1: Create a File

 The client wants to create a new file.

 It calls create() on the DistributedFileSystem.

Step 2: Contact NameNode

 The DistributedFileSystem sends a request to the NameNode to create the file.

 The NameNode checks if the file name already exists.
 If it's new, the file is created without any blocks yet.
 The client receives a stream called FSDataOutputStream to start writing.

AI&ML VVIT Page 31

Big Data Analytics

Step 3: Start Writing Data

 When the client writes data, it is:

o Broken into packets.
o Stored temporarily in a data queue.
 A DataStreamer sends these packets to DataNodes.
 The NameNode picks 3 DataNodes (default replication factor = 3).
o These 3 form a pipeline.

Step 4: Send Data through Pipeline

 The first DataNode receives the packet and saves it.

 It then forwards the same packet to the second DataNode.
 The second forwards it to the third DataNode.

Step 5: Wait for Acknowledgement (Ack)

 After each DataNode gets the packet, they send an acknowledgment (ack) back.
 Only after all 3 DataNodes confirm, the packet is removed from the "Ack Queue".

Step 6: Finish Writing

 When the client finishes sending all data, it calls close() on the stream.

Step 7: Confirm with NameNode

 Remaining packets (if any) are sent.

 After all acks are received, the NameNode is informed that the file is fully written and
stored.

🧠 Summary of Components:

 Client: Writes the file.

 NameNode: Keeps metadata, assigns DataNodes.
 DataNodes: Actually store the data.
 Pipeline: 3 DataNodes that replicate data from client → D1 → D2 → D3.

Image Walkthrough:

 Steps 1–3: Between Client and NameNode.

 Steps 4–5: Between Client and a chain of 3 DataNodes (pipeline).

AI&ML VVIT Page 32

Big Data Analytics

 Steps 6–7: Finish and confirm with NameNode.

5.10.4 Replica Placement Strategy

[Link] Hadoop Default Replica Placement Strategy

In Hadoop, when you save a file, it is split into blocks and replicated (copied) across multiple
machines (called DataNodes).
This is done to:

 Prevent data loss

 Improve data availability
 Increase read performance

📦 Hadoop’s Default Replica Strategy (3 Copies)

Hadoop creates 3 copies (replicas) of each block using this smart placement method:

✅ Replica 1 (First Copy):

 Placed on the same machine (node) where the client is writing from.
→ This helps reduce network traffic and makes writing faster.

AI&ML VVIT Page 33

Big Data Analytics

✅ Replica 2 (Second Copy):

 Placed on a different rack (a group of computers connected together).

→ If the first rack fails, the data is still safe in another rack.

✅ Replica 3 (Third Copy):

 Placed on the same rack as Replica 2, but on a different machine.

→ This provides quick access while still avoiding a single point of failure.

🔄 Pipeline Formation:

 After selecting where the replicas will go, a pipeline is created between the client and the
3 DataNodes.
 Data flows from the client → first DataNode → second → third, one after another.

🎯 Why This Strategy?

 Balances performance (by keeping data close to the client)

 Improves fault tolerance (by spreading data across racks)
 Ensures data is still available even if a machine or rack fails

5.10.5 Working with HDFS Commands

1. Show files/folders at the root of HDFS
bash
CopyEdit

AI&ML VVIT Page 34

Big Data Analytics

hadoop fs -ls /

🔹 Shows a list of files and directories in the root folder (/) of the Hadoop file system.

✅ 2. Show all files/folders in HDFS (Recursively)

bash
CopyEdit
hadoop fs -ls -R /

🔹 Lists all files and directories in the entire HDFS, including subfolders.

📁 Create a directory

✅ 3. Create a directory in HDFS

bash
CopyEdit
hadoop fs -mkdir /sample

🔹 Creates a new folder named sample in HDFS.

🔼 Copy file from Local to HDFS

✅ 4. Copy a file from your computer (local system) to HDFS

bash
CopyEdit
hadoop fs -put /root/sample/[Link] /sample/[Link]

🔹 Uploads [Link] from your computer’s /root/sample/ folder to HDFS at /sample/.

🔽 Copy file from HDFS to Local

✅ 5. Download a file from HDFS to your computer

bash
CopyEdit
hadoop fs -get /sample/[Link] /root/sample/[Link]

🔹 Downloads the file [Link] from HDFS to your local folder as [Link].

AI&ML VVIT Page 35

Big Data Analytics

🔁 Other copy commands

✅ 6. Another way to copy from local to HDFS

bash
CopyEdit
hadoop fs -copyFromLocal /root/sample/[Link] /sample/[Link]

🔹 Same as -put, it uploads a file to HDFS.

✅ 7. Copy file from HDFS to local system

bash
CopyEdit
hadoop fs -copyToLocal /sample/[Link] /root/sample/[Link]

🔹 Same as -get, it downloads a file from HDFS.

📜 View contents of a file

✅ 8. Display file contents on screen

bash
CopyEdit
hadoop fs -cat /sample/[Link]

🔹 Shows the contents of the file on your terminal.

🔄 Copy file within HDFS

✅ 9. Copy a file from one HDFS folder to another

bash
CopyEdit
hadoop fs -cp /sample/[Link] /sample1

🔹 Copies [Link] from /sample/ to /sample1/ directory within HDFS.

❌ Delete directory

✅ 10. Remove a folder from HDFS

bash
CopyEdit
hadoop fs -rm -r /sample1

AI&ML VVIT Page 36

Big Data Analytics

🔹 Deletes the folder /sample1 and all its contents from HDFS.

5.10.6 Special Features of HDFS

1. Data Replication

 HDFS automatically makes copies of each file (called replicas) and stores them on
different DataNodes.
 The client doesn’t need to worry about where each block of data is stored.
 HDFS chooses the closest copy (replica) of the data for reading, to make it fast and
efficient.

✅ Example:
If a file is split into blocks and stored in 3 places, HDFS will guide your computer to the nearest
one for faster access.

🚰 2. Data Pipeline

 When a client writes a file to HDFS, it sends each block to the first DataNode in a
chain (called a pipeline).
 That first DataNode passes the block to the second one, which then passes it to the
third, and so on.
 This way, multiple copies (replicas) of the data block are saved in one smooth flow.

✅ Example:
Block → DataNode 1 → DataNode 2 → DataNode 3
Each node stores the block before passing it along the line.

🔑 Summary:

 You don’t need to manually track or manage where data is saved.

 Data is copied and stored in a smart, fast way for reliability and speed.

5.11 PROCESSING DATA WITH HADООР

What is MapReduce?

AI&ML VVIT Page 37

Big Data Analytics

MapReduce is a software framework used in Hadoop to process huge amounts of data in

parallel across multiple computers (nodes).

🔧 How It Works: Step-by-Step

1. Input Split into Parts:

o The data is divided into chunks.
o Each chunk is processed independently by a Map task.
2. Map Phase:
o Converts each part of data into key-value pairs.
o These are like little labels that help organize the data.
3. Shuffle and Sort (Automatic):
o After the map phase, the data is shuffled and sorted by key.
o This means data with the same key is grouped together.
4. Reduce Phase:
o Combines the sorted data with the same key and produces the final result.
o For example, it can sum up values or count occurrences.
5. Final Output:
o The reduced result is saved into the Hadoop File System (HDFS).

⚙️Who Does What? (Daemons)

 JobTracker (Master):
o Schedules tasks (Map and Reduce).
o Monitors their progress.
o Re-runs tasks if something fails.
 TaskTracker (Slave):
o Executes the task assigned by the JobTracker.
o Reports progress back to the JobTracker.

💡 Extra Features

 Data Locality: Hadoop tries to run tasks on the same machine where data is stored to
save time.
 High Throughput: This system allows fast processing of large datasets.

AI&ML VVIT Page 38

Big Data Analytics

📦 Job Configuration

 A MapReduce program includes the logic (functions), data locations, and job settings.
 These are packaged as a job and sent to the JobTracker for execution.

🔁 In Short:

 MapReduce = Map (process) + Reduce (combine) + Hadoop magic (sort, schedule,

handle failure).
 It makes big data processing easy, efficient, and fault-tolerant.

5.11.1 MapReduce Daemons

1. Client

 The Client is the user or application that submits a job (a MapReduce program) to
Hadoop.

AI&ML VVIT Page 39

Big Data Analytics

📌 2. JobTracker (Master Daemon)

 It receives the job from the client.

 It then plans how the job will be executed:
o Which parts go to which nodes
o How many map/reduce tasks are needed
 It then sends the tasks to TaskTrackers (workers).
 JobTracker also monitors the entire job:
o If any task fails, it reassigns it to another TaskTracker.
o It listens for heartbeats from TaskTrackers to make sure they’re still working.

⚙️3. TaskTrackers (Slave Daemons)

 Each TaskTracker runs on a worker/slave node.

 It executes the actual map or reduce task given by the JobTracker.
 It creates multiple JVMs (Java processes) to run multiple tasks in parallel.
 It keeps sending heartbeat messages to JobTracker to say “I’m alive”.
 If a TaskTracker stops sending heartbeat, the JobTracker will assume it's failed and
move the task somewhere else.

🔁 Workflow from the Image

1. Client submits job ➝ JobTracker

2. JobTracker ➝ distributes tasks ➝ multiple TaskTrackers
3. Each TaskTracker ➝ runs Map and Reduce tasks
4. TaskTrackers report status back to JobTracker

📊 Summary Table

Component Role
Client Submits the MapReduce job
JobTracker Master that plans, distributes, and monitors all tasks
TaskTracker Worker that runs map/reduce tasks on each node
JVMs (inside TaskTracker) Used to run multiple map or reduce tasks in parallel
Heartbeat Message sent by TaskTracker to show it is still running

AI&ML VVIT Page 40

Big Data Analytics

5.11.2 How Does MapReduce Work?

How MapReduce Works

Let’s break it down step-by-step:

1. Input Splitting:
o Big input data is split into smaller chunks (pieces).
o Each chunk is processed independently.
2. Creating Master & Workers:
o A master (JobTracker) is created to manage the task.
o Multiple worker nodes (TaskTrackers) are started to perform map and reduce
tasks.
3. Map Phase:
o Each worker processes its assigned data chunk.
o It uses a map function to turn the data into key-value pairs.
4. Partitioning:
o Map output is divided by a partitioner.
o The partitioner decides which reducer should receive which part of the data.
5. Shuffle & Sort:
o Once map tasks are done, reduce tasks start.
o Reducers collect map outputs, shuffle (reorganize) and sort them by keys.
6. Reduce Phase:
o The reduce function processes sorted data by key.
o It combines values and produces final output.
7. Completion:
o When reducers finish, the JobTracker gives the output to the user.

✅ Summary (Easy to Remember):

Step What Happens?

Split Input data is divided into small chunks
Map Each chunk is processed → gives key-value output
Shuffle/Sort Outputs are grouped by key and prepared for reduction
Reduce Outputs are combined to give final results

AI&ML VVIT Page 41

Big Data Analytics

Figure 5.23 – MapReduce Programming Workflow

This figure explains how a MapReduce job is processed step-by-step:

Simple Explanation:

1. JobTracker (Master Controller):

The JobTracker assigns the computing tasks. It is the "brain" that controls everything.
2. Map Tasks:
o The input data is divided into small data segments.
o These are given to Map Tasks, which run on different nodes.
o For example, TaskTracker 1 processes Data Segment 1 and gives Map Output 1.
o Similarly, TaskTracker 2 processes Data Segment 2 and gives Map Output 2.
3. Reduce Task:
o TaskTracker 3 collects all the Map Outputs.
o It combines them and gives the final output.

✅ Key Point: Map tasks work in parallel. Once all map outputs are ready, reduce task collects
and combines them.

AI&ML VVIT Page 42

Big Data Analytics

Figure 5.24 – MapReduce Programming Architecture

This figure shows how clients, JobTracker, and TaskTrackers work together.

Simple Explanation:

1. Clients:
Users or applications that send jobs (tasks) to be executed.
2. JobTracker (Master):
Receives jobs from clients, plans the job, and assigns small tasks to different nodes.
3. TaskTrackers (Slaves):
o Each TaskTracker handles a few tasks (Map or Reduce).
o They report back to JobTracker about task progress.

✅ Key Point: JobTracker is like a manager and TaskTrackers are like workers who execute the
tasks.

5.11.3 MapReduce Example

Imagine you have 50 text files, and you want to count how many times each word appears
across all of them.

Instead of doing it manually, you can use MapReduce programming to automate this process.

AI&ML VVIT Page 43

Big Data Analytics

🔁 How it Works (Word Count Example):

Let’s say the files contain words like:

nginx
CopyEdit
apple banana apple mango
banana apple mango

MapReduce will do the following:

✅ Map Phase:

 It reads each line from all files.

 Breaks the lines into words.
 For each word, it outputs something like:
apple → 1, banana → 1, apple → 1, etc.

So the map output might look like:

nginx
CopyEdit
apple → 1
banana → 1
apple → 1
mango → 1
banana → 1
apple → 1
mango → 1

🔀 Shuffle and Sort Phase:

 Groups the same words together:

css
CopyEdit
apple → [1, 1, 1]
banana → [1, 1]
mango → [1, 1]

✅ Reduce Phase:

 Adds up the numbers for each word:

nginx
CopyEdit
apple → 3
banana → 2
mango → 2

AI&ML VVIT Page 44

Big Data Analytics

🔧 MapReduce Java Code Components:

To make this work in Java, you need 3 parts (classes):

1. Driver Class

o This is like the manager of the program.

o It configures the job: tells which input files to use, which mapper and reducer to
run, and where to save the result.

2. Mapper Class

o Reads the input line by line.

o Breaks lines into words.
o Sends out key-value pairs like: word → 1.

3. Reducer Class ➕

o Takes the grouped key-value pairs from the mapper.

o Adds the counts together for each word.
o Writes the final result like: word → total count.

✅ Summary:

Component Role

Driver Class Sets up and manages the MapReduce job

Mapper Class Processes input → turns words into (word, 1)

Reducer Class Adds up values → gives final word counts

[Link]: Driver Program

✅ Package Declaration
java
CopyEdit
package [Link];

 This means your program is part of the [Link] package (like a folder for organizing
Java files).

AI&ML VVIT Page 45

Big Data Analytics

✅ Import Statements

These lines bring in the tools and libraries needed for your Hadoop program:

java
CopyEdit
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];

 Path: Used to specify input and output file paths.

 IntWritable: A Hadoop type to represent integers.
 Text: A Hadoop type to represent strings.
 Job: Manages the whole MapReduce task.
 FileInputFormat: Reads input files.
 TextInputFormat: Used to read plain text files line-by-line.
 FileOutputFormat: Writes the final output to files.
 TextOutputFormat: Writes output as plain text.

✅ Main Class
java
CopyEdit
public class WordCounter {

 This is the main driver class for the MapReduce program.

✅ Main Method
java
CopyEdit
public static void main (String [] args) throws IOException,
InterruptedException, ClassNotFoundException {

 This is the starting point of the program.

 It can throw exceptions related to input/output or thread interruptions.
AI&ML VVIT Page 46
Big Data Analytics

✅ Create a Job
java
CopyEdit
Job job = new Job ();

 Creates a new MapReduce job instance.

✅ Set Job Name

java
CopyEdit
[Link] ("wordcounter");

 Gives your job a name — helpful for identifying it in logs or monitoring tools.

✅ Set the JAR File

java
CopyEdit
[Link] ([Link]);

 Tells Hadoop which .jar file to run by using the class containing the main() method.

✅ Set Mapper and Reducer Classes

java
CopyEdit
[Link] ([Link]);
[Link] ([Link]);

 Specifies which classes contain the Mapper and Reducer logic.

✅ Set Output Key and Value Types

java
CopyEdit
[Link] ([Link]);
[Link] ([Link]);

AI&ML VVIT Page 47

Big Data Analytics

 Says that:
o The output key is a word (Text).
o The output value is a count (IntWritable).

🛑 Note: setoutputValueClass should be setOutputValueClass (case-sensitive Java typo).

✅ Set Input Path

java
CopyEdit
[Link] (job, new Path ("/sample/[Link]"));

 Tells the job where to read the input file from.

 Here it will read /sample/[Link].

✅ Set Output Path

java
CopyEdit
[Link] (job, new Path ("/sample/wordcount"));

 Tells the job where to store the output.

 It will save the word count result in /sample/wordcount.

✅ Run the Job

java
CopyEdit
[Link] ([Link] (true)? 0 : 1);

 Runs the job and waits until it finishes.

 If the job finishes successfully, it exits with 0 (success).
 If it fails, it exits with 1 (failure).

Fixes Needed:

You should correct the following issues in your code:

1. This line has a colon : instead of a semicolon ;:

AI&ML VVIT Page 48

Big Data Analytics

java
CopyEdit
[Link] ([Link]): // WRONG

Change it to:

java
CopyEdit
[Link] ([Link]); // CORRECT

2. This line:

java
CopyEdit
[Link] ([Link]);

Should be:

java
CopyEdit
[Link] ([Link]);

3. The FileOutputFormat line is broken across lines:

java
CopyEdit
[Link] (job, new Path ("/sample/
wordcount")):

Should be:

java
CopyEdit
[Link] (job, new Path ("/sample/wordcount"));

[Link]: Map Class

1. Package Declaration
java
CopyEdit
package [Link];

 The code belongs to the [Link] package (like a folder or namespace to group related
classes).

✅ 2. Import Statements

AI&ML VVIT Page 49

Big Data Analytics

java
CopyEdit
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];

These import required classes from Java and Hadoop:

 IOException: Handles input/output errors.

 IntWritable: Hadoop's integer data type (used instead of regular int).
 LongWritable: Hadoop's long integer type (used for keys).
 Text: Hadoop’s string data type.
 Mapper: The base class for all mappers in Hadoop.

✅ 3. Mapper Method Declaration

java
CopyEdit
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {

This is the Map method which runs for each line of the input file:

 key: The position of the line in the file (line offset), not used here.
 value: The line of text itself (as Text).
 context: This allows the mapper to write output (key-value pairs) to the Hadoop
system.

✅ 4. Split the Line into Words

java
CopyEdit
String[] words = [Link]().split(",");

 Converts the line to a Java String.

 Splits the string into words using comma (,) as the separator.
 Stores the words in a string array called words.

📝 Example: If the line is "apple,banana,apple", it will become ["apple", "banana",

"apple"].

AI&ML VVIT Page 50

Big Data Analytics

✅ 5. Write Each Word with Count 1

java
CopyEdit
for (String word : words) {
[Link](new Text(word), new IntWritable(1));
}

 Loops through each word in the line.

 For each word, it writes a key-value pair:
o Key = the word (e.g., "apple")
o Value = 1 (because the word appears once in this line)

🧠 Hadoop will later group all identical keys (words) and add the counts in the Reduce phase.

✅ 6. Closing the Class

java
CopyEdit
}

 This ends the map method and the class (though the class declaration line itself is missing
from your code — see below).

⚠️Issues in Your Code

You are missing the class declaration! Here's the correct full version of the Mapper class:

java
CopyEdit
package [Link];

import [Link];
import [Link];
import [Link];
import [Link];
import [Link];

public class WordCounterMap extends Mapper<LongWritable, Text, Text,

IntWritable> {

protected void map(LongWritable key, Text value, Context context)

throws IOException, InterruptedException {

String[] words = [Link]().split(",");

for (String word : words) {

AI&ML VVIT Page 51

Big Data Analytics

[Link](new Text(word), new IntWritable(1));

}
}
}

✅ What This Code Does

If you have a text file with lines like:

CopyEdit
apple,banana,apple
banana,apple

This Mapper will output:

nginx
CopyEdit
apple 1
banana 1
apple 1
banana 1
apple 1

Each word is counted as it appears — the Reducer will combine them later.

[Link]: Reduce Class

1. Package Declaration
java
CopyEdit
package [Link];

This Java file is part of the [Link] package (like a folder for related files).

✅ 2. Import Statements
java
CopyEdit
import [Link];
import [Link];
import [Link];
import [Link];

AI&ML VVIT Page 52

Big Data Analytics

These lines import classes needed for the program:

 IOException: Handles file reading/writing errors.

 IntWritable: Hadoop's version of integer data.
 Text: Hadoop's version of string data.
 Reducer: The base class used to create a Reducer in MapReduce.

✅ 3. Reducer Class Declaration

java
CopyEdit
public class WordCounterRed extends Reducer<Text, IntWritable, Text,
IntWritable> {

This line defines your custom Reducer class:

 WordCounterRed is the name of the class.

 It extends Reducer, meaning it inherits and overrides its behavior.
 Text, IntWritable, Text, IntWritable:
o Input Key = Text (a word)
o Input Value = IntWritable (count for that word)
o Output Key = Text (the word)
o Output Value = IntWritable (total count)

✅ 4. Reduce Method Declaration

java
CopyEdit
@override
protected void reduce(Text word, Iterable<IntWritable> values, Context
context)
throws IOException, InterruptedException {

This is the reduce method, which runs for each unique word sent by the Mapper:

 word: the actual word (like "apple")

 values: a list of 1s from the Mappers (e.g., [1, 1, 1] for 3 apples)
 context: allows you to write output from the reducer

✅ 5. Counting the Total

AI&ML VVIT Page 53

Big Data Analytics

java
CopyEdit
Integer count = 0;
for(IntWritable val : values) {
count += [Link]();
}

 Starts a counter at 0.
 Loops through each value (val) in the list of counts.
 [Link]() gets the integer value (each is 1).
 Adds them together.

🧠 For example, if "banana" appears 4 times, this loop adds 1+1+1+1 = 4.

✅ 6. Writing the Final Output

java
CopyEdit
[Link](word, new IntWritable(count));

 Writes the final result to output.

 Key = the word (e.g., "banana")
 Value = total count (e.g., 4)

✅ 7. Closing the Class

java
CopyEdit
}
}

 Ends the reduce method and the class.

✅ What the Reducer Does (In Simple Words):

For each unique word (like “apple”), the Reducer:

1. Collects all the counts (which are all 1s).

2. Adds them up.
3. Writes the final word and total count to the output.

AI&ML VVIT Page 54

Big Data Analytics

🧾 Example:

If the Mapper output is:

nginx
CopyEdit
apple 1
apple 1
banana 1
apple 1
banana 1

Then the Reducer will output:

nginx
CopyEdit
apple 3
banana 2

5.12 MANAGING RESOURCES AND APPLICATIONS

WITH HADOOP YARN (YET ANOTHER RESOURCE
NEGOTIATOR)
Managing Resources and Applications with Hadoop, focusing on YARN and improvements in Hadoop
2.x:

What is YARN (Yet Another Resource Negotiator)?

 YARN is a key part of Hadoop 2.x architecture.

 It helps Hadoop manage resources (CPU, memory, etc.) across all applications.
 Earlier, Hadoop was mostly used for MapReduce jobs. But with YARN, Hadoop can
now support:
o Batch processing
o Streaming
o Interactive processing
o Graph processing
o Machine learning, and more
 Think of YARN as the operating system of Hadoop, which allocates and tracks
resources like a manager.

🔷 Problems in Hadoop 1.0

Before YARN, Hadoop had several limitations:

AI&ML VVIT Page 55

Big Data Analytics

1. ✅ Only one NameNode: This NameNode managed all file system information, which
created a single point of failure.
2. ❌ Only supported MapReduce: It couldn’t handle other processing types like real-time
or streaming.
3. 🕒 Slow for interactive jobs: It was not designed for quick, responsive analysis.
4. 🧠 Not good for memory-heavy tasks like machine learning or graph analysis.
5. Poor resource usage: Sometimes, map slots were full, but reduce slots were empty—
and vice versa. This led to wasted resources.

🔷 HDFS Limitation in Hadoop 1.x

 The NameNode kept all metadata (file and directory info) in main memory (RAM).
 RAM has a limit—so if there are too many files or blocks, the NameNode becomes
overloaded.
 This made Hadoop less scalable.

🔷 Solution in Hadoop 2.x: HDFS Federation

Hadoop 2.x introduced HDFS Federation to fix these problems:

1. ✅ Multiple NameNodes:
o Now, instead of just one NameNode, there can be many.
o Each NameNode manages a portion of the namespace, and they work
independently.
o This improves horizontal scalability—you can add more NameNodes as your
data grows.
2. ✅ Shared DataNodes:
o All NameNodes share the same DataNodes (the machines that store actual data
blocks).
o These DataNodes register with all NameNodes.

🔷 High Availability (HA) in Hadoop 2.x

Hadoop 2.x also solved the single-point-of-failure problem of the NameNode:

🔁 Active-Passive NameNode Setup:

 Active NameNode: This is the one currently in use.

 Passive (Standby) NameNode: This is ready to take over if the active one fails.
 Both NameNodes share metadata using NFS (Network File System).

AI&ML VVIT Page 56

Big Data Analytics

 Only the Active writes data; the Passive only reads updates and stays in sync.
 If the Active fails, the Passive becomes Active automatically, ensuring no downtime.

✅ Summary:

Feature Hadoop 1.x Hadoop 2.x

Resource Manager JobTracker (limited) YARN (flexible & efficient)

NameNode Single Multiple (Federation)

Availability No automatic failover Active-Passive HA

Processing Support Only MapReduce MapReduce + Streaming + Graph + ML

Resource Utilization Poor Optimized by YARN

Scalability Limited High (horizontally scalable)

 There are two NameNodes:

1. Active NameNode – currently managing the file system and processing all client
requests.
2. Passive NameNode – on standby, ready to take over if the Active one fails.
 In the middle is a Shared Edit Log:

o This is like a common record of all changes made to the Hadoop file system
(e.g., file creation, deletion, renaming).
o It is stored in a shared location, such as a Network File System (NFS).

🔁 How It Works:

AI&ML VVIT Page 57

Big Data Analytics

1. 📝 Active NameNode Writes:

o Every time there’s a change (e.g., a new file is added), the Active NameNode
writes that update to the Shared Edit Logs.
2. 📖 Passive NameNode Reads:
o The Passive NameNode continuously reads these edit logs to stay updated.
o It keeps a copy of the latest metadata, just like the Active NameNode.
3. 🔄 Failover Mechanism:
o If the Active NameNode fails, the Passive NameNode becomes Active
automatically.
o Since it already has up-to-date information from the shared log, it can continue
the work without downtime.

✅ Key Benefits:

 Ensures high availability (no single point of failure).

 Avoids system crashes or interruptions.
 Automatically switches control to the standby node in case of failure.

5.12.4 Hadoop 2 YARN: Taking Hadoop beyond Batch

Main Idea:

 In YARN, responsibilities are divided into separate parts:

o Resource management
o Job scheduling and tracking

🔧 YARN Components (Simple Explanation):

1. ResourceManager (RM) – Boss of the cluster

 Allocates resources to all applications.

 Two parts:
o Scheduler: Decides who gets how much CPU, memory, etc.
o Application Manager: Accepts jobs and helps start the ApplicationMaster.

2. NodeManager (NM) – Worker on each machine

 Starts and monitors "containers" (mini environments to run tasks).

 Reports resource usage (CPU, memory) to the RM.

AI&ML VVIT Page 58

Big Data Analytics

3. ApplicationMaster (AM) – Boss of one job

 Works only for one application (like one MapReduce job).

 Talks to ResourceManager to get resources.
 Talks to NodeManagers to actually run the job.

🧱 YARN Concepts Simplified:

🔹 Application:

 A job that is submitted to run on Hadoop (e.g., a MapReduce program).

🔹 Container:

 A small, isolated environment where tasks run.

 It can have custom amounts of:
o Memory (e.g., 2GB)
o CPU (e.g., 1 or more cores)
 Example:
o container_0 = 2GB, 1 CPU
o container_1 = 1GB, 6 CPU
 Replaces the old concept of fixed map/reduce slots for better flexibility.

🧠 Summary:

 Hadoop 1.0: Limited to batch processing using MapReduce.

 Hadoop 2.x + YARN:
o More flexible, supports many types of applications.
o Resource management is better and more dynamic.
o Enables interactive, real-time, and in-memory processing (like Spark).

AI&ML VVIT Page 59

Big Data Analytics

ext and images (Figure 5.27 and 5.28) in simple language to help you understand how
Hadoop 2.x and YARN improved over Hadoop 1.x.

✅ Figure 5.27 – Hadoop 1.x vs Hadoop 2.x

Hadoop 1.0 Hadoop 2.0

Has MapReduce and HDFS Still has MapReduce and HDFS

MapReduce does 2 jobs: But now the jobs are separated:

1. Manages cluster resources - MapReduce: Only for data processing

2. Processes data - YARN: New system that manages cluster

resources more efficiently

🔁 What's the problem with Hadoop 1.0?

AI&ML VVIT Page 60

Big Data Analytics

 MapReduce had to do both resource management and data processing.

 This caused bottlenecks and limited usage to only batch jobs.

🌟 What’s improved in Hadoop 2.0?

 YARN separates cluster management from data processing.

 This makes it easier to run different kinds of applications, not just MapReduce.

✅ Figure 5.28 – Hadoop YARN Architecture

 YARN allows different types of data processing to run on the same Hadoop cluster:
o Batch jobs (MapReduce)
o Interactive queries (TEZ)
o Online applications (HBase)
o Streaming data (Storm)
o In-memory data (Spark)
o Graph or Search jobs (Others)

➡ All these applications share the same storage (HDFS 2) and are managed by YARN.

YARN Architecture

AI&ML VVIT Page 61

Big Data Analytics

The image (Figure 5.29) shows how YARN (Yet Another Resource Negotiator) works behind
the scenes to run big data applications in a Hadoop cluster.

Let’s break down the steps shown in the image in a very simple way:

🔢 Step-by-Step Flow (Matched with Image Arrows):

1. Client Submits Application

 A Client sends an application to ResourceManager.

 This application includes the program to run and the instructions to launch a special
controller called the ApplicationMaster (AppMstr).

2. ResourceManager Starts AppMaster

 The ResourceManager finds a free spot (called a Container) in one of the

NodeManagers.
 It then launches the ApplicationMaster in that container.

3. AppMaster Registers with ResourceManager

 After starting, the AppMaster tells the ResourceManager that it is ready.

 This allows the Client to directly talk to AppMaster to check job progress.

4. AppMaster Asks for Resources

 The AppMaster requests more containers (i.e., space and CPU/memory) to run tasks of
the application.

5. ResourceManager Assigns Containers

 The ResourceManager assigns more containers by contacting NodeManagers.

 These containers are then ready to run different parts of the application.

6. NodeManager Runs the Code

 The AppMaster tells NodeManagers what to run inside those containers.

 NodeManagers execute the code and report status and progress back to the AppMaster.

7. Client Communicates with AppMaster

 The Client can ask the AppMaster directly about job status, progress, or errors during
execution.

AI&ML VVIT Page 62

Big Data Analytics

8. AppMaster Finishes Job

 When the job is done, the AppMaster deregisters itself from the ResourceManager and
shuts down.
 Its container is now free for other applications.

🔁 Roles Summary:

 Client: Submits the job.

 ResourceManager: Main coordinator. Assigns containers and tracks everything.
 NodeManager: Manages containers in a node.
 AppMaster: Manages one specific job (application). Handles resource requests and
communication.

📌 Visual Mapping:

Arrow Number Description

1 Client → ResourceManager: Submit job

2 ResourceManager → NodeManager: Start AppMaster

3, 4, 8 AppMaster ↔ ResourceManager: Register, request resources, deregister

5 ResourceManager → NodeManager: Allocate containers

6 AppMaster → NodeManager: Launch tasks in containers

7 Client ↔ AppMaster: Get status and updates

AI&ML VVIT Page 63

Big Data Analytics

MapReduce
8.1 INTRODUCTION

MapReduce in Simple Words

MapReduce is a programming model used in Hadoop to process large amounts of data by

splitting the work into smaller parts and running them in parallel across many computers in a
cluster.

🔄 How MapReduce Works:

1. A Job (Application) is divided into Map Tasks and Reduce Tasks.

2. These tasks are distributed across different computers (called DataNodes) in the Hadoop
cluster.
3. Each task handles only a small part of the total data, which is stored in HDFS (Hadoop
Distributed File System).

MAP Task Responsibilities:

The Map Task handles the initial processing of the data. This includes:

1. RecordReader – Reads data line by line from input files.

2. Mapper – Applies logic to process each line (e.g., split, filter, modify).
3. Combiner – An optional step to reduce the data size by summarizing it early (like a
mini-reducer).
4. Partitioner – Divides the data and sends it to different reducers.

✔️Output: Map Task produces intermediate key-value pairs (temporary data for next step).

🧮 REDUCE Task Responsibilities:

The Reduce Task takes all the intermediate key-value pairs and processes them to create the
final output.

Steps involved:

1. Shuffle – Collects and moves intermediate data from mappers to reducers.

2. Sort – Sorts the data by keys.

AI&ML VVIT Page 64

Big Data Analytics

3. Reducer – Applies logic to group and summarize the data.

4. Output Format – Writes final output to HDFS.

🚀 Data Locality (Important Concept):

 Hadoop sends the processing code to where the data already exists (on DataNodes),
instead of moving large data across the network.
 This reduces network usage and makes the process faster and more efficient.

📌 Summary:

Task What it does

Map Task Reads and processes data to create key-value pairs
Reduce Task Groups and summarizes map output to produce final results
Data Locality Processing happens where the data is stored to save time and bandwidth

8.2 MAPPER

What is a Mapper?

In MapReduce, a Mapper is like a worker that reads data and converts it into a different
format (called intermediate key-value pairs). These are temporary results that are used later by
the Reducer.

Each Mapper handles a small part of the big data file.

🔄 Mapper Process - Step by Step:

1. RecordReader

 This is the first step.

 It reads raw input data (bytes) and turns it into records that Mappers can understand.
 Each record is made up of:
o A Key (usually position or line number)
o A Value (actual content like a line of text)

🎯 Think of it like breaking a big file into small lines and numbering each line.

AI&ML VVIT Page 65

Big Data Analytics

2. Map Function

 This is the main logic of the Mapper.

 It takes each key-value pair from the RecordReader and processes it.
 It produces intermediate key-value pairs, depending on the logic (e.g., word count).

📌 Example: For a line “hello world”, the map function might output:

arduino
CopyEdit
("hello", 1), ("world", 1)

3. Combiner (Optional, but useful)

 Acts like a mini-reducer.

 It summarizes data locally, right after mapping.
 This helps reduce the amount of data that needs to be transferred across the network.

📌 Example: If the Mapper has many ("hello", 1) pairs, the Combiner may convert them into
("hello", 3) before sending.

4. Partitioner

 It decides which Reducer will handle which key.

 Keys that are the same (like all "hello" keys) go to the same Reducer.
 The Mapper’s output is split and stored locally until the Reducers pull the data.

🎯 This ensures that all related data is processed together in one place.

📌 Summary Table:

Component What It Does

RecordReader Converts input file to key-value records

Map Function Applies logic and generates intermediate key-value pairs

Combiner Optional: Reduces data size by combining values locally

Partitioner Sends each key to the correct Reducer based on logic

AI&ML VVIT Page 66

Big Data Analytics

8.3 REDUCER

MapReduce processes text data in steps to count how many times each word appears (Word
Count problem).

🔄 Step-by-Step Explanation with Image:

🧩 1. Input Split

 Big data is broken into chunks (called Input Splits).

 Each chunk is processed separately by a Mapper.

2. Mapper

 Each Mapper processes one Input Split.

AI&ML VVIT Page 67

Big Data Analytics

 It reads lines of text and outputs key-value pairs like:

o ("X", 1), ("Y", 1) → if the words “X” and “Y” appear in the input.

📦 Output example:

 Mapper 1: X 1, Y 1
 Mapper 2: Z 4, Z 5
 Mapper 3: X 6, Z 3
 Mapper 4: Y 6, Z 3

🔁 3. Combiner (Local Reducer)

 This is optional but helps reduce data early.

 It adds up repeated keys within the same Mapper.
 Example: Z 4 and Z 5 → becomes Z 9

📦 Output after Combiner:

 Mapper 2: Z 9 (4 + 5)
 Mapper 3: X 6, Z 3
 Mapper 4: Y 6, Z 3

🧩 4. Partitioner

 The Partitioner sends the same key to the same Reducer.

 For example:
o All “X” go to Reducer 1,
o All “Y” go to Reducer 2,
o All “Z” go to Reducer 3.

📦 Partitioned Data:

 X → Reducer 1
 Y → Reducer 2
 Z → Reducer 3

🔃 5. Shuffle and Sort

 This step brings all data for a key together at one Reducer.

AI&ML VVIT Page 68

Big Data Analytics

 It also sorts the keys for easy processing.

🧮 6. Reduce

 Reducers process one key at a time.

 They add up the values for each word:
o X: 1 + 6 = 7
o Y: 1 + 6 = 7
o Z: 9 + 3 + 3 = 15

📄 7. Output Format

 Final results are written to a file.

 Each line contains:

vbnet
CopyEdit
key value

📦 Output:

nginx
CopyEdit
X 7
Y 7
Z 15

✅ Summary Table:

Component Role

Mapper Converts input into key-value pairs

Combiner Adds values locally (optional)

Partitioner Sends same keys to same Reducer

Shuffle & Sort Gathers and sorts data by key

Reducer Combines all values for a key

Output Format Writes final results to file

AI&ML VVIT Page 69

Big Data Analytics

Component Role

8.4 COMBINER
What is a Combiner in MapReduce?

A Combiner is an optional component used in MapReduce to optimize performance. It helps

to reduce the amount of data transferred from the Mapper to the Reducer.

Think of it as a mini-reducer that works locally on each Mapper’s output before sending it over
the network.

🔄 Combiner vs Reducer – What’s the Difference?

Combiner Reducer

Works locally after each Mapper Works globally on the output of all Mappers

Output is intermediate data Output is final result written to HDFS

Reduces network traffic Produces final answer

Usually same logic as Reducer Final aggregation logic

Image 1: Writing and Running a MapReduce Word Count Program

🧭 Objective:

The goal is to write a MapReduce program that:

 Counts similar words in a file.

 Uses a Combiner to reduce the amount of data passed to the Reducer.

📂 Input Data:

The input file contains the following lines:

AI&ML VVIT Page 70

Big Data Analytics

pgsql
CopyEdit
Welcome to Hadoop Session
Introduction to Hadoop
Introducing Hive
Hive Session
Pig Session

💻 Act (What to Do in Code):

You set the Combiner class in your driver program:

java
CopyEdit
[Link]([Link]);

This tells Hadoop to use the Reducer logic as a Combiner to sum values locally on each
Mapper output before sending them to the Reducer.

You also set:

java
CopyEdit
[Link](job, new Path("/mapreducedemos/[Link]"));
[Link](job, new
Path("/mapreducedemos/output/wordcount/"));

This means:

 Hadoop will read data from /mapreducedemos/[Link]

 Hadoop will write the output to /mapreducedemos/output/wordcount/

▶️How to Run the Program:

Run this command:

bash
CopyEdit
hadoop jar <jar name> <driver class> <input path> <output path>

Example:

bash
CopyEdit
hadoop jar [Link]

AI&ML VVIT Page 71

Big Data Analytics

This runs the Word Count program using the JAR file.

🔹 Image 2: Output of the Program

📂 Directory Output:

The final output of the Reducer is stored by default in a file called:

CopyEdit
part-r-00000

Path:

swift
CopyEdit
/mapreducedemos/output/wordcount/part-r-00000

📝 Reducer Output Contents:

Here are the word counts generated by the MapReduce program:

Word Count

Hadoop 2

Hive 2

Introduction 1

Introducing 1

Pig 1

Session 3

to 2

Welcome 1

These results show how often each word appeared in the input file.

AI&ML VVIT Page 72

Big Data Analytics

✅ Summary of the Two Images:

Section What It Shows

Image 1 How to write the MapReduce program, use a combiner, and run the job

Image 2 Where output is stored and the final word count results

8.5 PARTITIONER
What is a Partitioner in MapReduce?

A Partitioner decides which Reducer should receive which set of data after the Mapper
finishes.

🔄 When Does It Work?

 The Partitioner works after the Map phase and before the Reduce phase.

⚙️What Does It Do?

 The Mapper creates key-value pairs.

 The Partitioner checks the key and decides which Reducer should handle that key.
 This ensures that keys with the same value go to the same Reducer, so the Reduce task
can process related data together.

🧮 Example:

Let’s say there are 3 Reducers, and the Mapper generates these keys:

arduino
CopyEdit
("apple", 1), ("banana", 1), ("apple", 1), ("cherry", 1)

The Partitioner might assign:

 All "apple" → Reducer 1

 All "banana" → Reducer 2

AI&ML VVIT Page 73

Big Data Analytics

 All "cherry" → Reducer 3

🔐 Default Partitioner:

 Hash Partitioner is used by default.

 It assigns reducers using a formula like:

scss
CopyEdit
hash(key) % number_of_reducers

 This helps distribute the data evenly across Reducers.

✅ Summary:

Term Meaning

Partitioner Assigns map output keys to reducers

When After Map phase, before Reduce phase

Why Ensures correct and efficient grouping of keys

Default HashPartitioner (based on key hash values)

First Image: Java Code ([Link])

This is a custom Partitioner class used in a MapReduce program in Hadoop. Its job is to
divide the data into different parts (called partitions) based on the first letter of each word.

What the code does:

1. It takes a word as a key.

2. Converts the first letter of the word to uppercase.
3. Based on that letter (A to Z), it assigns a partition number.
o For example:
 Words starting with A go to partition 1.
 Words starting with B go to partition 2.
 ...
 Words starting with Z go to partition 26.

AI&ML VVIT Page 74

Big Data Analytics

4. If the letter doesn't match A-Z, it assigns partition 0 by default.

📄 Second Image: MapReduce Task Description

This gives the task you are supposed to do:

✅ Objective:

 Write a MapReduce program to count how many times each word appears in a file.
 Use a partitioner (like the one in the first image) to group words based on their starting
alphabet.

📚 Note:

 You should refer to Chapter 5 of your Hadoop textbook to learn how to create the
Mapper, Reducer, and Driver classes.

📝 Input Data Example:

pgsql
CopyEdit
Welcome to Hadoop Session
Introduction to Hadoop
Introducing Hive
Hive Session
Pig Session

Summary:

You're writing a MapReduce job that:

 Counts word occurrences.

 Uses a custom partitioner to group words starting with the same letter.
 Processes text like the example input shown.

What is This About?

This page explains how to run your MapReduce program using the custom partitioner
([Link]) you saw earlier. It shows how the program divides the output
into 27 partitions — one for each letter (A to Z) and one default (0).
AI&ML VVIT Page 75
Big Data Analytics

🧾 Key Steps Explained Simply:

1. 🧑‍💻 Setting Number of Reducers & Partitioner:

In your driver program, you must set:

java
CopyEdit
[Link](27); // 26 letters + 1 default
[Link]([Link]);

 This tells Hadoop: "Use 27 reducers and divide words based on their first letter using our
custom partitioner."

2. 📂 Input and Output Paths:

Where your data comes from and where the result will be saved:

java
CopyEdit
[Link](job, new Path("/mapreducedemos/[Link]"));
[Link](job, new
Path("/mapreducedemos/output/wordcountpartitioner/"));

 Input File: /mapreducedemos/[Link]

 Output Folder: /mapreducedemos/output/wordcountpartitioner/

🧾 Output:

The result is stored in 27 files, named like:

 part-r-00000
 part-r-00001
 ...
 part-r-00026

Each file stores words starting with a specific alphabet letter. For example:

🟩 part-r-00008 → Stores words that start with the letter H

🔗 File path: /mapreducedemos/output/wordcountpartitioner/part-r-00008

So, if the word is Hadoop or Hive, it will be in this file.

AI&ML VVIT Page 76

Big Data Analytics

📸 What’s in the Screenshot (Table):

 It shows all the 27 output files, their sizes, modification time, permissions, and owners.
 It’s basically the file listing of the Hadoop output directory.
 Each file name corresponds to a specific letter based on your partitioner logic.

📝 Summary:

 Your MapReduce program splits the output based on the first letter of each word.
 It creates 27 output files, one for each letter + one default.
 The partitioner ensures that all words starting with the same letter go to the same reducer.
 This makes word counting organized and easy to manage.

8.6 SEARCHING
Goal (Objective)

You are writing a MapReduce program to search for a specific word (keyword) inside a file
and display where it was found.

📄 Input File: [Link]

Example data:

yaml
CopyEdit
1001,John,45
1002,Jack,39
1003,Alex,44
1004,Smith,38
1005,Bob,33

Each line has:

 Roll number
 Name
 Marks

AI&ML VVIT Page 77

Big Data Analytics

📁 File: [Link] (The Driver Program)

This is the main program that:

1. Sets up everything (Job, Mapper, Reducer).

2. Specifies input and output paths.
3. Sets the search keyword to "Jack".
4. Starts the MapReduce job.

🔧 Key Parts:
java
CopyEdit
[Link]().set("keyword", "Jack");

 This line tells the program: "Search for the word Jack."

🧠 File: [Link] (Mapper)

This file does most of the searching.

What it does:

1. Reads the input line by line.

2. Gets the file name and line content.
3. Checks if the keyword (e.g., Jack) is in the line.
4. If yes, it emits:
o Key → File name and keyword location
o Value → Position (where keyword appears in the line)

Output Example from Mapper:

makefile
CopyEdit
Key: [Link],Jack
Value: 5

It means the word "Jack" is in file [Link] and starts at position 5 in that line.

AI&ML VVIT Page 78

Big Data Analytics

🔄 File: [Link] (Reducer)

The reducer is very simple here. It just:

 Takes the output from the Mapper

 Writes it directly to the final result

📦 Output File: /mapreducedemo/output/search/part-r-00000

The final output:

CopyEdit
[Link],Jack 5

This means:

 The word "Jack" was found in the file [Link]

 It starts at position 5 in the line

✅ Summary in Simple Steps:

Step What Happens?

1️⃣ The user searches for a word (like "Jack")

2️⃣ Mapper checks each line for that word

3️⃣ If found, it outputs file name and position

4️⃣ Reducer just saves that result

5️⃣ Output shows where the keyword was found

8.7 SORTING – What is the Goal?

🎯 Objective:

Write a MapReduce program to sort student data by student name (not by roll number or
marks).

AI&ML VVIT Page 79

Big Data Analytics

📝 Input Data:

From the file [Link]:

yaml
CopyEdit
1001,John,45
1002,Jack,39
1003,Alex,44
1004,Smith,38
1005,Bob,33

Each line represents:

 Roll number, Name, Marks

💡 What We Want:

Sort all lines by the student's name:

 Alex
 Bob
 Jack
 John
 Smith

🧑‍💻 Code Explanation (Sorting in MapReduce):

📁 Java File: SortStdNames

This file contains the Mapper, Reducer, and Main (Driver) code.

🔹 1. Mapper (SortMapper class)

What it does:

 Reads each line from the input.

 Splits the line by commas.
 Uses the student name (token[1]) as the key.
 Uses the entire line as the value.

AI&ML VVIT Page 80

Big Data Analytics

java
CopyEdit
[Link](new Text(token[1]), new Text(token[0] + "," + token[1] + "," +
token[2]));

So for the line: 1002,Jack,39

It will emit:
Key: Jack
Value: 1002,Jack,39

🔸 2. Reducer (SortReducer class)

What it does:

 Receives data already sorted by student name (because Hadoop sorts keys before
sending them to reducer).
 Just writes it to the output.

java
CopyEdit
[Link]([Link](), details);

⚙️3. Main Method (Driver Code)

 Sets up the Hadoop job

 Specifies which classes to use (Mapper, Reducer, etc.)
 Input path: /mapreducedemo/[Link]
 Output path: /mapreducedemo/output/sorted

java
CopyEdit
[Link](job, new Path("/mapreducedemo/[Link]"));
[Link](job, new Path("/mapreducedemo/output/sorted"));

🧾 Output (Final Sorted Result)

After running the job, the sorted result looks like this:

yaml
CopyEdit
1003,Alex,44
1005,Bob,33
1002,Jack,39
1001,John,45
1004,Smith,38

AI&ML VVIT Page 81

Big Data Analytics

As you can see, the rows are now sorted alphabetically by student name.

✅ Summary in Simple Words:

Step What Happens

1️⃣ Read student records from file

2️⃣ Mapper emits student name as key (for sorting)

3️⃣ Hadoop automatically sorts keys (names)

4️⃣ Reducer outputs sorted records

5️⃣ Final output has students sorted by name

AI&ML VVIT Page 82