What is a DataNode?

 DataNode is a worker (slave) node in HDFS.

 It is responsible for storing the actual data in the form of blocks.
 Every DataNode runs on a separate machine.
 A Hadoop cluster usually contains many DataNodes and one
NameNode.

If HDFS is a library,

 NameNode = Librarian (knows where books are)

 DataNode = Bookshelves (stores the books)

Main Responsibilities of a DataNode

1. Storing Actual Data Blocks

 The primary responsibility of a DataNode is to store the actual
data in HDFS.
 When a file is uploaded into HDFS, it is not stored as a single file.
 Instead, the file is split into fixed-size blocks (by default, 128 MB
each). These blocks are then distributed across multiple DataNodes.
 Each DataNode stores these blocks on its local file system (Linux
file system).
 A single DataNode can store blocks from many different files,
and one file’s blocks are usually spread across multiple
DataNodes.

This design is very important because:

 It allows parallel processing of data

  It avoids dependency on a single machine
 It enables horizontal scalability

Example:
A 1 GB file → split into 8 blocks → blocks stored on different DataNodes
This makes large-scale data processing possible.

2. Handling Client Read Requests

When a user or application wants to read a file from HDFS, the

DataNode plays a key role in data delivery.

The process works like this:

 The client first contacts the NameNode to ask where the blocks of
the file are located.
 The NameNode responds with the addresses of DataNodes that
contain those blocks.
 The client then directly contacts the DataNode to read the data.

The DataNode:

 Reads the requested block from its local disk

 Streams the data back to the client
 Sends data in a continuous and efficient manner

This approach is efficient because:

 The NameNode is not overloaded

 Data is read from the nearest DataNode
 High throughput is achieved for big data workloads

3. Handling Client Write Requests

DataNodes are also responsible for writing data into HDFS.

When a client writes a file:

 The client requests permission from the NameNode

 The NameNode selects a set of DataNodes for block storage
 The client sends data to the first DataNode
 The data flows through a pipeline of DataNodes

Each DataNode in the pipeline:

 Receives a block of data

 Writes it to its local disk
 Forwards the data to the next DataNode (for replication)
 Sends an acknowledgment (ACK) once writing is successful
This pipeline mechanism ensures:

 Reliable data storage

 Efficient replication
 High write performance

Even if one DataNode fails during writing, HDFS automatically adjusts.

4. Managing Data Replication

Replication is one of the most important responsibilities of a

DataNode.

In HDFS:

 Each block is stored on multiple DataNodes

 Default replication factor = 3

The DataNode:

 Stores its own replica of a block

 Transfers replicas to other DataNodes
 Deletes replicas if instructed by the NameNode

If a DataNode fails:

 Data remains available from other replicas

 The NameNode orders remaining DataNodes to create new
replicas

This mechanism ensures:

 Fault tolerance
 High availability
 No data loss even if machines crash

5. Sending Heartbeats to the NameNode

To ensure the system is healthy, every DataNode sends a heartbeat

message to the NameNode at regular intervals (usually every 3
seconds).

A heartbeat message tells the NameNode:

 The DataNode is alive

 The DataNode is functioning correctly
 The DataNode has enough resources

If the NameNode does not receive heartbeats for a certain time:

  The DataNode is marked as dead
 All its blocks are considered unavailable
 Replication is triggered automatically

This responsibility is critical for:

 Failure detection
 Cluster health monitoring
 Automatic recovery

6. Sending Block Reports

In addition to heartbeats, a DataNode periodically sends a block report

to the NameNode.

A block report contains:

 Complete list of blocks stored on that DataNode

 Block IDs and metadata information

This helps the NameNode:

 Track exact block locations

 Ensure replication factor is maintained
 Detect missing or extra blocks

Block reports are usually sent:

 When a DataNode starts

 Periodically (e.g., once per hour)

Without block reports, the NameNode cannot manage HDFS efficiently.

How Data Is Stored in HDFS

1. File Is Divided into Large Blocks

HDFS divides a large file into fixed-size blocks (default 128 MB).

This approach provides:

Reduced Metadata Overhead

 Because blocks are large, the number of blocks created is small.

 This reduces the amount of metadata that the NameNode must
store, improving memory efficiency and system scalability.

Efficient Sequential Processing

Large blocks allow Hadoop jobs to read data sequentially instead of
randomly, which matches the design of big data analytics and improves
processing speed.

Fewer Disk Seeks

Large block sizes reduce frequent disk head movements, resulting in

faster read and write operations.

2. Blocks Are Distributed Across Multiple DataNodes

After splitting, blocks are stored on different DataNodes.

This distribution allows:

Parallel Data Access

Multiple DataNodes can serve different blocks of the same file at the same
time. This allows Hadoop to process large datasets faster using parallel
computation.

Load Balancing

Blocks are spread across many DataNodes so that no single node is

overloaded. This ensures balanced use of storage and processing
resources.

Better Performance

Since data is accessed from multiple nodes simultaneously, overall system

throughput increases and job execution time decreases.

3. Replication of Each Block

Each block is stored on multiple DataNodes (default replication factor =

3).

Replication ensures:

Fault Tolerance

If one DataNode fails, data can still be accessed from another replica,
preventing data loss.

High Availability

Multiple replicas ensure that data remains accessible even during

hardware or network failures.
Reliability

Replication guarantees consistent data access and system stability in

large clusters.

4. Metadata Is Stored Separately from Data

HDFS stores metadata on the NameNode, not on DataNodes.

This separation enables:

Faster Metadata Access

Since metadata is centralized, the NameNode can quickly locate blocks

and respond to client requests.

Simplified Data Management

Keeping metadata separate makes block tracking, replication control, and

failure handling easier.

Reduced DataNode Complexity

DataNodes focus only on storage and data transfer, improving efficiency.

[Link]-Once, Read-Many Storage Model

HDFS allows data to be written once and read multiple times.

This model provides:

Data Consistency

Since files are not modified after writing, consistency issues are avoided.

Simplified Architecture

No complex locking or update mechanisms are needed, making HDFS

simple and robust.

Optimized Analytics Processing

Batch processing jobs benefit from stable, unchanging datasets.

Fault Tolerance & Replication in HDFS

Replication Factor (Default = 3)

What it means

Replication factor indicates how many copies of each data block HDFS
stores. By default, HDFS keeps three copies of every block on different
DataNodes.

Why it is needed

In big data clusters, hardware failures are common. If data exists on only
one machine, failure of that machine would cause data loss. Replication
prevents this by keeping multiple copies.

How it helps

Even if one or two DataNodes fail, the remaining replica can still serve
data. This ensures data safety and continuous access.

Example

A 128 MB block is stored on:

 DataNode 1
 DataNode 2
 DataNode 3
If DataNode 1 fails, data is still available from DataNode 2 or 3.

2 DataNode Failure Handling

What happens during failure

When a DataNode crashes or becomes unreachable, HDFS does not stop

working. The system is designed to expect such failures.

How HDFS reacts

Since replicas of data exist on other DataNodes, HDFS continues to serve

data from healthy nodes without user interruption.

Why this is important

Applications continue running smoothly even when machines fail. Users

do not need to manually recover data.

Example

If a DataNode storing one replica goes down, Hadoop jobs still read data
from other replicas.

3. Heartbeat-Based Failure Detection

What is a heartbeat

A heartbeat is a small signal sent by every DataNode to the NameNode

at regular intervals (usually every 3 seconds).

Purpose of heartbeat

It tells the NameNode:

“I am alive and working properly.”

What if heartbeat stops

If the NameNode does not receive heartbeats for a certain time, it marks
that DataNode as dead.

Example

If DataNode 5 stops sending heartbeats, the NameNode immediately

starts recovery actions.
4. Automatic Re-Replication

Why re-replication is needed

When a DataNode fails, the number of replicas for some blocks reduces.
This makes data less safe.

How HDFS handles it

The NameNode automatically selects other healthy DataNodes and orders

them to create new replicas.

Key advantage

This process is automatic and happens in the background without user

involvement.

Example

If replication factor is 3 and one replica is lost, HDFS creates a new third
replica elsewhere.

[Link] Awareness

What is rack awareness

Rack awareness means HDFS understands how DataNodes are physically

arranged in racks in a data center.

Why it is important

If all replicas were stored in one rack, failure of that rack would cause data
loss.

How HDFS places replicas

HDFS stores replicas across different racks to protect against rack-level

failures.

Example

One replica in Rack 1, one in Rack 1, and one in Rack 2 ensures higher
safety.

6. High Availability of Data

Meaning of high availability

High availability means data is always accessible, even when failures

occur.

How HDFS achieves this

Because multiple replicas exist, clients can read data from any available
DataNode.

Automatic selection

If one DataNode is slow or unavailable, HDFS automatically reads from

another replica.

Example

A client reads data from the nearest available replica for faster access.

7. Reliable Write Operations

How data is written

When data is written, it flows through a pipeline of DataNodes. Each

DataNode writes its replica and forwards data to the next.

Confirmation mechanism

Each DataNode sends an acknowledgment after successful write.

Failure during write

If a DataNode fails during writing, the pipeline is reconfigured

automatically.

Result

Data is safely stored on multiple nodes before the write is considered

complete.

What is MapReduce?

 MapReduce is a programming model used to process very large

datasets in a distributed and parallel manner using Hadoop.
 It is designed to handle data that is too large to be processed
efficiently on a single machine.
 It divides a big job into smaller tasks
  In MapReduce, a large data processing job is broken down into
many smaller tasks.
 Instead of processing the entire dataset at once, the input data is
split into smaller chunks, and each chunk is handled separately.
 These smaller tasks are easier to manage, faster to process, and
can be executed independently.
 This division of work is what makes MapReduce suitable for handling
massive datasets.

Example:

A 1 TB log file is divided into hundreds of smaller parts so that each part
can be processed separately.

Processes them on multiple machines

 Once the job is divided into smaller tasks, MapReduce runs these
tasks simultaneously on multiple machines in the Hadoop cluster.
 Each machine processes a different part of the data at the same
time, instead of waiting for one machine to finish.
 This parallel execution significantly reduces processing time and
improves efficiency.
 Hadoop automatically manages task assignment and execution
across machines.

Example:

If 10 machines are available, 10 parts of the data are processed at the

same time instead of one after another.

Combines the results to produce the final output

 After all machines finish processing their individual tasks,

MapReduce collects and combines the partial results.
 This is done in the Reduce phase, where results from different
machines are aggregated to form the final output.
 The user receives a single meaningful result, even though the data
was processed in parts.
  This final combination ensures correctness and completeness of the
output.

Example:

Each machine counts words in its data chunk, and then all counts are
combined to get the total word count.

Main Components of MapReduce

MapReduce processes large datasets by dividing the work into two main
phases.

Each phase has a specific role, and together they complete the entire data
processing task.

1️⃣ Map Phase

The Map phase is the first step in MapReduce processing. Its main role is
to read the input data and convert it into key–value pairs that can be
processed further.

Reading Input Data

 In the Map phase, input data is read from HDFS.

 The data is automatically divided into input splits, and each split is
assigned to one Mapper.
 This allows many Mappers to work at the same time, each handling
a small part of the data.

Example:

A large log file is divided into multiple chunks, and each chunk is
processed by a separate Mapper.

Processing Data
  Each Mapper processes its input line by line or record by record.
 The Mapper applies logic such as filtering, parsing, or transforming
the raw data.
 This step prepares the data so that it becomes meaningful for
further processing.

Example:

Extracting words from a sentence or extracting fields from a log entry.

Generating Key–Value Pairs

 After processing, the Mapper outputs data in the form of key–value

pairs.
 The key represents an identifier (such as a word or ID), and the
value represents related information (such as count or occurrence).
 These key–value pairs are called intermediate output.

Example (Word Count):

Input line: "Big data is powerful"

Mapper output:

(Big, 1)

(data, 1)

(is, 1)

(powerful, 1)

Parallel Execution

 Many Map tasks run in parallel on different machines.

 This parallelism significantly reduces the total processing time.
 The user does not need to manage this — Hadoop handles task
scheduling automatically.

Output of Map Phase

 The output of the Map phase is not the final result.

 It is temporary data that will be grouped and processed by the
Reduce phase.
 Before reaching the Reducer, this data goes through Shuffle and
Sort (handled automatically by Hadoop).
The Shuffle and Sort phase is the middle stage of MapReduce.

It works between the Map phase and the Reduce phase and is
automatically handled by Hadoop.

Its main purpose is to group all intermediate data with the same key and
prepare it for reduction.

What is Shuffle?

 Shuffle is the process of moving intermediate data from Mappers to

Reducers.
 During this step, Hadoop transfers the key–value pairs produced by
the Map phase across the network.
 All values related to the same key are sent to the same Reducer, no
matter which Mapper produced them.
 This ensures that all data required for aggregation reaches the
correct Reducer.

Both pairs are shuffled to the same Reducer.

Why Shuffle is Important

 Shuffle ensures that data is properly grouped before reduction.

 Without shuffle, values of the same key would remain scattered
across different machines.
 This step enables correct aggregation and meaningful results.
 It also allows MapReduce to scale across large clusters.

What is Sort?
  Sort is the process of ordering keys before they are sent to the
Reducer.
 Hadoop automatically sorts keys in ascending order.
 Sorting ensures that Reducers receive data in a structured and
organized manner.
 It simplifies the Reduce logic and improves efficiency.

Grouping of Values

 After sorting, Hadoop groups all values of the same key together.
 Each Reducer receives:
 One key
 A list of all values associated with that key
 This grouping is essential for accurate computation in the Reduce
phase.

Reduce phase

The Reduce phase is the final step in MapReduce processing. Its main role
is to combine and summarize the intermediate data produced by the Map
phase.

Receiving Grouped Data

 Before the Reduce phase starts, Hadoop groups all values with the
same key together.
 Each Reducer receives one key and a list of corresponding values.
 This grouping ensures accurate aggregation.

Aggregating Data

 The Reducer applies logic such as sum, count, max, min, or average
to the grouped values.
 This step converts raw intermediate data into meaningful
information.
 The aggregation logic depends on the problem being solved.

Producing Final Output

 After aggregation, the Reducer generates the final key–value pairs.

 These results represent the completed processing of the dataset.
 The output is written back to HDFS.