What is ?

 Solution of Big Data Problem

 Hadoop is an Apache open source framework written
in java that allows distributed processing of large
datasets across clusters of computers using simple
programming models. The Hadoop framework
application works in an environment that provides
distributed storage and computation across clusters of
computers. Hadoop is designed to scale up from single
server to thousands of machines, each offering local
computation and storage
Hadoop
Why?
 Highly Economical
 High Reliability
 High Availability
 High Scalability
 High Throughput
 No Schema
 Easy to use
Modes of Hadoop
 Standalone
 Pseudo-distributed (single node cluster)
 Fully distributed (multiple node cluster)
Core components of Hadoop
 HDFS
Distribution and storage of Big Data across the cluster of
computers

 MapReduce
It is a Programming model that allows the processing of
data across the entire cluster at a very high level
Hadoop Ecosystem
Hadoop Ecosystem
Hadoop Ecosystem

Ingestion Storage Processing Analysis Management and

coordination

Apache Sqoop HDFS MapReduce Pig

(RDBMS Connector) (Structured data) (Data Processing) (Scripting) ZooKeeper
Apache Ambari
Apache Flume HBASE Yarn Hive Cloudera
(Data Collection) (Unstructured columnar (Cluster Resource (SQL query) Map R control System
store) management) Hortonworks
Oozie (Workflow
Apache Kafka MongoDB Spark Spark SQL monitoring)
(Streaming)

Apache Impala Apache Cassandra Spark GraphX

Apache Nifi Apache Mahout

(Machine Learning)

Storm Tableau
(streaming) (Visualization)

Change Data Capture Drill

(CDC) (Interactive analytics)
Hadoop Distributed File System
 Store datatypes in any format (structured, Semi-
structured, unstructured)
 Write once read many semantics
 Data processed in parallel
 Computations happened in datanodes where the data
resides
 Data is replicated across the nodes
 Highly scalable (Supports thousands of nodes in a
single cluster)
HDFS ARCHITECTURE
HDFS Storage Daemon's
 NameNode(Master)
 DataNode(Slave)
HDFS ARCHITECTURE
HDFS ARCHITECTURE
HDFS Architecture Components
1. NameNode (Master Node):
 Manages the HDFS cluster and stores metadata (file names, sizes, permissions, block
locations).
 Does not store actual data, only metadata (via edit logs and fsimage).
 Controls file operations like create, delete, and replicate, directing DataNodes
accordingly.
 Helps clients locate the nearest DataNode for fast access.
2. DataNode (Slave Node):
 Stores the actual data blocks on local disks.
 Handles read/write requests from clients.
 Sends heartbeats and block reports to the NameNode regularly.
 Data is replicated (default 3 copies) for reliability.
 More DataNodes = More storage & better performance; each should have high capacity
and stable connectivity.
File Block In HDFS
HDFS ARCHITECTURE
 Rack Awareness: A rack is a group of machines
(typically 30–40) in a Hadoop cluster. Large clusters
have many racks. Rack Awareness helps the
NameNode to:
 Choose the nearest DataNode for faster read/write
operations.
 Reduce network traffic by minimizing inter-rack data
transfer.
Map Reduce
Map Reduce
 MapReduce is a data processing model in Hadoop
built on top of the YARN framework. Its main feature
is distributed and parallel processing across a Hadoop
cluster, which significantly boosts performance when
handling Big Data. Since serial processing is inefficient
at large scale, MapReduce enables fast and efficient
data processing by splitting the workload into two
phases: Map and Reduce.
Map Reduce
Map Task Components
 RecordReader: Reads input data and converts it into key-value pairs. The key usually
represents location metadata, and the value contains the actual data.
 Mapper: A user-defined function that processes each key-value pair. It may output zero
or more new key-value pairs.
 Combiner (Optional): Acts as a local mini-reducer to summarize or group Mapper
outputs before sending them to the Reducer. It's used to reduce data transfer overhead.
 Partitioner: Determines which Reducer will receive which key-value pair. It uses the
formula: [Link]() % numberOfReducers to assign partitions.

Reduce Task Components

 Shuffle and Sort: Intermediate key-value pairs from the Mapper are transferred
(shuffled) to the Reducer and sorted by key. Shuffling starts as soon as some Mappers
finish, speeding up the process.
 Reducer: Receives grouped key-value pairs and performs operations such as aggregation
or filtering, based on business logic.
 OutputFormat: The final output is written to HDFS using a RecordWriter, where each
line represents a record with the key and value separated by a space.
References
 [Link]
architecture/
 [Link]
introduction/
 [Link]
hadoop-distributed-file-system/
 [Link]
 [Link]
system-hdfs
 [Link]
engineering/mapreduce-architecture/
 [Link]
 [Link]
m