BIG DATA ANALYTICS(BCS714D)
Module 2
Introduction to Hadoop: Introducing hadoop, Why hadoop, Why not RDBMS, RDBMS Vs
Hadoop, History of Hadoop, Hadoop overview, Use case of Hadoop, HDFS (Hadoop
Distributed File System),Processing data with Hadoop, Managing resources and
applications with Hadoop YARN(Yet Another Resource Negotiator). Introduction to Map
Reduce Programming: Introduction, Mapper, Reducer, Combiner, Partitioner,
Searching, Sorting, Compression.
1. Introduction to Hadoop
• What is Hadoop?
An open-source framework by Apache for distributed storage and processing of
large datasets using commodity hardware.
• Why Hadoop?
o Handles huge volumes of data (TBs–PBs).
o Stores structured, semi-structured, and unstructured data.
o Cost-effective (runs on clusters of cheap machines).
o Scalable & fault tolerant.
• Why not RDBMS?
o RDBMS designed for structured, small-to-medium data.
o Poor performance with unstructured or massive datasets.
o Schema-on-write (rigid schema).
o Expensive scaling (vertical).
o Hadoop: Schema-on-read, flexible, scalable (horizontal).
• RDBMS Vs Hadoop
Feature RDBMS Hadoop
Data Type Structured All types (structured, semi, unstructured)
Schema Schema-on-write Schema-on-read
Scalability Vertical Horizontal
Storage Limited Distributed (HDFS)
Processing ACID transactions Batch + real-time processing
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
Feature RDBMS Hadoop
Cost High Low (open source)
2. History of Hadoop
• Inspired by Google File System (GFS) and MapReduce papers.
• Developed by Doug Cutting and Mike Cafarella (2005).
• Initially part of Nutch search engine project.
• Later became an Apache project and evolved into today’s Hadoop ecosystem.
3. Hadoop Overview
• Core components:
o HDFS – Storage layer.
o YARN – Resource management.
o MapReduce – Processing engine.
• Ecosystem: Hive, Pig, Sqoop, Flume, Spark, Oozie, HBase, Zookeeper, etc.
4. Use Cases of Hadoop
• Social media analytics (Twitter, Facebook).
• Fraud detection in banking.
• Recommendation engines (Amazon, Netflix).
• Healthcare data analysis.
• Log processing in IT operations.
• IoT and sensor data management.
5. HDFS (Hadoop Distributed File System)
• Stores very large files across multiple machines.
• Follows Master-Slave architecture:
o NameNode (master) – manages metadata (namespace, file mapping).
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
o DataNodes (slaves) – store actual blocks of data.
• Features:
o Fault tolerance (replication of blocks).
o High throughput (parallel reads/writes).
o Write once, read many design.
1. Introduction
• HDFS = Hadoop Distributed File System.
• Designed to store very large files (GBs → PBs) reliably across clusters of commodity
machines.
• Inspired by Google File System (GFS).
• Provides high throughput, fault tolerance, and scalability.
2. Features of HDFS
• Distributed storage – Data is stored across multiple nodes.
• Block-based storage – Files are split into fixed-size blocks (default: 128 MB or 256
MB).
• Replication – Each block is replicated (default replication factor = 3) to ensure
reliability.
• Fault tolerance – If a DataNode fails, data can be retrieved from replicas.
• Scalability – Can easily add nodes to increase storage/processing capacity.
• Write-once, read-many – Simplifies consistency and improves throughput.
3. Architecture of HDFS
HDFS follows a Master-Slave Architecture:
a) NameNode (Master)
• Maintains metadata (file namespace, block mapping, replication info).
• Does not store actual data.
• Single point of control → but with High Availability (HA) configuration, can avoid
SPOF.
b) DataNodes (Slaves)
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
• Store actual blocks of data.
• Send heartbeat signals to NameNode (to confirm they are alive).
• Handle read/write requests from clients.
c) Secondary NameNode
• Not a backup of NameNode.
• Periodically merges edit logs + file system metadata (FsImage) to reduce
NameNode load.
4. HDFS Block Concept
• File is divided into blocks (default: 128 MB).
• Blocks are distributed across different DataNodes.
• Each block is replicated (3 copies) → stored on different machines for reliability.
• Replication policy:
o One replica on local node.
o Second replica on different rack.
o Third replica on same rack but different node.
5. Data Flow in HDFS
a) Write Operation
1. Client requests NameNode to write a file.
2. NameNode allocates blocks and selects DataNodes for replicas.
3. Client writes data → first DataNode → pipeline to other DataNodes.
4. After success, acknowledgment sent to NameNode.
b) Read Operation
1. Client requests NameNode for block locations.
2. NameNode returns list of DataNodes holding replicas.
3. Client reads directly from nearest DataNode.
6. Advantages of HDFS
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
• Handles massive data efficiently.
• Cost-effective (runs on commodity hardware).
• Reliable and fault tolerant.
• High throughput for batch processing.
7. Limitations of HDFS
• Not suitable for low-latency data access (not like RDBMS).
• Not efficient for small files (too much metadata load on NameNode).
• No support for concurrent writes / file updates.
• Depends heavily on NameNode (though HA reduces this risk).
8. Real-World Use Cases of HDFS
• Storing log files from web servers.
• Storing and processing clickstream data (e.g., e-commerce, social media).
• Storing IoT and sensor data.
• Large-scale scientific data processing (astronomy, genomics).
6. Processing Data with Hadoop
• Done via MapReduce programming model.
• Works on large datasets stored in HDFS.
• Breaks tasks into map tasks (parallel) and reduce tasks (aggregating results).
7. Managing Resources with YARN
• YARN = Yet Another Resource Negotiator.
• Separates resource management from processing.
• Components:
o ResourceManager (RM) – allocates cluster resources.
o NodeManager (NM) – manages resources on each node.
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
o ApplicationMaster (AM) – manages individual applications.
o Container – execution environment for tasks.
Introduction to MapReduce Programming
1. What is MapReduce?
• A programming model for parallel processing of large datasets.
• Two main functions:
o Map() → processes input and produces key-value pairs.
o Reduce() → aggregates values for the same key.
2. Components
• Mapper – Takes input, transforms into key-value pairs.
• Reducer – Merges intermediate values based on keys.
• Combiner – Mini-reducer, reduces data transfer between map and reduce phases.
• Partitioner – Decides how map output is divided among reducers.
3. MapReduce Operations
• Searching – Map filters relevant records, Reduce aggregates matches.
• Sorting – Data is sorted by keys automatically between Map & Reduce.
• Compression – Reduces storage & speeds up data transfer (e.g., Snappy, Gzip).
Question Bank
1. Explain RDBMS vs Hadoop with examples.
2. Describe the history and evolution of Hadoop.
3. With a neat diagram, explain HDFS architecture.
4. Discuss YARN architecture and its components.
5. What are the advantages of Hadoop over traditional systems?
6. Explain MapReduce programming with an example.
7. Write notes on Mapper, Reducer, Combiner, and Partitioner.
Dept Of CSE 2025-26
BIG DATA ANALYTICS(BCS714D)
8. Discuss how Hadoop is applied in real-world use cases.
9. Explain searching, sorting, and compression in MapReduce.
10. Compare traditional processing vs MapReduce-based processing.
Dept Of CSE 2025-26