Module 5 Questions

1. Discuss the architecture of Apache Hadoop in detail. Explain the roles of the NameNode,
DataNode, and the YARN framework in managing IoT big data.

2. Explain the Hadoop MapReduce programming model. Illustrate the Map and Reduce phases
and describe how it is utilized for Batch Data Analytics in IoT systems.

3. Define Apache Storm and explain its core components: Spouts, Bolts, and Topologies. How
does Storm facilitate real-time data analysis compared to traditional batch processing?

4. Describe the architecture of Apache Spark. Highlight the significance of RDDs (Resilient
Distributed Datasets) and explain why Spark is often preferred over Hadoop MapReduce for
iterative IoT data tasks.

5. Explain the role of Apache Oozie as a workflow scheduler in the Hadoop ecosystem. Describe
how it manages complex dependencies and job execution in an IoT analytics pipeline.

1. Architecture of Apache Hadoop

 Apache Hadoop is an open-source framework for distributed batch processing of big data.

 It uses the MapReduce programming model which parallelizes computations across large
clusters of commodity servers.

 Hadoop comprises two main components:

1. HDFS (Hadoop Distributed File System) – for distributed storage.

2. MapReduce/YARN – for distributed data processing and resource management.

🔹 NameNode

 The NameNode keeps the directory tree of all files in the file system and tracks where across
the cluster the file data is kept.

 It does not store the data itself, only metadata.

 Client applications talk to the NameNode whenever they wish to locate a file, or when they
want to add/copy/move/delete a file.

 The NameNode responds with a list of relevant DataNode servers where the data resides.

 Acts as both directory namespace manager and inode table for HDFS.

 There is a single NameNode in any DFS deployment.

4. Secondary NameNode

 Takes periodic snapshots of HDFS metadata

 Helps reduce NameNode restart time

 In case of NameNode failure, can be manually configured to bring up the cluster

. NameNode (Master Node)

 Stores metadata only, such as:

o File names

o Directory structure

o Block locations

o Permissions

 Manages file-to-block mapping

o Example: [Link] → Block A, Block B, Block C

 Assigns blocks to different DataNodes

 Does not store actual data

🔹 DataNode

 A DataNode stores actual data blocks in HDFS.

 A functional HDFS has multiple DataNodes, with data replicated across them for fault
tolerance.

 DataNodes connect to the NameNode on startup and respond to requests for filesystem
operations.
  Client applications can talk directly to a DataNode once the NameNode provides the location
of the data.

 MapReduce operations are assigned to TaskTracker instances near a DataNode, so

computation happens close to the data (data locality).

YARN (Yet Another Resource Negotiator)

 Introduced in Hadoop 2.x to overcome limitations of the traditional model.

 Separates resource management from job scheduling.

 ResourceManager:

o Allocates cluster resources.

 NodeManager:

o Manages resources on individual nodes.

 ApplicationMaster:

o Manages execution of a single application/job.

 Supports multiple processing models (MapReduce, Spark, Tez, etc.).

✅ Advantages of YARN over Traditional Hadoop

1. Scalability

o Handles thousands of nodes efficiently.

o Eliminates JobTracker bottleneck.

2. Flexibility

o Supports diverse frameworks (MapReduce, Spark, Tez, Flink).

o Not limited to MapReduce.

3. Resource Utilization

o Dynamic resource allocation.

o Better cluster efficiency.

4. Fault Tolerance

o No single point of failure like JobTracker.

o ApplicationMaster failures can be restarted independently.

5. Multi-tenancy

o Allows multiple applications to run simultaneously on the same cluster.

Hadoop MapReduce Programming Model

🔹 Definition

 MapReduce is a programming model in Hadoop used for parallel processing of large datasets
across distributed clusters.

 It divides tasks into two phases: Map and Reduce.

 Ideal for batch data analytics where massive IoT data streams are processed in chunks.

🔹 Map Phase

 Input data is split into smaller chunks.

 Each chunk is processed by a Mapper function.

 Mapper takes input as (key, value) pairs and produces intermediate (key, value) pairs.

 Example:

o Input: Sensor logs → (SensorID, Temperature)

o Mapper Output: (SensorID, 1) for counting occurrences.

🔹 Shuffle & Sort (Intermediate Step)

 The framework automatically groups and sorts intermediate key-value pairs.

 Ensures that all values for the same key are sent to the same Reducer.

🔹 Reduce Phase

 Reducer takes grouped intermediate results.

 Performs aggregation, summarization, or computation.

 Example:

o Input: (SensorID, [1,1,1,1])

o Reducer Output: (SensorID, 4) → total count of readings.

🔹 Utilization in Batch Data Analytics for IoT

[Link] Data Aggregation

o Collects millions of IoT sensor readings.

o Map phase → extracts relevant values.

o Reduce phase → aggregates (e.g., average temperature, total energy usage).

2. Log Analysis

o IoT devices generate logs continuously.

o MapReduce processes logs in batches to detect anomalies or usage patterns.

3. Scalability
 o Handles petabytes of IoT data by distributing tasks across clusters.

o Ensures fault tolerance via replication.

4. Batch Orientation

o Suitable for offline analytics (e.g., daily/weekly IoT reports).

o Not real-time, but highly efficient for large-scale historical data analysis.

. Architecture of Apache Spark

🔹 Overview

 Apache Spark is an open-source, distributed computing framework designed for fast,

in-memory data processing.

 It supports batch, streaming, interactive, and iterative workloads, making it highly suitable
for IoT big data analytics.

 Spark provides APIs in Python, Java, Scala, and R, and integrates with Hadoop ecosystem
components (HDFS, YARN, Hive).

🔹 Core Components of Spark Architecture

1. Driver Program

o The main program that defines the Spark application.

o Responsible for creating the SparkContext which coordinates tasks.

2. Cluster Manager
 o Allocates resources across applications.

o Examples: YARN, Mesos, or Spark’s built-in standalone manager.

3. Workers (Executors)

o Run on cluster nodes.

o Execute tasks assigned by the driver.

o Store data in memory for faster access.

4. Tasks

o Smallest unit of execution.

o Distributed across executors for parallel processing.

🔹 Significance of RDDs (Resilient Distributed Datasets)

 RDDs are the fundamental data structure in Spark.

 They are immutable, distributed collections of objects that can be processed in parallel.

 Key features:

1.

2.

3. In-Memory Computation → Data can be cached in memory for repeated use.

4.

Resilient Distributed Datasets (RDDs) are the fundamental data structure of

Apache Spark, designed to enable fast, fault-tolerant, parallel processing of
large datasets across clusters.

🔹 Key Characteristics of RDDs

 Immutable → Once created, RDDs cannot be changed. Any

transformation produces a new RDD.

 Distributed → Data is partitioned across multiple nodes in the

cluster for parallel computation.

 Resilient (Fault-Tolerant) → If a node fails, RDDs can be recomputed

using lineage information.

 Lazy Evaluation → Transformations are not executed immediately;

they are only computed when an action (like collect, count, save) is
called.

 Parallel Operations → Supports operations like map, filter, reduce,

join, etc., across partitions simultaneously.
🔹 Why Spark is Preferred over Hadoop MapReduce for Iterative IoT Data Tasks

1. Speed

o Spark performs in-memory computation, making it up to 100x faster than Hadoop

MapReduce for iterative tasks.

2. Iterative Processing

o IoT analytics often require repeated operations (e.g., machine learning, graph analysis).

o Spark caches RDDs in memory, avoiding repeated disk reads.

3. Unified Framework

o Supports batch, streaming, SQL queries, and machine learning in a single platform.

4. Ease of Use

o Provides high-level APIs and libraries (Spark SQL, MLlib, GraphX, Spark Streaming).

5. Flexibility

o Works with multiple data sources (HDFS, Cassandra, Kafka, Amazon S3).

Apache Storm

🔹 Definition

 Apache Storm is an open-source, distributed, real-time computation system.

 It processes unbounded streams of data reliably and with low latency.

 Storm is designed for real-time analytics, continuous computation, and event processing,
making it highly suitable for IoT applications where sensor data arrives continuously.
🔹 Core Components

1. Spouts

 Spouts are the data sources in a Storm topology.

 They read data from external sources such as Kafka, message queues, or IoT sensors.

 Emit data as streams of tuples into the topology.

 Example: A spout connected to IoT sensors emits (SensorID, Value, Timestamp) tuples.

2. Bolts

 Bolts are the processing units in Storm.

 They consume tuples from spouts or other bolts, perform computations, and emit new tuples.

 Functions of bolts include:

o Filtering

o Aggregation

o Joining streams

o Writing results to databases or dashboards

 Example: A bolt aggregates sensor values and computes average temperature per minute.

3. Topologies

 A topology is the network of spouts and bolts that defines the data flow in Storm.

 Unlike MapReduce jobs (which run and finish), Storm topologies run continuously until
terminated.

 Topologies are represented as directed acyclic graphs (DAGs) where:

o Nodes = Spouts/Bolts

o Edges = Data streams

🔹 Storm vs. Traditional Batch Processing

 Batch Processing (Hadoop MapReduce)

o Processes large datasets in chunks.

o Suitable for offline analytics (e.g., daily IoT reports).

o High latency due to disk I/O and batch scheduling.

 Real-Time Processing (Apache Storm)

o Processes data as soon as it arrives.

o Provides low latency and near real-time insights.

o Ideal for IoT scenarios like anomaly detection, fraud detection, and live dashboards.

o Continuous computation model ensures uninterrupted stream analysis.