1. Why Data Analytics for IoT?

 IoT systems generate huge amounts of data with high volume, velocity, and variety,
so traditional single-machine databases and tools are not enough.

 Analytics can be:

o Simple aggregation: mean, max, min, counts on different timescales.

o Machine learning: clustering (group similar data) and classification (assign to

predefined classes).

 Example: Forest fire detection system

o Many sensors’ nodes measure temperature, humidity, light, CO and send data
to the cloud.

o Cloud analytics predict whether a fire has occurred, using both real-time
analysis and batch aggregation to build prediction models.

2. MapReduce Programming Model

 MapReduce is a parallel programming model for large-scale data analysis.

 Data is represented as key–value pairs in both Map and Reduce phases.

Phases:

 Map phase:

o Input data split and sent to multiple mappers (on different nodes).

o Each mapper processes its part independently and produces intermediate

key–value pairs.

 Shuffle & Sort:

o Intermediate pairs are grouped by key and transferred to reducers.

 Reduce phase:

o Reducers aggregate values for each key (sum, mean, count, etc.) and write
final output to distributed storage.

Key idea: Move computation to where data resides (data locality) to reduce network transfer
and improve efficiency.

3. Hadoop Architecture (Hadoop 1.x)

Hadoop provides a distributed file system (HDFS) and a MapReduce execution framework
over a cluster.

Main components

 NameNode:

o Master for HDFS.

o Stores metadata: directory tree, file to block mapping, and locations of blocks
on DataNodes.

 Secondary NameNode:

o Not a hot standby.

o Periodically creates checkpoints of NameNode metadata to help recovery.

 DataNode:

o Stores actual data blocks.

o Sends heartbeats and block reports to NameNode; serves read/write

requests from clients and MapReduce tasks.

 JobTracker:

o Master for MapReduce.

o Receives job submissions from clients, talks to NameNode to find data

locations, schedules map/reduce tasks on TaskTrackers, tracks progress,
handles failures.

 TaskTracker:

o Runs on slave nodes.

o Has a configured number of map/reduce slots.

o Executes tasks in separate JVMs, sends heartbeats to JobTracker with

available slots and status.

Job execution workflow

1. Client submits job to JobTracker and gets a JobID.

2. JobTracker queries NameNode for data locations.

3. JobTracker assigns map tasks to TaskTrackers with data locality preference.

4. TaskTrackers run tasks, store intermediate results locally, and send heartbeats.
 5. After maps finish, reduce tasks are scheduled; reducers fetch intermediate data,
aggregate, and write output.

6. JobTracker updates final job status; client can poll status pages through web UI.

Hadoop cluster setup (high level)

 Install Java, download and unpack Hadoop on all nodes.

 Configure networking: hostnames (master, slave1, slave2, …), password-less SSH

using key pairs and authorized_keys.

 Configure key files ([Link], [Link], [Link], masters, slaves)

with NameNode URI, replication factor, JobTracker, and node roles.

 Format NameNode, start HDFS daemons and MapReduce daemons, then use web
UIs:

o NameNode: default port 50070.

o JobTracker: default port 50030.

4. Batch Analytics with Hadoop MapReduce (IoT Example)

 Use case: Forest fire detection batch analysis:

o Raw sensor readings (timestamp, temperature, humidity, light, CO) stored in a

file.

 Mapper ([Link]):

o Reads each line.

o Extracts key (e.g., time up to minute) and sensor values as CSV string.

o Emits: key -> "temp,humidity,light,CO".

 Reducer ([Link]):

o Groups values by key.

o Uses arrays and NumPy to compute mean of each sensor per key.

o Emits aggregated values (e.g., mean temp, humidity, light, CO per minute).

Running a streaming MapReduce job on Hadoop:

1. Copy data to HDFS (input directory).

2. Run [Link] with mapper and reducer scripts.

3. View output in output directory on HDFS.

5. Hadoop YARN (Hadoop 2.x)

YARN separates resource management from data processing, making Hadoop support
multiple engines (MapReduce, Tez, Storm, Spark, etc.).

Main YARN components

 ResourceManager (RM):

o Global resource manager.

o Has:

 Scheduler: allocates containers to applications based on policies.

 ApplicationsManager: manages ApplicationMaster lifecycles.

 NodeManager (NM):

o Runs on each node.

o Manages containers and monitors their resource usage.

 ApplicationMaster (AM):

o One per application.

o Negotiates resources from RM and works with NMs to run tasks, track
progress, handle failures.

 Container:

o A bundle of resources (CPU, memory, etc.) on a node for running a task or

AM.

YARN job execution (simplified)

1. Client asks RM for a new application ID, then submits an Application Submission
Context (ASC) with app jar, configuration, and AM resource requirements.

2. RM allocates a container and asks a NodeManager to start the AM.

3. AM registers with RM, periodically sends heartbeats, and requests containers for
tasks.

4. RM’s Scheduler grants containers; AM launches tasks on NMs.

5. After completion, AM informs RM and unregisters; client can check status or kill apps
through RM.

YARN cluster setup (overview)

  Download Hadoop 2.x, set environment variables (HADOOP_HOME, YARN_HOME,
etc.).

 Configure:

o [Link]: default filesystem, temp directory.

o [Link]: replication factor, permissions.

o [Link]: set [Link] = yarn.

o [Link]: resource manager addresses, shuffle service, etc.

 Start NameNode, DataNodes, ResourceManager, NodeManagers, and JobHistory

server; monitor with web UIs.

6. Apache Oozie (Workflow Scheduler)

 Oozie is a workflow scheduler for managing Hadoop jobs (MapReduce, Pig, Hive,
Sqoop, shell, Java, etc.).

 Workflows are defined as DAGs (Directed Acyclic Graphs) using an XML-based

language (hPDL).

Features

 Control dependencies: an action runs only when previous actions succeed.

 Supports error handling (kill actions, email notifications).

 Supports parameterization via property files.

Basic setup steps

 Require a working Hadoop cluster, plus additional tools (Maven, zip/unzip).

 Build Oozie from source, prepare WAR, deploy libraries to HDFS sharelib, create
Oozie DB, and start Oozie server.

 Access Oozie via:

o Command line client.

o Web console (default URL uses port 11000).

Example workflow (machine status/error counts)

 Input format: timestamp,statusErrorCode (e.g., 2014-07-01 20:03:18,115).

 Goal: output statusErrorCode,count.

 Map:
 o Extract code and emit code -> 1.

 Reduce:

o Sum counts per code, emit code -> totalCount.

 Workflow:

o A MapReduce action (streaming) with [Link] and [Link].

o Success -> send success email.

o Failure -> send failure email and kill action.

7. Apache Spark (Overview only)

 Spark is an in-memory cluster computing framework for fast data processing.

 Provides:

o Spark Core: RDD operations.

o Spark Streaming: mini-batch processing of streams.

o Spark SQL: SQL-like queries on structured data.

o MLlib: machine learning library.

o GraphX: graph processing.

 Often used where low latency and iterative algorithms are important compared to
disk-based MapReduce.

8. Apache Storm (Real-time Stream Processing)

 Storm is a distributed framework for real-time, fault-tolerant stream processing.

 Basic concepts:

o Spout: source of streams (reads from queues, APIs, sensors).

o Bolt: processes streams (filter, aggregate, classify, etc.).

o Topology: a directed graph linking spouts and bolts; runs continuously.

IoT example: Forest fire real-time detection

 Sensors send data via WebSocket/WAMP to a cloud controller, which stores in

MongoDB and pushes to ZeroMQ queues.

 A Storm Spout reads from ZeroMQ and emits tuples (sensor readings).
  A Storm Bolt:

o Loads a trained Decision Tree classifier.

o Predicts fire/no-fire label for incoming sensor windows.

o Emits prediction results for alerts.

Structural Health Monitoring (SHM) case study

 Use 3-axis accelerometer on structures (e.g., bridges) to measure vibrations.

 Pipeline:

o Native controller on Raspberry Pi reads accelerometer and publishes data via

WAMP.

o Central controller stores to MongoDB and forwards via ZeroMQ.

o Storm Spout reads vibration data; Bolt computes Short Time Fourier
Transform (STFT) to analyze frequency content over time.

9. Key Comparison Table

Hadoop
Aspect YARN Oozie Spark Storm
MapReduce

Batch Workflow Fast, in- Real-time

Main Resource & job
processing of scheduling for memory stream
purpose management
big data Hadoop analytics processing

Manages
Processing Disk-based, Orchestrates Iterative, Event-by-event,
containers, not
style high latency multiple jobs low latency very low latency
logic

ML, SQL,
Aggregation, Share cluster Multi-step IoT Alerts, online
Typical use graph on
ETL, logs across engines analytics flows classification
big data