LOVELY PROFESSIONAL UNIVERSITY

ECAP456
Introduction to Big Data

COMPLETE STUDY NOTES

All 14 Units · Every Topic Covered · Key Concepts · Definitions · Tables · Commands

14 100% 14+ All

Units Topics Covered Tools Covered Commands Included

Author: Rajni Bhalla, LPU · Editor: Sartaj Singh · ECAP456

TABLE OF CONTENTS

No. Unit Title Key Topics Covered

01 Introduction to Big Data Big Data basics, 5 V's, Applications, Tools, Challenges

02 Foundations of Big Data File Systems, DFS, Cloud Computing, MapReduce, BSP

03 Data Models Data Formats, Data Warehouse, Data Mart, Data Lake, Streaming

04 NoSQL Data Management RDBMS vs NoSQL, NoSQL Types, Aggregates, Sharding, Partitioner

05 Introduction to Hadoop Hadoop intro, HDFS/MapReduce/YARN, Ecosystem, Benefits

06 Hadoop Administration Installation, HDFS Goals, Architecture, Commands, Config files

07 Hadoop Architecture HDFS deep dive, YARN, High Availability, Daemons

08 Hadoop Master-Slave Architecture & MapReduce MapReduce Steps, Streaming, Multi-node Setup, Job/Task Tracker

09 Hadoop Node Commands HDFS Ops Commands, Adding/Removing Nodes, Block Pool

10 MapReduce Applications & MRUnit Testing MRUnit Testing, Shuffle & Sort, Schedulers, Failure Handling

11 Hadoop Ecosystem (Hive, Pig, HBase, Hive, Pig, HBase, ZooKeeper, Sqoop, Flume, InfoSphere
ZooKeeper)

12 Predictive Analytics & Data Visualisation ML Types, Linear Regression, Correlation, Data Visualisation

13 Data Analytics with R & Machine Learning ML Methods, Neural Networks, Naïve Bayes, RL, R Programming

14 Big Data Management using Splunk Splunk Features, Interfaces, SPL, Dashboards, Datameer, TEZ
 UNIT
01
Unit 01 — Introduction to Big Data

● 1.1 What is Big Data?

Big Data is a massive, ever-growing collection of structured, semi-structured, and unstructured data that traditional data
management tools cannot store or process effectively. Big Data analytics applies advanced techniques on datasets
ranging from terabytes to zettabytes.

■ Facebook absorbs 500 million+ terabytes of data daily — photo uploads, messages, posts.
■ Big Data is NOT just about volume — it's about extracting value from data to improve decisions.

Structured Data Organised in rows & columns. Easily queried with SQL. Examples: RDBMS,
spreadsheets.

Semi-Structured Data Has tags/markers but no strict schema. Examples: JSON, XML, CSV, emails.

Unstructured Data No predefined format. Examples: videos, images, audio, social media posts, PDFs.
(~80% of all data)

● 1.2 The 5 V's of Big Data

V Description

Volume Sheer amount of data — terabytes to zettabytes. Cheap cloud/Hadoop storage now feasible.

Velocity Speed of data generation and required processing. IoT, RFID, sensors demand near-real-time
handling.

Variety Diversity of data types: structured, semi-structured, and unstructured from many sources.

Variability Unpredictable, inconsistent data flows — seasonal spikes, event-driven surges.

Veracity Quality and reliability of data. Ensuring accuracy, cleansing, and linking across sources.

■ Note: A 6th V — Value — is sometimes added, referring to the business insights extracted from Big Data.

● 1.3 Applications of Big Data

■ Healthcare — Disease outbreak prediction, personalised treatment, hospital management.

■ Finance — Fraud detection, risk assessment, algorithmic trading, credit scoring.
■ Retail / E-Commerce — Recommendation engines, customer behaviour analysis, supply chain optimisation.
■ Social Media — Sentiment analysis, trend detection, targeted advertising.
■ IoT & Smart Cities — Traffic management, energy monitoring, predictive infrastructure maintenance.
■ Manufacturing — Predictive machine maintenance, quality control, demand forecasting.
■ Scientific Research — Genomics, climate modelling, particle physics, astronomy.

● 1.4 Types of Big Data

Social Data Likes, tweets, shares, comments — used for sentiment analysis.

Machine Data Server logs, RFID, IoT sensors — auto-generated by machines.

 Transactional Data ATM/POS/e-commerce transactions — billions of records daily.

Black Box Data Flight recorders: crew voices, microphone recordings, aircraft performance parameters.

Stock Exchange Data Buy/sell orders, price movements — real-time, extremely high velocity.

Power Grid Data Smart meter readings, consumption records across power networks.

● 1.5 Big Data Tools

Tool Description

Apache Hadoop Open-source framework: HDFS (storage) + MapReduce (processing). Foundation of Big Data.

Apache Spark In-memory distributed engine — 100× faster than MapReduce for iterative tasks.

HPCC System High-Performance Computing Cluster by LexisNexis — analytics platform.

Storm Distributed real-time fault-tolerant stream processing by Twitter.

Apache Cassandra Free, distributed, wide-column NoSQL DB — highly scalable & fault-tolerant.

CouchDB JSON document store accessible via HTTP/JavaScript.

MongoDB Most popular NoSQL document DB — flexible schema.

Apache HBase Column-oriented distributed DB on HDFS — based on Google BigTable.

Apache Hive SQL-like querying (HiveQL) on Hadoop — data warehousing layer.

Apache Pig High-level dataflow language (Pig Latin) for large-scale ETL.

Splunk Search, analyse, and visualise machine-generated data (logs, events).

● 1.6 Challenges of Big Data

■ Data Quality — Incomplete, inconsistent, or erroneous data degrades analytics results.

■ Data Storage — Managing petabytes/zettabytes requires scalable, cost-effective infrastructure.
■ Data Security & Privacy — Protecting sensitive data; GDPR/regulatory compliance.
■ Lack of Data Professionals — Critical shortage of data scientists, engineers, and analysts.
■ Processing Speed — Traditional RDBMS cannot handle real-time Big Data velocity.
■ Integration — Combining heterogeneous sources is technically complex.
■ Data Governance — Ensuring consistent, accurate, trustworthy data across the organisation.
UNIT
02
Unit 02 — Foundations of Big Data

● 2.1 File System vs Distributed File System

File System (FS) Manages data on a single machine. Files → Folders → Storage device (HDD, SSD). OS
uses file management system to access them. Extension shows file type
(EXE=executable, TXT=text).

Distributed File System Manages storage across a network of machines. Provides Location Transparency
(DFS) (access by name, not physical location) and Redundancy (multiple copies).

Feature File System Distributed File System

Location Single machine Multiple networked nodes

Scalability Limited to one disk Thousands of nodes

Fault Tolerance No built-in backup Data replicated across nodes

Access Local only Location-transparent

Examples NTFS, ext4, FAT32 HDFS, GFS, Amazon S3

● 2.2 Scalable Computing & Cloud Computing

Cluster Computing Group of interconnected computers working as one. Can communicate and cooperate to
store and compute over very large amounts of data.

Cloud Computing Delivers computing services over the internet on pay-per-use basis. Examples: AWS,
Azure, GCP.

Grid Computing Coordinates resources across multiple administrative domains for large scientific
computing goals.

■ New computing paradigms enabling Big Data: RFID systems, Sensor technologies, GPS, Mobile computing, IoT.

● 2.3 Programming Models for Big Data

MapReduce Google's model: Map phase (process → key-value pairs) then Reduce phase
(aggregate). Core of Hadoop processing.

Directed Acyclic Graph Directed graph with no cycles. Applications: Genealogy, Citation graphs, Job Scheduling.
(DAG) Used in Spark execution plans.

Message Passing Processes communicate by sending/receiving messages. Foundation of MPI

(High-Performance Computing).

Bulk Synchronous Parallel Computation split into supersteps: local compute → message exchange →
(BSP) synchronisation barrier. Used in Apache Giraph.

Workflow Systems Multi-step data processing pipelines with dependencies. Examples: Apache Oozie,
Apache Airflow.

● 2.4 MapReduce — Phases

Phase Description

Input Splits Input data divided into fixed-size chunks. Each split processed by one Mapper.

Map Mapper processes each record → emits intermediate (key, value) pairs.

Shuffle & Sort Framework automatically groups all values by key; transfers from Mappers to Reducers; sorted by key.

Reduce Reducer aggregates all values for each key → writes final output to HDFS.

■ Example — Word Count: Map emits (word, 1) per word. Reduce sums all 1s → (word, total_count).

■ Note: Number of Mappers ≈ input splits (driven by data size). Number of Reducers is user-configured.

● 2.5 Data Replication & Fault Tolerance

■ HDFS replicates each data block 3 times across different DataNodes (and ideally different racks).
■ Replication factor = 3: data safe even if 2 nodes fail simultaneously.
■ Replication enables fault tolerance AND high read concurrency (clients read from different replicas).
■ Trade-off: 3× more storage needed for same data volume.
 UNIT
03
Unit 03 — Data Models

● 3.1 Data Format & Data Model

Data Format Standard structure for encoding data for exchange. Common formats: JSON, XML, CSV,
Avro, Parquet.

JSON Key-value text format. Lightweight, human-readable, used in APIs and NoSQL
databases.

XML Tag-based hierarchical format. Verbose but flexible. Used in enterprise systems and web
services.

Data Model Abstract model defining how data is stored, organised, and accessed. Foundation for DB
design.

Applications must agree on data format (like routers needing standardised protocols to communicate). Standard
formats: JSON and XML are most common.

● 3.2 Data Warehouse

A Data Warehouse is a centralized repository of integrated, historical data for decision support and business
intelligence.

Property Description

Subject-Oriented Organised around business subjects (Sales, HR, Finance) — not operational processes.

Integrated Data from multiple sources integrated into consistent format and naming.

Time-Variant Historical data stored with time dimension. Supports trend analysis over years.

Non-Volatile Data appended only — never updated or deleted in real-time.

■ Examples: Amazon Redshift, Google BigQuery, Snowflake, Teradata, Azure Synapse.

● 3.3 Data Mart

A Data Mart is a subject-specific subset of a Data Warehouse, focused on one business area (Sales, Marketing, HR).

Type Source Approach

Dependent Central data warehouse Top-down (Inmon methodology)

Independent Operational/external sources Bottom-up (Kimball methodology) — no warehouse

needed

Hybrid Warehouse + other sources Combines both approaches

■ Note: Advantage: Faster query performance, simpler design, tailored for one department's specific analytical needs.

● 3.4 Data Lake

A Data Lake stores vast amounts of raw data in native format (structured, semi-structured, unstructured) until needed for
analytics.
Feature Data Lake Data Warehouse

Data Type All types (raw) Structured & processed

Schema Schema-on-read Schema-on-write

Storage Flat files/object storage Hierarchical tables/dimensions

Users Data scientists, ML engineers Business analysts, managers

Cost Low (commodity) Higher (specialised HW)

■ Uses flat file architecture — Amazon S3, HDFS, Azure Data Lake Storage.
■ Risk: Without governance, Data Lake becomes a 'Data Swamp' — unusable raw data without metadata.
■ Pipeline: Ingestion → Storage → Processing → Serving → Analytics.

● 3.5 Big Data Streaming

Streaming Data Continuously generated data from many sources (IoT, social media, financial tickers).
Processed incrementally as it arrives — not collected then processed.

Real-Time Processing Analysis within milliseconds to seconds of data arrival. Used in fraud detection, live
dashboards, autonomous vehicles.

Batch Processing Collect data over a period → process all at once. For non-time-critical tasks (nightly
reports, end-of-day reconciliation).

■ Use cases for streaming: Fraud detection (MasterCard identifies fraudulent merchants instantly), IoT smart
environment monitoring, Social media trending, Stock market analytics.

Streaming Data Inherently unstructured, fast rate of change, often cannot be stored before processing.
Characteristics

Sensor Data Sensors fail (battery dies), restart when replaced → systems must handle intermittent,
incomplete data streams. PocketLab = basic starter sensor solution.
 UNIT
04
Unit 04 — NoSQL Data Management

● 4.1 Relational Databases (RDBMS)

RDBMS organise data into tables (relations) with rows (tuples) and columns (attributes). Uses SQL. Enforces ACID
properties.

ACID Atomicity (all-or-nothing), Consistency (valid state transitions), Isolation (concurrent

transactions don't interfere), Durability (committed data persists).

■ Examples: MySQL, PostgreSQL, Oracle, SQL Server, IBM DB2.

■ Strengths: Strong consistency, complex joins, mature, well-understood.
■ Weaknesses: Poor horizontal scaling, rigid schema, poor on unstructured data, expensive for Big Data.

● 4.2 Why NoSQL?

■ Big Data volumes exceed single-server capacity — must scale horizontally across commodity machines.
■ Diverse data formats: JSON documents, key-value, graphs, wide columns — don't fit rigid relational tables.
■ High velocity: millions of reads/writes per second needed — RDBMS cannot keep up.
■ Evolving schemas: agile teams cannot predefine all data structures upfront.

■ Note: Carlo Strozzi coined 'NOSQL' in 1998 for his lightweight open-source relational DB that didn't use SQL.

● 4.3 Types of NoSQL Databases

Type Storage Model Best For Examples

Key-Value key → any value Caching, sessions Redis, DynamoDB

Document JSON/BSON docs CMS, e-commerce MongoDB, CouchDB

Column-Family Column-grouped rows Analytics, IoT Cassandra, HBase

Graph Nodes + Edges + Properties Social, fraud Neo4j, Neptune

Key-Value Simplest model (like a dictionary). Ultra-fast reads/writes. Limited querying — by key
only.

Document Self-contained JSON/BSON objects. Rich field-level querying. Schema-free.

Column-Family Column-by-column storage. Efficient analytical queries reading few columns from many
rows.

Graph Native graph storage — fast traversal of relationships between entities.

● 4.4 Aggregate Data Model

Aggregate A cluster of related data treated as a whole unit. Boundaries for ACID operations.
Key-Value, Document, Column-Family databases are all aggregate-oriented.

■ Aggregate orientation helps greatly with running on a cluster — the entire aggregate lives on one node, minimising
cross-node joins.
■ Graph databases are NOT aggregate-oriented — they optimise for inter-entity relationships.
■ Note: The clinching reason for aggregate orientation = it helps greatly with running on a cluster.

● 4.5 Distribution Models — Sharding & Replication

Sharding Partitioning data horizontally — different data subsets stored on different servers.
Enables horizontal scaling.

Replication Copying data to multiple servers. Improves read throughput and fault tolerance.

Master-Slave Replication One master handles writes; slaves replicate and serve reads. Write bottleneck at master.

Peer-to-Peer Replication All nodes equal — any node accepts reads/writes. High availability, but consistency
challenges.

■ Rebalancing sharding requires: (1) changing application code AND (2) migrating data — both are complex.
■ CAP Theorem: A distributed system can guarantee only 2 of 3: Consistency, Availability, Partition Tolerance.

● 4.6 Hadoop Partitioner

In MapReduce, the Partitioner determines which Reducer handles which intermediate key — ensuring all values for the
same key reach the same Reducer.

Hash Partitioner Default. Computes hash(key) % numReducers to assign partition. Distributes keys
evenly.

■ Number of partitions = number of Reducers. Each Reducer processes data from one partition.
■ Poor partitioning → data skew → some Reducers overloaded → job bottleneck.
■ Custom Partitioner: Extend [Link] class to control data distribution.
 UNIT
05
Unit 05 — Introduction to Hadoop

● 5.1 What is Hadoop?

Apache Hadoop is an open-source Java framework for distributed storage and processing of massive datasets across
clusters of commodity computers using simple programming models.

■ Created by Doug Cutting (inspired by Google GFS + MapReduce papers, 2006).

■ Core philosophy: Move computation TO the data — not data to computation.
■ Scales from 1 server to thousands of machines, each offering local computation and storage.

Problem Hadoop's Solution

Storing massive data HDFS — Distributed File System across commodity hardware

Processing massive data MapReduce — Parallel batch computation framework

Resource management YARN — Yet Another Resource Negotiator (Hadoop 2+)

Coordination ZooKeeper — Distributed coordination service

● 5.2 Hadoop Core Components

HDFS Storage layer. Files split into 128 MB blocks, distributed across DataNodes. Default
replication = 3.

MapReduce Processing layer. Parallel batch framework: Map → Shuffle/Sort → Reduce.

YARN (Yet Another Resource management since Hadoop 2.x. Manages CPU, memory, scheduling.
Resource Negotiator)

Hadoop Common Shared utilities and libraries used by all Hadoop modules.

● 5.3 Benefits of Hadoop

■ Scalability — Linear scale-out: add nodes to increase capacity and throughput.

■ Fault Tolerance — Data replication; failed tasks automatically re-executed on healthy nodes.
■ Cost-Effective — Runs on commodity hardware. No expensive specialised servers needed.
■ Flexibility — Handles structured, semi-structured, and unstructured data without predefined schema.
■ High Throughput — Parallel distributed processing for massive data volumes.
■ Open Source — Free, large community, no vendor lock-in.
■ Data Locality — Computation moved to data node, reducing network traffic significantly.

● 5.4 Hadoop Ecosystem

Component Purpose

HDFS Distributed file storage — the foundation layer

MapReduce Parallel batch data processing

YARN Resource management and job scheduling (Hadoop 2+)

Component Purpose

Apache Hive SQL-like HiveQL queries; data warehousing on Hadoop

Apache Pig High-level Pig Latin scripting for ETL on Hadoop

Apache HBase Real-time NoSQL columnar DB on HDFS

Apache Sqoop Import/Export between Hadoop and relational databases

Apache Flume Collect and move large log streams into HDFS

Apache Oozie Workflow scheduler for chained Hadoop jobs

ZooKeeper Distributed coordination, naming, configuration, locking

Apache Mahout Scalable ML library (collaborative filtering, clustering)

Eclipse Java IDE for MapReduce development (one of 3 biggest IDEs)

JAQL JSON query language for processing JSON-based documents

● 5.5 Parallel Computing

Parallel Computing Simultaneous use of multiple processors to solve a problem.

Parallel Programming Creating programs that execute computations concurrently across processors.

Apache Pig High-level platform for processing massive datasets. Concerned with 2 nodes. Developed
by Yahoo.

Apache Hive Data warehouse tool. Reads and writes large datasets. Created by Facebook.

Lucene Simple, powerful Java-based search library (used within Hadoop ecosystem for text
search).

Eclipse Java IDE — one of the 3 most popular IDEs globally. Used for Hadoop/MapReduce
development.
 UNIT
06
Unit 06 — Hadoop Administration

● 6.1 Prerequisites

■ OS: Linux (Ubuntu/CentOS preferred). Windows: use VirtualBox or VMware with CentOS.
■ Java: JDK 8 — Hadoop is entirely Java-based. Main prerequisite.

$ java -version # Verify Java installation

■ SSH: Password-less SSH configured between all master and slave nodes.
■ Hadoop Package: Hadoop 2.7.3 or later.

JRE Java Runtime Environment — to run Java programs.

JDK Java Development Kit — to compile + run Java. Includes JRE. Select 'Both JRE and JDK'
(Java SE package).

$ useradd hadoop # Create dedicated Hadoop Linux user

● 6.2 Hadoop Cluster Modes

Mode Description

Local/Standalone Default mode. No HDFS. Runs as single Java process. Used for debugging MapReduce
programs.

Pseudo-Distributed All daemons on single machine as separate Java processes. Simulates a cluster on one machine.

Fully Distributed Production mode. Multiple machines each running specific daemons — master + multiple slaves.

● 6.3 HDFS Goals & Architecture

■ Goals of HDFS:
◆ Fault Detection and Recovery — Automatic failure detection; automatic re-replication of lost blocks.
◆ Data Duplication — Each 128 MB block replicated 3 times on different DataNodes.
◆ Data Redundancy — Data available even if multiple nodes fail simultaneously.
◆ High Throughput — Optimised for streaming reads of large files (write-once, read-many).
◆ Portability — Runs on inexpensive commodity hardware across platforms.

NameNode (Master) Manages filesystem namespace: file names, directory structure, block locations,
permissions. Stores fsimage + edit logs on local disk.

DataNode (Slave) Stores actual data blocks. Sends heartbeats to NameNode every 3 seconds. Reports
block list on startup.

Secondary NameNode Periodically merges fsimage with edit logs (checkpoint). NOT a backup NameNode!

Block Size Default = 128 MB. A 1 GB file = 8 blocks distributed across DataNodes.

■ Note: When Primary NameNode fails, Secondary NameNode is utilised. In HA mode, Standby NameNode takes over
automatically.

● 6.4 Key HDFS Commands

Command Description

hadoop namenode -format Format HDFS filesystem — run ONCE at initial setup only

[Link] / [Link] Start/Stop NameNode + DataNodes

[Link] / [Link] Start/Stop all daemons (deprecated — use separate commands)

hadoop fs -ls / List HDFS root directory

hadoop fs -put file /path Upload local file to HDFS

hadoop fs -get /path file Download HDFS file to local

hadoop fs -mkdir /path Create directory in HDFS

hadoop fs -rm /path/file Delete a file in HDFS

hadoop fs -cat /path/file Display file content from HDFS

hadoop fs -stat %b /path Show block size (%r = replication, %s = file size)

hadoop fs -checksum /path Verify file integrity

hadoop fsck / Check HDFS health — report corrupt/missing blocks

● 6.5 Hadoop Configuration Files

[Link] Environment variables for all daemons — JAVA_HOME path, heap sizes.

[Link] Core shared properties — HDFS URI ([Link]), I/O settings.

[Link] HDFS properties — replication factor, block size, NameNode/DataNode dirs.

[Link] YARN properties — ResourceManager address, NodeManager memory.

[Link] MapReduce properties — framework name (yarn), history server address.

■ Note: JAVA_HOME is configured in [Link]. Java environment variables are set there.
 UNIT
07
Unit 07 — Hadoop Architecture

● 7.1 HDFS Properties

■ Huge Volumes — Stores petabytes of data without performance issues.

■ Data Access — Write-once, read-many-times pattern for high-throughput streaming reads.
■ Cost-Effective — Runs on commodity (inexpensive) hardware from any vendor.
■ A 1 GB file split into 8 × 128 MB blocks, each replicated 3 times across different DataNodes.
■ HDFS is NOT suitable for: low-latency data access, small files, multiple concurrent writes to same file.

● 7.2 Hadoop Three Layers

Layer Role

HDFS Storage — stores data as blocks across DataNodes. Fault-tolerant, scalable.

MapReduce Processing — parallel batch computation. Divides jobs into Map + Reduce tasks.

YARN Resource Management — allocates CPU & RAM; schedules applications cluster-wide.

● 7.3 YARN Architecture

Resource Manager (RM) Cluster-level master. Manages and allocates all cluster resources. Contains Scheduler
and Applications Manager. Runs on master node. Uses 'yarn' user.

Node Manager (NM) Per-node slave daemon. Manages node resources, launches/monitors Containers,
reports to ResourceManager.

Application Master (AM) Per-application process. Negotiates resources with RM. Coordinates task execution with
NodeManagers. Monitored by RM.

Container A resource allocation (CPU + memory) on a specific node. Each task runs inside a
Container.

Job History Service Daemon storing completed MapReduce job info. Run as separate daemon. Uses
'mapred' user.

■ Note: Slave computers run TWO daemons: DataNode (storage) + NodeManager (compute).

● 7.4 Hadoop High Availability (HA)

Without HA, the NameNode is a Single Point of Failure (SPOF). Hadoop HA eliminates this with two NameNodes:

Active NameNode Primary — handles all client requests in production.

Standby NameNode Hot standby — continuously synchronised with Active via QJM (Quorum Journal
Manager) or NFS shared storage.

Automatic Failover ZooKeeper monitors Active NameNode; automatically promotes Standby on failure.

■ HA requires: ZooKeeper ensemble (3 or 5 nodes), Journal Nodes (3+), and fencing to prevent split-brain.
 ● 7.5 Hadoop Daemons Summary

Daemon Node Role

NameNode Master HDFS namespace manager — metadata, block locations

Secondary NameNode Separate Checkpoints NameNode fsimage+edits periodically

DataNode Every slave Stores actual HDFS data blocks; heartbeat to NN

ResourceManager Master YARN — manages cluster CPU/RAM resources

NodeManager Every slave YARN — manages resources on each individual node

Job History Server Separate Stores completed MapReduce job history

■ HDFS metadata is stored persistently in: Fsimage (namespace snapshot) and Edit Log (change log).
■ Hadoop divides files into Blocks — the physical representation of data. All blocks same size except possibly the
last block.
 UNIT Unit 08 — Hadoop Master-Slave Architecture &
08 MapReduce

● 8.1 MapReduce Execution — Four Steps

Step Description

1. Input Splits Input data divided into fixed-size chunks. Each split → one Mapper. Split size ≈ HDFS block size.

2. Map User-defined map() processes each record, emits intermediate (key, value) pairs.

3. Shuffle & Sort Framework groups all values by key; transfers from Mappers to Reducers; sorts by key.

4. Reduce User-defined reduce() aggregates values per key → writes final output to HDFS.

Mapper Processes input key-value → emits intermediate key-value pairs. One Mapper per input
split.

Reducer Receives all values for each unique key post-shuffle. Aggregates → produces final
output.

waitForCompletion() Method that causes the call to return only when the job finishes, returning success/failure
status.

■ Note: numReduceTasks = 0: No Shuffle or Sort performed. Map output written directly to HDFS. Even Map phase is
faster.

● 8.2 Hadoop Streaming

Hadoop Streaming allows MapReduce jobs in ANY language (Python, Ruby, Bash, Perl, etc.) using stdin/stdout as
interface.

■ Mapper: reads from stdin line-by-line, writes to stdout.

■ Reducer: reads sorted stdin, writes to stdout.

hadoop jar [Link] \

-input /input/path -output /output/path \

-mapper [Link] -reducer [Link]

■ Mandatory streaming arguments: input directory, output directory, mapper executable.

■ You CAN specify any executable as the mapper and/or reducer — not just Java classes.

● 8.3 Multi-Node Cluster Setup

■ Step 1: Install Java JDK 8 on all nodes (master + slaves).

■ Step 2: Create dedicated Hadoop user on all nodes:

useradd hadoop

■ Step 3: Configure password-less SSH:

ssh-keygen -t rsa -P '' # Generate RSA key pair

■ Step 4: Edit /etc/hosts on ALL machines — map hostnames to IPs.

■ Step 5: Configure Hadoop XML files (core-site, hdfs-site, yarn-site, mapred-site) on all nodes.
 ■ Step 6: Add slave hostnames to 'slaves' file on master.
■ Step 7: Format HDFS and start daemons from master:

hadoop namenode -format

[Link] && [Link]

■ Step 8: Verify with JPS — lists running Java daemons:

jps # Shows NameNode, DataNode, ResourceManager, NodeManager, etc.

● 8.4 Job Tracker vs Task Tracker (Hadoop 1.x) — Legacy

Component Role

Job Tracker (Master) Receives job submissions. Assigns Map/Reduce tasks to Task Trackers. Monitors progress.
Reschedules failed tasks.

Task Tracker (Slave) Executes individual Map or Reduce tasks. Reports progress via heartbeats to Job Tracker.

■ Note: Hadoop 2.x (YARN): Job Tracker → ResourceManager + ApplicationMaster. Task Tracker → NodeManager.

● 8.5 Hadoop Subprojects

Hadoop Common Utilities and libraries shared by all Hadoop subprojects.

HDFS Distributed File System — provides application data access across the cluster.

MapReduce Software framework for processing huge distributed datasets on compute clusters.

YARN Framework for resource management and job scheduling across the Hadoop cluster.
 UNIT
09
Unit 09 — Hadoop Node Commands

● 9.1 HDFS Daemon Commands

Command Action

hadoop namenode -format Format HDFS — initialise NameNode directory. ONCE only (destroys data if
repeated!)

[Link] Start NameNode + DataNodes

[Link] Stop NameNode + DataNodes

[Link] Start ResourceManager + NodeManagers

[Link] Stop ResourceManager + NodeManagers

[Link] Start all daemons (DEPRECATED — use separate commands)

[Link] Stop all daemons (DEPRECATED)

[Link] Start MapReduce daemons: JobTracker + TaskTrackers (Hadoop 1.x)

[Link] Stop MapReduce daemons (Hadoop 1.x)

jps List running Java processes — verify which daemons are active

● 9.2 HDFS File Operations

Command Description

hadoop fs -ls /path List files/directories

hadoop fs -mkdir /path Create directory

hadoop fs -put local /hdfs Upload local → HDFS (copyFromLocal)

hadoop fs -get /hdfs local Download HDFS → local (copyToLocal)

hadoop fs -cat /path/file Display file content

hadoop fs -rm /path/file Delete file

hadoop fs -rm -r /path/dir Delete directory recursively

hadoop fs -mv /src /dst Move/rename within HDFS

hadoop fs -cp /src /dst Copy within HDFS

hadoop fs -du /path Show disk usage

hadoop fs -df -h HDFS filesystem usage (human readable)

hadoop fs -chmod 755 /path Change permissions

hadoop fs -chown user /path Change ownership

hadoop fs -chgrp group /path Change group ownership

hadoop fs -stat %b /path Block size (%r=replication, %s=size, %u=owner, %y=last modified)

hadoop fsck / HDFS health check — missing/corrupt blocks report

hadoop dfsadmin -report Cluster state: node count, capacity, used space
Command Description

hadoop dfsadmin -refreshNodes Reload node config (after decommissioning)

● 9.3 Adding a New DataNode

■ Install Java + Hadoop on new node.

■ Create hadoop user; configure SSH access from master.
■ Copy Hadoop configuration files to new node.
■ Update /etc/hosts on ALL cluster nodes with new node's hostname/IP.

ping [Link] # Verify hostname resolution

■ Start DataNode daemon on new node:

$HADOOP_HOME/bin/[Link] start datanode

■ Add hostname to 'slaves' file on master.

■ Verify: JPS on new node should show 'Datanode'.

● 9.4 Decommissioning (Removing) a DataNode

■ Step 1: Add node hostname to [Link] file in [Link].

■ Step 2: hadoop dfsadmin -refreshNodes — NameNode begins replicating blocks away from this node.
■ Step 3: Monitor: hadoop dfsadmin -report — wait for decommissioning to complete.
■ Step 4: Safely shut down and remove the DataNode from the cluster.

■ Note: HDFS decommissioning guarantees safe node removal WITHOUT data loss.

● 9.5 HDFS Internal Concepts

Block Pool Set of blocks belonging to a single namespace (one NameNode). DataNodes act as
common storage for blocks.

ClusterID Unique identifier assigned during namenode formatting. Identifies all nodes in this cluster.

Namespace Layer Manages HDFS file/directory structure — names and hierarchy.

Block Storage Layer Manages physical storage of data blocks on DataNodes.

[Link]() Method that clears ALL keys from the Hadoop configuration object.
 UNIT
10
Unit 10 — MapReduce Applications & MRUnit Testing

● 10.1 Testing in Hadoop

Testing Type Description

Unit Testing Test individual modules in isolation. JUnit framework for Java. Catches defects early.

Functional Testing Test entire system end-to-end — full job from input to output.

Stress Testing Test under extreme high load to find performance bottlenecks.

Load Testing Test under expected production load to ensure SLAs are met.

JUnit Standard unit testing framework for Java. Works with MRUnit.

MRUnit Apache MRUnit — Java library for testing Hadoop MapReduce jobs WITHOUT a full
cluster. Runs on single machine.

● 10.2 MRUnit — Core Classes

MRUnit Class Purpose

MapDriver Tests Mapper in isolation. Provide input key-value → verify output key-value pairs.

ReduceDriver Tests Reducer in isolation. Provide key + list of values → verify output.

MapReduceDriver Tests complete pipeline. Provide input → verify final output after Map and Reduce.

■ MRUnit tests run on single machine without a cluster — fast, easy CI/CD integration.
■ Based on JUnit — standard @Test, @Before, @After annotations work normally.

● 10.3 Shuffle and Sort — Detailed

Shuffling Transfer of intermediate Map output to the appropriate Reducer. System sorts and
transfers map output to reducer as input. Starts before all Mappers finish (pipeline
parallelism).

Sorting All intermediate key-value pairs automatically sorted by KEY before Reducers begin.
Values are NOT sorted.

■ Shuffle and Sort occur simultaneously — entirely managed by the MapReduce framework; no user code needed.
■ Secondary Sorting: Use custom comparator when values also need to be sorted (not default behaviour).

■ Note: numReduceTasks = 0 → Neither Shuffle nor Sort is performed. Map output goes directly to HDFS. Even faster Map
phase.

● 10.4 Hadoop Schedulers

FIFO Scheduler Default. Jobs executed in submission order. Simple but unfair in multi-user clusters.

Capacity Scheduler For large multi-tenant clusters. Resources partitioned among organisations. Minimum
capacity guaranteed per org. Pluggable scheduler in ResourceManager.
Fair Scheduler Each application receives equal resource share over time. Better for interactive
workloads. Scheduling based on memory by default.

● 10.5 Failure Handling in MapReduce

Task Failure Map/Reduce task fails (exception, JVM crash). Hadoop retries up to 4 times (default) on
different nodes before failing the job.

Job Failure Job fails if task exceeds max retry limit. Check outcome with waitForCompletion() —
returns true on success.

Node Failure ResourceManager detects via missed heartbeats. Tasks redistributed to healthy nodes.
Data re-replicated from surviving replicas.

NameNode Failure Critical in non-HA. With HA, ZooKeeper automatically promotes Standby NameNode.

● 10.6 YARN Concepts

YARN (Yet Another Resource management layer of Hadoop 2+. Manages CPU and RAM for all nodes,
Resource Negotiator) applications, and queues.

■ YARN is responsible for: Scheduling tasks, Monitoring tasks, Re-executing failed tasks.
■ Decomposing a data processing application into Mappers and Reducers is sometimes non-trivial (challenging).
 UNIT Unit 11 — Hadoop Ecosystem (Hive, Pig, HBase,
11 ZooKeeper)

● 11.1 Apache Hive

Apache Hive is an open-source data warehousing and SQL analytics system on top of Hadoop. Created by Facebook;
maintained by Apache. Used by Netflix, Amazon (Elastic MapReduce), and many enterprises.

■ NOT for real-time queries or row-level updates. Designed for OLAP batch processing.
■ HiveQL (HQL) — SQL-like language automatically compiled to MapReduce / Tez / Spark jobs.

Hive Component Role

Metastore Central repository for table schemas, partitions, and data locations. Uses relational DB
(MySQL/Derby).

Driver Manages full query lifecycle — compilation → optimisation → execution.

Compiler Parses HiveQL, creates DAG execution plan. Fetches metadata from Metastore.

Optimizer Optimises logical plan for better performance.

Executor Executes plan using MapReduce, Tez, or Spark.

User Interface Hive CLI, Beeline (JDBC CLI), Web UI, HiveServer2, JDBC/ODBC drivers.

Hive Services HiveServer2 (remote connections), Beeline (JDBC CLI), Metastore service, Hive CLI.

■ Note: Schema is stored in Metastore (relational DB). Processed data is saved to HDFS.

● 11.2 Apache Pig

Apache Pig is a high-level platform for processing massive datasets on Hadoop. Developed by Yahoo researchers.
Language: Pig Latin — a dataflow scripting language.

■ Pig compiles Pig Latin scripts automatically into MapReduce jobs.

■ Concerned with 2 nodes. Allows complex data transformations without needing to write Java.

Component Role

Parser Initial parsing and syntax validation of Pig Latin. Generates DAG.

Optimizer Optimises logical plan — filter pushdown, etc.

Compiler Compiles optimised plan into series of MapReduce jobs.

Execution Engine Submits MapReduce jobs to Hadoop cluster.

Pig Execution Modes Local Mode (single machine for testing), MapReduce Mode (production on cluster).

Pig Latin Data Types int, long, float, double, chararray, bytearray, boolean, tuple, bag, map.

Pig Operators Relational (LOAD, STORE, FILTER, FOREACH, JOIN, GROUP, ORDER, DISTINCT) +
Diagnostic (DUMP, DESCRIBE, EXPLAIN, ILLUSTRATE).

JOIN operator Performs inner equijoin of 2+ relations on common field values.

● 11.3 Apache HBase

HBase is a distributed, scalable, column-oriented NoSQL database built on HDFS. Modeled after Google Bigtable.
Provides real-time random read/write access to HDFS data.

■ Horizontally scalable — add RegionServers to increase capacity.

■ HBase Tables are separated into Regions HORIZONTALLY by row key range.

HBase Component Role

HMaster Master — coordinates RegionServers, schema changes, load balancing.

RegionServer Serves reads/writes for assigned Regions. Manages MemStore and HFiles.

ZooKeeper Coordination — cluster state, distributed locks, master election.

HFile Physical storage format for HBase data on HDFS.

MemStore In-memory write buffer in RegionServer — flushed to HFile when full.

Block Cache In-memory read cache — caches recently accessed HFile blocks.

■ Note: HBase is NOT a replacement for HDFS — it provides random access ON TOP of HDFS (which is sequential).

● 11.4 Apache ZooKeeper

ZooKeeper is a centralised coordination service for distributed applications — configuration, naming, synchronisation,
and group services.

■ Used by: HBase, Hadoop HA, Storm, Kafka, Solr, and many other distributed systems.
■ Services: Leader election, Distributed locking, Configuration management, Service registry.
■ ZooKeeper ensemble = 3 or 5 nodes (odd number for quorum majority voting).

● 11.5 IBM InfoSphere Streams

Platform for analysing massive volumes of streaming data in real-time. Analyses data-in-motion — continuously
generated data from markets, sensors, social media.

■ Language: Streams Processing Language (SPL). Can extend with C, C++, or Java code.
■ Scale-out: Single server to unlimited nodes; millions of events/second with microsecond latency.

Toolkits Complex Event Processing (CEP), Financial Services, Standard, Data Mining.

■ Tools: InfoSphere Streams Debugger, drag-and-drop graphical editor, instance graph (visual monitoring), data
visualisation with charts.

● 11.6 Supporting Ecosystem Tools

Apache Sqoop Transfers data between Hadoop (HDFS/Hive) and relational databases using
MapReduce for parallel bulk transfer.

Apache Flume Collects, aggregates, and moves large log data streams into HDFS. Architecture: Source
→ Channel → Sink.

Apache Oozie Workflow scheduler for chained Hadoop jobs. Defines job dependencies as DAGs.

Apache Mahout Scalable ML library on Hadoop — collaborative filtering, clustering, classification.

 UNIT
12
Unit 12 — Predictive Analytics & Data Visualisation

● 12.1 Machine Learning — Overview

Machine Learning (ML) is an application of AI that enables systems to automatically learn and improve from experience
without being explicitly programmed. ML algorithms train on historical data to recognise patterns and make predictions.

ML Category Description

Supervised Learning Trains on labelled data (input-output pairs). Examples: Linear Regression, Decision Trees, Neural
Networks, Naïve Bayes, SVM, Random Forest.

Unsupervised Learning Unlabelled data — discovers hidden patterns. Examples: K-Means Clustering, Association
Analysis, PCA.

Semi-Supervised Learning Small labelled + large unlabelled data. Middle-ground approach — labelling is expensive.

Reinforcement Learning Agent learns by interacting with environment. Maximises cumulative reward. Used in robotics,
game-playing, autonomous systems.

● 12.2 Linear Regression

Linear Regression Supervised ML algorithm. Models relationship between dependent variable and one or
more independent variables by fitting a linear equation.

Simple Linear Regression One independent variable → best-fit line: y = mx + b. Studies relationships between two
continuous (quantitative) variables.

Multiple Linear Regression Two or more independent variables: y = b■ + b■x■ + b■x■ + … + b■x■.

Least Square Error Strategy to determine best-fit line — minimises sum of squared differences between
actual and predicted values.

Independent Variable (IV) Feature manipulated/controlled by researcher. Input variable. Can be directly
manipulated.

Dependent Variable (DV) Variable being measured/predicted. Output variable. Changes in response to IV.

Correlation Measures the linear link between two variables. Does NOT reveal causation or complex
non-linear correlations. Positive correlation (both increase); negative (inverse).

● 12.3 Data Visualisation

Data Visualisation is the graphical representation of information and data using visual elements (charts, graphs, maps)
to help identify trends, outliers, and patterns.

Chart Type Best Used For

Bar Chart Comparing categories — distributions across discrete groups

Line Chart Trends over time — time-series data

Scatter Plot Relationships/correlation between two continuous variables; reveals curvilinear patterns

Pie Chart Parts of a whole — proportions/percentages

Histogram Frequency distribution of a single continuous variable

Chart Type Best Used For

Box Plot Data distribution — median, quartiles, outliers

Heat Map Data density or correlations between two variables

Univariate Analysis Analysis of one variable at a time.

Bivariate Analysis Study of relationship between two variables simultaneously.

Multivariate Analysis (MVA) Analysis of multiple variables simultaneously — finds patterns and relationships in
complex datasets.

Matplotlib (Matplot) Primary Python library for data visualisation — making charts and graphs.

PDF (Probability Density Mathematical function describing probability of a continuous random variable taking a
Function) given value.

● 12.4 Applications of Predictive Analytics

Predictive Analytics uses historical data, statistical modelling, data mining, and ML to forecast future events.

Customer Targeting Identify high-potential customers using engagement factors: Recency, Frequency,
Monetary value (RFM), plus past campaign response.

Churn Prevention Predict which customers will leave and when. Retaining existing customers is far cheaper
than acquiring new ones.

Sales Forecasting Estimate future demand using historical sales, seasonality, market events, pricing, and
economic factors.

Market Analysis Analyse preferences, product attributes, and market demand to improve
products/services.

Risk Assessment Identify financial risks — credit, operational, market. Evaluate customer/firm risk using
socio-demographic and financial data.

Financial Modelling Convert market behaviour assumptions into numerical forecasts for investment decisions.
UNIT
13
Unit 13 — Data Analytics with R & Machine Learning

● 13.1 Machine Learning Methods — Detailed

Supervised Learning Labelled data (input-output pairs). Goal: learn function mapping X → Y. Predict Y for new
inputs.

■ Supervised algorithms: Linear Regression, Logistic Regression, Decision Trees, Random Forest, SVM, Neural
Networks, Naïve Bayes, KNN.

Unsupervised Learning No labels. Discovers hidden structure in unlabelled data.

■ Unsupervised algorithms: K-Means, DBSCAN, Hierarchical Clustering, PCA, Autoencoders, Association Rules.

Semi-Supervised Learning Tiny labelled + large unlabelled dataset. Useful when labelling is
expensive/time-consuming.

Reinforcement Learning Agent takes actions in an environment to maximise cumulative reward. Uses MDP
(RL) (Markov Decision Process) framework.

● 13.2 Neural Networks & Naïve Bayes

Neural Networks Computational models inspired by the human brain. Interconnected neurons in layers:
Input → Hidden → Output. Recognise complex patterns in images, text, audio. Deep
Learning = many hidden layers.

Naïve Bayes Probabilistic classifier based on Bayes' theorem with 'naive' assumption of feature
independence. Simple, fast. Excellent for text classification (spam, sentiment analysis).
Family of probabilistic classifiers.

● 13.3 Recommendation Systems

Content-Based Filtering Uses item features to recommend similar items based on user's past actions or explicit
feedback. Considers user's taste profile.

Collaborative Filtering Automatic prediction for a user based on preferences of similar users. Unsupervised
learning technique. Uses collective behaviour patterns.

■ Note: Content-based = 'what you liked'. Collaborative = 'what people like you liked'.

● 13.4 Reinforcement Learning Details

MDP (Markov Decision Mathematical framework: States, Actions, Rewards, Transitions, Discount factor. Basis of
Process) RL.

Q-Learning Model-free RL algorithm. Learns action values without requiring environment model.
Handles stochastic transitions.

■ RL Applications: Robotics/industrial automation, autonomous vehicles, game playing (AlphaGo, Chess), aircraft
control, personalised education, business strategy planning.

● 13.5 R Programming Language

R is a statistical computing and data visualisation language. Developed by Ross Ihaka and Robert Gentleman.
Resembles the S programming language on the surface.

Packages R functionality is divided into packages. Loaded with library() function. Published on
CRAN.

RStudio Primary IDE for R — console, script editor, environment viewer, plot viewer. Used for
statistical analysis.

C, C++ Advanced users can extend R with C, C++ code for performance-critical sections.

■ R Data Types: numeric, integer, character, logical, complex.

■ R Data Structures: vector, matrix, array, data frame, list.

■ Note: Data Frame is NOT a data type — it is a data STRUCTURE holding tabular data with mixed column types.

R Package/Use Description

ggplot2 Publication-quality data visualisation

dplyr / tidyr Data wrangling and manipulation

caret / randomForest Machine learning model training

Bioconductor Bioinformatics — genomics analysis

Shiny Interactive web applications in R

● 13.6 Clustering & Association Analysis

Clustering Groups data points so intra-group similarity > inter-group similarity. Unsupervised.
Examples: K-Means, DBSCAN, Hierarchical.

Association Analysis Finding frequent itemsets and association rules in large datasets. Market basket analysis
— 'customers who buy X also buy Y'. Two types: frequent item sets and association
rules.

Sentiment Analysis Systematic identification, extraction, quantification, and study of emotional states from
text using NLP, text analysis, and computational linguistics.
 UNIT
14
Unit 14 — Big Data Management using Splunk

● 14.1 What is Splunk?

Splunk is a software platform for searching, analysing, and visualising machine-generated data (logs, events, metrics).
Processes unstructured, semi-structured, and structured data. Has built-in data type recognition, field separator
detection, and search optimisation.

■ Machine data sources: web servers, IoT devices, mobile app logs, server CPU logs, network equipment.
■ Evolution: Simple log analysis tool → now a general analytical tool for Big Data of all kinds.
■ Prerequisite knowledge: SQL querying language, familiarity with computer application logs.

● 14.2 Splunk Product Categories

Product Description

Splunk Enterprise For large organisations with significant IT infrastructure. On-premises. Full-featured — websites,
apps, devices, sensors.

Splunk Cloud Cloud-hosted SaaS with same functionality as Enterprise. Available via Splunk or AWS
Marketplace.

Splunk Light Lightweight — real-time search, reporting, and alerts from a single location. Fewer features; for
smaller environments.

● 14.3 Splunk Core Features

Feature Description

Data Ingestion Accepts JSON, XML, and unstructured machine data (web/app logs). User models structure as
desired.

Data Indexing Indexes imported data for faster searching and querying — stores time-stamped events in an
index.

Data Searching Uses indexed data to create metrics, forecast trends, spot patterns via SPL.

Alerts Sends emails or RSS feeds when specified criteria are met in examined data.

Dashboards Panel-based displays: charts, tables, reports, pivot tables. Linked to saved searches.

Data Model Hierarchical datasets based on domain knowledge. Non-SPL users access data via Pivot.

Forwarders Lightweight agents feeding data from remote machines into central Splunk instance.

Reports Saved searches — schedulable, shareable, addable to dashboards.

● 14.4 Splunk Interface & Search

Search & Reporting App Primary Splunk app — run SPL searches, create reports, view dashboards.

Add Data / Upload Data Interface for data ingestion — specify source, format, index.

Administrator Menu Customise and modify administrator account and system settings.
 SPL (Search Processing Splunk's query language. Components: Search Terms, Commands, Functions,
Language) Arguments, Clauses.

index=main sourcetype=access_log status=200 | stats count by uri | sort -count

Analytics Stages Design/edit time (create searches, dashboards) and Execution/run time (data searched,
results returned).

● 14.5 Splunk Data Pipeline

■ Stage 1 — Input: Data enters via Universal Forwarder, Heavy Forwarder, or direct upload.
■ Stage 2 — Parsing: Data broken into events (timestamped records). Field extraction performed.
■ Stage 3 — Indexing: Parsed events indexed and stored on disk for fast retrieval.
■ Stage 4 — Searching: SPL queries executed against indexed data to return results.

● 14.6 Installing Splunk on Windows

■ Download MSI installer from [Link]/download.

■ Run installer → Accept License → Customise or use default settings.
■ Default install path: C:\Program Files\Splunk (= %SPLUNKHOME%).
■ MAX_PATH restriction in Windows API = 260 characters. Use short paths (e.g., C:\Splunk) to avoid issues.
■ Restrictions: Cannot install 32-bit on 64-bit Windows. Cannot install on unsupported OS (e.g., Windows Server
2003).
■ Disable antivirus on-access scanning for Splunk dirs — indexer requires heavy disk I/O.
■ Windows services installed: splunkd (operations) + splunkweb (legacy web, optional).
■ After install: Set admin password → Launch browser → Complete setup.

■ Note: User must be specified in domain\username format. Must be valid Active Directory user with local admin privileges.

● 14.7 Datameer & Apache TEZ

Datameer is an all-in-one Hadoop analytics solution that integrates seamlessly with the Hadoop ecosystem.

Data Ingestion Ingest data into Hadoop from various sources.

Data Preparation Cleanse, transform, and enrich ingested data. Feature engineering for ML.

Querying Run Hive, Tez, or Spark queries without writing code.

Visualisation Built-in visualisation for query results.

Apache TEZ Execution framework on YARN. Splits workloads into smaller pieces. Executes complex
DAG jobs more efficiently than standard MapReduce. Used by Hive and Pig on YARN.

ECAP456 — Introduction to Big Data · Complete Study Notes · Lovely Professional University
Author: Rajni Bhalla · Editor: Sartaj Singh · All 14 Units Covered