Big Data Analytics — Complete Exam Notes

[Link] VIII Sem | RTU | By Aditya Khabya

UNIT 1: Introduction to Big Data

What is Big Data?

Big Data refers to data collections so massive and complex that traditional tools like Excel or
relational databases cannot process them efficiently. It involves data produced by devices,
applications, social media, sensors, and more.

🧒 5th Grader Example: Imagine your school library has 100 books — your teacher can
find any book easily. Now imagine ALL the books in ALL the libraries in the WORLD. No
single teacher or computer can manage that alone. That's Big Data!

The 3 V's of Big Data ⭐ (Asked EVERY Year!)

V Meaning Real-World Example

Volume Huge amount of data Facebook generates 4 petabytes/day

Velocity Speed of data generation Twitter gets 500 million tweets/day

Variety Different types/formats Videos, text, images, sensor logs

Additional V's:

Veracity — Truthfulness and quality of data

Value — Usefulness of data after analysis
🧒 5th Grader Example: Think of collecting LEGO bricks. Volume = how many bricks you
have. Velocity = how fast new bricks arrive. Variety = bricks of different shapes, sizes,
and colors!

Types of Big Data

1. Structured — Organized in rows and columns (e.g., SQL database, Excel table)
2. Unstructured — No fixed format (e.g., emails, videos, social media posts)
3. Semi-Structured — Mix of both (e.g., XML, JSON files)
Big Data Challenges
Capturing and storing massive data volumes
Searching and sharing across distributed systems
Analysis at scale
Privacy and security concerns
Presentation of results

Traditional Approach vs Big Data (Hadoop)

Traditional Big Data (Hadoop)

Single machine Cluster of thousands of machines

RDBMS (Oracle, MySQL) HDFS + MapReduce

Fails at very large scale Scales linearly

Expensive high-end servers Cheap commodity hardware

Types of Data Analysis

1. Descriptive Analysis — What happened? (uses past data, dashboards)
2. Diagnostic Analysis — Why did it happen? (finds root cause)
3. Predictive Analysis — What is likely to happen? (forecasting)
4. Prescriptive Analysis — What should we do? (combines all insights)

Big Data Applications (Industries)

Banking and Securities | Healthcare | Retail | Transportation | Government | Education |
Media and Entertainment

UNIT 2: Hadoop & HDFS ⭐⭐⭐

What is Hadoop?
Hadoop is an open-source, Java-based framework for storing and processing Big Data across
clusters of commodity computers. It was created by Doug Cutting (named after his son's toy
elephant 🐘) and is based on Google's MapReduce and Google File System (GFS) papers.

🧒 5th Grader Example: Hadoop is like a TEAM of ants carrying a huge watermelon. One
ant alone can't do it, but thousands of ants working together can carry it easily!
Hadoop Core Modules
Module Function

Hadoop Common Shared Java libraries and utilities

HDFS Distributed file system for storing data

MapReduce Framework for parallel data processing

YARN Resource management and job scheduling

Google File System (GFS) — The Inspiration

Created by Google for its own data storage needs
Files divided into fixed 64 MB chunks
Cluster has 1 Master Node + many Chunk Servers
Each chunk replicated a minimum of 3 times (fault tolerance)
Master tracks all metadata; chunk servers send periodic heartbeat messages
Not implemented in OS kernel — runs as a user-space library

HDFS Architecture ⭐⭐⭐ (Always Draw in Exam!)

[Client]
|
[Nam eNode] ← Master: stores m etadata (file nam es, block locations)
/ | \
[DN1] [DN2] [DN3] ← DataNodes: store actual data blocks

Block Replication:
Each file is split into blocks (default 64 MB or 128 MB). Every block is copied to 3 DataNodes
automatically.

🧒 5th Grader Example: NameNode is like the school PRINCIPAL who keeps a record of
where every student sits. DataNodes are the CLASSROOMS where students actually
sit!

HDFS Key Components

Component Role

NameNode Master — stores filesystem metadata, manages namespace

DataNode Slave — stores actual data blocks

Secondary NameNode Takes periodic checkpoints of NameNode's edit logs (NOT a backup!)
 Component Role

Block Default size = 64 MB (128 MB in newer versions)

Replication Factor Default = 3 copies of each block

HDFS Properties:

Write once, read many times (optimized for large reads)

Runs on commodity hardware
Not suitable for low-latency data access

Hadoop Building Blocks — Full Picture

Component Type Function

NameNode HDFS Master Manages file metadata

DataNode HDFS Slave Stores data blocks

Secondary NameNode HDFS Support Checkpoints NameNode edits

JobTracker MapReduce Master Schedules and monitors jobs

TaskTracker MapReduce Slave Executes map/reduce tasks on each node

Hadoop Cluster Modes

1. Local (Standalone) Mode — Single machine, no HDFS, for testing and debugging
2. Pseudo-Distributed Mode — All Hadoop daemons on one machine, simulates a real
cluster
3. Fully Distributed Mode — Real multi-machine cluster (production use)

HDFS Read and Write Operations

Read: Client → asks NameNode for block locations → reads directly from nearest DataNode

Write: Client → informs NameNode → writes data to a pipeline of DataNodes → DataNodes

replicate automatically

UNIT 3: MapReduce ⭐⭐⭐

What is MapReduce?
MapReduce is a programming model that breaks a large job into small parallel tasks
distributed across many nodes.

🧒 5th Grader Example: Your teacher wants to count all words in 30 student essays.
Instead of doing it alone, she gives each student their own essay to count (MAP
phase), then she collects everyone's totals and adds them up (REDUCE phase). Done
in minutes instead of hours!

MapReduce Flow (Must Memorize!)

Input Data (stored in HDFS)

↓
[MAP Phase]
Converts data into Key-Value pairs
↓
[SHUFFLE & SORT]
Groups sam e keys together (done autom atically by Hadoop)
↓
[REDUCE Phase]
Aggregates values for each key
↓
Output Data (stored in HDFS)

Word Count Example:

Input: "hello world hello hadoop"

After MAP: (hello,1), (world,1), (hello,1), (hadoop,1)
After SHUFFLE: (hello,[1,1]), (world,[1]), (hadoop,[1])
After REDUCE: (hello,2), (world,1), (hadoop,1)

MapReduce Components
Component Role

Driver Code Sets up the job — input/output paths, Mapper/Reducer class

Mapper Takes raw input, emits key-value pairs

Reducer Takes grouped key-value pairs, produces final output

Combiner "Mini-Reducer" — locally aggregates Mapper output to reduce network traffic

Partitioner Decides which Reducer handles which key

RecordReader Reads input splits and converts to key-value pairs for Mapper
Combiner vs Partitioner ⭐ (Frequently Asked!)
Combiner Partitioner

Reduce data transferred between Map and

Purpose Route keys to the correct Reducer
Reduce

When it
After Map, before Shuffle During the Shuffle phase
runs

Send keys A–M to Reducer 1, N–Z to Reducer

Example Sum local word counts before sending to network
2

Weather Dataset — Classic MapReduce Example

Data: Weather records from NCDC (temperature, year, quality code)
Mapper: Extracts (year, temperature) pairs, filters invalid readings
Reducer: Finds maximum temperature per year
Output: (1950, 22°C), (1949, 111°C)...
Mapper Code:

public class MaxTem pMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

private static final int MISSING = 9999;
public void m ap(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = [Link]();
String year = [Link](15, 19);
int airTem p = [Link]([Link](87, 92));
String quality = [Link](92, 93);
if (airTem p != MISSING && quality.m atches("[01459]")) {
[Link](new Text(year), new IntWritable(airTem p));
}
}
}

Reducer Code:

public class MaxTem pReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int m axValue = Integer.MIN_VALUE;
for (IntWritable val : values) {
m axValue = Math.m ax(m axValue, [Link]());
}
[Link](key, new IntWritable(m axValue));
}
}
JobTracker vs TaskTracker
JobTracker TaskTracker

1 per cluster (Master) 1 per node (Slave)

Schedules and monitors jobs Executes individual Map/Reduce tasks

Single point of failure in MRv1 Reports task status to JobTracker

UNIT 4: Hadoop I/O — Writable Interface ⭐⭐

What is Serialization?
Serialization = Converting an object into bytes for network transmission or disk storage.
Deserialization = Converting bytes back into objects.

🧒 5th Grader Example: Serialization is like packing your toys in a box to mail them.
Deserialization is opening the box and getting your toys back exactly as they were!

The Writable Interface ⭐⭐⭐

Hadoop uses its own serialization format called Writables — faster and more compact than
Java's default serialization.

The interface has exactly two methods:

public interface Writable {

void write(DataOutput out) throws IOException; // serialize
void readFields(DataInput in) throws IOException; // deserialize
}

Writable Wrapper Classes (Table — Always in Exams!)

Java Primitive Hadoop Writable Size (bytes)

boolean BooleanWritable 1

byte ByteWritable 1

short ShortWritable 2

int IntWritable 4

int (variable) VIntWritable 1–5

float FloatWritable 4

long LongWritable 8

double DoubleWritable 8
 Java Primitive Hadoop Writable Size (bytes)

String Text Variable

byte[ ] BytesWritable Variable

null NullWritable 0

WritableComparable ⭐

- `WritableCom parable<T>` = `Writable` + `Com parable<T>`

MapReduce keys must be WritableComparable because they need to be sorted

RawCom parator — compares byte streams without deserializing → huge performance gain
WritableCom parator — default general-purpose implementation of RawCom parator

Custom Writable Implementation ⭐⭐ (Asked in Every Year!)

Use when built-in types are not enough. Example: TextPair (stores two strings).

Steps:

1. Implement WritableCom parable<T>

2. Provide a default (no-arg) constructor — required by MapReduce framework
3. Override write() and readFields()
4. Override hashCode(), equals(), toString(), com pareTo()

public class TextPair im plem ents WritableCom parable<TextPair> {

private Text first;
private Text second;

public TextPair() { set(new Text(), new Text()); }

public TextPair(String first, String second) {
set(new Text(first), new Text(second));
}

public void write(DataOutput out) throws IOException {

[Link](out);
[Link](out);
}

public void readFields(DataInput in) throws IOException {

[Link](in);
[Link](in);
}

public int com pareTo(TextPair tp) {

int cm p = [Link] pareTo([Link]);
if (cm p != 0) return cm p;
 return [Link] pareTo([Link]);
}
}

Custom Comparators (For Speed!)

Implement RawCom parator to compare serialized byte arrays directly — avoids the overhead of
creating objects during the sort phase of MapReduce.

public int com pare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
// Com pare bytes directly without deserializing
}

UNIT 5: Apache Pig ⭐⭐

What is Pig?
Apache Pig is a high-level platform for processing Big Data that hides the complexity of
writing Java MapReduce programs. It was developed at Yahoo! Research in 2006 and became
an Apache project in 2007.

🧒 5th Grader Example: Writing MapReduce is like building LEGO without instructions —
hard! Pig Latin is the instruction manual — you say WHAT you want done, and Pig
figures out HOW to do it with MapReduce underneath!

Pig Architecture (2 Main Parts)

1. Pig Latin — The high-level scripting language you write
2. Pig Latin Compiler — Converts Pig Latin scripts into MapReduce jobs automatically
Pig programs run on MapReduce v1, MapReduce v2 (YARN), or Apache Tez without code
changes.

Pig Latin Application Flow

LOAD ← Read data from HDFS

↓
TRANSFORM ← FILTER, GROUP, JOIN, ORDER, FOREACH...
↓
DUMP or STORE ← Output to screen or save to HDFS

Sample Pig Script:

 A = LOAD '[Link]' USING PigStorage(',')
AS (nam e:chararray, grade:int);
B = FILTER A BY grade > 60;
C = ORDER B BY grade DESC;
DUMP C;

⚡ Pig uses lazy evaluation — no data moves until a DUMP or STORE is encountered!

Pig Data Types — ABCs of Pig Latin ⭐⭐ (Asked Every Year!)

Type Description Example

Atom Single scalar value "Diego" , 42

Tuple Record with ordered fields (like a table row) (Diego, Gom ez, 6)

Bag Collection of tuples (non-unique allowed) {(Diego,6), (Maria,8)}

Map Collection of key-value pairs [nam e#Diego, age#6]

Pig Latin Operators

Operator Purpose

LOAD Read data from file system

STORE Write results to file system

DUMP Print results to screen

FILTER Remove rows not matching condition

GROUP Group records by a key

JOIN Join two datasets

ORDER Sort records by a field

DISTINCT Remove duplicate records

FOREACH Apply expression to each record

LIMIT Restrict number of output records

DESCRIBE Show schema of a relation (debugging)

EXPLAIN Show MapReduce execution plan

Pig Execution Modes
Mode Command Use

Local Mode pig -x local Single machine, local filesystem, for testing

Distributed Mode pig or pig -x m apreduce Full Hadoop cluster, production

Pig Script Interfaces

1. Grunt Shell — Interactive command-line shell for Pig
2. Script Files — Write .pig script file and execute it
3. Embedded — Embed Pig code inside Java programs using PigServer API

Pig vs MapReduce
Feature Pig MapReduce

Language level High-level Pig Latin Low-level Java

Lines of code ~10 lines ~100 lines

Ease of learning Easy for non-Java users Requires Java expertise

Data flexibility Handles any data type Fixed key-value model

Debugging DUMP, DESCRIBE, EXPLAIN Complex Java debugging

UNIT 6: Apache Hive ⭐⭐⭐

What is Hive?
Apache Hive is a data warehouse tool built on top of Hadoop that allows querying Big Data
using an SQL-like language called HiveQL (HQL). It was created at Facebook by Jeff
Hammerbacher. HiveQL queries are automatically converted into MapReduce jobs.

🧒 5th Grader Example: Hive is like having a SEARCH ENGINE for your huge messy
room (HDFS). Instead of digging through everything yourself, you just ask "Where are
my red shoes?" in plain language, and it finds them!

Hive Architecture ⭐⭐⭐ (Draw in Exam!)

[User / Application]
↓
[Hive Clients] ← CLI, Web UI, JDBC, ODBC, Thrift Server
↓
[Hive Driver] ← Com piler → Optim izer → Executor
↓
 [Metastore] ← Stores table schem as, m etadata (Derby / MySQL)
↓
[MapReduce / YARN] ← Executes the actual job
↓
[HDFS] ← Actual data stored here

Hive Clients (3 Types)

1. Thrift Server — Allows non-Java clients to connect to Hive
2. JDBC — Java Database Connectivity interface
3. ODBC — Open Database Connectivity interface

Hive Data Types

Primitive Types:

Type Description Example

TINYINT 1-byte integer 1

SMALLINT 2-byte integer 100

INT 4-byte integer 1000

BIGINT 8-byte integer 9999999

FLOAT Single-precision float 3.14

DOUBLE Double-precision float 3.14159

STRING Character string 'hello'

BOOLEAN True/false value TRUE

TIMESTAMP Date and time 2024-01-01 10:00:00

Complex Types:

Type Example

ARRAY ARRAY<STRING>

MAP MAP<STRING, INT>

STRUCT STRUCT<nam e:STRING, age:INT>

HiveQL — DDL Commands (Database and Table Management)

-- Create and use a database

CREATE DATABASE m ydb;
USE m ydb;
DROP DATABASE m ydb;

-- Create a table
CREATE TABLE students (
id INT,
nam e STRING,
m arks FLOAT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE;

-- Alter table (add colum n)

ALTER TABLE students ADD COLUMNS (city STRING);

-- Drop table
DROP TABLE students;

HiveQL — DML Commands (Data Operations)

-- Load data from local file

LOAD DATA LOCAL INPATH '/hom e/data/[Link]'
INTO TABLE students;

-- Load data from HDFS

LOAD DATA INPATH '/hdfs/data/[Link]'
INTO TABLE students;

-- Select with filter

SELECT nam e, m arks FROM students WHERE m arks > 60;

-- Group By with aggregate

SELECT city, COUNT(*) AS total FROM students GROUP BY city;

-- Order By
SELECT * FROM students ORDER BY m arks DESC;

-- Join two tables

SELECT [Link] e, [Link]
FROM students a JOIN enrollm ent b ON ([Link] = b.student_id);

-- Lim it output
SELECT * FROM students LIMIT 10;
Schema on Read vs Schema on Write
Hive (Schema on Read) RDBMS (Schema on Write)

Schema applied when reading data Schema enforced when loading data

NULL shown if data doesn't match Error thrown if data doesn't match

Flexible and fast to load Strict but consistent

Ideal for exploratory analysis Ideal for OLTP applications

Hive vs Pig ⭐⭐ (Asked Every Single Year!)

Feature Hive Pig

Language HiveQL (SQL-like, declarative) Pig Latin (dataflow, procedural)

Target Users SQL developers and analysts Programmers and data engineers

Data Types Structured data best All types including unstructured

Created By Facebook Yahoo

Best For Data warehouse queries ETL pipelines and transformations

Schema Schema on Read Schema on Read

Hive vs Traditional RDBMS

Hive RDBMS

Schema on Read Schema on Write

No ACID transactions Full ACID transactions

High-latency batch queries Low-latency queries

Handles petabytes Handles GB to TB

Runs on commodity hardware Requires high-end servers

Best for analytics/reporting Best for OLTP

Quick Revision — Definitions Cheatsheet

Term One-Line Definition

Big Data Data too large/complex for traditional tools

3 V's Volume, Velocity, Variety

Hadoop Open-source distributed computing framework

HDFS Hadoop's distributed file system

 Term One-Line Definition

NameNode HDFS master — stores metadata

DataNode HDFS slave — stores actual data blocks

MapReduce Parallel data processing model (Map + Shuffle + Reduce)

Mapper Converts input to key-value pairs

Reducer Aggregates key-value pairs to final output

Combiner Mini-reducer to cut network traffic

Partitioner Routes keys to correct reducer

Writable Hadoop's serialization interface

WritableComparable Writable + Comparable (used for keys)

RawComparator Compares bytes directly without deserializing

Pig High-level data processing platform on Hadoop

Pig Latin Pig's scripting language

Hive SQL-like querying layer on Hadoop

HiveQL Hive's query language (SQL dialect)

Metastore Hive's metadata/schema storage

GFS Google File System — inspiration for HDFS

YARN Yet Another Resource Negotiator (resource manager)

Serialization Object → bytes (for transmission or storage)

🔥 2026 Guess Paper — High Probability Questions

PART A — Expected (2 marks each)

1. What are the 3 V's of Big Data? Give examples.
2. Differentiate NameNode and DataNode.
3. What is MapReduce?
4. Differentiate Combiner and Partitioner.
5. What is the Writable Interface in Hadoop?
6. Define WritableComparable.
7. What is Pig Latin?
8. List Pig Script interfaces.
9. What are Hive clients? List different types.
10. Define Hive data types / difference between Hive and Pig.
PART B — Expected (4–8 marks each)
1. Explain HDFS architecture with a neat diagram.
2. Explain MapReduce workflow with Word Count example.
3. Explain Writable class hierarchy in Hadoop I/O.
4. Explain Pig architecture and application flow (ABCs of Pig Latin).
5. Explain the role of Combiner in MapReduce with example.
6. Write HiveQL queries to create a table, load data, filter, and group data.
7. Explain the main features of Big Data.

PART C — Expected (10–15 marks each)

1. Explain Hadoop building blocks (NameNode, DataNode, JobTracker, TaskTracker) with
block diagram.
2. Explain HDFS architecture with neat diagram and HDFS read/write operations.
3. Explain MapReduce with Weather Dataset (Mapper + Reducer code included).
4. Explain Custom Writable implementation with complete TextPair example.
5. Explain Hive architecture with diagram + HiveQL queries for data analysis.

Exam Tips
Always draw diagrams for HDFS architecture, Hive architecture, MapReduce flow — adds
significant marks.
Word Count problem is the hello-world of MapReduce — know Mapper and Reducer
code cold.
Hive vs Pig comparison table appears every year — memorize it.
3 V's always comes in Part A — never skip it.
For Part C answers: write a definition → draw a diagram → explain each component (2–3
lines each) → give a practical example. This structure alone can score 12/15.
Mention fault tolerance and replication factor (3) whenever discussing HDFS.
For Pig questions, always mention the ABCs: Atom, Tuple, Bag, Map — guaranteed marks.
For Hive questions, include at least one HiveQL query — even in 8-mark answers.

Notes prepared from RTU BDA syllabus and past papers (2022–2025). Best of luck, Aditya! 🎯