CBCPC403

Big Data Analytics

[Link] 4th Semester

Complete Exam Notes

Mid-Semester & End-Semester PYQs

All Units | Diagrams | Key Points

Mid-Semester Time: 1.5 Hr Max Marks: 20 March 2025

End-Semester Time: 3 Hr Max Marks: 50 April-May 2025

PYQ Topics Covered in This PDF

■ Define Big Data & Evolution of Databases
■ 5 Vs of Big Data
■ Real-Time vs Offline Big Data Analysis
■ Big Data Analytics – Key Technologies
■ Examples where Big Data Analytics is used
■ NoSQL vs SQL – Difference & Types
■ Hadoop & HDFS – Architecture & MapReduce
■ Hive (HiveQL), Pig (Pig Latin, Pig Data Model)
■ Healthcare Data & Analytics Types
■ Depression Detection Lexicon
■ Social Media Hate Speech Detection
■ Event Prediction using Big Data
 UNIT I
Introduction to Big Data

★ PYQ: MID Q1a: Define Big Data and explain evolution of databases
★ PYQ: END Q1a: Define Big Data and Explain Characteristics with suitable example

1.1 What is Big Data?

Big Data refers to extremely large datasets that cannot be processed by traditional data
management tools due to their volume, variety, and velocity. It includes structured,
semi-structured, and unstructured data generated from diverse sources like social media, IoT
devices, sensors, web logs, and transactions.

Key Characteristics — The 5 Vs:

Value

Volume Variety

BIG DATA
5 Vs

Velocity Veracity

Figure 1: The 5 Vs of Big Data

Volume Enormous scale of data (terabytes to zettabytes). Ex:

Facebook generates 4 PB/day.

Velocity Speed at which data is generated & processed. Ex:

Twitter – 500M tweets/day.

Variety Different types: structured (tables), semi-structured

(JSON/XML), unstructured (video, images).

Veracity Quality & trustworthiness of data. Handling noise, bias,

and inconsistency.

Value Extracting useful insights from data. The ultimate goal of

Big Data processing.

1.2 Evolution of Databases

 1960s 1970s 1990s 2000s 2010s+
File Relational Data NoSQL Big Data
Systems DB Warehouses DBs & Cloud

Figure 2: Evolution Timeline of Databases

1960s – File Systems Flat files; no relationships; data redundancy.

1970s – Relational DB SQL, tables, keys (Oracle, MySQL). ACID properties.

1990s – Data Warehouses OLAP, analytical reporting, star/snowflake schema.

2000s – NoSQL DBs Flexible schema, horizontal scaling (MongoDB,

Cassandra).

2010s+ – Big Data & Cloud Hadoop, Spark, cloud storage (AWS S3, Azure), ML
pipelines.

★ PYQ: MID Q2a: Explain difference between Real-Time and Offline Big Data Analysis
★ PYQ: END Q1b: Differentiate between Real-Time and Offline Big Data Analytics architectures

1.3 Real-Time vs Offline Big Data Analysis

Real-Time Analysis Offline Analysis

• Low latency (ms-sec) • High latency (hours/days)

• Stream processing • Batch processing

VS
• Kafka, Spark Streaming • Hadoop MapReduce

• Fraud detection, IoT • Reports, ML training

Figure 3: Real-Time vs Offline Analysis

Feature Real-Time Analysis Offline (Batch) Analysis

Latency Milliseconds – Seconds Hours – Days

Processing Stream processing Batch processing

Tools Spark Streaming, Kafka, Flink Hadoop MapReduce, Hive

Data Size Smaller windows Full historical dataset

Use Cases Fraud detection, IoT, Stock trading Reports, ML training, ETL

Cost Higher (real-time infra) Lower (scheduled jobs)

★ PYQ: END Q1c: What is time series analysis and how is it used to forecast low dimensional numerical data?
1.4 Time Series Analysis
Time Series Analysis involves analyzing data points collected over time intervals to identify
patterns, trends, seasonality, and cycles. It is used to forecast future values of low-dimensional
numerical data.

Key Components:
• Trend: Long-term increase or decrease in data (e.g., rising sales over years).
• Seasonality: Regular, repeating patterns (e.g., holiday sales spikes).
• Cyclicity: Long irregular fluctuations (e.g., economic cycles).
• Noise/Residuals: Random variation after removing trend and seasonality.
Common methods: ARIMA, Exponential Smoothing, LSTM (Deep Learning), Prophet (Facebook).
★ PYQ: MID Q2b: Discuss key technologies involved in Big Data processing
★ PYQ: MID Q2c: Provide examples where Big Data Analytics is used

1.5 Key Technologies in Big Data Processing

Technology Purpose Examples

Hadoop Distributed storage & batch processing HDFS, MapReduce, YARN

Spark In-memory fast processing Spark SQL, MLlib, Streaming

Kafka Distributed messaging/streaming Real-time event streaming

Hive SQL on Hadoop HiveQL queries on HDFS

Pig Data flow scripting on Hadoop ETL using Pig Latin

NoSQL DBs Flexible large-scale storage MongoDB, Cassandra, HBase

Cloud Storage Scalable distributed storage AWS S3, Google GCS, Azure

Real-World Examples of Big Data Analytics:

Healthcare Patient monitoring, disease outbreak detection, drug
discovery.

Finance Fraud detection, algorithmic trading, credit scoring.

Retail Recommendation systems (Amazon), customer

segmentation.

Social Media Sentiment analysis, hate speech detection, trend

analysis.

Transport Route optimization (Google Maps), predictive

maintenance.

Government Smart city planning, election analytics, civil unrest

prediction.
 UNIT II
NoSQL Big Data Management & Big Data Tools

★ PYQ: MID Q3a: Explain difference between SQL and NoSQL databases
★ PYQ: MID Q3b: What are different types of NoSQL databases? Provide examples
★ PYQ: END Q2a: How does NoSQL help in managing Big Data efficiently?

2.1 SQL vs NoSQL – Key Differences

Feature SQL NoSQL

Schema Fixed schema Dynamic/flexible schema

Scalability Vertical scaling Horizontal scaling

Data Model Tables with rows & cols Key-value, Document, Column, Graph

ACID Full ACID compliance Eventual consistency (BASE)

Query Language SQL API-based / query language varies

Best For Structured data, OLTP Big Data, unstructured, distributed

Examples MySQL, Oracle, PostgreSQL MongoDB, Cassandra, Redis, HBase

2.2 Types of NoSQL Databases

Key-Value Document Column Graph

Store Store Family Database

Examples: Examples: Examples: Examples:

Redis MongoDB Cassandra Neo4j
DynamoDB CouchDB HBase Amazon
Neptune

Figure 4: Four Types of NoSQL Databases

Key-Value Store Simplest model. Data stored as key-value pairs. Fast lookups. Ex:
Redis, DynamoDB, Memcached. Used for: session management,
caching.

Document Store Stores semi-structured documents (JSON/BSON). Flexible schema.

Ex: MongoDB, CouchDB. Used for: content management, catalogs.

Column Family Stores data in columns instead of rows. Efficient for sparse data. Ex:
Cassandra, HBase. Used for: IoT, time-series.

Graph Database Stores entities (nodes) and relationships (edges). Ex: Neo4j, Amazon
Neptune. Used for: social networks, fraud detection.

★ PYQ: MID Q4b: Define Hadoop and explain its significance in Big Data ecosystem
★ PYQ: MID Q5b: What is HDFS? How does it work in context of Hadoop?
★ PYQ: MID Q5a: What is MapReduce? Explain with context of Hadoop
2.3 Hadoop Ecosystem

Applications / Data Ingestion (Flume, Kafka, Zookeeper)

Hive Pig HBase Spark Sqoop

MapReduce (Processing Engine) YARN (Resource Manager)

HDFS (Distributed File System)

Figure 5: Hadoop Ecosystem Architecture

Hadoop is an open-source framework (Apache) for distributed storage and processing of large
datasets across clusters of computers using simple programming models. It is designed to
scale from a single server to thousands of machines.

HDFS – Hadoop Distributed File System:

• Splits files into blocks (default 128 MB) and distributes across DataNodes.
• NameNode: Master node – stores metadata (file names, block locations).
• DataNode: Stores actual data blocks. Multiple replicas (default 3) for fault tolerance.
• Follows Write-Once-Read-Many (WORM) model.
• Designed for commodity hardware – failures are expected and handled automatically.

MapReduce – Processing Engine:

Input Map Shuffle Reduce Output

Data Phase & Sort Phase Data

Figure 6: MapReduce Processing Phases

Input Splitting Input data is divided into fixed-size chunks called

InputSplits.

Map Phase Map function processes each key-value pair →

intermediate key-value pairs.

Shuffle & Sort Framework groups all values with the same key
together and sorts them.
 Reduce Phase Reduce function aggregates/summarizes the grouped
intermediate values.

Output Final output written back to HDFS.

Example: Word Count — Map emits (word,1) for each word; Shuffle groups by word; Reduce
sums counts → (word, total_count).

★ PYQ: END Q2b: Write short notes on HiveQL queries, Pig data model, Pig Latin and Pig Latin scripts

2.4 Apache Hive & HiveQL

Apache Hive is a data warehouse system built on top of Hadoop. It provides an SQL-like
interface called HiveQL (HQL) to query data stored in HDFS without writing MapReduce
programs.

HiveQL Examples:
CREATE TABLE sales (id INT, product STRING, amount FLOAT) ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','; -- Load data LOAD DATA INPATH '/data/[Link]' INTO TABLE
sales; -- Query SELECT product, SUM(amount) AS total FROM sales GROUP BY product;

Hive Architecture Hive Client → Hive Server → Driver → Compiler → Metastore →

Execution Engine → HDFS

Hive vs SQL HiveQL is similar to SQL but runs on MapReduce (or Tez/Spark).
No row-level updates.

When to use Hive Large-scale ETL, batch analytics, ad-hoc queries on HDFS data.

2.5 Apache Pig & Pig Latin

Apache Pig is a high-level platform for creating MapReduce programs using a scripting
language called Pig Latin. It abstracts the complexity of MapReduce.

Pig Latin Processing Pipeline:

LOAD FILTER GROUP FOREACH STORE

data from remove group by GENERATE save to
HDFS unwanted rows key transform HDFS

Figure 7: Pig Latin Data Processing Pipeline

Pig Data Model:

Atom Smallest unit (single value). Ex: 'John', 42

Tuple Ordered set of fields. Like a row. Ex: (John, 25, Delhi)
 Bag Collection of tuples. Ex: {(John,25),(Mary,30)}

Map Set of key-value pairs. Ex: [name#John, age#25]

Pig Latin Script Example – MapReduce Word Count:

-- Load data from HDFS words = LOAD '/input/[Link]' AS (line:chararray); -- Tokenize each
line into words tokens = FOREACH words GENERATE FLATTEN(TOKENIZE(line)) AS word;
-- Group all occurrences of each word grpd = GROUP tokens BY word; -- Count occurrences
word_count = FOREACH grpd GENERATE group AS word, COUNT(tokens) AS count; -- Store
result back to HDFS STORE word_count INTO '/output/wordcount';

MapReduce Shuffle, Sort, Task Execution in Pig:

• Shuffle: After Map phase, all records with same key are moved to the same reducer.
• Sort: Records are sorted by key within each reducer partition.
• Task Execution: Each task (map/reduce) runs in a separate JVM on the cluster node.
 UNIT III
Big Data and Healthcare

★ PYQ: END Q3a: What is lexicon for depression detection and how is it utilized in analytics?
★ PYQ: END Q3b: Differentiate between following types of analytics with examples (Descriptive, Diagnostic,
Predictive, Prescriptive)

3.1 Types of Analytics (Pyramid Model)

Prescriptive Analytics
What should we do?
Predictive Analytics
What will happen?
Diagnostic Analytics
Why did it happen?
More Descriptive Analytics
Value What happened?

Figure 8: Analytics Hierarchy – Value Pyramid

Descriptive Summarizes past data. Answers: 'What happened?' Tools: Reports,

Analytics dashboards, BI tools. Ex: Monthly sales report, patient admission
counts.

Diagnostic Examines causes of past outcomes. Answers: 'Why did it happen?'

Analytics Techniques: Drill-down, data mining, correlation. Ex: Why did patient
readmissions increase last month?

Predictive Analytics Forecasts future outcomes. Answers: 'What will happen?' Techniques:
ML, regression, time series. Ex: Predict disease outbreak, stock price
forecasting.

Prescriptive Recommends actions. Answers: 'What should we do?' Techniques:

Analytics Optimization, AI, simulation. Ex: Recommend best treatment plan for a
patient.

3.2 Depression Detection using Big Data

Automatic Depression Detection uses machine learning and NLP on large datasets (social
media posts, clinical notes, speech patterns) to identify individuals at risk of depression.

Lexicon for Depression Detection:

A Lexicon is a dictionary/word list where each word is associated with a sentiment score or
depression indicator. Words like 'hopeless', 'worthless', 'alone' carry high depression scores,
while positive words carry low/negative scores.

• Step 1 – Data Collection: Social media posts (Twitter, Reddit), EHR data, speech audio.
• Step 2 – Preprocessing: Tokenization, stop-word removal, stemming/lemmatization.
 • Step 3 – Lexicon Application: Map words to depression scores using lexicons (LIWC, ANEW,
custom).
• Step 4 – Feature Extraction: Frequency of depressive words, sentiment polarity, behavioral
patterns.
• Step 5 – Classification: SVM, Random Forest, or Deep Learning (LSTM) to classify depression
level.
• Step 6 – Output: Risk score → alert clinicians or recommend counseling.

How Lexicon is utilized in Analytics:

LIWC Lexicon Linguistic Inquiry and Word Count – maps words to psychological
categories.

ANEW Affective Norms for English Words – rates words on valence, arousal,
dominance.

Custom Lexicon Domain-specific word lists built from clinical data for depression
keywords.

Usage Lexicon scores are used as features in ML models to classify

posts/texts as depressive or not.
 UNIT IV
Social Media Data Analytics – Hate Speech

★ PYQ: END Q4a: What is the significance of social media data in detecting hate speech?
★ PYQ: END Q4b: Describe the classification guidelines used for identifying hate speech in social media.
★ PYQ: END Q4c: Discuss the mining techniques used for detecting hate speech in social media data.

4.1 Hate Speech Detection System

Social Media Classification
Data Collection Model

Hate Speech Detection

NLP Action /
Processing Alert
Figure 9: Hate Speech Detection Pipeline

4.2 Categories of Hate Posts on Social Media

Racism / Ethnicity Hate based on race, skin color, national origin.

Religion Targeting religious beliefs or communities.

Gender / Sexism Discriminatory language based on gender or sexual

orientation.

Political Hate Extreme political ideologies promoting violence.

Cyberbullying Targeted harassment of individuals online.

4.3 Classification Guidelines

• Explicit Hate: Direct offensive language, slurs, explicit threats.
• Implicit Hate: Coded language, dog whistles, sarcasm used to spread hate.
• Context-Aware: Same word may be hate in one context but not another (reclaimed slurs).
• Platform Guidelines: Twitter Rules, Facebook Community Standards define hate categories.
• Annotation Standards: Human annotators label data using inter-rater reliability measures.

4.4 Mining Techniques for Hate Speech Detection

Keyword-based Simple lexicon matching. Fast but many false positives (low
precision).

TF-IDF + ML Convert text to numerical features; train classifiers (SVM, Naive

Bayes, Logistic Regression).

Word Embeddings Word2Vec, GloVe – semantic representations capture context better

than keywords.
 Deep Learning LSTM, BERT, RoBERTa – capture long-range context.
State-of-the-art accuracy.

Graph Mining Analyze social networks – hate spreaders form clusters. Community
detection.

Multimodal Combine text + images + metadata (user history) for richer detection.

4.5 Hate Content in Business – Related Notions

Businesses must comply with content moderation laws. Hate speech affects brand reputation,
advertiser safety, and platform user retention. Lexicon for Hate Speech is similar to
depression lexicons — a curated word list with offensiveness scores used to flag posts
automatically.
 UNIT V
Event Prediction in Big Data Era

★ PYQ: END Q5a: Describe the process of event detection and prediction using Big Data techniques.
★ PYQ: END Q5b: Why is event prediction important in today's data-driven world?
★ PYQ: END Q5c: Discuss the role of Big Data in predicting civil unrest events.
★ PYQ: END Q5d: What techniques are commonly used for forecasting future events with Big Data?

5.1 Event Prediction Pipeline

Data Feature ML Model Event Civil Unrest

Collection Engineering Training Prediction Alert

Figure 10: Big Data Event Prediction Pipeline

5.2 Types of Events in Big Data

Physical Events Natural disasters, accidents, earthquakes (sensor data).

Social Events Protests, elections, festivals (social media data).

Cyber Events DDoS attacks, data breaches (network logs).

Economic Events Stock crashes, inflation spikes (financial data).

Civil Unrest Riots, strikes, political demonstrations (Twitter, news

feeds).

5.3 Event Detection & Prediction Techniques

Burst Detection Identifies sudden spikes in topic frequency (e.g., keyword
'protest' trending).

Sentiment Analysis Negative sentiment surge in region → potential civil unrest

indicator.

Topic Modeling (LDA) Latent Dirichlet Allocation discovers emerging topics in large text
corpora.

Machine Learning Random Forest, SVM, Neural Networks trained on historical

event data.

Deep Learning (LSTM) Captures temporal patterns in sequential social media data.

Graph Analysis Track information spread and influencer networks to detect

coordination.
 Geo-spatial Analysis Cluster events by location; heatmaps show event concentration
zones.

5.4 Civil Unrest Event Prediction

Civil Unrest refers to protests, riots, and political demonstrations. Big Data enables
governments and researchers to predict such events days in advance by analyzing social
media, news, and economic indicators.

• Data Sources: Twitter, Facebook, news articles, economic reports, historical event databases
(ACLED, GDELT).
• Features used: Tweet volume by location, hashtag trends, sentiment polarity, unemployment rate,
political events.
• EMBERS System: Early Model-Based Event Recognition using Surrogates – a famous civil unrest
prediction system.
• Prediction Horizon: Systems can predict events 1-7 days in advance with reasonable accuracy.
• Challenges: Data noise, rapidly changing situations, misinformation, language/dialect variation.

5.5 Why Event Prediction is Important Today

Public Safety Law enforcement can allocate resources proactively
before an event escalates.

Disaster Response Early warning systems save lives in natural disasters.

Business Companies can adjust supply chains, pricing, and

logistics before disruptions.

Politics Governments monitor public sentiment to address

grievances before escalation.

Media News organizations can prepare coverage for likely

upcoming events.

■ Quick Revision – Key Terms Glossary

Term One-line Definition

HDFS Hadoop Distributed File System – stores data in 128MB blocks across cluster nodes.

MapReduce Programming model: Map → Shuffle/Sort → Reduce for parallel data processing.

HiveQL SQL-like query language to query data stored in HDFS using Hadoop.

Pig Latin Scripting language for ETL data flows; compiled to MapReduce jobs.

NoSQL Non-relational databases for flexible, scalable storage of unstructured data.

YARN Yet Another Resource Negotiator – cluster resource manager in Hadoop 2.x.

Spark In-memory fast processing engine; 100x faster than MapReduce for iterative jobs.

5 Vs Volume, Velocity, Variety, Veracity, Value – defining characteristics of Big Data.

Lexicon Word list with sentiment/emotion scores used in NLP-based analytics.

LDA Latent Dirichlet Allocation – topic modeling algorithm for text corpora.
EMBERS Civil unrest prediction system using social media surrogates.

LIWC Linguistic Inquiry and Word Count – psychological word category lexicon.

ACLED Armed Conflict Location & Event Data – historical civil unrest database.

BERT Bidirectional Encoder Representations from Transformers – NLP deep learning model.

■ Exam Tips: For 2-mark questions, give a crisp definition + one example. For 5-mark
questions, include a definition, 3-4 key points, a comparison table or diagram, and a real-world
use case. Always draw diagrams for Hadoop ecosystem, MapReduce, 5Vs, and Analytics
Pyramid.