BIG DATA ANALYTICS

COMPREHENSIVE TEXTBOOK NOTES (EXAM ORIENTED)

Module 1: Introduction to Big Data

Page 1 of 15
 Q2. Explain the evolution of Big Data from traditional data processing
systems to modern data platforms.

The evolution of data processing systems into modern Big Data platforms occurred over
several decades, driven by the exponential growth in data volume, velocity, and variety. This
evolution can be categorized into three distinct phases:

Phase 1: Traditional Database Management Systems (1970s - 1990s)

In the early days, data processing was synonymous with Relational Database Management
Systems (RDBMS) like Oracle, IBM DB2, and SQL Server.

• Focus: Day-to-day transactional processing (OLTP) and maintaining ACID (Atomicity,

Consistency, Isolation, Durability) properties.

• Data Type: Strictly structured data organized into rigid tables with rows and columns.

• Scale: Data volumes were relatively small, ranging from Megabytes to early Gigabytes.

• Limitation: These systems relied on a centralized architecture. To scale, organizations had

to purchase larger, highly expensive mainframes (Vertical Scaling / Scale-up), which
eventually hit physical and financial limits.

Phase 2: Business Intelligence and Data Warehousing (1990s - Late 2000s)

As businesses realized the value of historical data for decision-making, Data Warehousing
emerged.

• Focus: Online Analytical Processing (OLAP), generating complex reports, and finding
historical trends.

• Architecture: Involved complex ETL (Extract, Transform, Load) pipelines. Data from
various RDBMS was extracted, rigorously cleaned/transformed into a predefined schema,
and loaded into an Enterprise Data Warehouse (EDW).

• Limitation: Relied heavily on "Schema-on-Write" (data must be structured before storage).

As the internet boomed, unstructured data (emails, logs, images) began to dominate,
which traditional ETL and EDW systems could not handle efficiently or cost-effectively.

Phase 3: Modern Big Data Platforms (2010s - Present)

The advent of web 2.0, social media, and IoT generated data at an unprecedented scale,
necessitating a paradigm shift.

• Architecture: Shifted from centralized to distributed systems. Frameworks like Apache

Hadoop and Apache Spark process data across clusters of thousands of commodity
(standard, cheap) computers.

• Scaling: Horizontal Scaling (Scale-out). Need more power? Simply add more cheap servers
to the cluster.

Page 2 of 15
• Processing Paradigm: Embraced "Schema-on-Read." Data is dumped into Data Lakes (like
HDFS or Amazon S3) in its raw, unstructured format. Structure is only applied when the
data is queried, allowing for immense flexibility.

• Technology: Introduction of NoSQL databases (MongoDB, Cassandra) to handle massive

variety and velocity without the strict constraints of relational tables.

Page 3 of 15
 Q5. Compare Traditional Business Intelligence (BI) with Big Data in
terms of architecture, processing, and scalability.

Traditional Business Intelligence (BI) and Big Data Analytics represent two fundamentally
different approaches to handling and analyzing data. The primary distinctions lie in how they
scale, the types of data they handle, and their underlying architectures.

Comparison Traditional Business Intelligence

Big Data Analytics
Parameter (BI)

Distributed architecture. Relies on

Centralized architecture. Relies on
clusters of commodity hardware
Architecture large, monolithic servers and
processing data in parallel (e.g.,
Enterprise Data Warehouses (EDW).
Hadoop).

Scale-Up (Vertical Scaling):

Scale-Out (Horizontal Scaling):
Increasing capacity requires buying
Increasing capacity simply involves
larger, more expensive hardware
Scalability adding more standard, inexpensive
components (CPU, RAM) for a single
nodes/servers to the existing cluster.
machine. Highly cost-prohibitive at
Highly cost-effective.
large scales.

Schema-on-Read: Data is ingested

Schema-on-Write: Data must be
and stored in its raw, native format
strictly validated, cleaned, and
Processing in a Data Lake. Structure and
transformed to fit a predefined
Approach schema are applied dynamically
tabular schema before it can be
only when the data is queried for
loaded into the database.
analysis.

Structured, Semi-structured (JSON,

Highly structured data originating
Data Types XML), and Unstructured (Logs, Text,
from internal business applications
Handled Images, Sensor Data, Social Media
(CRM, ERP, Transactional DBs).
feeds).

Predictive and Prescriptive

Descriptive and Diagnostic analytics.
analytics. Focuses on forecasting the
Analytics Focuses on looking backward (What
future (What will happen?) and
Focus happened? Why did it happen?) using
automating decisions using Machine
historical dashboards.
Learning.

Page 4 of 15
 Q6. Explain the architecture of a typical data warehouse environment
and compare it with a Hadoop environment.

1. Typical Data Warehouse Architecture

A Data Warehouse is a centralized repository of integrated data from one or more disparate
sources, used for reporting and data analysis.

• Data Sources: Operational systems (ERP, CRM), flat files, legacy systems.

• ETL Layer (Extract, Transform, Load): The most resource-intensive phase. Data is
extracted from sources, cleansed, transformed into a standard format, and loaded into the
warehouse.

• Storage Layer: The Enterprise Data Warehouse (EDW) stores the historical, structured
data.

• Data Marts: Sub-sections of the EDW designed for specific departments (e.g., Sales Data
Mart, HR Data Mart).

• BI Tools: Front-end applications used by analysts to generate reports and dashboards.

2. Hadoop Environment Architecture

Hadoop is an open-source framework that allows for the distributed processing of large data
sets across clusters of computers using simple programming models.

• Data Ingestion: Tools like Apache Flume (for streaming logs) and Apache Sqoop (for
relational data) ingest raw data into the system.

• Storage Layer (HDFS): The Hadoop Distributed File System splits massive files into
smaller blocks (e.g., 128 MB) and distributes them across multiple DataNodes, replicating
them for fault tolerance.

• Resource Management (YARN): Yet Another Resource Negotiator manages computing

resources in clusters and uses them for scheduling users' applications.

• Processing Layer (MapReduce/Spark): Frameworks that process the data locally on the
nodes where the data resides, minimizing network congestion.

[Reference Note for Exam Diagram]

Draw two side-by-side block diagrams. Left side: "Data Sources → ETL Engine → Data
Warehouse → BI Tools". Right side: "Unstructured Data → Hadoop Cluster (HDFS + YARN) →
Analytics".

Page 5 of 15
 Q7. How can Big Data systems coexist with traditional data warehouses?
Explain with examples.

A common misconception is that Hadoop and Big Data platforms completely replace
traditional Enterprise Data Warehouses (EDWs). In reality, in a modern enterprise
architecture, they coexist synergistically. Big Data systems handle the heavy lifting of raw,
massive data, while EDWs handle high-performance, structured business reporting.

Key Methods of Coexistence:

1. ETL Offloading:

◦ The Problem: ETL processing on a traditional EDW is highly expensive in terms of

compute resources. If the EDW is spending 80% of its processing power transforming
data, it cannot serve queries quickly.

◦ The Solution: The Hadoop cluster acts as a pre-processing engine. Raw data is dumped
into Hadoop. Hadoop uses its cheap, distributed compute power to clean, filter, and
aggregate the data. Only the finalized, structured, high-value data is then sent to the
EDW.

2. Data Archiving (Cold Storage):

◦ The Problem: EDW storage is expensive (e.g., thousands of dollars per Terabyte).
Storing 10-year-old historical data in an EDW is not cost-effective.

◦ The Solution: Historical, infrequently accessed "cold" data is moved from the EDW into
Hadoop (HDFS), which runs on cheap commodity disks. The data remains queryable
via tools like Hive, but frees up premium space on the EDW.

3. Exploratory Data Science vs. Operational Reporting:

◦ Data Scientists use the Hadoop Data Lake to explore raw, unstructured data (like social
media text or image files) to train Machine Learning models.

◦ Business Analysts use the EDW to run standard, recurring financial and operational
reports using BI tools like Tableau or PowerBI.

Page 6 of 15
 Q9. Classify different types of analytics and explain each with examples.

Data analytics is categorized into four primary types, representing a progression from basic
reporting to advanced artificial intelligence. They move from hindsight (what happened) to
foresight (what will happen) and optimization (what to do).

1. Descriptive Analytics (What happened?):

This is the most basic form of analytics. It involves aggregating and summarizing
historical data to provide insight into past events. It uses simple math (averages, counts,
percentages).
Example: A monthly sales report showing that revenue dropped by 10% in Q3, or a
dashboard showing the total number of website visitors yesterday.

2. Diagnostic Analytics (Why did it happen?):

This goes a step further by investigating the root cause of the events identified in
descriptive analytics. It involves data discovery, drill-down, and correlations.
Example: Drilling down into that Q3 sales report to discover that the 10% drop in revenue
was specifically caused by a severe supply chain delay in the Southern region.

3. Predictive Analytics (What is likely to happen?):

This type utilizes historical data, statistical modeling, and Machine Learning algorithms to
forecast future trends and probabilities.
Example: A bank using past credit history and transaction patterns to predict the
likelihood of a customer defaulting on a loan, or a manufacturing plant predicting when a
specific machine part will fail based on vibration sensor data.

4. Prescriptive Analytics (What should we do about it?):

The most advanced tier. It not only predicts future outcomes but also suggests the optimal
course of action to capitalize on a prediction or mitigate a risk. It heavily relies on AI and
optimization algorithms.
Example: Google Maps predicting traffic congestion (Predictive) and then automatically
recalculating and suggesting the fastest alternative route to your destination
(Prescriptive).

Page 7 of 15
 Q10. What are the major challenges faced in Big Data management and
processing?

While Big Data offers immense value, managing and processing it presents significant
technical and organizational challenges, often categorized by the "V's" of Big Data.

• 1. The Volume Challenge (Data Storage): Generating terabytes and petabytes of data
daily requires massive storage infrastructure. Traditional relational databases cannot
handle this scale. Organizations must architect complex distributed storage systems (like
HDFS or Cloud Object Storage) which require constant maintenance and scaling.

• 2. The Variety Challenge (Data Integration): Data no longer comes in neat rows and
columns. Integrating structured data (SQL), semi-structured data (JSON, XML), and
unstructured data (video, audio, text logs) into a single cohesive system for analysis is
incredibly complex and requires sophisticated ingestion and parsing tools.

• 3. The Velocity Challenge (Real-Time Processing): Data is generated at blistering speeds

(e.g., thousands of credit card transactions per second or constant IoT telemetry).
Processing this data in real-time or near-real-time to detect fraud or system anomalies
requires specialized streaming frameworks (like Apache Kafka or Spark Streaming) rather
than traditional batch processing.

• 4. Data Quality and Veracity: With massive volumes of unstructured data, the signal-to-
noise ratio drops. Ensuring the data is accurate, clean, and trustworthy (Veracity) is
difficult. If an ML model is trained on poor quality data, the predictions will be flawed
("Garbage in, Garbage out").

• 5. Security and Privacy: Securing a distributed system is much harder than securing a
centralized server. Nodes can be vulnerable. Furthermore, complying with strict data
privacy regulations (like GDPR or HIPAA) while processing massive datasets requires
advanced anonymization and access control techniques.

• 6. Skill Gap: There is a persistent shortage of highly skilled professionals (Data Engineers,
Data Scientists, Hadoop Administrators) capable of designing, managing, and extracting
value from complex Big Data ecosystems.

Page 8 of 15
 Q11. Explain the CAP Theorem and discuss its implications in
distributed systems.

The CAP Theorem, also known as Brewer's Theorem, is a fundamental principle in theoretical
computer science that applies to distributed data stores. It states that it is impossible for a
distributed data store to simultaneously provide more than two out of the following three
guarantees:

• Consistency (C): Every read operation receives the most recent write or an error. In a
consistent system, all nodes see the exact same data at the same time. If data is updated on
Node A, a user querying Node B will immediately see that updated data.

• Availability (A): Every request receives a (non-error) response, without the guarantee
that it contains the most recent write. The system is always on and responsive, even if
some nodes are down.

• Partition Tolerance (P): The system continues to operate despite an arbitrary number of
messages being dropped (or delayed) by the network between nodes. In a distributed
system over a network, partitions (network failures) are a reality and cannot be avoided.

Implications in Distributed Systems:

Because network failures (Partitions) are inevitable in any distributed system, a system must
support Partition Tolerance (P). Therefore, architects cannot choose CA. When a network
partition occurs, the system must choose between:

1. Choosing Consistency over Availability (CP Systems): If the network fails between
nodes, the system will return an error or timeout rather than return potentially outdated
data. The system becomes unavailable to ensure data remains consistent.
Examples: MongoDB, HBase, Redis. (Good for financial transactions).

2. Choosing Availability over Consistency (AP Systems): If the network fails, the system
will return the most recent version of the data it has on the queried node, even if it is stale
(not the absolute latest write). The system remains available but sacrifices strict
consistency (often opting for "Eventual Consistency").
Examples: Cassandra, CouchDB, DynamoDB. (Good for social media feeds or shopping
carts).

Page 9 of 15
 Q12. What is NoSQL? Explain its different types with examples.

NoSQL (often interpreted as "Not Only SQL") refers to a broad class of database management
systems that do not use the traditional relational tabular structure (rows and columns) found
in RDBMS. NoSQL databases are designed specifically for distributed architectures, high
scalability, and handling massive volumes of unstructured or semi-structured data.

Different Types of NoSQL Databases:

1. Key-Value Stores:

The simplest type of NoSQL database. Every item is stored as an attribute name (key),
together with its value. They are highly performant for simple lookups.
Characteristics: Extremely fast, highly scalable, no complex querying.
Use Case: Session management in web apps, caching, user profiles.
Examples: Redis, Amazon DynamoDB, Riak.

2. Document Databases:

Stores data in documents similar to JSON (JavaScript Object Notation) or BSON. Each
document contains pairs of fields and values. The values can typically be a variety of types
including strings, numbers, booleans, arrays, or objects.
Characteristics: Flexible schema, intuitive for developers (maps directly to objects in code).
Use Case: Content management systems, e-commerce catalogs, real-time analytics.
Examples: MongoDB, Couchbase, CouchDB.

3. Column-Family Stores (Wide-Column Stores):

Stores data in columns rather than rows. Instead of a rigid table, it uses a concept of a
keyspace containing column families. Each row can have a different number of columns.
Optimized for queries over large datasets.
Characteristics: High write performance, excellent for time-series data.
Use Case: IoT sensor data, logging, recommendation engines.
Examples: Apache Cassandra, Apache HBase.

4. Graph Databases:

Designed to store and navigate relationships. Data is stored in nodes (entities like people
or places) and edges (the relationships between them).
Characteristics: Ideal for traversing complex, highly connected data where the
relationships are as important as the data itself.
Use Case: Social networks (who follows whom), fraud detection rings, recommendation
engines.
Examples: Neo4j, Amazon Neptune.

Page 10 of 15
 Q14. What are the key features of Hadoop? Discuss its advantages.

Apache Hadoop is a foundational open-source framework that allows for the distributed
processing of massive data sets across clusters of computers. It is designed to scale up from
single servers to thousands of machines.

Key Features of Hadoop:

• Distributed Storage (HDFS): The Hadoop Distributed File System splits large files into
blocks (default 128 MB) and distributes them across the nodes in a cluster. This allows for
the storage of files that are larger than the capacity of any single machine.

• Distributed Processing (MapReduce/YARN): Instead of moving massive amounts of data

over the network to a central processor, Hadoop sends the processing logic (the code) to
the nodes where the data physically resides.

• Data Locality: This is the core concept of Hadoop. Moving computation is cheaper than
moving data. Hadoop ensures that tasks are executed on the node containing the required
data blocks.

• Schema-on-Read: Data can be ingested into Hadoop without any prior validation or
structuring. The schema is applied only at the time the data is read/queried.

Advantages of Hadoop:

• Cost-Effective: Hadoop runs on commodity hardware (standard, off-the-shelf servers),

eliminating the need to buy expensive, proprietary supercomputers or SAN (Storage Area
Network) infrastructure.

• Highly Scalable: It utilizes horizontal scaling. If more storage or compute power is

needed, administrators can simply add more nodes to the cluster without downtime.

• Fault Tolerant and Resilient: Hardware failures are treated as the norm, not the
exception. HDFS automatically replicates data blocks (usually 3 times) across different
nodes and racks. If a node fails, Hadoop automatically redirects processing to a replica
node without user intervention.

• Flexibility: It can store and process structured, semi-structured, and completely

unstructured data seamlessly.

Page 11 of 15
 Q17. What are Hadoop distributions? Explain how Hadoop works in a
distributed environment.

1. Hadoop Distributions:
While Apache Hadoop is open-source and free, deploying, securing, and managing a raw, open-
source cluster of hundreds of nodes from the command line is highly complex. A Hadoop
Distribution is a packaged, enterprise-ready version of Hadoop provided by a vendor. These
vendors take the core Apache components (HDFS, YARN, MapReduce, Hive, Spark), integrate
them, test them for compatibility, and add proprietary management interfaces, security
protocols, and technical support.
Major Examples: Cloudera (CDH), Hortonworks (HDP - now merged with Cloudera), MapR (now
part of HPE), and cloud distributions like Amazon EMR and Google Cloud Dataproc.

2. How Hadoop Works in a Distributed Environment (Architecture):

Hadoop operates on a strict Master-Slave architecture.

The Storage Layer (HDFS):

• NameNode (The Master): The "brain" of the filesystem. It holds the metadata (the
directory tree and tracking of which data blocks are stored on which specific DataNodes).
It does not store the actual data.

• DataNodes (The Slaves): The workhorses. These are the thousands of machines that store
the actual physical data blocks and handle read/write requests from the clients. They
constantly send "Heartbeats" and "Block Reports" to the NameNode to prove they are alive
and report what data they hold.

The Processing Layer (YARN & MapReduce):

• ResourceManager (The Master): The ultimate authority that allocates compute resources
(CPU, Memory) across all the applications running in the cluster.

• NodeManager (The Slaves): An agent running on every single machine in the cluster. It
launches and monitors containers (the isolated environments where tasks actually
execute) and reports resource usage back to the ResourceManager.

• ApplicationMaster: When a user submits a job (like a MapReduce task), an

ApplicationMaster is created just for that specific job. It negotiates resources from the
ResourceManager and then works with the NodeManagers to execute and monitor the
specific tasks.

The Workflow: A client submits a file. The NameNode breaks it into 128MB blocks and scatters
them across DataNodes, ensuring 3x replication. When a user runs a query, the

Page 12 of 15
ResourceManager finds where the data blocks are via the NameNode, and sends the processing
code directly to those specific DataNodes (Data Locality) to process the data in parallel.

Module 2: Hadoop Architecture & Processing

Q2. Differentiate between RDBMS and Hadoop in terms of storage,

processing, scalability, and use cases.

Feature RDBMS (Relational Database) Hadoop Ecosystem

Data is stored in highly

structured, normalized tables Data is stored in distributed blocks across
Storage
consisting of defined rows and a cluster (HDFS). Can store any format:
Structure
columns. Adheres to strict text, video, logs, JSON, or tabular data.
schemas.

Transactional processing (OLTP). Batch processing (MapReduce) and

Processing Optimized for rapid, continuous Analytical processing. Optimized for
Model read/write operations and reading massive datasets sequentially; not
immediate consistency (ACID). designed for low-latency transactions.

Scale-up (Vertical). Requires Scale-out (Horizontal). Requires adding

Scalability buying a larger, more powerful more standard, commodity servers to the
central server (expensive). cluster (highly cost-effective).

High data integrity at the point of Schema-on-Read. Integrity is handled by

Data
entry (Schema-on-Write). Data the application querying the data,
Integrity
must fit the rules to be saved. allowing raw data ingestion.

Banking systems, E-commerce Log analysis, training Machine Learning

Primary
shopping carts, Inventory models, Data Lakes, recommendation
Use Cases
management, ERP systems. engines, IoT data aggregation.

Page 13 of 15
 Q3. What are the two major challenges in distributed computing?
Explain them with examples.

Distributed computing, while powerful, introduces significant complexities that do not exist in
single-machine systems. The two primary challenges are:

1. Partial Failures:

In a single machine, if a critical component fails, the whole system usually halts (a total
failure). In a distributed system with 1,000 nodes, it is statistically guaranteed that nodes
will fail constantly (hardware crash, network cable cut, disk failure). The challenge is that
a part of the system fails while the rest continues to operate.
Example: A client is writing a 1GB file to a cluster. The file is being split into blocks. If the
specific node receiving block #4 suddenly loses power, the master node must instantly
detect the failure, find a new healthy node, and reroute block #4 without the client
application crashing or the file becoming corrupted.

2. Concurrency and Synchronization:

When multiple nodes attempt to access, modify, or compute shared resources or data
simultaneously over a network, ensuring the data remains consistent is immensely
difficult due to unpredictable network delays.
Example: Imagine a distributed banking system. Node A processes a $50 withdrawal, and
Node B simultaneously processes a $50 withdrawal on the same $100 account. Because of
network latency, both nodes might read the balance as $100, approve the withdrawals,
and update the balance to $50. The bank just lost $50. Distributed systems require complex
locking mechanisms (like Apache ZooKeeper) to manage this concurrency.

Page 14 of 15
 Q5. What are the key aspects of Hadoop that make it suitable for Big
Data processing?

Hadoop was purpose-built to solve the exact problems posed by Big Data. Its suitability is
driven by three core architectural design principles:

• 1. Data Locality (Compute to Data): In traditional systems, data is moved over the
network from the storage disk to the CPU for processing. With Big Data (e.g., Petabytes),
moving data causes massive network bottlenecks. Hadoop reverses this: it sends the
processing logic (a small kilobyte script) over the network to the specific nodes where the
massive data blocks physically reside. Processing occurs locally, eliminating network
congestion.

• 2. Assumption of Hardware Failure (Fault Tolerance): Hadoop does not rely on

expensive, highly reliable hardware. It assumes cheap commodity hardware will fail
frequently. It ensures reliability purely through software by automatically replicating
every data block across multiple distinct nodes (default 3x replication). If a node dies, no
data is lost, and tasks are automatically redirected.

• 3. Linear and Horizontal Scalability: Hadoop scales linearly. If you have a 10-node
cluster processing 1TB of data in 1 hour, and your data grows to 2TB, you simply add 10
more identical nodes, and the processing time remains 1 hour. There is theoretically no
upper limit to this scale-out architecture.

Page 15 of 15
 BIG DATA ANALYTICS

Detailed Study Guide & Notes (With Textbook Image References)

Module 2: Hadoop Architecture & Processing (Continued)

Q6. Explain the main components of Hadoop with a neat diagram.

Apache Hadoop is a robust, open-source framework designed to store and process massive datasets
across clusters of commodity hardware. Its architecture is built upon four primary modules:

• Hadoop Common (Core): The foundational library containing essential Java archives (JAR files),
scripts, and utilities required to start Hadoop. It provides the base file system, OS-level
abstractions, and Java API libraries.
• Hadoop Distributed File System (HDFS): The storage layer. A highly fault-tolerant, distributed
file system designed to run on low-cost hardware. It breaks large files into fixed-size blocks
(default 128 MB) and distributes them across cluster nodes.
• YARN (Yet Another Resource Negotiator): The resource management layer introduced in
Hadoop 2.x. It acts as the "operating system" of the cluster, dynamically allocating system
resources (CPU, Memory) to various applications so multiple engines can share the cluster
simultaneously.
• Hadoop MapReduce: The processing layer. A software framework for writing applications that
process vast amounts of data in parallel, divided into a Map phase (filtering/sorting) and Reduce
phase (aggregating).

📸 Refer to Textbook Image: Main Components of Hadoop Ecosystem Search Unit 2

Image Glimpse: Draw a layered block diagram. The bottom layer is a wide block labeled "HDFS
(Distributed Storage)". The middle layer is "YARN (Resource Management)". The top layer contains
multiple blocks sitting on YARN, specifically "MapReduce (Processing)", alongside other potential
ecosystem tools like Hive, Pig, or Spark.

Page 1 of 5
Q11. Explain the replica placement strategy in Hadoop and its importance.

HDFS is designed to reliably store massive amounts of data across clusters of commodity hardware,
which are prone to failure. To guarantee data availability, Hadoop uses a **Rack Awareness** replica
placement strategy (default 3 blocks):

• Replica 1 (Local Node): Placed on the local DataNode where the client application is running to
optimize initial write speed.
• Replica 2 (Off-Rack): Placed on a DataNode residing in a completely different server rack than
the first replica.
• Replica 3 (Same Rack, Different Node): Placed on a different DataNode, but within the same
rack as the second replica.

Importance:
- High Fault Tolerance: If an entire server rack fails (e.g., network switch dies), data is not lost because
Replica 1 is on a completely different rack.
- Bandwidth Optimization: Placing the third replica on the same rack as the second means the data
only travels across the main network switch once during replication, saving massive amounts of cross-
rack network bandwidth.

Q12. Explain the MapReduce programming model as a software framework in

Hadoop.

MapReduce abstracts the complexities of distributed programming away from the developer, who only
needs to write the Mapper and Reducer functions. It operates through a strict sequence of phases
using (key, value) pairs:

1. Input and Splitting Phase: Raw input data in HDFS is logically divided into "Input Splits", each
assigned to a single Mapper task.
2. Map Phase: The Mapper reads data line-by-line, processes it, and generates intermediate (key,
value) pairs, temporarily storing them on the local disk.
3. Shuffle and Sort Phase: Hadoop automatically gathers all intermediate pairs from all Mappers,
sorts them, and groups all values associated with the exact same key together.
4. Reduce Phase: The Reducer receives a key and a list of values (key, list of values) . It
applies aggregation logic (summing, averaging) to produce a single, finalized (key, value)
output.
5. Output Phase: Final results from Reducers are written back to HDFS.

Page 2 of 5
Q15. Explain the Word Count program in MapReduce with an example involving
multiple (e.g., 50) files and illustrate with a diagram.

Scenario: We have 50 large text files stored in HDFS.

• Input: 50 files are divided into Input Splits, resulting in 50 Mapper tasks.
• Mapper Logic: Each Mapper reads its file, tokenizes sentences into words, and emits a key-value
pair where the word is the key and '1' is the value. Example: (Deer, 1), (Bear, 1) .
• Shuffle and Sort: The framework gathers outputs from all 50 files, sorts them, and groups
identical keys. Example: (Bear, [1, 1, 1, 1...]) .
• Reducer Logic: The Reducer receives the grouped data and iterates through the list of 1s,
summing them up to find total frequency. Example output: (Bear, 4) .

📸 Refer to Textbook Image: MapReduce Word Count Flow Search Unit 2

Image Glimpse: Draw a flow diagram moving left to right. Left: "Input Files (50 files)". Arrows point to
multiple "Mapper" boxes (showing words being split into word, 1). Arrows then crisscross into a "Shuffle/
Sort" phase where identical words are grouped. Finally, arrows point to "Reducer" boxes where the 1s are
summed up, leading to the "Final Output" block.

Q16. Explain Hadoop 2 HDFS architecture and its two major components.

HDFS in Hadoop 2.x follows a strict Master/Slave architecture with High Availability.

• NameNode (Master): The centerpiece that stores the Metadata (directory tree, block tracking). It
does not store user data. Hadoop 2 features an Active/Standby NameNode configuration. If the
Active node crashes, Zookeeper triggers a failover, and the Standby instantly takes over to
prevent downtime.
• DataNodes (Slaves): The commodity servers that store the physical 128 MB data blocks. They
handle read/write requests and send "Heartbeats" (every 3 seconds) and "Block Reports" to the
NameNode to prove they are alive and report what data they hold.

Q17. Describe the architecture of YARN and explain its components and working.

YARN decouples resource management from task monitoring to prevent bottlenecks.

• ResourceManager (RM): The master daemon containing a "Scheduler" that strictly allocates CPU
and memory to various applications across the cluster.
• NodeManager (NM): The slave daemon on every worker node. It launches application containers,
monitors resource usage, and reports back to the RM.
• ApplicationMaster (AM): Spawned specifically for a single job. It negotiates resources from the
RM and works with NodeManagers to execute and monitor tasks.
• Container: The fundamental unit of allocation (e.g., 2 GB RAM, 1 CPU core) on a specific node.

Page 3 of 5
Working: A client submits an app. The RM allocates one Container to start the AM. The AM calculates
needed resources, negotiates more Containers from the RM, contacts NMs to launch tasks inside
those containers, and monitors them until completion.

Module 3: MongoDB Operations & Queries

Theory Questions

Q1. Why is JSON used in MongoDB? Explain its role and structure with a suitable
diagram.

MongoDB stores data as individual documents using the BSON (Binary JSON) format rather than rigid
RDBMS tables.

• Object-Oriented Mapping: JSON perfectly mirrors the structure of objects in modern

programming languages (Java, Python), eliminating the need for complex Object-Relational
Mapping (ORM).
• Dynamic Schema: Allows flexible structure. Two JSON documents in the same collection can
have completely different fields, allowing businesses to adapt models instantly.
• Embedded Data: Arrays and nested documents allow all related data (e.g., user, address, phone)
to be stored in a single document, making reads incredibly fast compared to SQL joins.

📸 Refer to Textbook Image: JSON Document Structure vs. RDBMS Search Unit 3
Image Glimpse: Contrast two visuals. Left: Standard RDBMS structure showing three linked tables
(Users, Addresses, Contacts). Right: A single MongoDB document enclosed in curly braces {} showing the
Address and Contacts data *nested* directly inside the User document as arrays [] and sub-documents.

Q6. Explain the aggregation framework in MongoDB with a suitable diagram.

The Aggregation Framework processes data records and returns computed results (similar to SQL's
GROUP BY ). It operates as a Data Processing Pipeline where documents pass through stages to be
filtered, aggregated, or transformed.

• $match : Filters documents (like WHERE in SQL).

• $group : Groups documents by a key and applies accumulators (like $sum, $avg).
• $sort , $project (reshape), $limit (pagination).

Page 4 of 5
 📸 Refer to Textbook Image: MongoDB Aggregation Pipeline Search Unit 3
Image Glimpse: Draw a horizontal or vertical conveyor belt/pipeline. Input documents (Doc 1, Doc 2, Doc
3) enter the first block labeled $match (a filter). Surviving documents move to a $group block (merging
documents). Finally, they move to a $sort block, exiting as the final aggregated output.

Q9. How do cursors work in MongoDB? Explain their behaviour and types.

A query does not instantly return all matching documents (which could exhaust RAM). Instead,
MongoDB returns a Cursor, a pointer to the result set on the server.

• Behavior: The client iterates through the cursor, fetching documents in manageable batches
(default 101 documents or 1MB). Cursors timeout automatically after 10 minutes of inactivity.
• Standard Cursor: Iterates through a static result set and closes at the end.
• Tailable Cursor: Used with Capped Collections. Remains open after returning initial results and
automatically pushes newly inserted documents to the client (ideal for real-time logging).

Q10. How does MongoImport work in MongoDB? Explain its usage with syntax and
example.

A command-line tool used to import data from JSON, CSV, or TSV files directly into MongoDB (run
from the OS terminal, not mongosh).

Syntax: mongoimport --db <db_name> --collection <col_name> --file <file_path>

Example (Importing an employees CSV file with headers):

mongoimport --db CompanyDB --collection staff --type csv --headerline --file /
users/data/[Link]

Q11. How does MongoExport work in MongoDB? Explain its usage with syntax and
example.

A command-line tool used to extract data from a MongoDB collection and export it into a human-
readable JSON or CSV file.

Syntax: mongoexport --db <db_name> --collection <col_name> --out <output_file>

Example (Exporting only "IT" department staff to JSON):

mongoexport --db CompanyDB --collection staff --query '{"department": "IT"}' --out
/users/data/it_staff.json

Page 5 of 5
 BIG DATA ANALYTICS

Detailed Study Guide & Notes - Part 3 (With Textbook Image References)

Module 3: MongoDB Operations & Queries

Practical Queries: Database Operations

Q1. Consider the restaurant database with the following attributes: Name,
address (building, street, area, pin code), id, cuisine, nearby landmarks, online
delivery (yes/no), and famous for. Write MongoDB queries for the given
scenarios.

Note on Schema Assumption: Since address is explicitly defined with sub-fields (building, street,
area, pin code), it is structured as an embedded document in MongoDB. We will use dot-notation
(e.g., "[Link]" ) to query these specific fields.

i. List the name and cuisine of all restaurants in Bengaluru.

[Link](
{ "[Link]": /Bengaluru/i },
{ "Name": 1, "cuisine": 1, "_id": 0 }
)

ii. List the name, address, and famous dish of restaurants that provide online delivery.

[Link](
{ "online delivery": "yes" },
{ "Name": 1, "address": 1, "famous for": 1, "_id": 0 }
)

iii. List all restaurant details except nearby landmarks where the cuisine is Italian.

[Link](
{ "cuisine": "Italian" },
{ "nearby landmarks": 0 }
)

Page 1 of 7
iv. List restaurants located on MG Road.

[Link](
{ "[Link]": /MG Road/i }
)

v. Delete restaurants that do not have a famous dish field.

[Link](
{ "famous for": { $exists: false } }
)

Q7. Consider the student database with the following attributes: Name, USN,
department, year, marks, address (city), phone number. Write MongoDB queries
for the given scenarios.

i. List the name and USN of all students in CSE department.

[Link](
{ "department": "CSE" },
{ "Name": 1, "USN": 1, "_id": 0 }
)

ii. List the name and marks of students scoring above 80.

[Link](
{ "marks": { $gt: 80 } },
{ "Name": 1, "marks": 1, "_id": 0 }
)

iii. List all details except phone number where year is 3rd.

[Link](
{ "year": "3rd" },
{ "phone number": 0 }
)

iv. List students who scored less than 40.

[Link](
{ "marks": { $lt: 40 } }
)

Page 2 of 7
 v. Delete students where address is missing.

[Link](
{ "address": { $exists: false } }
)

Module 4: Hive & Pig

Q2. Explain the different data units in Hive such as databases, tables, partitions,
and buckets.

Apache Hive organizes data into a distinct hierarchical structure to optimize querying and data
management over HDFS. The four primary data units are:

• Databases (Namespaces): The highest level of organization. Databases are essentially

namespaces that group logically related tables together, preventing naming collisions. In HDFS,
a database is physically represented as a master directory.

• Tables: The core structural component that imposes a schema onto the raw data stored in
HDFS. Hive supports two types of tables:

◦ Managed (Internal) Tables: Hive fully manages both the schema and the physical data.
Dropping the table deletes both.

◦ External Tables: Hive only manages the schema metadata. The actual data resides outside
Hive's default warehouse directory. Dropping an external table deletes only the schema,
leaving the data intact.

• Partitions: A technique to divide a large table into smaller, manageable chunks based on the
values of one or more specific columns (e.g., partitioning sales data by Year and Month ).
Physically, partitions are sub-directories within the table's main directory. This drastically speeds
up queries by pruning irrelevant directories (e.g., querying for "March" skips all other month
folders).

• Buckets (Clusters): A further subdivision of data within a partition. Bucketing takes a specified
column, applies a hashing algorithm to it, and distributes the data evenly across a pre-defined
number of files (buckets). This optimizes sampling operations and makes joins between tables
significantly more efficient.

Page 3 of 7
 📸 Refer to Textbook Image: Hive Data Model / Architecture Search Unit 4

Image Glimpse: Look for a hierarchical tree or nested folder diagram. The top layer will show a
"Database" cylinder. Below it will be "Tables". The "Tables" block will point down to multiple "Partitions"
(often labeled as sub-directories like year=2023). Finally, the Partitions point down to specific "Buckets"
or "Files" (Bucket 1, Bucket 2).

Q4. Explain the different types of Metastore in Hive and their roles.

The Metastore is the central repository of Hive's metadata. It stores the definitions of databases,
tables, columns, partitions, and their physical mapping to HDFS. Hive offers three deployment
architectures for the Metastore depending on the scale and user requirements:

1. Embedded Metastore:

In this mode, the Hive service and the Metastore service run within the same Java Virtual
Machine (JVM). It uses an embedded Derby database to store the metadata on the local file
system.
Limitation: It only supports a single active user/session at a time. It is exclusively used for unit
testing and local development, never for production.

2. Local Metastore:

The Hive service and the Metastore service still run in the same JVM, but the metadata is stored
in an external relational database (like MySQL or PostgreSQL) running on a separate machine
or process.
Advantage: This setup supports multiple concurrent users and is suitable for smaller production
environments.

3. Remote Metastore:

The most scalable and secure architecture. The Hive service, the Metastore service, and the
underlying database run on completely separate JVMs/machines. Client applications connect to
the Metastore server via the Thrift protocol.
Advantage: This is the standard for enterprise production clusters. It offloads processing,
ensures high availability, and allows centralized security management.

Page 4 of 7
Q5. Explain the various data types supported in Hive with examples.

Hive supports a rich set of data types, broadly categorized into Primitive types and Complex
(Collection) types. Primitive types are used for standard scalar values.

• Numeric Types:

◦ TINYINT (1-byte integer), SMALLINT (2-byte), INT (4-byte), BIGINT (8-byte).

◦ FLOAT (single-precision), DOUBLE (double-precision).

◦ DECIMAL (user-defined precision and scale, critical for financial data).

• String Types:

◦ STRING (unbounded variable-length character string).

◦ VARCHAR (variable-length string with a maximum length).

◦ CHAR (fixed-length string).

• Date/Time Types:

◦ TIMESTAMP (represents a point in time, independent of timezone).

◦ DATE (represents a year, month, and day without a time component).

• Miscellaneous Types:

◦ BOOLEAN (represents TRUE or FALSE ).

◦ BINARY (array of bytes for storing images or compressed data).

Example Table Definition:

CREATE TABLE employees (id INT, name STRING, salary DECIMAL(10,2), is_active
BOOLEAN);

Q10. Explain collection data types in Hive such as ARRAY, MAP, and STRUCT.

Unlike traditional SQL databases, Hive excels at handling nested, semi-structured data natively
through its Complex or Collection data types. These types allow a single column to hold multiple
values or complex objects.

• ARRAY:

An ordered sequence of elements that must all be of the exact same data type. Elements are
accessed using a zero-based index.
Syntax: ARRAY<data_type>

Page 5 of 7
 Example: A column skills ARRAY<STRING> storing ["Java", "Python", "Hadoop"] .
Accessed via skills[0] .

• MAP:

An unordered collection of key-value pairs. Keys must be primitive types, while values can be
any type. Accessed using array notation with the key.
Syntax: MAP<primitive_type, data_type>
Example: A column deductions MAP<STRING, FLOAT> storing {"Tax": 500.0,
"Insurance": 200.0} . Accessed via deductions["Tax"] .

• STRUCT:

A record object containing a fixed number of named fields. Each field can be a different data
type. This is analogous to a 'struct' in C. Elements are accessed using dot notation.
Syntax: STRUCT<col_name: data_type, ...>
Example: A column address STRUCT<street: STRING, pin: INT> storing {"MG Road",
560001} . Accessed via [Link] .

Q11. Explain the two types of partitioning in Hive briefly.

Partitioning is the process of physically dividing tables into sub-directories to optimize read
performance. Hive supports two distinct methods for inserting data into partitioned tables:

1. Static Partitioning:

In static partitioning, the user explicitly hardcodes the partition value when loading or inserting
the data. It requires a separate INSERT statement for every single partition. It is ideal when you
have a small number of known partitions and you want strict control over data placement.

Example: LOAD DATA INPATH '[Link]' INTO TABLE sales PARTITION (year=2023,
country='India');

2. Dynamic Partitioning:

In dynamic partitioning, the partition values are calculated automatically by Hive based on the
value of the last column(s) of the SELECT statement feeding the insertion. A single INSERT
statement can dynamically create hundreds of partitions on the fly. It is essential when loading
massive datasets with unpredictable partition keys.

Example: INSERT INTO TABLE sales PARTITION (country) SELECT item, revenue,
country FROM staging_table;

Page 6 of 7
Q15. How does Pig work with Hadoop? Explain its execution process.

Apache Pig is a high-level scripting platform built on top of Hadoop. Writing raw MapReduce jobs in
Java is complex and requires hundreds of lines of code. Pig solves this by introducing Pig Latin, an
easy-to-use scripting language. A 10-line Pig Latin script is equivalent to roughly 200 lines of
MapReduce Java code. Pig does not execute on its own; it acts as a compiler that translates Pig
Latin scripts into a sequence of MapReduce jobs that Hadoop can execute.

The Execution Process of Pig:

1. Script Submission: The developer writes a script in Pig Latin and submits it to the Pig
execution environment (via the Grunt shell, a script file, or embedded inside Java).

2. Parser: The Pig Parser reads the script, checks for syntax errors, performs type checking, and
verifies the schema. It outputs a Directed Acyclic Graph (DAG) representing the logical flow of
operations.

3. Optimizer: The Logical Optimizer applies various rules to make the script run faster. For
example, it might push FILTER operations earlier in the pipeline to reduce the amount of data
processed in subsequent steps.

4. Compiler: The Compiler translates the optimized logical plan into a Physical Plan. This physical
plan is a series of actual MapReduce jobs.

5. Execution Engine: Finally, Pig submits these MapReduce jobs to the Hadoop cluster (YARN
and HDFS) for distributed execution. The results are either dumped to the screen or stored back
into HDFS.

📸 Refer to Textbook Image: Apache Pig Architecture / Execution Flow Search Unit 4

Image Glimpse: Look for a flowchart starting from the top. It begins with "Pig Latin Script" or "Grunt
Shell", moving into a box labeled "Parser". An arrow points to the "Optimizer", then to the "Compiler",
and finally into an "Execution Engine". The bottom of the diagram will show the compiled jobs interacting
with "Hadoop / MapReduce" and "HDFS".

Page 7 of 7
 BIG DATA ANALYTICS

Detailed Study Guide & Notes - Part 4 (With Textbook Image References)

Module 4: Hive & Pig (Continued)

Q16. Explain the philosophy of Pig with a suitable diagram.

Apache Pig was created at Yahoo to simplify the complexity of writing native Java MapReduce jobs.
Its philosophy is built around a few core tenets:

• "Pigs eat anything": Pig is designed to handle all types of data. It can ingest and process
structured, semi-structured, and completely unstructured data without requiring a strict schema
to be defined beforehand.

• Data Flow over Data Schema: Unlike SQL, which is declarative (you declare what you want),
Pig Latin is a data flow language. You write scripts that define exactly step-by-step how the data
should be moved, transformed, filtered, and aggregated.

• Extensibility: Pig relies heavily on User-Defined Functions (UDFs). If Pig lacks a built-in
function, developers can write their own in Java, Python, or Ruby and seamlessly invoke them in
the Pig script.

📸 Refer to Textbook Image: Apache Pig Philosophy/Architecture Search Unit 4

Image Glimpse: A flowchart showing the execution lifecycle. It starts with a "Pig Latin Script", moving
into a "Parser/Optimizer" block, which then translates into a "Compiler" block. This compiler generates a
series of MapReduce DAGs (Directed Acyclic Graphs) that are sent to the underlying "Hadoop/HDFS"
cluster at the bottom.

Q19 & Q24. Explain different data types in Pig with examples. Explain complex
data types in Pig, such as tuples and maps.

Pig supports a rich variety of data types, divided into Scalar (primitive) types and Complex types.

1. Scalar Data Types:

• int (32-bit integer) - Example: 10

Page 1 of 7
 • float (32-bit floating point) - Example: 10.5f

• chararray (String/character array) - Example: 'Hadoop'

• bytearray (Array of bytes, default if type is not specified)

2. Complex Data Types (Q24):

Complex types are used to represent nested structures, making Pig highly effective for semi-
structured data like JSON or log files.

• Tuple: An ordered set of fields, similar to a row in a relational database. Fields inside a tuple can
be of any type. Enclosed in parentheses () .
Example: (John, 35, 'Manager')

• Bag: A collection of tuples, similar to a table in a relational database. Bags do not guarantee
order and can contain duplicate tuples. Enclosed in braces {} .
Example: {(John, 35), (Alice, 28), (Bob, 40)}

• Map: A set of key-value pairs. The key must strictly be a chararray (string), but the value can
be of any data type. Enclosed in brackets [] .
Example: ['name'#'John', 'age'#35]

Q23. Explain evaluation functions in Pig, such as AVG, MAX, and COUNT.

Evaluation functions in Pig are built-in aggregate functions used to perform mathematical or
statistical operations on data. They are typically used in conjunction with the GROUP or COGROUP
operators, which group data into a Bag.

• COUNT(): Computes the total number of elements (tuples) in a bag. It ignores NULL values.
Usage: C = FOREACH grouped_data GENERATE group, COUNT(my_data);

• AVG(): Computes the mathematical average (mean) of numeric values in a single-column bag.
It requires the data to be cast as numeric (int, float, double).
Usage: A = FOREACH grouped_data GENERATE group, AVG(my_data.salary);

• MAX() / MIN(): Returns the highest or lowest value from a bag of numeric or chararray data.
Usage: M = FOREACH grouped_data GENERATE group, MAX(my_data.temperature);

Q27. Explain parameter substitution in Pig with syntax and an example.

Parameter substitution allows developers to pass variables to a Pig script at runtime. This prevents
hardcoding values (like file paths or specific dates) into the script, making the code dynamic,
reusable, and easier to automate in production pipelines.

Page 2 of 7
Syntax & Execution: You define parameters in the script using the $ symbol. You pass the values
from the command line using the -param flag or via a parameter file using -param_file .

Example:

Inside the Pig Script ([Link]):

A = LOAD '$INPUT_FILE' USING PigStorage(',') AS (id:int, name:chararray);

B = FILTER A BY id == $TARGET_ID;
STORE B INTO '$OUTPUT_DIR';

Execution via Terminal:

pig -param INPUT_FILE='/data/[Link]' -param TARGET_ID=101 -param OUTPUT_DIR='/out

Q28. Explain the Word Count example using Pig with steps.

The Word Count program in Pig Latin is significantly shorter than Java MapReduce. It involves
loading text, splitting sentences into words, grouping identical words, and counting them.

-- Step 1: Load the text file from HDFS into a single column named 'line'
lines = LOAD '[Link]' AS (line:chararray);

-- Step 2: TOKENIZE splits the sentence into a bag of words.

-- FLATTEN breaks the bag open to create a separate tuple/row for each word.
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;

-- Step 3: Group the tuples by the exact word.

-- This creates a structure like: (Hadoop, {(Hadoop), (Hadoop), (Hadoop)})
grouped_words = GROUP words BY word;

-- Step 4: Iterate through the grouped data.

-- Output the word (which is now the 'group' key) and count the elements in the bag.
word_count = FOREACH grouped_words GENERATE group, COUNT(words);

-- Step 5: Display the result on the screen or store to HDFS.

DUMP word_count;

Page 3 of 7
Module 5: Spark & Web/Text Mining

Q1. Explain the main components of Apache Spark architecture with a neat
diagram.

Apache Spark utilizes a master-worker architecture designed for distributed, in-memory data
processing.

• Driver Program: The central control node. It contains the SparkContext (or SparkSession ),
which acts as the entry point to the cluster. The Driver converts the user's code into tasks and
schedules them on the executors.

• Cluster Manager: An external service (like YARN, Apache Mesos, or Spark's Standalone
Manager) responsible for acquiring and allocating resources (CPU, Memory) across the cluster.

• Worker Nodes: The physical or virtual machines that do the actual computation.

• Executors: JVM processes running on Worker Nodes. They run the individual Tasks sent by the
Driver and store intermediate data in a Cache (memory or disk) to speed up subsequent
operations.

📸 Refer to Textbook Image: Apache Spark Architecture Search Unit 5

Image Glimpse: A diagram showing a "Driver Program" box at the top, pointing down to a central
"Cluster Manager" block. The Cluster Manager then points down to multiple "Worker Node" boxes.
Inside each Worker Node, there is an "Executor" box which contains smaller boxes labeled "Task" and
"Cache".

Q2. Analyze the main features of Apache Spark and their impact on big data
processing.

Spark was developed to overcome the slow, disk-bound limitations of Hadoop MapReduce.

• In-Memory Computation: Spark keeps intermediate data in RAM rather than writing it to disk
after every step. Impact: Makes iterative algorithms (like Machine Learning) up to 100x faster
than MapReduce.

• Lazy Evaluation: Spark does not execute transformations (like map or filter) immediately. It
builds a Directed Acyclic Graph (DAG) of logical instructions. Impact: Highly optimizes the
execution plan before actually running it, saving time and resources.

Page 4 of 7
 • Fault Tolerance (via RDDs): Uses Resilient Distributed Datasets. If a node fails and data in
RAM is lost, Spark does not need to replicate data like HDFS. Impact: It uses the DAG lineage
to perfectly recompute the lost data from the original source.

• Polyglot Integration: Supports code written in Java, Scala, Python (PySpark), and R. Impact:
Broadens adoption among Data Scientists who prefer Python/R over Java.

Q3. Explain the five-layer architecture for running applications using the Apache
Spark stack.

The Spark Stack is layered to provide maximum flexibility and integrate with existing Big Data
ecosystems.

1. Storage Layer: Spark does not have its own persistent file system. It sits on top of existing
storage systems like HDFS, Amazon S3, Azure Data Lake, or local file systems.

2. Resource Management Layer: Spark relies on cluster managers to allocate resources. It can
use Hadoop YARN, Apache Mesos, Kubernetes, or its own Standalone Scheduler.

3. Spark Core (Engine Layer): The foundation of the framework. It handles memory
management, task scheduling, fault recovery, and interacts directly with the cluster manager. It
provides the RDD API.

4. Higher-Level APIs (Libraries Layer): Built on top of Spark Core, these provide specialized
functionality:

◦ Spark SQL: For querying structured data using SQL.

◦ Spark Streaming: For real-time micro-batch processing.

◦ MLlib: Machine Learning library.

◦ GraphX: For graph computation.

5. Application Layer: The user-written code that invokes the Spark context and utilizes the
libraries to solve specific business logic.

Q7. Differentiate and explain User Defined Functions (UDF), Vectorized UDFs,
and Grouped Vectorized UDFs with examples.

In Spark (particularly PySpark), UDFs are custom functions written by developers when built-in SQL
functions aren't enough.

• Standard UDFs: Applies a function to the data row-by-row. In PySpark, this is notoriously slow
because data must be continuously serialized and deserialized between the JVM (where Spark
runs) and the Python runtime for every single row.

Page 5 of 7
 • Vectorized UDFs (Pandas UDFs): Also known as Scalar Pandas UDFs. Instead of working
row-by-row, they execute the function on a batch or block of data (a Pandas Series) at once. It
uses Apache Arrow for zero-copy memory transfer, making it vastly faster than standard UDFs
for Python users.

• Grouped Vectorized UDFs: Applies the Split-Apply-Combine paradigm. It splits data into
groups (like a SQL GROUP BY ), applies a Pandas function to each entire group (as a
DataFrame), and then combines the results back into a new Spark DataFrame. Essential for
complex aggregations.

Q8. Define and explain Text Mining and its core concepts.

Text Mining (or Text Analytics) is the process of deriving high-quality, actionable information from
unstructured text data using Natural Language Processing (NLP) and machine learning algorithms.

Core Concepts:

• Tokenization: Breaking down a continuous stream of text into individual, meaningful units called
tokens (words, phrases, symbols).

• Stop Word Removal: Filtering out extremely common words (e.g., "the", "is", "at") that carry
little to no semantic value for analytics.

• Stemming & Lemmatization: Reducing words to their root or dictionary form. (e.g., converting
"running", "ran", "runs" all to the base word "run").

• TF-IDF (Term Frequency - Inverse Document Frequency): A statistical measure used to

evaluate how important a word is to a document within a larger collection. It penalizes words
that appear too frequently across all documents.

Q15 & Q16. Explain the taxonomy of Web Mining with a diagram. Explain and
apply web content mining concepts in real-world applications.

Web Mining Taxonomy (Q15): Web mining is the application of data mining techniques to discover
patterns from the World Wide Web. It is categorized into three main branches:

1. Web Content Mining: Extracting useful information directly from the contents of web pages
(text, images, audio, tables).

2. Web Structure Mining: Analyzing the link structure of the web. It views the web as a graph
where pages are nodes and hyperlinks are edges.

3. Web Usage Mining: Analyzing user interactions and server access logs to discover patterns in
how users navigate a website.

Page 6 of 7
 📸 Refer to Textbook Image: Taxonomy of Web Mining Search Unit 5

Image Glimpse: A hierarchical tree diagram. The top root is "Web Mining". It splits into three main
branches: "Web Content Mining", "Web Structure Mining", and "Web Usage Mining". Sometimes sub-
branches are shown (e.g., Content mining splitting into IR view and DB view).

Real-World Applications of Web Content Mining (Q16):

• Sentiment Analysis: Mining customer reviews on Amazon or tweets to determine if public

sentiment regarding a new product is positive, negative, or neutral.

• Search Engine Indexing: Google bots crawling billions of web pages, extracting the text
content, and categorizing it to serve relevant search results.

• Spam Filtering: Analyzing the content of incoming emails or blog comments to automatically
flag and block promotional or malicious content.

Q21. Analyze dead ends and explain their implementation in web graph mining
systems.

In the context of Web Structure Mining and algorithms like PageRank (which ranks pages based on
inbound links), a Dead End is a web page that has incoming links but no outgoing links.

The Problem: If a web surfer (or the algorithm's random walker) arrives at a dead end, they have
nowhere to click next. In mathematical terms, the probability pools into that node and cannot
distribute back into the network, breaking the iterative PageRank calculation.

Implementation / Solution: Web graph mining systems solve this using a mechanism called
Taxation or Teleportation.
The algorithm modifies the rules: When a user reaches a page, they have a probability β (usually
0.85) to click a random outgoing link. They have a probability 1-β (0.15) to get bored, ignore the
links, and "teleport" by typing a completely random, new URL into the browser.
If the user hits a dead end, they are forced to teleport. This ensures the algorithm never gets stuck
and probability continues to flow smoothly through the web graph.

Page 7 of 7