BAD601: BIG DATA ANALYTICS

MODULE 1: INTRODUCTION TO BDA

Q1: Define Big Data and exhaustively explain its core
characteristics (Volume, Velocity, Variety, Veracity,
Volatility, Variability) and associated enterprise
challenges.
ANS;
Introduction:
Big Data refers to extremely large and complex sets of data that traditional data
processing tools cannot handle efficiently. It is characterized not only by the
amount of data but also by the speed, diversity, and quality of data that
organizations use to gain valuable insights and make better decisions.

Core Characteristics of Big Data

1. Volume

BAD601: BIG DATA ANALYTICS 1

 Refers to the enormous amount of data generated every second from multiple
sources such as social media, sensors, transactions, etc.
Example: Facebook generates over 500 terabytes of data daily from users
uploading posts, images, and videos.
2. Velocity
Refers to the speed at which new data is generated and processed to meet
demand. From batch processing in past systems to real-time streaming now.
Example: Stock market data streams where milliseconds matter for trading
decisions.
3. Variety
Refers to the different types of data — structured (databases), semi-
structured (XML, JSON), and unstructured data (videos, emails, social media
posts).
Example: Customer feedback from a call center (voice recordings), emails,
and online reviews.
4. Veracity
Refers to the trustworthiness, accuracy, and quality of the data. High veracity
means data is clean and reliable enough for decision-making.
Example: Data errors or biases in sensor readings can cause wrong
predictions in IoT applications.
5. Volatility
Refers to how long the data remains relevant or valid for use before it
becomes outdated or needs to be archived or deleted.
Example: Real-time traffic data is only useful for a short period; after that, it
becomes obsolete.
6. Variability
Refers to the inconsistency in data flow and the various meanings data can
take depending on context or the way it is processed over time.

BAD601: BIG DATA ANALYTICS 2

 Example: A social media trend may spike rapidly and disappear just as fast,
causing irregular data bursts.

Enterprise Challenges Associated with Big Data

Managing Massive Data Volumes: Traditional storage is insufficient;

enterprises must migrate to scalable cloud and distributed storage.
Integration from Multiple Sources: Data comes in from different platforms
and formats, requiring complex ETL (Extract, Transform, Load) processes to
unify.
Ensuring Data Quality (Veracity): Inaccurate or incomplete data harms
decision-making, so ongoing cleansing and validation are necessary.
Real-Time Processing (Velocity): Businesses need quick analysis on live data
streams, which requires advanced tools and infrastructure.
Security and Privacy Risks: Sensitive data must be protected against
breaches using encryption and access control.
Skilled Workforce Shortage: Lack of trained professionals to handle big data
tools slows implementation and insight generation.
Cost and Scalability Issues: High expenses in storing and processing big data
demand strategic planning to optimize resources.

Q2: Differentiate between traditional Business

Intelligence (BI) architectures and modern Big Data
environments using appropriate architectural
concepts.
ANS:
Introduction:
Business Intelligence (BI) and Big Data environments both help organizations
process and analyze data to make better decisions. However, traditional BI
focuses mostly on structured data and batch processing, while Big Data

BAD601: BIG DATA ANALYTICS 3

 environments handle huge volumes, varieties, and speeds of data using modern
architectures.

Core Differences

1. Data Storage
Traditional BI: Uses a centralized Data Warehouse where all structured data is
stored after ETL (Extract, Transform, Load).
Example: A retail company stores sales transactions in a relational database.
Big Data: Uses a distributed file system like Hadoop Distributed File System
(HDFS) across many servers that scale horizontally. It stores structured, semi-
structured, and unstructured data.
Example: A social media platform stores user posts, images, and videos
distributed over many nodes.
2. Data Types Managed
Traditional BI: Mainly deals with structured data from ERP, CRM, and legacy
systems.
Big Data: Handles variety - structured, semi-structured (XML, JSON), and
unstructured data (videos, logs, emails).
3. Data Processing Mode
Traditional BI: Mostly batch processing — data is analyzed offline after
storage.
Example: Monthly sales report generation after the month ends.
Big Data: Supports both real-time and batch processing, allowing immediate
insights and actions.
Example: Real-time fraud detection in banking transactions.
4. Scalability
Traditional BI: Scales vertically (adding more power to a single server). Limited
by hardware capacity.
Big Data: Scales horizontally (adding more servers to the cluster), making it
more flexible and cost-effective for large data sets.

BAD601: BIG DATA ANALYTICS 4

 5. Data Processing Approach
Traditional BI: Move data to code — data is extracted and loaded into separate
analytical tools.
Big Data: Move code to data — analytical processing is done near or on the
data nodes to reduce data movement and improve speed.
6. Architecture Complexity and Use
Traditional BI: Simpler architecture with well-defined processes ideal for
historical data analysis and reporting.
Big Data: Complex architecture involving data lakes, distributed computing,
and advanced analytics for handling fast and diverse data.

Q3: Elaborate on the classification of analytics,

specifically detailing Descriptive, Predictive, and
Prescriptive paradigms with relevant industry use
cases.
ANS:

BAD601: BIG DATA ANALYTICS 5

 Introduction:
Analytics helps organizations make better decisions by examining data in different
ways. It can be classified into Descriptive, Predictive, and Prescriptive analytics,
each serving a unique purpose in understanding and acting on data.

1. Descriptive Analytics
Definition: Focuses on summarizing and describing past data to understand
what happened.
Purpose: To get insights from historical data using reports, dashboards, and
basic statistics.
Techniques: Data aggregation, data visualization, basic reporting.
Example Industry Use Case:
Retail stores analyze sales data from last month to find which products sold
the most. This helps them understand customer buying patterns and stock
accordingly.
Key Question Answered: What happened?
2. Predictive Analytics
Definition: Uses historical data and statistical/machine learning models to
forecast what might happen in the future.
Purpose: To predict trends, customer behaviors, risks, and opportunities.
Techniques: Regression analysis, classification, time series forecasting,
clustering.
Example Industry Use Case:
Banks use predictive models to forecast the probability of loan default by
analyzing the customer’s financial history and credit scores. This helps reduce
bad debts.
Key Question Answered: What could happen?
3. Prescriptive Analytics

BAD601: BIG DATA ANALYTICS 6

 Definition: Goes beyond predictions and suggests the best actions to take for
desired outcomes. Uses optimization and simulation techniques.
Purpose: To recommend decisions that can affect future results and maximize
benefits or minimize risks.
Techniques: Optimization algorithms, decision analysis, what-if simulations.
Example Industry Use Case:
An airline uses prescriptive analytics to decide optimal ticket pricing and
routing based on predicted demand to maximize revenue.
Key Question Answered: What should we do?

Q4: Critically analyze the CAP Theorem and discuss its

direct relevance and application within distributed Big
Data storage systems.
ANS:

BAD601: BIG DATA ANALYTICS 7

 Introduction:
The CAP Theorem is a fundamental concept in distributed computing that explains
the trade-offs among three key properties: Consistency, Availability, and Partition
Tolerance. Understanding CAP helps design Big Data storage systems that
balance these properties based on business needs.

CAP Theorem - Definition:

Proposed by Eric Brewer, CAP Theorem states that a distributed system can
achieve only two out of the following three guarantees at the same time:
Consistency (C): All nodes see the same data simultaneously.
Availability (A): Every request receives a response, regardless of individual
node failure.
Partition Tolerance (P): The system continues to operate despite network
failures splitting nodes into isolated groups.
Key Properties Explained:
Consistency (C):
Ensures data is the same across all nodes at any transaction point.
Example: In banking, all branches seeing the same account balance after a
transaction.
Availability (A):
System responds to all requests even if some nodes are down.

BAD601: BIG DATA ANALYTICS 8

 Example: An online store that continues taking orders even if some servers
fail.
Partition Tolerance (P):
The system keeps working despite communication breakdowns between
nodes.
Crucial in Big Data systems as network failures are common in distributed
environments.
How CAP Theorem Works in Distributed Big Data Storage
Big Data storage uses multiple nodes to store huge volumes of data
distributed across servers. Network failures (Partition) are inevitable. Hence,
Partition Tolerance is a must-have property.
This leaves a choice between:
Consistency + Partition Tolerance (CP) Systems:
Prioritize all nodes seeing the same data, but some requests may be delayed
or denied during partitions.
Example: Traditional relational databases or HBase, where consistency is
critical.
Availability + Partition Tolerance (AP) Systems:
System always responds, even if data might be inconsistent temporarily.
Example: DNS services or Cassandra, where availability matters more than
immediate consistency.
Direct Relevance & Application:
No Big Data system can guarantee all three simultaneously because of CAP.
So systems choose based on use case priority:
If consistency is critical (finance, healthcare), a CP system is preferred,
sacrificing availability during network issues.
If availability is key (social media, content delivery), an AP system is chosen,
accepting eventual consistency.

BAD601: BIG DATA ANALYTICS 9

 Distributed file systems like Hadoop HDFS prioritize partition tolerance but
focus on consistency and availability based on configuration and data
replication.
Cloud storage systems use CAP trade-offs to ensure fault tolerance and
responsiveness.

Q5: Provide a comparative analysis of SQL, NoSQL,

and NewSQL database architectures concerning
scalability, schema rigidity, and ACID compliance.
ANS:
Introduction:
Databases are essential for storing and managing data efficiently. Over time, three
main types of database architectures evolved — SQL, NoSQL, and NewSQL. Each
has its own way of handling scalability, data structure (schema), and ACID
properties, which are crucial for applications.

Core Explanation

1. Scalability:

SQL (Relational Databases):

Scale vertically, meaning better hardware (CPU, RAM) on a single server is
used to handle more load.
Limited horizontal scaling (across multiple servers) due to strict consistency
and joins.
Example: Oracle DB in banking uses vertical scaling for reliability.
NoSQL Databases:
Designed for horizontal scaling, which means distributing data across many
servers easily (scale-out).
Suited for massive volumes of data and large user loads.

BAD601: BIG DATA ANALYTICS 10

 Example: Cassandra or MongoDB runs on a cluster of cheap servers to handle
social media data.
NewSQL Databases:
Attempt to combine horizontal scaling of NoSQL with relational model
features.
Designed to handle OLTP workloads at scale with ACID guarantees.
Example: Google Spanner provides SQL interface with cloud-scale horizontal
efficiency.

2. Schema Rigidity:

SQL Databases:
Have a fixed schema, meaning the structure of data (tables, columns) must be
defined before inserting data.
Changes to schema require careful database migration, which can be
complex.
Example: MySQL requires predefined tables and data types for customer
records.
NoSQL Databases:
Offer flexible or dynamic schema, where data can be stored without a fixed
structure.
Allows storing varied and evolving data formats easily.
Example: MongoDB stores JSON-like documents where fields can vary by
record.
NewSQL Databases:
Usually maintain fixed schema like traditional SQL for relational integrity and
transaction safety.
Allows ACID compliance while supporting evolving apps through schema
evolution techniques.
Example: VoltDB supports relational schema but can scale like NoSQL.

BAD601: BIG DATA ANALYTICS 11

 3. ACID Compliance:

SQL Databases:
Provide full ACID compliance ensuring reliable transactions: Atomicity,
Consistency, Isolation, Durability.
Critical for applications requiring data correctness, like banking.
Example: PostgreSQL guarantees all ACID properties for financial transactions.
NoSQL Databases:
Often relax ACID properties for better availability and partition tolerance (CAP
theorem).
May use eventual consistency instead of immediate consistency to enhance
performance and scalability.
Example: Cassandra uses eventual consistency for fast writes in distributed
systems.
NewSQL Databases:
Support full ACID compliance like SQL but scale out horizontally like NoSQL.
Provide transactional guarantees in distributed environments.
Example: Google Spanner supports strong consistency with transactions at
global scale.

Q6: Detail the primary technologies utilized in Big Data

environments, including In-Memory Analytics, In-

BAD601: BIG DATA ANALYTICS 12

 Database Processing, Symmetric Multiprocessor
Systems (SMP), and Massively Parallel Processing
(MPP).
ANS:
Introduction:
Big Data environments require advanced technologies to handle huge volumes of
data efficiently and quickly. Some primary technologies that support high-speed
processing, analysis, and storage include In-Memory Analytics, In-Database
Processing, Symmetric Multiprocessor Systems (SMP), and Massively Parallel
Processing (MPP).
In-Memory Analytics
Definition: Stores data primarily in the RAM instead of slower hard disks for
faster access and analysis.
Why it matters: Accessing data from RAM is much quicker than from disks,
which drastically reduces query time.
How it works: Relevant datasets or pre-aggregated tables are loaded into RAM
so CPU fetches small subsets rapidly.
Example: Retailers use in-memory analytics to quickly analyze customer
purchase patterns during peak shopping hours to adjust pricing or promotions
instantly.
In-Database Processing (In-Database Analytics)
Definition: Runs analytical computations directly inside the database instead of
exporting data to external programs.
Why it matters: Eliminates time-consuming data transfers, speeding up
analytical processes and reducing errors.
How it works: Analytical functions are embedded as stored procedures or
built-in functions inside the database engine.
Example: Banks use in-database processing to detect fraud patterns by
running complex queries directly on transactional data stored within the
database.

BAD601: BIG DATA ANALYTICS 13

 Symmetric Multiprocessor Systems (SMP)
Definition: A system architecture where multiple processors share the same
memory and operate under a single OS instance.
Why it matters: Enables parallel processing since multiple CPUs can work on
different threads or tasks simultaneously with shared access to memory.
How it works: All processors communicate via a common memory bus for
faster data sharing but can become limited when scaling beyond a few
processors due to contention.
Example: Traditional enterprise servers running multi-threaded applications
like ERP systems use SMP to improve performance for concurrent user
access.

Massively Parallel Processing (MPP)

Definition: Uses many independent processors (nodes), each with its own
memory and storage, working in parallel on partitioned data.
Why it matters: Easily scales out by adding more nodes; ideal for handling
huge datasets typical in Big Data.
How it works: Large datasets are split into partitions distributed to different
nodes where processing happens simultaneously, and results are combined at
the end.
Example: Hadoop clusters use MPP to process petabytes of web log data
quickly by breaking the data into chunks processed by many nodes.

BAD601: BIG DATA ANALYTICS 14

 Q7: Define NoSQL and categorize the different types of
NoSQL databases (Document, Key-Value, Column-
family, Graph), highlighting their specific operational
use cases.
ANS:
Introduction:
NoSQL stands for "Not Only SQL." It represents a class of databases designed to
handle large volumes of structured, semi-structured, and unstructured data with
flexibility and scalability. NoSQL databases are often used where traditional
relational databases face limitations due to scale or variety of data.

Definition of NoSQL

NoSQL: A group of database technologies that do not require fixed schemas,

support horizontal scaling, and are designed for large-scale data storage and
real-time web applications.
Main features: Flexible schema, scalability, high availability, and support for
diverse data types.

BAD601: BIG DATA ANALYTICS 15

 Why NoSQL? Handles big data better by allowing dynamic data structures
unlike rigid relational models.
Types of NoSQL Databases and Use Cases
a) Document Databases
Description: Store data as semi-structured documents, typically in JSON,
BSON, or XML formats.
Operation: Each document stores related information as key-value pairs inside
it.
Use Case: Well-suited for content management, blogging platforms, storing
user profiles.
Example: MongoDB is widely used for e-commerce websites to store product
catalogs as documents.
b) Key-Value Stores
Description: Data is stored as pairs of keys and values, where the key is
unique, and the value is the actual data.
Operation: Optimized for fast retrieval of simple data, often used for caching
and session management.
Use Case: Perfect for shopping cart data where quick access and update is
needed.
Example: Redis is used by online retailers for managing real-time session data.
c) Column-family Stores
Description: Organize data into columns rather than rows; each column family
contains rows with different columns, allowing sparse data storage.
Operation: Efficient for queries involving large datasets and aggregation
operations.
Use Case: Ideal for data warehousing, IoT sensor data, and analyzing time-
series data.
Example: Apache Cassandra is used by telecom companies to handle call
detail records.

BAD601: BIG DATA ANALYTICS 16

 d) Graph Databases
Description: Store data in nodes and edges, representing entities and
relationships respectively.
Operation: Designed for complex relationship queries that traditional
databases struggle with.
Use Case: Social networks, recommendation engines, fraud detection.
Example: Amazon Neptune is used in social media platforms to model and
query friend relationships.

Q8: Explain the Shared Nothing Architecture and

enumerate its inherent advantages regarding fault
isolation and horizontal scalability.
ANS:
Introduction:
Shared Nothing Architecture is a distributed computing architecture where each
node (computer) works independently without sharing memory or disk storage. It
is widely used in big data systems and distributed databases to provide high
performance and scalability.

BAD601: BIG DATA ANALYTICS 17

 What is Shared Nothing Architecture?

Definition: A system design where each node is autonomous and does not
share memory or disk resources with other nodes.
Contrast: Unlike shared memory or shared disk architectures, here each node
owns its resources exclusively.
Applications: Used in distributed databases, big data platforms like Hadoop,
and high transaction rate multiprocessor systems.
Advantages of Shared Nothing Architecture
a) Fault Isolation
Independent Nodes: Since nodes do not share memory or disk, failure of one
node does not directly affect others.
Fault Containment: Any fault is contained within the failing node only,
preventing system-wide crashes.
Example: If one server in a Hadoop cluster fails, other servers continue
working without interruption, ensuring data availability.
b) Horizontal Scalability
Adding Nodes: New nodes can be added easily without impacting the existing
system.
No Resource Bottlenecks: Since no shared resource exists, adding nodes
increases processing power almost linearly.
Load Balancing: Data and workload are spread across nodes, improving
overall throughput.
Example: Web services like Netflix scale their data services by adding more
independent nodes to handle increasing user requests.

Q9: Trace the evolution of Big Data analytics from

Analytics 1.0 (Descriptive) through Analytics 3.0
(Prescriptive), detailing the technological shifts.

BAD601: BIG DATA ANALYTICS 18

 ANS:
Introduction:
Big Data Analytics has evolved significantly over time, moving from simple
descriptive analysis to advanced prescriptive analytics. This evolution is marked
by improvements in data sources, techniques, and technologies that have
enhanced decision-making in organizations.

Analytics 1.0 (Descriptive Analytics)

Era: Early stage before 2010

Focus: Understand what happened by analyzing past structured data using
descriptive statistics.
Technology Used: Relational databases, SQL queries, and basic statistical
methods.
Data Type: Mostly structured and internal data.
Example: A company reports last month's sales figures and customer counts
using spreadsheets or SQL reports.

Analytics 2.0 (Descriptive + Predictive Analytics)

BAD601: BIG DATA ANALYTICS 19

 Era: Around 2010 to 2012
Focus: Describe past as well as predict future events using advanced
statistics. Key questions are what will happen? and why will it happen?
Technological Shift: Integration of big data from external sources like social
media and sensors.
Use of Hadoop clusters and advanced predictive models.
Data Type: Structured and semi-structured data, arriving at high speed and
volume.
Example: E-commerce websites predict customer buying behavior from
recent browsing data plus historic sales.

Analytics 3.0 (Descriptive + Predictive + Prescriptive Analytics)

Era: 2012 to present

Focus: Not only predicts what and when but also recommends what actions to
take, combining past data and real-time insights for better decision-making.
Key Questions: What will happen? When? Why? What should be done?
Technological Shift: Blend of big data with legacy systems (ERP, CRM) and
third-party data.
Utilizes in-memory analytics, machine learning, artificial intelligence, and agile
analytic methods.
Data Type: Internal + external; structured + unstructured data sources.
Example: Ride-sharing apps analyze real-time demand and suggest dynamic
pricing to drivers and passengers.

Q10: Discuss the strategic importance of Big Data

Analytics across various industry sectors, focusing on
real-time proactive decision-making, cost reduction,
and social media listening.

BAD601: BIG DATA ANALYTICS 20

 ANS:
Introduction:
Big Data Analytics (BDA) plays a vital role in modern industries by providing
actionable insights. It helps organizations make quick decisions, reduce costs, and
understand consumer behavior better through social media listening.

Strategic Importance Across Industries:

Healthcare: Using BDA for real-time patient monitoring and early diagnosis,
helping doctors take proactive steps.
Example: Wearable devices sending real-time health data to predict
emergencies.
Retail: Analyzing customer data and social media to personalize offers and
improve customer experience, which boosts sales.
Example: Online stores recommending products based on recent browsing
and social media trends.
Finance: Detecting fraudulent transactions instantly and managing risks
efficiently with real-time analytics.
Example: Banks flagging suspicious activities immediately to prevent fraud.

BAD601: BIG DATA ANALYTICS 21

 Real-Time Proactive Decision-Making:

Definition: Making decisions instantly based on live data, rather than waiting
for reports.
Benefit: Enables organizations to respond quickly to market changes or
operational issues.
Example: E-commerce platforms adjusting prices dynamically during flash
sales based on real-time demand.

Cost Reduction:

Definition: Cutting down operational costs by optimizing processes using data

insights.
Benefit: Helps avoid wastage of resources and improves efficiency.
Example: Manufacturing companies predicting machine failures early to
reduce downtime and maintenance costs.

Social Media Listening:

Definition: Analyzing social media content to gauge public opinion and

sentiment.
Benefit: Helps brands to improve products and services based on consumer
feedback.
Example: Brands monitoring Twitter for complaints or compliments to act
swiftly and improve their image.

BAD601: BIG DATA ANALYTICS 22