1.

1 Classification of Digital Data

Digital data is mainly classified into three types:
 Structured data
 Semi-structured data
 Unstructured data

1. Unstructured Data
 Unstructured data does not follow any fixed format or data model.
 It is not easy for computer programs to process.
 About 80–90% of an organization’s data is unstructured.
 This type of data is growing very rapidly.
Examples:
 Text documents
 Emails (email body)
 Images and videos
 Audio files
 PowerPoint presentations
 Chat messages
 Research papers and white papers

2. Semi-Structured Data
 Semi-structured data does not have a fixed structure like tables.
 It contains some form of organization or tags.
 It is partially readable by machines.
 Metadata is available but not enough for easy processing.
Examples:
 Emails
 XML files
 HTML files
  JSON documents

3. Structured Data
 Structured data is well organized.
 It is stored in rows and columns.
 It is easy to store, search, and analyze using computer programs.
 Relationships exist between different data items.
Examples:
 Data stored in databases
 Student records
 Employee details
 Banking transactions

Role of RDBMS
 Since the 1980s, most enterprise data has been stored in Relational Database
Management Systems (RDBMS).
 RDBMS uses tables, rows, columns, primary keys, and foreign keys.
 Over time, RDBMS became more efficient, reliable, and cost-effective.
 Structured data is mainly stored in RDBMS.

Growth of Unstructured Data

 With the growth of the Internet, large amounts of data started coming from outside
organizations.
 This data includes social media, emails, videos, and online content.
 Most of this data is unstructured.
 According to Gartner, nearly 80% of enterprise data today is unstructured.
 Only about 10% of data is structured or semi-structured.

1.1.1 Structured Data

Structured data is the data that follows a predefined structure or schema. When data is
stored according to a fixed format, it is called structured data.

What is Structured Data?

 Data is said to be structured when it conforms to a predefined schema.
 A schema defines how data is stored, organized, and accessed.
 Structured data is easy to store, search, and process using computer programs.

Structured Data and Data Model

 Structured data follows a data model.
 A data model defines the types of data that will be stored for business use.
 Most structured data is stored in Relational Database Management Systems
(RDBMS).

Structured Data in RDBMS

 An RDBMS conforms to the relational data model wherein the data is stored in
rows/columns. Refer Table 1.1.
 Data is stored in the form of tables (relations).
 Each table contains Row / Record / Tuple and Column / Attribute / Field
 The number of rows/records/tuples in a relation is called the cardinality of a relation
and the number of columns is referred to as the degree of a relation.

 The first step in database design is to create a relation (table).

 Decide the required fields/columns to store data.
 Specify the data type for each column (integer, real, character, date, Boolean, etc.).
 Define constraints to maintain data integrity.
 Common constraints include UNIQUE, NOT NULL, and business rules (e.g., value
should not be less than 50).
 Domain constraints restrict allowed values (e.g., only “CS”, “IS”, or “MS”).
 An Employee table is an example of storing employee details in an organization.
 The table schema includes table name, meaningful column names, data types,
lengths, and constraints.
 A well-designed table strictly follows the relational data model.

 It goes without saying that each record in the table will have exactly the same
structure. Let us take a look at a few records in Table 1.3.
 In an RDBMS, tables can be related to each other. For example, the Employee table
is related to the Department table. This relationship is created using a common
column called DeptNo.
 It is not necessary that both tables have the same column name. The relationship
depends on the values stored in the column, not on the column name.
 This relationship is called referential integrity. "DeptNo". Given in Figure 1.3 is a
depiction of referential integrity constraint (primary - foreign key) with the
"Department" table being the referenced table and "Employee" table being the
referencing table.

[Link] Sources of Structured Data

 When data is highly structured, it can be stored in an RDBMS. Many RDBMS

software systems are available for this purpose.
 available RDBMS [Oracle Corp. - Oracle, IBM - DB2, Microsoft - Microsoft SQL Server,
EMC- Greenplum, Teradata - Teradata, MySQL (open source), PostgreSQL (advanced
open source), etc.].
 Refer Figure 1.4. These databases are typically used to hold transaction/operational
data generated and collected by day-to-day business activities. In other words, the
data of the On-Line Transaction Processing (OLTP) systems are generally quite
structured.
[Link] Ease of Working with Structured Data

Structured data is organized in a predefined schema (rows and columns), which makes it
easy to store, manage, and analyze using traditional Relational Database Management
Systems (RDBMS). The ease of working with structured data can be explained as follows
(Refer Figure 1.5):

1. Insert/update/delete: The Data Manipulation Language (DML) operations provide the

required ease with data input, storage, access, process, analysis, etc.
2. Security
Security of information is ensured using encryption and tokenization, which protect data
throughout its lifecycle. Only authorized users can decrypt and access sensitive
information, helping organizations maintain control and compliance.
[Link]:
An index is a data structure that improves data retrieval speed, especially for SELECT
operations, by using extra storage space and causing additional write overhead, which is
acceptable due to faster search performance.
[Link]:
In traditional RDBMS, scalability is achieved by scaling up the database server through
increasing storage capacity and processing power (CPU, memory, and peripheral
storage).

[Link] Processing:
RDBMS supports ACID properties of transactions:
Atomicity: A transaction is atomic, means that either it happens in its entirety or none of
it at all.
 Consistency: The database moves from one consistent state to another consistent state.
In other words, if the same piece of information is stored at two or more places, they are
in complete agreement.
Isolation: The resource allocation to the transaction happens such that the transaction
gets the impression that it is the only transaction happening in isolation.
Durability: All changes made to the database during a transaction are permanent and
that accounts for the durability of the transaction.

1.1.2 Semi-Structured Data

 data is also referred to as self-describing structure.

Refer Figure 1.6. It has the following
 Does not follow strict relational data models or table
structures.
 Uses tags to represent semantic elements.
 Tags help in defining hierarchies of records and fields.
 No clear separation between data and schema;
structure depends on the purpose.
 Entities of the same type may have different attributes
or different attribute order, which is acceptable.

[Link] Sources of Semi-Structured Data

The main sources of semi-structured data are XML and JSON:

 XML (eXtensible Markup Language):
is hugely popularized by web services developed utilizing the Simple Object Access
Protocol (SOAP) principles..

 JSON (JavaScript Object Notation):

Used for data exchange between a server and web applications. Popular in REST-based
web services. Databases like MongoDB and Couchbase store data natively in JSON
format.

An example of HTML is as follows:

<HTML>
<HEAD>
<TITLE>Place your title here</TITLE>
</HEAD>
<BODY BGCOLOR="FFFFFF">
<CENTER><IMG SRC="[Link]" ALIGN="BOTTOM"></CENTER>
<HR>
<a href="[Link] Name</a>
<H1>this is a Header</H1>
<H2>this is a sub Header</H2>
Send me mail at <a href="[Link]
support@[Link]</a>.
<P>a new paragraph!
<P><B>a new paragraph!</B>
<BR><B><I>this is a new sentence without a paragraph break, in bold italics.</I></B>
<HR>
<BODY>
</HTML>
Sample JSON document
id:9,
BookTitle: "Fundamentals of Business Analytics",
AuthorName: "Seema Acharya",
Publisher: "Wiley India",
YearofPublication: "2011”

1.1.3 Unstructured Data:

Unstructured data does not conform to any predefined data model. It includes various
types of text and other data whose structure is highly unpredictable, as shown in Table
1.4. Figure 1.8 illustrates additional sources of unstructured data.

[Link] Issues with "Unstructured" Data

· Unstructured data does not conform to a predefined data model and is not organized
in a predefined manner.
· In some cases, data classified as unstructured may still show implied structure.
· As shown in Figure 1.9, data may be placed in the unstructured category despite
having some structure or being highly structured.
· It is often argued that a text file should be considered semi-structured data.
· A text file has:
 A file name
 Metadata such as owner, creation date, and file size
· This metadata provides limited structural information.
· For analysis, the content of the text file is more important than file properties.
· Metadata usually does not contribute significantly to the processing or analysis task.
· The content of text files:
 Lacks a fixed schema
 Is not machine-interpretable
 Varies across files
· Hence, despite having metadata, text files are rightly classified as unstructured data

[Link] How to Deal with Unstructured Data?

· Today, unstructured data accounts for nearly 80% of the data generated in most
enterprises.
· The proportion of unstructured data is rapidly increasing, as illustrated in Figure 1.10.
· There is a clear shift in balance toward unstructured data.
· Due to its large volume, unstructured data cannot be ignored by organizations.
· Figure 1.11 highlights various approaches used to deal with unstructured data.

Techniques Used to Interpret Unstructured Data

 The following techniques are used to find patterns in and interpret unstructured
data.
1. Data Mining
 Deals with large datasets.
 Uses techniques from:
o Artificial Intelligence
o Machine Learning
o Statistics
o Database Systems
 Helps to discover consistent patterns and systematic relationships between
variables.
 Represents the analysis step in the Knowledge Discovery in Databases (KDD) process.
Popular Data Mining Algorithms
 Association Rule Mining
o Also known as market basket analysis or affinity analysis.
o Answers the question: “What goes with what?”
o Identifies products that are frequently purchased together.
o Example:
 If a customer buys bread, they are likely to buy eggs or cheese.
 Regression Analysis
o Used to predict the relationship between variables.
o The variable to be predicted is called the dependent variable.
o Variables used for prediction are called independent variables.

2. Collaborative Filtering
 Predicts a user’s preferences based on the preferences of similar users. Table 1.5
 Uses known preferences of a user and compares them with other user profiles.
 Helps in recommendation and personalization.
 Example:
o Based on similar users’ learning preferences, User 4 is predicted to prefer
video-based learning rather than textual learning.

3. Text Analytics / Text Mining

 Text data is largely unstructured and difficult to process algorithmically.
 Text mining extracts high-quality and meaningful information from text.
 Uses statistical pattern learning to identify patterns and trends.
 Includes tasks such as:
 o Text categorization
o Text clustering
o Sentiment analysis
o Concept/entity extraction
4. Natural Language Processing (NLP)
 Related to human–computer interaction.
 Enables computers to understand and process human (natural) language input.
5. Noisy Text Analytics
 Extracts structured or semi-structured information from noisy unstructured data.
 Sources include:
o Chats, blogs, wikis, emails, message boards, text messages
 Noisy text may contain:
o Spelling mistakes
o Abbreviations and acronyms
o Non-standard words
o Missing punctuation and letter case
o Filler words like “uh”, “um”
6. Manual Tagging with Metadata
 Involves manually assigning metadata to unstructured data.
 Helps in adding semantic meaning for better understanding and analysis.
7. Part-of-Speech (POS) Tagging
 Also called POS, POST, or grammatical tagging.
 Tags each word in a sentence with its grammatical role.
 Examples: noun, verb, adjective, etc.
8. Unstructured Information Management Architecture (UIMA)
 An open-source platform developed by IBM.
 Used for real-time content analytics.
 Processes text and other unstructured data to discover:
o Latent meanings
 o Relevant relationships within data

CHAPTER 2 : Introduction to Big Data

2.1 CHARACTERISTICS OF DATА

1. Composition
 Refers to the structure of data
 Includes:
o Data sources
o Granularity
o Data types (structured, semi-structured, unstructured)
o Nature of data (static or real-time streaming)

2. Condition
 Refers to the state and quality of data
 Determines whether data:
o Can be used directly for analysis, or
o Requires cleansing, enhancement, or enrichment
 Deals with missing, noisy, or inconsistent data
3. Context
 Refers to the background and meaning of data.
o Where, why, and when data is generated
o Sensitivity of data
o Events associated with the data
The other following are six other characteristics of data which are : 1. Accuracy 2. Validity 3.
Reliability 4. Timeliness 5. Relevance 6. Completeness
Small Data vs Big Data
Small Data
 Refers to data before the Big Data revolution
 Characterized by certainty
 Data sources are known and well-defined
 No major changes in data composition or context
 Clear understanding of:
o Why the data was generated
o Where and when it was generated
o How it will be used
o What questions it can answer

Big Data
 Characterized by high complexity
 Involves multiple and unknown data sources/sets
 Data volume is very large and rapidly increasing
 Generated and processed at high speed
 Includes a wide variety of data:
o Internal and external
o Behavioural and social

2.2 EVOLUTION OF BIG DATA

  1970s and before were the era of mainframes
 Data during this period was primitive and structured
 1980s and 1990s saw the evolution of relational databases
 This period was the era of data-intensive applications
 The World Wide Web (WWW) and Internet of Things (IoT) led to an onslaught of
data
 Data generated includes structured, unstructured, and multimedia data. Refer Table
2.1.

2.3 DEFINITION OF BIG DATA

 When asked “Define Big Data”, the following responses are commonly heard:
1. Data that goes beyond human and technical infrastructure needed for
storage, processing, and analysis
2. What is BIG today may become NORMAL tomorrow
3. Data measured in terabytes, petabytes, or zettabytes
4. Big Data is about the 3 Vs
 Refer Figure 2.2
 All the above responses are correct
 Big Data is not just one of these, but all of these and more

Standard Definition of Big Data (Gartner)

Big data is high-volume, high-velocity, and high-variety information assets that demand cost
effective, innovative forms of information processing for enhanced insight and decision
making.

The 3Vs concept was proposed by the Gartner analyst Doug Laney in a 2001 MetaGroup
research publication, titled, 3D Data Management: Controlling Data Volume, Variety and
Velocity.

For the sake of easy comprehension, we will look at the definition in three parts. Refer
Figure 2.3.
Part I of the definition "big data is high-volume, high-velocity, and high-variety information
assets"
talks about voluminous data (humongous data) that may have great variety (a good mix of
structured, semi-structured, and unstructured data) and will require a good speed/pace for
storage, preparation, processing, and analysis.
Part II of the definition "cost effective, innovative forms of information processing" talks
about embracing new techniques and technologies to capture (ingest), store, process,
persist, integrate, and visualize the high-volume, high-velocity, and high-variety data.
Part III of the definition "enhanced insight and decision making" talks about deriving deeper,
richer, and meaningful insights and then using these insights to make faster and better
decisions to gain business value and thus a competitive edge.
Data → Information → Actionable intelligence → Better decisions → Enhanced business
value.

2.4 CHALLENGES WITH BIG DATА

Refer Figure 2.4. Following are a few challenges with big data:
1. Exponential growth of data
o Data is growing at a very fast rate
o Most of today’s data has been generated in the last 2–3 years
o Data volume will continue to increase
 o Key questions include:
 Will all this data be useful for analysis?
 Should we analyze all data or only a subset?
 How do we separate knowledge from noise?
2. Cloud computing and virtualization
o Cloud computing and virtualization are here to stay
o Cloud helps manage big data infrastructure due to:
 Cost efficiency
 Elasticity
 Easy upgrading and downgrading
o Creates challenges in deciding whether to host big data solutions outside the
enterprise
3. Data retention period
o Challenge in deciding how long big data should be retained
o Some data is useful for long-term decision making
o Some data becomes irrelevant or obsolete within hours of generation
4. Shortage of skilled professionals
o There is a dearth of skilled data science professionals
o High proficiency in data sciences is required
o This skill gap affects effective implementation of big data solutions
5. Technical and operational challenges
o Challenges related to:
 Data capture
 Storage
 Preparation
 Search
 Analysis
 Transfer
 Security
 Visualization
 o Big data exceeds the storage capacity of traditional database tools
o No explicit definition of how large data must be to qualify as big data
o Data is:
 Too large
 Moves very fast
 Does not fit traditional database structures
o Data is highly dynamic and must be ingested quickly
6. Data visualization challenge
o Data visualization is emerging as a separate discipline
o There is a shortage of business visualization experts

2.5 WHAT IS BIG DATА?

Big data is data that is big in volume, velocity, and variety. Refer Figure 2.5.

2.5.1 Volume
We have seen it grow from bits to bytes to petabytes and exabytes. Refer Table 2.2 and
Figure 2.6.

Bits → Bytes → Kilobytes → Megabytes → Gigabytes → Terabytes → Petabytes → Exabytes

→ Zettabytes → Yottabytes
[Link] Where Does This Data Get Generated?
 Big data is generated from multiple sources
 Examples of data types:
o Unstructured data:
 XLS, DOC, PDF files
 YouTube videos
 Internet chat conversations
 CCTV footage
 Weather forecast reports
o Semi-structured data:
 Customer feedback forms on online retail websites
 Refer Figure 2.7 for sources of big data

Sources of Big Data

1. Internal Data Sources
 Data present within an organization’s firewall
 Typical internal data sources: Data present within an organization's firewall. It is as
follows:
 Data storage: File systems, SQL (RDBMSs - Oracle, MS SQL Server, DB2, MySQL,
PostgreSQL, etc.), NoSQL (MongoDB, Cassandra, etc.), and so on.
  Archives: Archives of scanned documents, paper archives, customer correspondence
records, patients' health records, students' admission records, students' assessment
records, and so on.
2. External data sources:
 Data residing outside an organization's firewall. It is as follows:
 Public Web: Wikipedia, weather, regulatory, compliance, census, etc.
3. Both(internal + external data sources)
 Sensor data: Car sensors, smart electric meters, office buildings, air conditioning
units, refrigerators, and so on.
 Machine log data: Event logs, application logs, Business process logs, audit logs,
clickstream data, etc.
 Social media: Twitter, blogs, Facebook, LinkedIn, YouTube, Instagram, etc. Business
apps: ERP, CRM, HR, Google Docs, and so on.
 Media: Audio, Video, Image, Podcast, etc. Docs: Comma separated value (CSV),
Word Documents, PDF, XLS, PPT, and so on.

2.5.2 Velocity
 Refers to the speed at which data is generated and processed
 We have moved from the days of batch processing (remember our payroll
applications) to real-time processing.
 Batch → Periodic → Near real time → Real-time processing

2.5.3 Variety
 Refers to the diversity of data types and sources of data.
 We will study this under three categories: Structured data, semi-structured data and
unstructured data.
1. Structured data: From traditional transaction processing systems and RDBMS, etc.
2. Semi-structured data: For example Hyper Text Markup Language (HTML),
eXtensible Markup Language (XML).
3. Unstructured data: For example, unstructured text documents, audios, videos,
emails, photos, PDFs, social media, etc.

2.7 WHY BIG DATA?

 The more data available for analysis, the higher the analytical accuracy
 Increased analytical accuracy leads to greater confidence in decision making
  Improved decisions result in:
o Enhanced operational efficiencies
o Reduction in cost
o Reduction in time
o Innovation of new products
o Innovation of new services
o Optimization of existing services
 Refer Figure 2.8
 More data → More accurate analysis → Greater confidence in decision making →
Greater operational efficiencies, cost reduction, time reduction, new product
development, and optimized offerings, etc.

2.9 TRADITIONAL BUSINESS INTELLIGENCE (BI) VERSUS BIG DATA

Differences Between Traditional BI and Big Data

1. Data storage
o In traditional BI, all enterprise data is stored on a central server
o In big data, data resides in a distributed file system
o Distributed file systems scale horizontally (scale in/out), while traditional
database servers scale vertically
2. Mode of analysis
o Traditional BI analyzes data mainly in offline mode
o Big data supports analysis in both real-time and offline modes
3. Data type and processing approach
o Traditional BI deals primarily with structured data
 o Data is moved to processing functions (move data to code)
o Big data deals with structured, semi-structured, and unstructured data
o Processing functions are moved to data (move code to data)

2.10 A TYPICAL DATA WAREHOUSE ENVIRONMENT

Typical Data Warehouse (DW) Environment
 Operational/transactional/day-to-day business data comes from ERP (Enterprise
Resource Planning) systems, CRM (Customer Relationship Management) systems,
legacy systems, and third-party applications.
 Data formats may vary: RDBMS (Oracle, MS SQL Server, DB2, MySQL, Teradata),
spreadsheets (.xls, .xlsx), .csv files, text (.txt) files.

 Data may originate from the same or different geographies

 Data is integrated, cleaned, transformed, and standardized using ETL (Extraction,
Transformation, and Loading)
 Transformed data is loaded into Enterprise Data Warehouse (EDW – enterprise
level) or Data Marts (DM – business/unit/functional level)
 Business Intelligence (BI) tools support decision making via ad-hoc queries, SQL
(Structured Query Language), dashboards, and data mining Refer Figure 2.9

2.11 A TYPICAL HADOOP ENVIRONMENT

 Hadoop deals with disparate data sources, such as web logs, images, audio, videos,
social media data, documents, PDFs, etc.
 Data includes both internal data (within company firewall) and external data
(outside firewall)
 Data is stored in Hadoop Distributed File System (HDFS)
 If needed, data can be repopulated back to operational systems, fed to Enterprise
Data Warehouse (EDW), Data Marts (DM), or Operational Data Store (ODS) for
further processing and analysis. Refer Figure 2.10
CHAPTER 3: Big Data Analytics

3.2 WHAT IS BIG DATA ANALYTICS?

Big Data Analytics is…

 Technology-enabled analytics
o Uses analytics and visualization tools from vendors such as IBM, Tableau, SAS,
R Analytics, Statistica, World Programming Systems (WPS), etc.
o Helps in processing and analyzing big data
 Gaining meaningful, deeper, and richer insights
o Helps steer the business in the right direction
o Enables understanding customer demographics
o Supports cross-selling and up-selling
o Helps in better leveraging vendors and suppliers
 Improved decision making and competitive edge
o Provides findings that enable faster and better decisions
o Helps gain a competitive advantage over competitors
 Collaboration among key communities
o Requires a tight handshake between IT, business users, and data scientists
o Refer Figure 3.3
  Handling very large and diverse datasets
o Works with data whose volume and variety exceed traditional storage and
processing capabilities
o Infrastructure limitations of enterprises are addressed
 Moving code to data
o Programs for distributed processing are small in size (a few KBs)
o Data sizes are very large (Terabytes, Petabytes today, Exabytes and Zettabytes
in the future)

3.5 CLASSIFICATION OF ANALYTICS

There are basically two schools of thought:
1. Those that classify analytics into basic, operationalized, advanced, and
monetized.
2. 2 Those that classify analytics into analytics 1.0, analytics 2.0, and analytics
3.0.
3.5.1 First School of Thought
1. Basic analytics: This primarily is slicing and dicing of data to help with basic business
insights. This is about reporting on historical data, basic visualization, etc.
2. Operationalized analytics: It is operationalized analytics if it gets woven into the
enterprise's business processes.
3. Advanced analytics: This largely is about forecasting for the future by way of predictive
and prescriptive modeling.
4. Monetized analytics: This is analytics in use to derive direct business revenue.

3.5.2 Second School of Thought

Let us take a closer look at analytics 1.0, analytics 2.0, and analytics 3.0. Refer Table 3.1.
Figure 3.6 shows the subtle growth of analytics from Descriptive → Diagnostic→ Predictive
→ Prescriptive analytics.

3.8 WHY IS BIG DATA ANALYTICS IMPORTANT?

Approaches to Data Analysis

The analysis of data can be broadly classified into reactive and proactive approaches:
1. Reactive – Business Intelligence (BI):
Business Intelligence (BI) helps organizations make faster and better decisions by
delivering the right information to the right person at the right time in the right
format. It focuses on analyzing historical data and presenting results through
enterprise dashboards, reports, alerts, and notifications. BI supports both pre-
defined reports and ad-hoc queries.
 2. Reactive – Big Data Analytics:
Analysis is performed on very large datasets; however, the approach remains reactive
since it relies on static, historical data.
3. Proactive – Analytics:
Proactive analytics supports future-oriented decision making using data mining,
predictive modeling, text mining, and statistical analysis. This approach mainly uses
traditional database management techniques, leading to limitations in storage and
processing when handling big data.
4. Proactive – Big Data Analytics:
This approach involves analyzing massive volumes of data (terabytes, petabytes,
exabytes) to extract relevant information. It includes high-performance analytics to
gain rapid insights and solve complex problems by leveraging large-scale data.
3.12 TERMINOLOGIES USED IN BIG DATA ENVIRONMENTS

To understand the big data environment, it is important to become familiar with key
terminologies.
3.12.1 In-Memory Analytics
 Accessing data from non-volatile storage like hard disks is slow; pre-processing and
storing aggregates (cubes, aggregate tables, query sets, etc.) helps but requires
advance planning and reprocessing if data needs change.
 In-memory analytics addresses this by storing all relevant data in Random Access
Memory (RAM) instead of secondary storage.
 This results in faster data access, rapid deployment, better insights, and minimal IT
involvement.

3.12.2 In-Database Processing

 In-database processing, also called in-database analytics, integrates data
warehouses with analytical systems.
 Data from On Line Transaction Processing (OLTP) systems is cleaned through
Extraction, Transformation, and Loading (ETL) and stored in Enterprise Data
Warehouse (EDW) or data marts.
 Computations are executed within the database itself, eliminating data export,
saving time, and are supported by leading database vendors.

3.12.3 Symmetric Multiprocessor System (SMP)

  Symmetric Multiprocessor System (SMP) has two or more identical processors
sharing a single common main memory and controlled by a single operating system.
 All processors have full access to I/O devices and are tightly coupled.
 Each processor has its own cache memory and they are connected through a system
bus. Refer Figure 3.9.

3.12.4 Massively Parallel Processing (MPP)

 Massively Parallel Processing (MPP) uses multiple processors working in parallel,
each with its own operating system and dedicated memory, to execute different
parts of the same program.
 Processors communicate through a messaging interface, making MPP systems more
complex to program.
 MPP differs from Symmetric Multiprocessing (SMP), where processors share the
same memory and operating system (tightly coupled multiprocessing).

3.12.5 Difference Between Parallel and Distributed Systems

Here’s your text with figure numbers added while keeping everything else the same:

 Parallel Database Systems:

o Tightly coupled; processors cooperate for query processing (Refer Fig 3.10).

o Users are unaware of parallelism; processors share common memory (Refer Fig 3.11)
or use message passing.

 Distributed Database Systems:

 o Loosely coupled; composed of individual machines, each running its own
applications (Refer Fig 3.12).

o Data is distributed across multiple machines, requiring access to several machines to

answer a query (Refer Fig 3.13).

3.12.6 Shared Nothing Architecture

Three Common Multiprocessor Architectures
1. Shared Memory (SM): Multiple processors share a common central memory.
2. Shared Disk (SD): Multiple processors share a common set of disks but have private
memory.
3. Shared Nothing (SN): Neither memory nor disk is shared among processors; each
processor has its own memory and disk.
[Link] Advantages of a "Shared Nothing Architecture"
1. Fault Isolation: Faults in a single node are contained to that node and only
communicated via messages, preventing system-wide impact.
2. Scalability: Since memory and disks are not shared, nodes do not need to wait for
access to shared resources, allowing the system to scale easily without
synchronization bottlenecks.

3.12.7 CAP Theorem Explained

 CAP Theorem also called as Brewer's Theorem

 In a distributed computing environment (interconnected nodes sharing data), it is
impossible to guarantee all three of the following simultaneously.
 Refer Figure 3.14. You can at most have two out of three; one must be sacrificed:

[Link] CAP Theorem

Let us spend some time understanding the earlier mentioned terms.
1. Consistency (C): Every read fetches the most recent write.
 2. Availability (A): All reads and writes always succeed, returning a response in
reasonable time.
3. Partition Tolerance (P): The system continues to function even if network partitions
occur.
Real-Life Example (Training Institute "XYZ"):
 50 instructors report to a training coordinator.
 Schedules are maintained by an office administrator (Amey).
 When an additional admin (Joey) is added, updates may go to one admin but not the
other → inconsistency.
 Plan: Both admins update all schedules or share updates via email when one is
absent → Consistency & Availability.
 If admins stop communicating temporarily → system is Partition Tolerant but not
fully Available.

Key Insight: In a distributed system, you can at most guarantee two of the three properties
simultaneously:

Choice Description When to Choose

CA (Consistency + Always correct and available; may Critical transactions requiring

Availability) fail under partition atomicity

CP (Consistency + Consistent and tolerates Systems needing strict

Partition Tolerance) partitions; may not always respond correctness

AP (Availability + Always responds; may return Systems allowing flexible,

Partition Tolerance) slightly stale data eventually consistent data

Decision Guideline:
 Choose Availability over Consistency if your business can tolerate delayed updates.
 Choose Consistency over Availability if strict, atomic reads/writes are required.
CAP Theorem – Summary
 At most two out of three guarantees can be achieved at a time:
1. Consistent: Always get the latest updated information.
2. Available: Schedule info always accessible if admins are present.
3. Partition Tolerant: System works even if communication between admins is
lost.
 Choosing Between Consistency (C) and Availability (A):
1. Prioritize Availability if some flexibility in data synchronization is acceptable.
2. Prioritize Consistency if atomic reads and writes are critical.
 Database Examples Following CAP Combinations:
1. AP: Availability + Partition Tolerance
2. CP: Consistency + Partition Tolerance
3. CA: Consistency + Availability
 Reference Figure: Refer Fig 3.15 for databases adhering to two of the three CAP
guarantees.

CHAPTER 4: The Big Data Technology Landscape

4.1 NoSQL (NOT ONLY SQL)

NoSQL Databases
  Origin:
o Term first coined by Carlo Strozzi (1998) for his lightweight, open-source
relational database without SQL interface.
o Johan Oskarsson (2009) reintroduced the term at an event on open-source
distributed networks.
o Eric Evans and other developers popularized it to describe non-relational
databases.
 Key Features: Open-source, Non-relational, Distributed, Schema-less, Cluster-
friendly, Designed for 21st-century web applications.

4.1.1 Where is it Used?

NoSQL Databases: Widely used in big data and real-time web applications (Refer Fig 4.1),
used to store log data, social media data, and other types of data that cannot be easily
stored or analyzed in RDBMS.

4.1.2 What is it?

NoSQL stands for Not Only SQL. These are non-relational, open source, distributed
databases. They are hugely popular today owing to their ability to scale out or scale
horizontally and the adeptness at dealing with a rich variety of data: structured, semi-
structured and unstructured data. Refer Figure 4.2 for additional features of NoSQL.
1. Are non-relational: They do not adhere to relational data model, In fact, they are
either key-value pairs or document-oriented or column-oriented or graph-based
databases.
2. Are distributed: They are distributed meaning the data is distributed across several
nodes in a cluster constituted of low-cost commodity hardware..
3. Offer no support for ACID properties (Atomicity, Consistency, Isolation, and
Durability): Follow CAP theorem; may compromise consistency for availability and
partition tolerance.
 4. Provide no fixed table schema: Flexible structure; no fixed table schema required at
storage.

4.1.3 Types of NoSQL Databases

we have already stated that NoSQL databases are non-relational. They can be broadly
classified into the following:
1. Key-value or the big hash table.
2. Schema-less.
Refer Figure 4.3. Let us take a closer look at key-value and few other types of schema-less
databases:
1. Key-value: It maintains a big hash table of keys and values. For example,
Dynamo, Redis, Riak, etc.
Sample Key-Value Pair in Key-Value Database.

2. Document: It maintains data in collections constituted of documents. For

example, MongoDB, Apache CouchDB, Couchbase, MarkLogic, etc.

Sample Document in Document Database

{
"Book Name": "Fundamentals of Business Analytics",
"Publisher": "Wiley India",
"Year of Publication": "2011"
}
 3. Column: Each storage block has data from only one column. For example:
Cassandra, HBase, etc.

4. Graph: They are also called network database. A graph stores data in nodes.
For example, Neo4j, HyperGraphDB, etc.
Sample Graph in Graph Database

Refer Table 4.1 for popular schema-less databases.

4.1.4 Why NoSQL?

Key Features of NoSQL Databases
1. Scale-out architecture instead of monolithic architecture of relational databases.
2. Can house large volumes of structured, semi-structured, and unstructured data.
3. Dynamic schema: Allows insertion of data without a pre-defined schema, supporting
faster development, easy code integration, and less database administration.
4. Auto-sharding: Automatically distributes data across servers, balances load, and
handles server failures without disruption.
 5. Replication: Supports replication, ensuring high availability, fault tolerance, and
disaster recovery.

4.1.5 Advantages of NoSQL

Let us enumerate the advantages of NoSQL. Refer Figure 4.4.
1. Can easily scale up and down: Supports rapid, elastic scaling, including cloud
deployment.

o Cluster scale: Distributes database across 100+ nodes in multiple data

centers.
o Performance scale: Handles over 100,000 reads/writes per second.
o Data scale: Can store 1 billion+ documents.
2. Doesn't require a pre-defined schema: Flexible schema allows documents in a
collection to have different key-value pairs.
Example:
{_id: 101, "BookName": "Fundamentals of Business Analytics", "AuthorName":
"Seema Acharya", "Publisher": "Wiley India"}
{_id: 102, "BookName": "Big Data and Analytics"}
3. Cheap and easy to implement: Provides scalability, high availability, and fault
tolerance while reducing operational costs.
 4. Relaxes data consistency requirement: Adheres to CAP theorem (Consistency,
Availability, Partition tolerance); most NoSQL databases compromise consistency for
availability and partition tolerance but follow eventual consistency.
5. Data replication and partitioning:
o Sharding: Automatically distributes data across multiple servers, balances
load, and handles server failures transparently. Servers can be added or
removed without downtime. Data and query loads are automatically balanced
across servers.
o Replication: Stores multiple copies across nodes or data centers, ensuring
high availability and fault tolerance.

4.1.6 What We Miss With NoSQL?

Even with NoSQL addressing scale and schema flexibility, a few features of conventional
RDBMS are missed (Refer Figure 4.5):

1. Joins: NoSQL does not support joins, though embedded documents (e.g., in
MongoDB) can partially compensate.
2. ACID Transactions: NoSQL lacks full ACID property support but follows Brewer’s CAP
theorem.
3. Standard SQL Interface: NoSQL does not have standard SQL; databases like
MongoDB and Cassandra provide their own query languages (MQL, CQL).
4. Easy Integration: Integration with other SQL-based applications is limited compared
to RDBMS.

4.1.7 Use of NoSQL in Industry

NoSQL is being put to use in varied industries. They are used to support analysis for
applications such as web user data analysis, log analysis, sensor feed analysis, making
recommendations for upsell and cross-sell. etc. Refer Figure 4.6.

4.1.8 NoSQL Vendors

Refer Table 4.2 for few popular NoSQL vendors.

4.1.9 SQL versus NoSQL

Refer Table 4.3 for few salient differences between SQL and NoSQL.
4.1.10 What is NewSQL?
 NewSQL is a modern RDBMS that combines the best of SQL and NoSQL.
 It offers the scalable performance of NoSQL for OLTP while maintaining ACID
guarantees of traditional databases.
 Supports relational data model and uses SQL as the primary interface.

[Link] Characteristics of NewSQL

Refer Figure 4.7 to learn about the characteristics of NewSQL. NewSQL is based on the
shared nothing architecture with a SQL interface for application interaction.
4.1.11 Comparison of SQL, NoSQL, and NewSQL

Refer Table 4.4 for a comparative study of SQL, NoSQL and NewSQL.

4.2 HADOOP

 Open-source project of the Apache Foundation, written in Java.

 Developed by Doug Cutting in 2005, named after his son's toy elephant.
 Originally created to support distribution for the Nutch text search engine.
 Built on concepts from Google’s MapReduce and Google File System.
 Core part of computing infrastructure for companies like Yahoo, Facebook, LinkedIn,
Twitter.
 Refer Figure 4.8.

4.2.1 Features of Hadoop

  Hadoop Distributed File System (HDFS) and MapReduce form the core components.
 Optimized for massive structured, semi-structured, and unstructured data using
commodity hardware.
 Follows a shared-nothing architecture.
 Replicates data across multiple machines for fault tolerance.
 Designed for high throughput, not low latency; best for batch processing large
datasets.
 Complements On-Line Transaction Processing (OLTP) and On-Line Analytical
Processing (OLAP); not a replacement for RDBMS.
 Not suitable for tasks that cannot be parallelized or have interdependent data.
 Not efficient for processing small files; performs best with large datasets.

4.2.2 Key Advantages of Hadoop

Refer Figure 4.9 for a quick look at the key advantages of Hadoop. Some of them are as
follows:
1. Stores data in native format: HDFS is schema-less; structure is applied only during
processing.
2. Scalable: Can store and distribute massive datasets (thousands of TBs) across
hundreds of inexpensive servers operating in parallel.
3. Cost-effective: Scale-out architecture reduces storage and processing costs per
terabyte.
4. Resilient to failure: Fault-tolerant; data is replicated across multiple nodes ensuring
availability in case of node failure.
5. Flexible: Handles structured, semi-structured, and unstructured data; useful for log
analysis, data mining, recommendation systems, and more.
6. Fast: High processing speed due to the “move code to data” paradigm.
 Follows a shared-nothing architecture.

4.2.3 Versions of Hadoop

There are two versions of Hadoop available:
1. Hadoop 1.0
2. Hadoop 2.0
Let us take a look at the features of both. Refer Figure 4.10.

[Link] Hadoop 1.0

Hadoop Components and Limitations
1. Main Parts of Hadoop:
 Data Storage Framework: Hadoop Distributed File System (HDFS): General-purpose,
schema-less file system. Stores data in its original format, providing flexibility and
agility for business use.

 Data Processing Framework (MapReduce): Functional programming model with two

main functions:
o MAP: Processes key-value pairs to generate intermediate data.
o REDUCE: Aggregates intermediate data to produce final output.
o Works in isolation, enabling distributed, highly-parallel, fault-tolerant, and
scalable processing.
There were, however, a few limitations of Hadoop 1.0. They are as follows:

1. Required expertise in MapReduce and programming languages like Java.

2. Supported only batch processing – good for log analysis and large-scale data mining
but unsuitable for other projects.
3. Tightly coupled with MapReduce, forcing vendors to either rewrite functionality in
MapReduce or process data outside Hadoop, causing inefficiencies.
Next Step: Evaluate how Hadoop 2.0 addresses these limitations.

42.3.2 Hadoop 2.0

  HDFS (Hadoop Distributed File System) continues to serve as the data storage
framework.
 A new resource management framework called YARN (Yet Another Resource
Negotiator) has been introduced.
 YARN supports any application capable of dividing itself into parallel tasks. It
coordinates the allocation of tasks for submitted applications, enhancing flexibility,
scalability, and efficiency.
 YARN replaces the old JobTracker with an ApplicationMaster and the TaskTracker
with a NodeManager.
 ApplicationMaster can run any application, not just MapReduce, meaning
MapReduce expertise is no longer mandatory.
 Hadoop 2.0 supports both batch processing and real-time processing.
 Alternative data processing operations such as data standardization and master data
management can now be performed natively in HDFS.

4.2.4 Overview of Hadoop Ecosystems

The components of the Hadoop ecosystem are shown in Figure 4.11.
There are components available in the Hadoop ecosystem for data ingestion, processing, and
analysis.
Data Ingestion → Data Processing → Data Analysis
Components that help with Data Ingestion are:
1. Sqoop
2. Flume
Components that help with Data Processing are:
1. MapReduce
2. Spark
Components that help with Data Analysis are:
1. Pig
2. Hive
3. Impala

HDFS (Hadoop Distributed File System)

 Distributed storage unit of Hadoop.
 Provides streaming access to file system data, along with file permissions and
authentication.
 Based on GFS (Google File System).
 Enables scaling from a single cluster node to hundreds or thousands of nodes.
 Handles large datasets on commodity hardware.
 Highly fault-tolerant: stores files across multiple machines in a redundant fashion for
data recovery in case of failure.

HBase
 Stores data in HDFS.
 First non-batch component of the Hadoop ecosystem.
 Functions as a database on top of HDFS.
 Provides quick random access to stored data with low latency compared to HDFS.
 A NoSQL, non-relational, column-oriented database.
 Table structure:
o Tables can have thousands of columns and multiple rows.
o Each row can have several column families.
 o Each column family can have several columns, and each column can have
several key-values.
 Based on Google BigTable.
 Widely used by Facebook, Twitter, Yahoo, etc.

Difference between HBase and Hadoop/HDFS

HDFS vs HBase
1. Nature: HDFS is a file system, whereas HBase is a Hadoop database (like NTFS vs
MySQL).
2. Read/Write Model: HDFS is WORM (Write Once, Read Many). Latest versions allow
appending but rarely used. HBase supports real-time random read and write.
3. Foundation: HDFS is based on Google File System (GFS); HBase is based on Google
BigTable.
4. Data Access: HDFS supports full table scan or partition table scan; HBase supports
random small-range scan or table scan.
5. Performance with Hive: Hive on HDFS is fast; Hive on HBase can be 4–5 times
slower.
6. APIs for Access: HDFS data access is via MapReduce jobs; HBase supports Java APIs,
REST, Avro, Thrift APIs.
7. Storage Flexibility: HDFS has rigid storage; HBase supports dynamic storage.
8. Latency: HDFS has high latency operations; HBase has low latency operations.
9. Use Case: HDFS is suitable for batch analytics; HBase is suitable for real-time
analytics.

Hadoop Ecosystem Components for Data Ingestion

1. Sqoop: Stands for SQL to Hadoop. Its main functions are:
a) Import data from RDBMS (MySQL, Oracle, DB2, etc.) to Hadoop file system (HDFS,
HBase, Hive).
 b) Export data from Hadoop file system (HDFS, HBase, Hive) to RDBMS (MySQL,
Oracle, DB2).
Uses of Sqoop:
a) Connector-based architecture allows plug-ins to connect to external systems.
b) Provisions data from external systems onto HDFS and populates tables in Hive and HBase.
c) Integrates with Oozie to schedule and automate import/export tasks.
2. Flume: An important log aggregator in the Hadoop ecosystem.
o Aggregates logs from different machines and stores them in HDFS.
o Developed by Cloudera.
o Designed for high-volume ingestion of event-based data into Hadoop.
o Default destination (sink) is HDFS; can also write to HBase or Solr.

Hadoop Ecosystem Components for Data Processing

1. MapReduce:
o Programming paradigm for distributed and parallel processing of huge
datasets, based on Google MapReduce (2004).
o Input data comes from HDFS.
o Two main phases:
a) Map phase – converts input data into key-value pairs (intermediate
dataset).
b) Reduce phase – aggregates and consolidates the intermediate dataset into
a smaller set of tuples.
o Result is stored back in HDFS.
2. Spark:
o Open-source big data processing framework, developed at UC Berkeley
AmpLab (2009), written in Scala, became open-source in 2010.
o Provides in-memory computing, making it 10–100x faster than disk-based
processing.
o Can process datasets too large for memory via disk-based fallback.
 o Accesses data from HDFS, bypassing MapReduce, and can work with or
without Hadoop (via YARN or standalone).
o Supports multiple programming languages: Scala, Python, Java, R.
o Spark Libraries:
a) Spark SQL – SQL querying on data from multiple sources.
b) Spark Streaming – real-time data analysis.
c) MLlib – machine learning and advanced statistical operations.
d) GraphX – parallel graph computation.
o Often used with Hadoop: Hadoop for storing unstructured data and batch
processing, Spark for high-speed in-memory analytics and real-time
processing.

Hadoop Ecosystem Components for Data Analysis

1. Pig:
o High-level scripting language for Hadoop, alternative to MapReduce.
o Two parts:
a) Pig Latin – SQL-like scripting language; scripts are converted into
MapReduce jobs running on YARN. Initially developed by Yahoo.
 Supports loading data from HDFS and performing grouping, filtering,
sorting, joining, etc.
 Ideal for building ETL pipelines, processing, and analyzing huge
datasets.
b) Pig Runtime – the runtime environment for executing Pig scripts.
2. Hive:
o Data warehouse software built on top of Hadoop.
o Performs summarization, querying, and analysis.
o Uses HiveQL (HQL), a declarative SQL-like language.
o Converts HiveQL queries into MapReduce jobs executed on Hadoop.

Difference between Hive and RDBMS

 1. SQL Support: Both Hive and traditional RDBMS (MySQL, MS SQL Server, PostgreSQL)
support SQL interfaces, but Hive is primarily a data warehouse (D/W), not a
transactional database.
2. Schema Enforcement:
o RDBMS: Schema enforced at write time; data not conforming to schema is
rejected → slower load, faster queries.
o Hive: Schema enforced at read time (schema-on-read) → faster data load,
slower query retrieval.
3. Data Model:
o Hive: Write once, read many times; batch-oriented system; better suited for
OLAP.
o RDBMS: Read and write many times; supports OLTP with frequent inserts,
updates, deletions.
4. Data Type:
o Hive: Handles static, non-real-time data; query latency exists.
o RDBMS: Handles dynamic, real-time data; fast query response.
5. Scalability & Storage:
o Hive: Scales easily at low cost using HDFS; not the owner of data.
o RDBMS: Manages and owns data; scaling is costlier.
6. Computing Model:
o Hive: Uses parallel computing for query execution.
o RDBMS: Uses serial computing.

We summarize the difference in Table 4.5.

Difference between Hive and HBase
1. Hive: MapReduce-based SQL engine that runs on top of Hadoop.
HBase: Key-value NoSQL database that runs on top of HDFS.
2. Processing Type:
o Hive: Batch processing of big data.
o HBase: Real-time data streaming.

Other Hadoop Ecosystem Components

 Impala: High-performance SQL engine running on Hadoop cluster; ideal for
interactive analysis; very low latency (milliseconds); supports Impala SQL.
 ZooKeeper: Coordination service for distributed applications.
 Oozie: Workflow scheduler system to manage Hadoop jobs.
 Mahout: Scalable machine learning and data mining library.
 Chukwa: Data collection system for managing large distributed systems.
  Ambari: Web-based tool for provisioning, managing, and monitoring Hadoop
clusters.

4.2.5 Hadoop Distributions

Hadoop is an open-source Apache project. Anyone can freely download the core aspects of
Hadoop. The core components include:
1. Hadoop Common
2. Hadoop Distributed File System (HDFS)
3. Hadoop YARN (Yet Another Resource Negotiator)
4. Hadoop MapReduce
Several companies, such as IBM, Amazon Web Services, Microsoft, Teradata, Hortonworks,
Cloudera, etc., have packaged Hadoop into more easily consumable distributions or
services.
Although each company has a slightly different strategy, the key essence remains the same:
the ability to distribute data and workloads across potentially thousands of servers,
making big data manageable.
Refer Figure 4.12 for a few Hadoop distributions.

4.2.6 Hadoop versus SQL

Table 4.6 lists the differences between Hadoop and SQL.
4.2.7 Integrated Hadoop Systems Offered by Leading Market Vendors
Refer Figure 4.13 to get a glimpse of the leading market vendors offering integrated Hadoop
systems.

4.2.8 Cloud-Based Hadoop Solutions

 Amazon Web Services provides a comprehensive, end-to-end portfolio of cloud
computing services to help manage big data, with emphasis on reducing costs,
scaling to meet demand, and accelerating the speed of innovation.
 The Google Cloud Storage connector for Hadoop enables MapReduce jobs to be
performed directly on data stored in Google Cloud Storage without copying data to
local disk or running it in Hadoop Distributed File System (HDFS).
 The connector simplifies Hadoop deployment, reduces cost, offers performance
comparable to HDFS, and improves reliability by eliminating the single point of failure
of the NameNode. Refer Figure 4.14.