1.

1 Classification of Digital Data

Digital data is mainly classified into three types:
 Structured data
 Semi-structured data
 Unstructured data

1. Unstructured Data
 Unstructured data does not follow any fixed format or data model.
 It is not easy for computer programs to process.
 About 80–90% of an organization’s data is unstructured.
 This type of data is growing very rapidly.
Examples:
 Text documents
 Emails (email body)
 Images and videos
 Audio files
 PowerPoint presentations
 Chat messages
 Research papers and white papers

2. Semi-Structured Data
 Semi-structured data does not have a fixed structure like tables.
 It contains some form of organization or tags.
 It is partially readable by machines.
 Metadata is available but not enough for easy processing.
Examples:
 Emails
 XML files
 HTML files
  JSON documents

3. Structured Data
 Structured data is well organized.
 It is stored in rows and columns.
 It is easy to store, search, and analyze using computer programs.
 Relationships exist between different data items.
Examples:
 Data stored in databases
 Student records
 Employee details
 Banking transactions

Role of RDBMS
 Since the 1980s, most enterprise data has been stored in Relational Database
Management Systems (RDBMS).
 RDBMS uses tables, rows, columns, primary keys, and foreign keys.
 Over time, RDBMS became more efficient, reliable, and cost-effective.
 Structured data is mainly stored in RDBMS.

Growth of Unstructured Data

 With the growth of the Internet, large amounts of data started coming from outside
organizations.
 This data includes social media, emails, videos, and online content.
 Most of this data is unstructured.
 According to Gartner, nearly 80% of enterprise data today is unstructured.
 Only about 10% of data is structured or semi-structured.

1.1.1 Structured Data

Structured data is the data that follows a predefined structure or schema. When data is
stored according to a fixed format, it is called structured data.

What is Structured Data?

 Data is said to be structured when it conforms to a predefined schema.
 A schema defines how data is stored, organized, and accessed.
 Structured data is easy to store, search, and process using computer programs.

Structured Data and Data Model

 Structured data follows a data model.
 A data model defines the types of data that will be stored for business use.
 Most structured data is stored in Relational Database Management Systems
(RDBMS).

Structured Data in RDBMS

 An RDBMS conforms to the relational data model wherein the data is stored in
rows/columns. Refer Table 1.1.
 Data is stored in the form of tables (relations).
 Each table contains Row / Record / Tuple and Column / Attribute / Field
 The number of rows/records/tuples in a relation is called the cardinality of a relation
and the number of columns is referred to as the degree of a relation.

 The first step in database design is to create a relation (table).

 Decide the required fields/columns to store data.
 Specify the data type for each column (integer, real, character, date, Boolean, etc.).
 Define constraints to maintain data integrity.
 Common constraints include UNIQUE, NOT NULL, and business rules (e.g., value
should not be less than 50).
 Domain constraints restrict allowed values (e.g., only “CS”, “IS”, or “MS”).
 An Employee table is an example of storing employee details in an organization.
 The table schema includes table name, meaningful column names, data types,
lengths, and constraints.
 A well-designed table strictly follows the relational data model.

 It goes without saying that each record in the table will have exactly the same
structure. Let us take a look at a few records in Table 1.3.
 In an RDBMS, tables can be related to each other. For example, the Employee table
is related to the Department table. This relationship is created using a common
column called DeptNo.
 It is not necessary that both tables have the same column name. The relationship
depends on the values stored in the column, not on the column name.
 This relationship is called referential integrity. "DeptNo". Given in Figure 1.3 is a
depiction of referential integrity constraint (primary - foreign key) with the
"Department" table being the referenced table and "Employee" table being the
referencing table.
[Link] Sources of Structured Data

 When data is highly structured, it can be stored in an RDBMS. Many RDBMS

software systems are available for this purpose.
 available RDBMS [Oracle Corp. - Oracle, IBM - DB2, Microsoft - Microsoft SQL Server,
EMC- Greenplum, Teradata - Teradata, MySQL (open source), PostgreSQL (advanced
open source), etc.].
 Refer Figure 1.4. These databases are typically used to hold transaction/operational
data generated and collected by day-to-day business activities. In other words, the
data of the On-Line Transaction Processing (OLTP) systems are generally quite
structured.

[Link] Ease of Working with Structured Data

Structured data is organized in a predefined schema (rows and columns), which makes it
easy to store, manage, and analyze using traditional Relational Database Management
Systems (RDBMS). The ease of working with structured data can be explained as follows
(Refer Figure 1.5):

1. Insert/update/delete: The Data Manipulation Language (DML) operations provide the

required ease with data input, storage, access, process, analysis, etc.
2. Security
Security of information is ensured using encryption and tokenization, which protect data
throughout its lifecycle. Only authorized users can decrypt and access sensitive
information, helping organizations maintain control and compliance.
[Link]:
An index is a data structure that improves data retrieval speed, especially for SELECT
operations, by using extra storage space and causing additional write overhead, which is
acceptable due to faster search performance.
[Link]:
In traditional RDBMS, scalability is achieved by scaling up the database server through
increasing storage capacity and processing power (CPU, memory, and peripheral
storage).

[Link] Processing:
RDBMS supports ACID properties of transactions:
Atomicity: A transaction is atomic, means that either it happens in its entirety or none of
it at all.
Consistency: The database moves from one consistent state to another consistent state.
In other words, if the same piece of information is stored at two or more places, they are
in complete agreement.
Isolation: The resource allocation to the transaction happens such that the transaction
gets the impression that it is the only transaction happening in isolation.
Durability: All changes made to the database during a transaction are permanent and
that accounts for the durability of the transaction.

1.1.2 Semi-Structured Data

 Semi-structured
features:
 data is also referred to as self-describing structure.
Refer Figure 1.6. It has the following

 Does not follow strict relational data models or table

structures.
 Uses tags to represent semantic elements.
 Tags help in defining hierarchies of records and fields.
 No clear separation between data and schema;
structure depends on the purpose.
 Entities of the same type may have different attributes
or different attribute order, which is acceptable.

[Link] Sources of Semi-Structured Data

 The main sources of semi-structured data are XML and JSON:

 XML (eXtensible Markup Language):

is hugely popularized by web services developed utilizing the Simple Object Access
Protocol (SOAP) principles..

 JSON (JavaScript Object Notation):

Used for data exchange between a server and web applications. Popular in REST-based
web services. Databases like MongoDB and Couchbase store data natively in JSON
format.

An example of HTML is as follows:

<HTML>
<HEAD>
<TITLE>Place your title here</TITLE>
</HEAD>
<BODY BGCOLOR="FFFFFF">
<CENTER><IMG SRC="[Link]" ALIGN="BOTTOM"></CENTER>
<HR>
<a href="[Link] Name</a>
<H1>this is a Header</H1>
<H2>this is a sub Header</H2>
Send me mail at <a href="[Link]
support@[Link]</a>.
<P>a new paragraph!
<P><B>a new paragraph!</B>
<BR><B><I>this is a new sentence without a paragraph break, in bold italics.</I></B>
<HR>
<BODY>
</HTML>
Sample JSON document
id:9,
BookTitle: "Fundamentals of Business Analytics",
AuthorName: "Seema Acharya",
Publisher: "Wiley India",
YearofPublication: "2011”

1.1.3 Unstructured Data:

Unstructured data does not conform to any predefined data model. It includes various
types of text and other data whose structure is highly unpredictable, as shown in Table
1.4. Figure 1.8 illustrates additional sources of unstructured data.

[Link] Issues with "Unstructured" Data

· Unstructured data does not conform to a predefined data model and is not organized
in a predefined manner.
· In some cases, data classified as unstructured may still show implied structure.
· As shown in Figure 1.9, data may be placed in the unstructured category despite
having some structure or being highly structured.
· It is often argued that a text file should be considered semi-structured data.
· A text file has:
 A file name
 Metadata such as owner, creation date, and file size
· This metadata provides limited structural information.
· For analysis, the content of the text file is more important than file properties.
· Metadata usually does not contribute significantly to the processing or analysis task.
· The content of text files:
 Lacks a fixed schema
 Is not machine-interpretable
 Varies across files
· Hence, despite having metadata, text files are rightly classified as unstructured data
[Link] How to Deal with Unstructured Data?
· Today, unstructured data accounts for nearly 80% of the data generated in most
enterprises.
· The proportion of unstructured data is rapidly increasing, as illustrated in Figure 1.10.
· There is a clear shift in balance toward unstructured data.
· Due to its large volume, unstructured data cannot be ignored by organizations.
· Figure 1.11 highlights various approaches used to deal with unstructured data.
Techniques Used to Interpret Unstructured Data
 The following techniques are used to find patterns in and interpret unstructured
data.
1. Data Mining
 Deals with large datasets.
 Uses techniques from:
o Artificial Intelligence
o Machine Learning
o Statistics
o Database Systems
 Helps to discover consistent patterns and systematic relationships between
variables.
 Represents the analysis step in the Knowledge Discovery in Databases (KDD) process.
Popular Data Mining Algorithms
 Association Rule Mining
o Also known as market basket analysis or affinity analysis.
o Answers the question: “What goes with what?”
 o Identifies products that are frequently purchased together.
o Example:
 If a customer buys bread, they are likely to buy eggs or cheese.
 Regression Analysis
o Used to predict the relationship between variables.
o The variable to be predicted is called the dependent variable.
o Variables used for prediction are called independent variables.

2. Collaborative Filtering
 Predicts a user’s preferences based on the preferences of similar users.
 Uses known preferences of a user and compares them with other user profiles.
 Helps in recommendation and personalization.
 Example:
o Based on similar users’ learning preferences, User 4 is predicted to prefer
video-based learning rather than textual learning.
3. Text Analytics / Text Mining
 Text data is largely unstructured and difficult to process algorithmically.
 Text mining extracts high-quality and meaningful information from text.
 Uses statistical pattern learning to identify patterns and trends.
 Includes tasks such as:
o Text categorization
o Text clustering
o Sentiment analysis
 o Concept/entity extraction
4. Natural Language Processing (NLP)
 Related to human–computer interaction.
 Enables computers to understand and process human (natural) language input.
5. Noisy Text Analytics
 Extracts structured or semi-structured information from noisy unstructured data.
 Sources include:
o Chats, blogs, wikis, emails, message boards, text messages
 Noisy text may contain:
o Spelling mistakes
o Abbreviations and acronyms
o Non-standard words
o Missing punctuation and letter case
o Filler words like “uh”, “um”
6. Manual Tagging with Metadata
 Involves manually assigning metadata to unstructured data.
 Helps in adding semantic meaning for better understanding and analysis.
7. Part-of-Speech (POS) Tagging
 Also called POS, POST, or grammatical tagging.
 Tags each word in a sentence with its grammatical role.
 Examples: noun, verb, adjective, etc.
8. Unstructured Information Management Architecture (UIMA)
 An open-source platform developed by IBM.
 Used for real-time content analytics.
 Processes text and other unstructured data to discover:
o Latent meanings
o Relevant relationships within data

CHAPTER 2 : Introduction to Big Data

2.1 CHARACTERISTICS OF DATА

1. Composition
 Refers to the structure of data
 Includes:
o Data sources
o Granularity
o Data types (structured, semi-structured, unstructured)
o Nature of data (static or real-time streaming)

2. Condition
 Refers to the state and quality of data
 Determines whether data:
o Can be used directly for analysis, or
o Requires cleansing, enhancement, or enrichment
 Deals with missing, noisy, or inconsistent data
3. Context
 Refers to the background and meaning of data.
o Where, why, and when data is generated
o Sensitivity of data
o Events associated with the data
Small Data vs Big Data
Small Data
 Refers to data before the Big Data revolution
 Characterized by certainty
 Data sources are known and well-defined
 No major changes in data composition or context
 Clear understanding of:
o Why the data was generated
o Where and when it was generated
o How it will be used
o What questions it can answer

Big Data
 Characterized by high complexity
 Involves multiple and unknown data sources/sets
 Data volume is very large and rapidly increasing
 Generated and processed at high speed
 Includes a wide variety of data:
o Internal and external
o Behavioural and social

2.2 EVOLUTION OF BIG DATA

 1970s and before were the era of mainframes
 Data during this period was primitive and structured
 1980s and 1990s saw the evolution of relational databases
 This period was the era of data-intensive applications
 The World Wide Web (WWW) and Internet of Things (IoT) led to an onslaught of
data
 Data generated includes structured, unstructured, and multimedia data. Refer Table
2.1.
2.3 DEFINITION OF BIG DATA

 When asked “Define Big Data”, the following responses are commonly heard:
1. Data that goes beyond human and technical infrastructure needed for
storage, processing, and analysis
2. What is BIG today may become NORMAL tomorrow
3. Data measured in terabytes, petabytes, or zettabytes
4. Big Data is about the 3 Vs
 Refer Figure 2.2
 All the above responses are correct
 Big Data is not just one of these, but all of these and more

Standard Definition of Big Data (Gartner)

Big data is high-volume, high-velocity, and high-variety information assets that demand cost
effective, innovative forms of information processing for enhanced insight and decision
making.

The 3Vs concept was proposed by the Gartner analyst Doug Laney in a 2001 MetaGroup
research publication, titled, 3D Data Management: Controlling Data Volume, Variety and
Velocity.
For the sake of easy comprehension, we will look at the definition in three parts. Refer
Figure 2.3.

Part I of the definition "big data is high-volume, high-velocity, and high-variety information
assets"
talks about voluminous data (humongous data) that may have great variety (a good mix of
structured, semi-structured, and unstructured data) and will require a good speed/pace for
storage, preparation, processing, and analysis.
Part II of the definition "cost effective, innovative forms of information processing" talks
about embracing new techniques and technologies to capture (ingest), store, process,
persist, integrate, and visualize the high-volume, high-velocity, and high-variety data.
Part III of the definition "enhanced insight and decision making" talks about deriving deeper,
richer, and meaningful insights and then using these insights to make faster and better
decisions to gain business value and thus a competitive edge.
Data → Information → Actionable intelligence → Better decisions → Enhanced business
value.

24 CHALLENGES WITH BIG DATА

Refer Figure 2.4. Following are a few challenges with big data:
1. Exponential growth of data
o Data is growing at a very fast rate
o Most of today’s data has been generated in the last 2–3 years
o Data volume will continue to increase
o Key questions include:
 Will all this data be useful for analysis?
 Should we analyze all data or only a subset?
 How do we separate knowledge from noise?
2. Cloud computing and virtualization
o Cloud computing and virtualization are here to stay
 o Cloud helps manage big data infrastructure due to:
 Cost efficiency
 Elasticity
 Easy upgrading and downgrading
o Creates challenges in deciding whether to host big data solutions outside the
enterprise
3. Data retention period
o Challenge in deciding how long big data should be retained
o Some data is useful for long-term decision making
o Some data becomes irrelevant or obsolete within hours of generation
4. Shortage of skilled professionals
o There is a dearth of skilled data science professionals
o High proficiency in data sciences is required
o This skill gap affects effective implementation of big data solutions
5. Technical and operational challenges
o Challenges related to:
 Data capture
 Storage
 Preparation
 Search
 Analysis
 Transfer
 Security
 Visualization
o Big data exceeds the storage capacity of traditional database tools
o No explicit definition of how large data must be to qualify as big data
o Data is:
 Too large
 Moves very fast
 Does not fit traditional database structures
 o Data is highly dynamic and must be ingested quickly
6. Data visualization challenge
o Data visualization is emerging as a separate discipline
o There is a shortage of business visualization experts

2.5 WHAT IS BIG DATА?

Big data is data that is big in volume, velocity, and variety. Refer Figure 2.5.
2.5.1 Volume
We have seen it grow from bits to bytes to petabytes and exabytes. Refer Table 2.2 and
Figure 2.6.
Bits → Bytes → Kilobytes → Megabytes → Gigabytes → Terabytes → Petabytes → Exabytes
→ Zettabytes → Yottabytes

[Link] Where Does This Data Get Generated?

 Big data is generated from multiple sources
 Examples of data types:
o Unstructured data:
 XLS, DOC, PDF files
 YouTube videos
 Internet chat conversations
 CCTV footage
 Weather forecast reports
o Semi-structured data:
 Customer feedback forms on online retail websites
 Refer Figure 2.7 for sources of big data
Sources of Big Data
1. Internal Data Sources
 Data present within an organization’s firewall
 Typical internal data sources: Data present within an organization's firewall. It is as
follows:
 Data storage: File systems, SQL (RDBMSs - Oracle, MS SQL Server, DB2, MySQL,
PostgreSQL, etc.), NoSQL (MongoDB, Cassandra, etc.), and so on.
 Archives: Archives of scanned documents, paper archives, customer correspondence
records, patients' health records, students' admission records, students' assessment
records, and so on.2. External Data Sources
2. External data sources:
 Data residing outside an organization's firewall. It is as follows:
 Public Web: Wikipedia, weather, regulatory, compliance, census, etc.
3. Both(internal + external data sources)
 Sensor data: Car sensors, smart electric meters, office buildings, air conditioning
units, refrigerators, and so on.
 Machine log data: Event logs, application logs, Business process logs, audit logs,
clickstream data, etc.
 Social media: Twitter, blogs, Facebook, LinkedIn, YouTube, Instagram, etc. Business
apps: ERP, CRM, HR, Google Docs, and so on.
  Media: Audio, Video, Image, Podcast, etc. Docs: Comma separated value (CSV),
Word Documents, PDF, XLS, PPT, and so on.

2.5.2 Velocity
 Refers to the speed at which data is generated and processed
 We have moved from the days of batch processing (remember our payroll
applications) to real-time processing.
 Batch → Periodic → Near real time → Real-time processing

2.5.3 Variety
 Refers to the diversity of data types and sources of data.
 We will study this under three categories: Structured data, semi-structured data and
unstructured data.
1. Structured data: From traditional transaction processing systems and RDBMS, etc.
2. Semi-structured data: For example Hyper Text Markup Language (HTML),
eXtensible Markup Language (XML).
3. Unstructured data: For example, unstructured text documents, audios, videos,
emails, photos, PDFs, social media, etc.

2.7 WHY BIG DATA?

 The more data available for analysis, the higher the analytical accuracy
 Increased analytical accuracy leads to greater confidence in decision making
 Improved decisions result in:
o Enhanced operational efficiencies
o Reduction in cost
o Reduction in time
o Innovation of new products
o Innovation of new services
o Optimization of existing services
 Refer Figure 2.8
 More data → More accurate analysis → Greater confidence in decision making →
Greater operational efficiencies, cost reduction, time reduction, new product
development, and optimized offerings, etc.
2.9 TRADITIONAL BUSINESS INTELLIGENCE (BI) VERSUS BIG DATA

Differences Between Traditional BI and Big Data

1. Data storage
o In traditional BI, all enterprise data is stored on a central server
o In big data, data resides in a distributed file system
o Distributed file systems scale horizontally (scale in/out), while traditional
database servers scale vertically
2. Mode of analysis
o Traditional BI analyzes data mainly in offline mode
o Big data supports analysis in both real-time and offline modes
3. Data type and processing approach
o Traditional BI deals primarily with structured data
o Data is moved to processing functions (move data to code)
o Big data deals with structured, semi-structured, and unstructured data
o Processing functions are moved to data (move code to data)

2.10 A TYPICAL DATA WAREHOUSE ENVIRONMENT

Typical Data Warehouse (DW) Environment
 Operational/transactional/day-to-day business data comes from ERP (Enterprise
Resource Planning) systems, CRM (Customer Relationship Management) systems,
legacy systems, and third-party applications.
 Data formats may vary: RDBMS (Oracle, MS SQL Server, DB2, MySQL, Teradata),
spreadsheets (.xls, .xlsx), .csv files, text (.txt) files.
 Data may originate from the same or different geographies
  Data is integrated, cleaned, transformed, and standardized using ETL (Extraction,
Transformation, and Loading)
 Transformed data is loaded into Enterprise Data Warehouse (EDW – enterprise
level) or Data Marts (DM – business/unit/functional level)
 Business Intelligence (BI) tools support decision making via ad-hoc queries, SQL
(Structured Query Language), dashboards, and data mining Refer Figure 2.9

211 A TYPICAL HADOOP ENVIRONMENT

 Hadoop deals with disparate data sources, such as web logs, images, audio, videos,
social media data, documents, PDFs, etc.
 Data includes both internal data (within company firewall) and external data
(outside firewall)
 Data is stored in Hadoop Distributed File System (HDFS)
 If needed, data can be repopulated back to operational systems, fed to Enterprise
Data Warehouse (EDW), Data Marts (DM), or Operational Data Store (ODS) for
further processing and analysis. Refer Figure 2.10