BDA MODULE – 1

Classification of data, Characteristics, Evolution and definition of Big data, What is Big data, Why Big data,
Traditional Business Intelligence Vs Big Data, Typical data warehouse and Hadoop environment.

Big Data Analytics: What is Big data Analytics, Classification of Analytics, Importance of Big Data Analytics,
Technologies used in Big data Environments, Few Top Analytical Tools , NoSQL, Hadoop.

TB1: Ch 1: 1.1, Ch2: 2.1-2.5,2.7,2.9-2.11, Ch3: 3.2,3.5,3.8,3.12, Ch4: 4.1,4.2

1. Explain classification of Big data with a diagram.

1.1.1 Structured Data

Structured data is the data which is in an organized form (E.g. in rows and columns) and can be
easily used by a computer program. Data stored in databases is an example of structured data.

Sources of Structured Data

• Structured data is typically stored in Relational Database Management Systems

(RDBMS).
• Examples of RDBMS include Oracle, IBM DB2, Microsoft SQL Server, Greenplum,
MySQL

Ease of Working with Structured Data

1. Insert / Update / Delete : Data Manipulation Language (DML) operations make data
input, update, storage, processing and deleting easy.
2. Security: Encryption and tokenization methods ensure the security of information
throughout its lifecycle. Only authorized individuals can decrypt and view sensitive data,
helping organizations maintain control.
3. Transaction Processing (ACID Properties): RDBMS supports:
o Atomicity: A transaction is executed completely or not executed at all.
o Consistency: The database moves from one consistent state to another.
o Isolation: Each transaction appears to run independently.
o Durability: Changes made during a transaction are permanent.
 1.1.2 Semi-Structured Data

Semi-structured data is the data which does not conform to a data model but has some
structure. However, it is not in a form that can be easily used by a computer program.
Examples of semi-structured data include emails, XML, and markup languages such as
HTML.

Features of Semi-Structured Data

Semi-structured data has the following features:

1. It does not conform to the data models that are typically associated with relational
databases or other forms of data tables.
2. It uses tags to identify and separate the data elements.
3. Tags also define the hierarchy between records and fields.
4. Data and schema are mixed together.

Sources of Semi-Structured Data

The major sources of semi-structured data are XML and JSON.

1. XML (eXtensible Markup Language): XML is widely popularized by web services

developed using Simple Object Access Protocol (SOAP) principles.

2. JSON (JavaScript Object Notation): JSON is used to transmit data between a server and a
web application.

1.1.3 Unstructured Data

Unstructured data is the data which does not conform to a data model or is not in a form which
can be used easily by a computer program. About 80-90% data of an organization is in this
format, for example, memos, PowerPoint presentations, images, videos, letters, etc
➢ How to Deal with Unstructured Data
Today, unstructured data constitutes approximately 80% of the data generated in an
enterprise. Due to its large volume, it cannot be ignored. The following techniques are used to
find patterns in or interpret unstructured data:

1. Data Mining

Data mining deals with large data sets and uses methods from artificial intelligence, machine
learning, statistics, and database systems to discover patterns or relationships between
variables.

Some popular data mining techniques are:

• Association rule mining: It determines “what goes with what”.

• Regression analysis: Predicts the relationship between variables. The variable to be
predicted is the dependent variable, and the variables used for prediction are
independent variables.

2. Text Analytics / Text Mining: Text is largely unstructured and amorphous. Text mining is
the process of extracting high-quality and meaningful information from text using statistical
analysis.

3. Noisy Text Analytics: This involves extracting structured or semi-structured

information from noisy unstructured data such as chats, blogs, wikis, emails, text messages.
Such data may contain spelling mistakes, abbreviations, missing punctuation, missing letter
case, and filler words.

4. Part-of-Speech (POS) Tagging: Also called grammatical tagging, it is the process of

tagging each word in a sentence as noun, verb, adjective, etc.
2. What is Big Data? Explain its characteristics & challenges with eg.

Big data is high-volume, high-velocity, and high-variety information assets that demand
cost-effective, innovative forms of information processing for enhanced insight and decision
making.

Big Data characteristics are:

➢ Volume

Volume refers to the amount of data generated and stored. Data has grown rapidly over time,
starting from very small units to extremely large ones. We have seen data grow from bits to
bytes to petabytes and exabytes.

The hierarchy of data size is as follows:

Bits → Bytes → Kilobytes → Megabytes → Gigabytes → Terabytes ,etc. This shows the
enormous scale of data handled in Big Data systems.

➢ Velocity

Velocity refers to the speed at which data is generated, processed, and analyzed. We have
moved from the days of batch processing (such as payroll applications) to real-time
processing.

The types of processing based on velocity are:

Batch → Periodic → Near real-time → Real-time processing

➢ Variety

Variety refers to the wide range of data types and sources in Big Data. Data can be
categorized into three types:

1. Structured data: Structured data is the data which is in an organized form (E.g. in rows
and columns) and can be easily used by a computer program. Examples of RDBMS
include Oracle, MySQL, etc
2. Semi-structured data: Semi-structured data is the data which does not conform to a
data model but has some structure. Examples include HTML and XML
3. Unstructured data: Unstructured data is the data which does not conform to a data
model or is not in a form which can be used easily by a computer program, for example,
memos, PowerPoint presentations, images, videos, letters, etc
➢ The following are some challenges associated with big data:

1. Data growth : Data today is growing at an exponential rate. Most of the data available has
been generated in the last 2–3 years, and this growth will continue. Key questions include
whether all the data is useful, whether to use all data or a subset, and how to separate
knowledge from noise. Example: Daily WhatsApp messages.
2. Infrastructure and cloud computing: Cloud computing and virtualization are used for
managing infrastructure due to cost-efficiency, elasticity, and easy scalability. But they make
hosting decisions more difficult. Example: Storing data on Google Drive or AWS.
3. Data retention : Deciding how long data should be retained is difficult. Some data is useful
for long-term decisions, while some becomes irrelevant within hours of being generated.
Example: Old emails vs live traffic data.
4. Skill shortage: There is a shortage of skilled professionals with high proficiency in data
sciences, which is essential for implementing big data solutions.
5. Data management challenges: Challenges exist in data capture, storage, preparation,
search, analysis, transfer, security, and visualization. Big data datasets are often beyond the
storage capacity of traditional database software tools, move very fast, and do not fit
traditional database structures. Example: Sensor data from machines.
6. Data visualization: Data visualization is emerging as a separate discipline, and there is a
shortage of business visualization experts. Example: Creating charts from large sales data.
[Link] is big data analytics? Explain classification of analytics
Big Data Analytics is about using technology-enabled analytics to process and analyze big data in order to
improve the business insights. Characteristics of Big data are:

• Technology-enabled analytics: Many tools like IBM, Tableau, and others are used to process and
analyze big data.

• Meaningful business insights: Big Data Analytics helps us understand the customers more better by
using analytics.

• Competitive advantage: It helps organizations make faster and better decisions by working together
across IT teams, business team, and data scientists.

• Handling large datasets: Big Data Analytics deals with very large and diverse data that traditional
storage systems cannot handle.

3.5 CLASSIFICATION OF ANALYTICS

1. First school of thought: Analytics is classified into basic, operationalized, advanced, and monetized
analytics.
2. Second school of thought: Analytics is classified into Analytics 1.0, Analytics 2.0, and Analytics 3.0.

3.5.1 First School of Thought

1. Basic analytics: This involves slicing and dicing of data to obtain basic business insights. It mainly
focuses on historical data, basic reporting, and visualization.
2. Operationalized analytics: Analytics becomes operationalized when it is woven into an
organiztions’s business processes.
3. Advanced analytics: This focuses on forecasting the future using predictive and prescriptive
modeling.
4. Monetized analytics: This refers to analytics used to derive direct business revenue.

3.5.2 Second School of Thought: This school of thought classifies analytics into
Analytics 1.0, Analytics 2.0, and Analytics 3.0.
Aspect Analytics 1.0 Analytics 2.0 Analytics 3.0
Era Mid-1950s to 2009 2005 to 2012 2012 to present
Type of Descriptive statistics Descriptive + Predictive Descriptive + Predictive +
analytics statistics Prescriptive statistics
Focus Reporting past events Using past data to predict Predicting future outcomes and
and occurrences future outcomes recommending actions
Nature of data Small and structured data Big data, mostly Blend of big data and traditional
unstructured, high velocity data
Technologies Relational databases Hadoop, SQL-to-Hadoop In-memory analytics,etc
used
Outcome Understanding past Forecasting future trends Actionable insights and
performance recommendations
[Link] CAP Theorem (Brewer’s Theorem).
• The CAP Theorem is a rule for distributed systems. The CAP Theorem states that a distributed
system can provide only two out of the following three guarantees at the same time
• At best, only two out of the following three can be achieved; one must be sacrificed:
1. Consistency
2. Availability
3. Partition Tolerance

➢ CAP Properties Explained

1. Consistency:
Whenever data is updated, all users see the same latest data. No matter which server or node is accessed,
the information returned is always up to date.
2. Availability:
The system always responds to read and write requests. Even if some parts of the system fail, users still
receive a response within a short time.
3. Partition Tolerance:
The system continues to work even when there are network problems. It can handle communication
failures between nodes without stopping completely.

Real-Life Example Explanation (Training Institute Case)

• In a training institute, two administrators manage schedules.

• If only one updates the schedule without informing the other → Consistency issue.
• If updates are delayed because one is unavailable → Availability issue.
• If both stop communicating temporarily → Partition Tolerance issue.

Choosing Between Consistency and Availability

1. Choose availability over consistency when the business can tolerate delayed data synchronization.
2. Choose consistency over availability when correct and updated data is critical for business operations.

Possible CAP Combinations

1. AP (Availability + Partition Tolerance)
2. CP (Consistency + Partition Tolerance)
3. CA (Consistency + Availability)
[Link] traditional Business Intelligence (bi) with Big Data.
1. In a traditional BI environment, all enterprise data is housed in a central server, whereas in a big
data environment, data resides in a distributed file system. The distributed file system scales
horizontally by scaling in or out, while a traditional database server scales vertically.
2. In traditional BI, data is generally analyzed in an offline mode, whereas in big data, data is analyzed
in both real-time and offline modes.
3. Traditional BI deals mainly with structured data, and data is taken to the processing functions (move
data to code). In contrast, big data deals with variety, including structured, semi-structured, and
unstructured data, and the processing functions are taken to the data (move code to data).

6. What is NoSQL(NOT ONLY SQL)? Explain its types with eg.

NoSQL is a type of database that is non-relational, distributed, schema-less, designed to handle large
volumes of structured, semi-structured, and unstructured data and follow the CAP theorem instead of
strict ACID properties.

Few features of NoSQL databases are as follows:

1. They are open source.

2. They are non-relational.
3. They are distributed.
4. They are schema-less.
5. They are cluster friendly.
6. They are born out of 21st century web applications.

➢ Types of NoSQL Databases

NoSQL databases are non-relational and schema-less. They can be broadly classified into the following types:

1. Key-Value Databases:
o Maintains a big hash table of keys and values.
o Examples: Dynamo, Redis,etc.
o Sample Key-Value Pair: First Name – Last Name This shows data stored as a pair, where one
value (key) is linked to another value (data).
2. Document Databases:
o The system keeps data in collections, and each collection contains multiple documents (like
records).
o Examples: MongoDB, MarkLogic.
o Sample Document:

{
"Book Name": "Fundamentals of Business Analytics",
"Publisher": "Wiley India",
"Year of Publication": "2017"
}
 3. Column Databases:
o It means that data is stored column by column, not row by row.
o A storage block will contain values of only one column (for example, only names or only ages),
and not a full record.
o Examples: Cassandra, HBase.
4. Graph Databases:
o Also called network databases.
o Data is stored in individual nodes, with links showing relationships between them.
o Examples: Neo4j, HyperGraphDB.

➢ Advantages of NoSQL

• Supports large data volumes

• Flexible schema, no strict adherence required.
• Cost-effective and easy to implement.
• Relaxed consistency: Follows CAP theorem with eventual consistency.

➢ NoSQL Vendors : Amazon: DynamoDB, Google: BigTable, Adobe: Photoshop

[Link] is NewSQL? What are the Characteristics of NewSQL?

NewSQL is a modern relational database system that combines the best of SQL and NoSQL. It provides the
scalable performance of NoSQL for Online Transaction Processing (OLTP) while still maintaining the ACID
guarantees of traditional RDBMS. It supports the relational data model and uses SQL as its primary interface.

Characteristics of NewSQL:

1. SQL Interface – Applications interact with the database using standard SQL.
2. ACID Support – Ensures transactional integrity with Atomicity, Consistency, Isolation, and Durability.
3. High Per-Node Performance – Designed to provide higher performance per server compared to
traditional RDBMS.
4. Scale-Out Architecture – Follows a shared-nothing architecture, allowing horizontal scaling.
5. Relational data model: NewSQL databases use tables, rows, and columns and follow relational
database principles, just like traditional SQL systems.

Comparative Study of SQL, NoSQL, and NewSQL

Feature SQL NoSQL NewSQL

Adherence to ACID properties Yes No Yes

Schema rigidity Yes No Maybe

Adherence to relational model Yes No Yes

Scalability Scale up (vertical scaling) Scale out (horizontal scaling) Scale out

Community support Huge Growing Slowly growing

[Link] is Hadoop and its advantages.
Hadoop is an open-source framework. Written in Java, it was first made to support the text search engine
Nutch. Hadoop is based on Google’s MapReduce and Google File System technologies and is now widely
used by companies like Yahoo, Facebook, LinkedIn, and Twitter.

1. Schema-less storage: Stores data in its native format; structure applied during processing.
2. Scalable and cost-effective: Handles very large datasets across many servers with reduced cost per
terabyte.
3. Fault-tolerant and flexible: Resilient to failures and works with all types of data (structured, semi-
structured, unstructured).

[Link] the Hadoop Ecosystem Components for Data Processing

and Data Analysis
The Hadoop ecosystem provides different components for Data Ingestion, Data Processing, and Data
Analysis to handle big data efficiently.

2. Data Processing Components

These are the components which helps in the processing large datasets stored in Hadoop:

(a) MapReduce

• MapReduce is a distributed programming model for parallel processing of big data.

• It works in two phases:
o Map phase: Converts input data into key–value pairs.
o Reduce phase: Aggregates and processes intermediate data.
• Input data is read from HDFS, and output is written back to HDFS.
• Suitable for batch processing of large datasets.

(b) Spark

• Spark is an open-source, fast data processing framework.

• Provides in-memory computing, making it much faster than MapReduce.
• Can work with Hadoop using YARN or run independently.
• Supports programming in Python, Java, and R.
• Spark libraries include:
o Spark SQL – SQL-based querying
o MLlib – machine learning properties
o GraphX – graph processing
3. Data Analysis Components
These components help analyze and query data stored in Hadoop.

(a) Pig

• Pig is a high-level scripting language for Hadoop.

• Uses Pig Latin, a SQL-like language.
• Pig scripts are converted into MapReduce jobs.
• Used for ETL operations, filtering, grouping, sorting, and joining data.
• Easier for developers who find MapReduce complex.

(b) Hive

• Hive is a data warehouse tool built on Hadoop.

• Supports HiveQL (HQL), which is similar to SQL.
• Converts SQL queries into MapReduce jobs.
• Used for data summarization, querying, and analysis.
• Preferred by users with a SQL background.