UNIT II : Big Data Analytics 9

Classification of analytics – Data Science – Terminologies in Big Data – CAP

Theorem – BASE Concept. NoSQL: Types of Databases – Advantages – NewSQL –
SQL vs. NOSQL vs NewSQL

Classification of analytics
Descriptive analytics
Descriptive analytics is a statistical method that is used to search and summarize historical data
inorder to identify patterns or meaning.
Objective: Understand what has happened in the past.
Function: Summarizes historical data to identify patterns or trends.
Tools: Dashboards, reports, data visualization.
Techniques: Data aggregation, data mining, slicing and dicing.

Examples:

o Website traffic reports.

Monthly sales analysis.
Social media engagement statistics.

Data aggregation and data mining are two techniques used in descriptive analytics to
discover historical data. Data is first gathered and sorted by data aggregation in order to make
the datasets more manageable by analysts.
Data mining describes the next step of the analysis and involves a search of the data to identify
patterns and meaning. Identified patterns are analyzed to discover the specific ways that
learners interacted with the learning content and within the learning environment.

Advantages:
 Quickly and easily report on the Return on Investment (ROI) by showing how
performance achieved business or target goals.
 Identify gaps and performance issues early - before they become problems.
  Identify specific learners who require additional support, regardless of how many
students or employees there are.
 Identify successful learners in order to offer positive feedback or additional
resources.
 Analyze the value and impact of course design and learning resources.

Diagnostic Analytics
Objective: Understand why something happened.
Function: Drills deeper into data to find root causes or relationships.
Tools: OLAP (Online Analytical Processing), SQL queries, statistical analysis.
Techniques: Data discovery, correlation analysis, root cause analysis.

Examples:
Identifying why product sales declined in a specific region.
Investigating causes of customer complaints.

Predictive analytics
Predictive Analytics is a statistical method that utilizes algorithms and machine learning to
identify trends in data and predict future behaviors
The software for predictive analytics has moved beyond the realm of statisticians and is
becoming more affordable and accessible for different markets and industries, including the
field of learning & development.
For online learning specifically, predictive analytics is often found incorporated in the
Learning Management System (LMS), but can also be purchased separately as specialized
software.

For the learner, predictive forecasting could be as simple as a dashboard located on the main
screen after logging in to access a course. Analyzing data from past and current progress,
visual indicators in the dashboard could be provided to signal whether the employee was on
track with training requirements.
Advantages:

 Personalize the training needs of employees by identifying their gaps, strengths,

and weaknesses; specific learning resources and training can be offered to support
individual needs.
 Retain Talent by tracking and understanding employee career progression and
forecasting what skills and learning resources would best benefit their career paths.
Knowing what skills employees need also benefits the design of future training.
 Support employees who may be falling behind or not reaching their potential by
offering intervention support before their performance puts them at risk.
 Simplified reporting and visuals that keep everyone updated when predictive
forecasting is required.
Objective: Predict what is likely to happen in the future.
Function: Uses historical data, machine learning, and statistical models to
forecast outcomes.
Tools: Machine learning platforms (e.g., Python, R, SAS, Spark MLlib).
Techniques: Regression, classification, time series forecasting.

Examples:
Predicting customer churn.
Forecasting demand for a product.
Credit risk assessment.

Prescriptive analytics
Prescriptive analytics is a statistical method used to generate recommendations and make
decisions based on the computational findings of algorithmic models.
Objective: Recommend actions to influence future outcomes.
Function: Suggests the best course of action using optimization and
simulation.
Tools: Optimization engines, simulation tools, AI-based decision systems.
Techniques: Linear programming, decision analysis, reinforcement learning.
 Examples:
Route optimization for delivery.
Dynamic pricing models.

Personalized marketing strategies.

Generating automated decisions or recommendations requires specific and unique algorithmic

models and clear direction from those utilizing the analytical technique. A recommendation
cannot be generated without knowing what to look for or what problem is desired to be solved.
In this way, prescriptive analytics begins with a problem.

Example
A Training Manager uses predictive analysis to discover that most learners without a particular
skill will not complete the newly launched course. What could be done? Now prescriptive
analytics can be of assistance on the matter and help determine options for action. Perhaps an
algorithm can detect the learners who require that new course, but lack that particular skill, and
send an automated recommendation that they take an additional training resource to acquire
the missing skill.

The accuracy of a generated decision or recommendation, however, is only as good as the

quality of data and the algorithmic models developed. What may work for one company’s
training needs may not make sense when put into practice in another company’s training
department. Models are generally recommended to be tailored for each unique situation and
need.

Descriptive vs Predictive vs Prescriptive Analytics

Descriptive Analytics is focused solely on historical data.
You can think of Predictive Analytics as then using this historical data to develop statistical
models that will then forecast about future possibilities.
Prescriptive Analytics takes Predictive Analytics a step further and takes the possible
forecasted outcomes and predicts consequences for these outcomes.
 Data Science
Data Science is an interdisciplinary field that uses statistics, computer science, and domain
knowledge to extract insights and knowledge from structured and unstructured data.
Goals:
 Extract meaningful patterns
 Make predictions and informed decisions
 Automate decision-making with AI/ML
Components of Data Science:
1. Data Collection – Gathering raw data from various sources (databases, sensors, APIs).
2. Data Cleaning & Preprocessing – Handling missing values, duplicates, and converting
data into usable formats.
3. Data Exploration & Visualization – Understanding patterns using graphs, plots, and
statistical summaries.
4. Feature Engineering – Creating new variables/features to improve model performance.
5. Modeling & Machine Learning – Applying algorithms to make predictions or
classifications.
6. Evaluation & Validation – Checking model performance using metrics like accuracy,
precision, recall, etc.
7. Deployment & Communication – Deploying the model in production and
communicating insights.
Key Disciplines Involved:
 Statistics & Probability – For data analysis and hypothesis testing
 Machine Learning & AI – For predictions and automation
 Programming – Mainly Python, R, SQL
 Big Data Technologies – Hadoop, Spark for large-scale data
 Data Visualization – Matplotlib, Seaborn, Power BI, Tableau
Types of Analytics in Data Science
1. Descriptive Analytics – Explains what happened
2. Diagnostic Analytics – Explains why it happened
3. Predictive Analytics – Predicts what is likely to happen
4. Prescriptive Analytics – Suggests what should be done

Data Science Lifecycle

1. Problem Definition
2. Data Collection
3. Data Cleaning and Preparation
4. Exploratory Data Analysis (EDA)
5. Modeling & Evaluation
6. Deployment & Monitoring

Applications of Data Science

 Business: Sales forecasting, customer segmentation
 Healthcare: Disease prediction, drug discovery
 Finance: Fraud detection, credit scoring
 E-commerce: Recommendation systems, pricing optimization
 Government & Public Sector: Smart city planning, crime prediction

Terminologies Used In Big Data Environments

  As-a-service infrastructure
Data-as-a-service, software-as-a-service, platform-as-a-service – all refer to the idea that rather
than selling data, licences to use data, or platforms for running Big Data technology, it can be
provided “as a service”, rather than as a product. This reduces the upfront capital investment
necessary for customers to begin putting their data, or platforms, to work for them, as the
provider bears all of the costs of setting up and hosting the infrastructure. As a customer, as-a-
service infrastructure can greatly reduce the initial cost and setup time of getting Big Data
initiatives up and running.

 Data science
Data science is the professional field that deals with turning data into value such as new
insights or predictive models. It brings together expertise from fields including statistics,
mathematics, computer science, communication as well as domain expertise such as business
knowledge. Data scientist has recently been voted the No 1 job in the U.S., based on current
demand and salary and career opportunities.
 Data mining
Data mining is the process of discovering insights from data. In terms of Big Data, because it
is so large, this is generally done by computational methods in an automated way using
methods such as decision trees, clustering analysis and, most recently, machine learning. This
can be thought of as using the brute mathematical power of computers to spot patterns in data
which would not be visible to the human eye due to the complexity of the dataset.
 Hadoop
Hadoop is a framework for Big Data computing which has been released into the public
domain as open source software, and so can freely be used by anyone. It consists of a number
of modules all tailored for a different vital step of the Big Data process – from file storage
(Hadoop File System – HDFS) to database (HBase) to carrying out data operations (Hadoop
MapReduce – see below). It has become so popular due to its power and flexibility that it has
developed its own industry of retailers (selling tailored versions), support service providers and
consultants.
 Predictive modelling
At its simplest, this is predicting what will happen next based on data about what has happened
previously. In the Big Data age, because there is more data around than ever before,
predictions are becoming more and more accurate. Predictive modelling is a core component
of most Big Data initiatives, which are formulated to help us choose the course of action which
will lead to the most desirable outcome. The speed of modern computers and the volume of
data available means that predictions can be made based on a huge number of variables,
allowing an ever-increasing number of variables to be assessed for the probability that it will
lead to success.
 MapReduce
MapReduce is a computing procedure for working with large datasets, which was devised due
to difficulty of reading and analysing really Big Data using conventional computing
methodologies. As its name suggest, it consists of two procedures – mapping (sorting
information into the format needed for analysis – i.e. sorting a list of people according to their
age) and reducing (performing an operation, such checking the age of everyone in the dataset
to see who is over 21).
 NoSQL
NoSQL refers to a database format designed to hold more than data which is simply arranged
into tables, rows, and columns, as is the case in a conventional relational database. This
database format has proven very popular in Big Data applications because Big Data is often
messy, unstructured and does not easily fit into traditional database frameworks.
 Python
Python is a programming language which has become very popular in the Big Data space due
to its ability to work very well with large, unstructured datasets
R Programming
R is another programming language commonly used in Big Data, and can be thought of as
more specialised than Python, being geared towards statistics. Its strength lies in its powerful
handling of structured data. Like Python, it has an active community of users who are
constantly expanding and adding to its capabilities by creating new libraries and extensions.
 Recommendation engine
A recommendation engine is basically an algorithm, or collection of algorithms, designed to
match an entity (for example, a customer) with something they are looking for.
Recommendation engines used by the likes of Netflix or Amazon heavily rely on Big Data
technology to gain an overview of their customers and, using predictive modelling, match
them with products to buy or content to consume. The economic incentives offered by
recommendation engines has been a driving force behind a lot of commercial Big Data
initiatives and developments over the last decade.
 Real-time
Real-time means “as it happens” and in Big Data refers to a system or process which is able to
give data-driven insights based on what is happening at the present moment. Recent years have
seen a large push for the development of systems capable of processing and offering insights in
real-time (or near-real-time), and advances in computing power as well as development of
techniques such as machine learning have made it a reality in many applications today.
 Reporting
The crucial “last step” of many Big Data initiative involves getting the right information to the
people who need it to make decisions, at the right time. When this step is automated, analytics
is applied to the insights themselves to ensure that they are communicated in a way that they
will be understood and easy to act on. This will usually involve creating multiple reports based
on the same data or insights but each intended for a different audience (for example, in-depth
technical analysis for engineers, and an overview of the impact on the bottom line for c-level
executives).
 Spark
Spark is another open source framework like Hadoop but more recently developed and more
suited to handling cutting-edge Big Data tasks involving real time analytics and machine
learning. Unlike Hadoop it does not include its own filesystem, though it is designed to work
with Hadoop’s HDFS or a number of other options. However, for certain data related processes
it is able to calculate at over 100 times the speed of Hadoop, thanks to its in-memory
processing capability. This means it is becoming an increasingly popular choice for projects
involving deep learning, neural networks and other compute-intensive tasks.
 Structured Data
Structured data is simply data that can be arranged neatly into charts and tables consisting of
rows, columns or multi-dimensioned matrixes. This is traditionally the way that computers
have stored data, and information in this format can easily and simply be processed and mined
for insights. Data gathered from machines is often a good example of structured data, where
various data points – speed, temperature, rate of failure, RPM etc. – can be neatly recorded and
tabulated for analysis.
 Unstructured Data
Unstructured data is any data which cannot easily be put into conventional charts and tables.
This can include video data, pictures, recorded sounds, text written in human languages and a
great deal more. This data has traditionally been far harder to draw insight from using
computers which were generally designed to read and analyze structured information.
However, since it has become apparent that a huge amount of value can be locked away in this
unstructured data, great efforts have been made to create applications which are capable of
understanding unstructured data – for example visual recognition and natural language
processing.
 Visualization
Humans find it very hard to understand and draw insights from large amounts of text or
numerical data – we can do it, but it takes time, and our concentration and attention is limited.
For this reason effort has been made to develop computer applications capable of rendering
information in a visual form – charts and graphics which highlight the most important insights
which have resulted from our Big Data projects. A subfield of reporting (see above),
visualizing is now often an automated process, with visualizations customized by algorithm to
be understandable to the people who need to act or take decisions based on them.

Basic availability, Soft state and Eventual consistency

Basic availability implies continuous system availability despite network failures and
tolerance to temporary inconsistency.
Soft state refers to state change without input which is required for eventual consistency.
Eventual consistency means that if no further updates are made to a given
updated database item for long enough period of time , all users will see the
same value for the updated item.

Top Analytics Tools:

* R is a language for statistical computing and graphics. It also used for big
data analysis. It provides a wide variety of statistical tests.

Features:

 Effective data handling and storage facility,

 It provides a suite of operators for calculations on arrays, in
particular, matrices,
 It provides coherent, integrated collection of big data tools for data
analysis
 It provides graphical facilities for data analysis which display either
on-screen or on hardcopy

* Apache Spark is a powerful open source big data analytics tool. It offers
over 80 high-level operators that make it easy to build parallel apps. It is
used at a wide range of organizations to process large datasets.

Features:

 It helps to run an application in Hadoop cluster, up to 100 times faster

in memory, and ten times faster on disk
 It offers lighting Fast Processing
  Support for Sophisticated Analytics
 Ability to Integrate with Hadoop and Existing Hadoop Data

* Plotly is an analytics tool that lets users create charts and dashboards to
share online.

Features:

 Easily turn any data into eye-catching and informative graphics

 It provides audited industries with fine-grained information on data
provenance
 Plotly offers unlimited public file hosting through its free community
plan

* Lumify is a big data fusion, analysis, and visualization platform. It helps

users to discover connections and explore relationships in their data via a
suite of analytic options.

Features:

 It provides both 2D and 3D graph visualizations with a variety of

automatic layouts
 It provides a variety of options for analyzing the links between
entities on the graph
 It comes with specific ingest processing and interface elements for
textual content, images, and videos
 It spaces feature allows you to organize work into a set of projects, or
workspaces
 It is built on proven, scalable big data technologies
 * IBM SPSS Modeler is a predictive big data analytics platform. It offers
predictive models and delivers to individuals, groups, systems and the
enterprise. It has a range of advanced algorithms and analysis techniques.

Features:

 Discover insights and solve problems faster by analyzing structured

and unstructured data
 Use an intuitive interface for everyone to learn
 You can select from on-premises, cloud and hybrid deployment
options
 Quickly choose the best performing algorithm based on model
performance

* MongoDB is a NoSQL, document-oriented database written in C, C++, and

JavaScript. It is free to use and is an open source tool that supports multiple
operating systems including Windows Vista ( and later versions), OS X
(10.7 and later versions), Linux, Solaris, and FreeBSD.

Its main features include Aggregation, Adhoc-queries, Uses BSON format,

Sharding, Indexing, Replication, Server-side execution of javascript,
Schemaless, Capped collection, MongoDB management service (MMS), load
balancing and file storage.

Features:

 Easy to learn.
 Provides support for multiple technologies and platforms.
 No hiccups in installation and maintenance.
  Reliable and low cost.

CAP Theorem:

CAP Theorem in Big Data is a pivotal concept that underpins the architecture of
distributed systems in the age of massive data sets and real-time processing.
Conceived by Eric Brewer, the CAP Theorem presents a triad of critical
properties, namely Consistency, Availability, and Partition Tolerance.
One property often must be prioritised at the expense of the others. This
framework plays a significant role in guiding the design choices for systems
dealing with vast and complex data, ensuring they strike the right balance
between data accuracy, system responsiveness, and fault tolerance. In this blog,
you will learn about how the CAP Theorem in Big Data plays an important role
in distributed computing and database systems, significantly impacting the
design and operation of Big Data systems.
Understanding What is the CAP Theorem in Big Data
a) Consistency
b) Availability
c) Partition Tolerance
d) Big Data and CAP Theorem
Example of the CAP Theorem in Big Data
a) MongoDB
b) Cassandra

Understanding What is the CAP Theorem in Big Data

The CAP Theorem, often referred to as Brewer's theorem after its creator, Eric
Brewer, is a fundamental concept in the world of distributed systems, and its
implications are especially pertinent in Big Data. It articulates the inherent
trade-offs that distributed databases and systems must navigate among three key
properties, which are:

1) Consistency
Consistency, the first element of the CAP Theorem, signifies that every read
operation in a distributed system will return the most recent write or an error. In
other words, all nodes within the system exhibit the same data value at any
given time. Achieving strong consistency is crucial in applications where data
accuracy is paramount, such as financial transactions or healthcare records.
2) Availability
The second property, Availability, indicates that every request, whether it's a
read or write operation, receives a response, and that response is not an error. In
essence, the system is always operational and responsive to client requests.
High availability is essential for systems that cannot tolerate downtime, like e-
commerce platforms or real-time analytics.
3) Partition Tolerance
Partition tolerance relates to the system's ability to function reliably despite
network partitions or communication breakdowns. Network partitions can occur
due to factors like hardware failures, congestion, or geographical distribution,
leading to nodes being unable to communicate. A partition-tolerant system
will continue to operate, ensuring nodes can communicate even under
challenging network conditions.
The CAP theorem declares that, in a distributed system, you can't
simultaneously achieve all three properties. Instead, you must prioritise two out
of the three, and the choice of which two significantly impacts the system's
behaviour:
a) CA or Consistency and Availability: Prioritising both Consistency and
Availability means that the system maintains strong data consistency and high
responsiveness but sacrifices Partition Tolerance. It can work well in stable
network conditions, but it may become problematic during network partitions.
b) CP or Consistency and Partition Tolerance: Emphasising Consistency and
Partition Tolerance ensures strong data consistency and the ability to withstand
network partitions, but it might result in periods of unavailability during
partition events.
c) AP or Availability and Partition Tolerance: Focusing on Availability and
Partition Tolerance aims for high system availability and the ability to operate
under network partitions. However, this might come at the cost of relaxing
strong consistency, allowing for temporary data inconsistencies.
Predict market trends and demographics by signing up for our Big Data for
Data Engineering Training now!
Big Data and CAP Theorem
In Big Data, distributed systems are prevalent, given the vast amounts of data
that need to be processed, stored, and analysed. The CAP Theorem provides
valuable guidance in making architectural decisions for these systems:
a) Data Consistency in Big Data: Big Data applications often prioritise
eventual consistency over strong consistency. In scenarios like real-time
analytics or recommendation engines, it's acceptable for data to temporarily be
inconsistent across nodes as long as it converges to a consistent state over time.
This approach improves system availability.
b) High Availability in Big Data: High availability is a paramount requirement
for Big Data systems, as they often deal with massive workloads and must serve
data without interruptions. Technologies like Hadoop and Spark typically adopt
the AP model, focusing on availability and partition tolerance.
c) Partition Tolerance in Big Data: Big Data systems inherently require
partition tolerance due to the large-scale distribution of data. Technologies like
Apache Kafka, used for streaming data, focus on partition tolerance and fault
tolerance.
Looking at an Example of the CAP Theorem in Big Data
Consider the example of an e-commerce application that combines real-time
product availability with uninterrupted customer access, handling network
disruptions seamlessly. It prioritises high availability and partition tolerance to
ensure 24/7 shopping while sacrificing strict consistency for occasional, minor
delays in updating product quantities during peak traffic, striking an effective
balance.
Here are the various ways the e-commerce application demonstrates the CAP
Theorem in Big Data:

a) Consistency: The E-commerce application ensures that product

inventory remains accurate in real-time, preventing overselling and maintaining
order consistency.
b) Availability: Customers can access the website 24/7, browse products, and
make purchases without encountering downtime or errors.
c) Partition Tolerance: The system handles high network loads during peak
shopping seasons and occasional network disruptions without affecting users'
ability to complete transactions. Data replication across multiple servers ensures
data availability even if a server goes down temporarily.
d) Balancing act: The application strikes a balance by prioritising Availability
and Partition Tolerance or AP to keep the platform accessible while sacrificing
strong Consistency for some scenarios, such as momentarily displaying product
quantities that may not be perfectly up-to-date during peak traffic.
Furthermore, MongoDB and Cassandra are two prominent NoSQL databases
that exemplify the CAP Theorem's application in Big Data. Both databases
address the need for scalable, distributed Data Management, but they do so with
different priorities and architectural approaches.
1) MongoDB
MongoDB, often associated with the AP or Availability and Partition Tolerance
end of the CAP spectrum, is a widely used document-oriented NoSQL database.
It prioritises high availability and fault tolerance, making it a valuable choice
for various Big Data applications. Here are the key aspects demonstrated by
MongoDB in Big Data:

a) Availability: MongoDB is designed to minimise downtime. With its

distributed architecture and replica sets, it ensures high availability by allowing
multiple copies of data across nodes. In the event of a node failure, the system
can promptly switch to a secondary node, preventing service interruptions.
b) Partition Tolerance: MongoDB excels in handling network partitions. Its
partition tolerance is a result of data replication and automatic failover
mechanisms. Even during network disruptions, the database remains accessible.
c) Consistency: While MongoDB provides strong consistency at the document
level, it offers flexibility in terms of consistency. Developers can choose the
level of consistency they need for specific queries, balancing it with availability
and performance. This means that MongoDB can lean towards eventual
consistency when required.
2) Cassandra
Cassandra, in contrast, aligns more with the AP side of the CAP theorem,
focusing on Availability and Partition Tolerance. It is built to deliver high
scalability and fault tolerance in distributed environments, making it an ideal
choice for managing Big Data. When comparing it to other databases,
understanding the differences in their design and functionality can help
highlight their strengths. For instance, a Couchbase vs Cassandra comparison
reveals unique aspects of how each handles data storage and retrieval in large-
scale systems.
a) Availability: Cassandra focuses on maintaining a high level of availability,
which is crucial in scenarios like e-commerce and social media platforms where
downtime can have significant repercussions. It employs a peer-to-peer
architecture and data replication to ensure that data remains accessible, even in
the face of node failures.
b) Partition Tolerance: Partition tolerance is a core feature of Cassandra. Its
decentralised design and support for data distribution across multiple nodes
make it inherently resilient to network disruptions. Data can still be read and
written to the database during partition events.
c) Consistency: Cassandra provides tunable consistency levels, allowing users
to choose between different consistency models. It can offer strong consistency
for critical operations while relaxing consistency for less crucial tasks,
promoting high availability.

Overview of SQL vs. NoSQL vs. NewSQL:

These are the types of databases, and in the upcoming paragraph, you will
understand each of them. SQL Databases can also be termed Relational
Database management Systems, also known as RDBMS, a classical approach to
storing and operating historical data. In such a system, the information follows a
structured approach utilizing tables or relations.

With the advent of Big Data, the structured approach fails miserably to cater to
the needs of the humongous information processing that tends to be
unstructured. Over time, SQL has undergone many iterations to support the vast
data processing and pipelines. However, answering big data systems that expect
a quick response and supreme scalability is still inefficient.

NewSQL is a term that’s used to describe product offerings that support

the relational data model while delivering the same scalable performance of
NoSQL database systems. Source: A NewSQL Database For Enterprises

A new approach known as NoSQL was introduced to work around the

limitations imposed by the former. Its system was designed to provide fast
scalability when dealing with unstructured data platforms or handling Big Data
applications. Its databases use a key-value pair, Documents, graph databases, or
wide–column stores without a typical schema. It is also horizontally scalable as
opposed to only vertical scaling in RDBMS.

It showed great promise as an ideal database system for Big Data applications,
but like anything else, it fell short due to some major drawbacks, as discussed
below. This is where NewSQL came alive. NewSQL is the latest development
in the world of database systems. NewSQL is a relational database with the
scalable properties of it.

What is SQL Relational Database?

The term SQL is both a language and a type of database. It stood for structured
query language and pioneered database design philosophy. Since the mid-’80s,
it has been a standard for managing and querying relational datasets; however,
the early beginnings of the relational model can be dated back to the ’60s and
’70s when the urgent need to distinguish between the application data and
application code emerged, allowing the developers to focus on other aspects of
the program development such as access to and manipulation of data at hand.
 IBM’s IMS was the first fully functional relational database designed for a
different purpose to organize data for the Apollo space exploration program.
The relational database is a collection of time-varying, normalized relations of
assorted degrees. The following intuitive correspondence can be made.
 A relation is a file.
 Each file contains only one record type
 The records have no particular order
 Every field has a single-value
 The records have a unique identifying field, or composite field, called the
primary key field.
What are the Concepts of SQL Relational Databases?
The concepts of SQL Relational Databases are below:
ACID
Atomicity, consistency, isolation, and durability are needed to maintain the
reliability of transactions.
1. Atomicity – completion of the transaction as a whole or none at all
2. Consistency – assures the stable state of the database with or without
changes
3. Isolation – multiple transactions do not interfere with each other
4. Durability – permanent effect on the database by the changes

Normalization
A process of designing efficient databases

 1NF – Split the table by separating repeating and nonrepeating attributes.

All domains are simple, and all elements are atomic
 2NF – Remove partial dependency between attributes. No attribute
should functionally depend on one part of an aggregated primary key.
 3NF – Remove transitive dependency between table attributes. No prime
attribute is functionally dependent on the non-prime attribute.
Scalability
The capability of the database to handle growing amounts of data. Vertical
scaling helps to enhance the existing capacity of the database server. Most SQL
databases support vertical scaling. They can, however, scale up, not scale out.
Domains
A domain is a named set of scalar values, all of the same type. They help to
impose semantic constraints. Rely on traditional features and utilize a defined
data schema. Support JOIN functionality, engineered for data integrity

Not-only-SQL are the databases that let the developers store/manage

unstructured data and perform complex analytical operations on it.
Source: Overview of NoSQL Databases

What are the Drawbacks?

Although RDBMS provides exclusive features, it suffers greatly from some
major drawbacks.
The Rigidity of Data Modeling
One of the relational database's biggest limitations is the rigidity of organizing
the data into a particular structure in tables and relations. Since all the data
cannot be conveniently fitted into tables, this approach can’t be applied to all-
natural data and stored as trees and graphs. However, RDBMS works around
this limitation by modeling this data in a normalized manner with parent-child
relations, which is still insufficient.
Diversity
The complexity of the data also creates a limitation in the relational database.
These databases are made to organize the data by common characteristics.
Complex numbers, images, and multimedia data are hard to store, access, and
process.
Inefficient Usage of Space
When defining the relation schema, we define the attributes' size. Not all the
records have data that uses the full space; some have a short length. Every
record needn’t necessarily fit into the given data type, again resulting in space
wastage.
Heavy Weight Changes
Any changes required for one record must be applied to all records, resulting in
a heavyweight change. The changes can be expensive and infeasible depending
on the size and number of records present. Thus, it was challenging to change
the schema of an existing database.
Inefficient for Big Data
SQL is not suitable for volume, velocity, and variety of data, rendering it highly
inefficient for a cloud-based application

What is NoSQL?
These problems became the impetus for the NoSQL movement in the mid-to-
late 2000s. The key working strategy is to forgo strong transactional guarantees
and relational models of DBMS in favour of eventual consistency and
alternative data models like key-value pairs and graphs.

This was done following the belief that these aspects of existing DBMSs inhibit
their ability to scale out and achieve the high availability needed to support web
apps on the go. The two most well-known systems that followed the creed were
Google’s BigTable and Amazon’s Dynamo, which were restricted to use inside
their organizations, leading to organizations creating their open-source clones
like Facebook’s Cassandra and Powerset’s HBase. By the end of the 2000s, a
diverse set of scalable and affordable DBMS existed.

What are the Concepts of NoSQL?

The concepts are listed below:
Lack of schema
Support for structured, semi-structured, and unstructured data. No specific
schema must be defined before data is entered into its databases. New fields can
be added, and it also supports nested data implementation and retrieval.
Developers can use data type and query options requisite for the specific
application, resulting in faster development. Faster development time is
considerably reduced due to the absence of complex SQL queries or join
statements.
Auto Balancing
Data is divided among multiple servers automatically, with no assistance
required from applications.
Integrated Caching
Its database caches data in system memory to increase data throughput and
performance in advance. With a simple data model and query language, it offers
high scalability and reliability.
The BASE Principle For Transaction
The base is to NoSql as what ACID is to SQL. It ensures that NoSQL databases
are reliable despite the loss of consistency. The base stands for Basically
Available Soft-state, Eventually consistent. Eventually consistent – The system
can become consistent, and information is updated wherever necessary.

Big Data Architecture helps design the Data Pipeline with the various
requirements of the Batch and Stream Processing System. Source: Big Data
Architecture

What are the drawbacks?

The drawbacks of NoSQL are listed below:
Lack of Consistency
Since these systems prefer availability over consistency, they fail miserably
when consistency is the most important thing in financial transactions. The non-
synchronization of data nodes risks system failure.
Lack of Analytics
For analytics, you require a relational model to process the data, which means
the whole database needs to be converted using some relational model. This
leads to increased cost overhead.
Lack of Standardization
No specific language
Security
Doesn’t provide security at the elemental level of data.
Transactional Nature
It is important to facilitate fraud detection before completing the transaction and
check for balance while on a call. NoSQL fails when the database needs to
compete against high-volume transactions per day, as these require a highly
scalable, consistent database.

What is NewSQL?
The early counter-measures to the above approaches were a powerful single-
node machine that can handle all the transactions and a custom-built
middleware system to distribute queries over traditional DBMS nodes. But both
of them are prohibitively expensive to carry out. As a result, there was a need
for an intermediate database system that combines NoSQL systems' distributed
architectures with multiple node concurrency and a whole new storage
mechanism.

Thus Newsql can be defined as a class of modern relational DBMSs that seek to
provide the same scalable performance of NoSQL for OLTP workloads and
simultaneously guarantee ACID compliance for transactions as in RDBMS. In
other words, these systems want to achieve NoSQL's scalability without
discarding the relational model with SQL and the legacy DBMS transaction
support.
 Concepts of NewSQL
 Main Memory storage of OLTP databases enables in-memory
computations of databases.
 Scaling out by splitting a database into disjoint subsets called partitions or
shards involves executing a query into multiple partitions and then combining
them into a single result.
 Its systems preserve the ACID properties of databases.
 Enhanced concurrency control system benefits traditional ones.
 The presence of a secondary index allows it to support faster query
processing times.
 High availability and Strong data durability are only possible with
replication mechanisms.
 Configure its systems to provide synchronous updates of data over the
WAN.
 It minimizes downtime and provides fault tolerance with its crash
recovery mechanism.

What is the difference Between SQL, NoSQL, and

NewSQL?
Feature SQL NoSQL NewSQL

No, it doesn't follow a Yes, since the

Yes, it follows
Relational relational model. It was relational model is
relational modelling
Property designed to be entirely equally essential for
to a large extent.
different from that. real-time analytics.

Yes, ACID
properties are No, rather provides for Yes, Acid properties
ACID
fundamental to their CAP support are taken care of.
application

SQL Support for SQL No support for old SQL Yes, proper support
 and even enhanced
functionalities for Old
SQL

Fully functionally
It supports such
Inefficient for OLTP supports OLTP
OLTP databases, but it is not
databases. databases and is
the best suited.
highly efficient

Vertical + Horizontal
Scaling Vertical scaling Only Vertical scaling
scaling

Can handle simple Highly efficient in

Better than SQL for
Query queries with ease processing complex
processing complex
Handling and fails when they queries and smaller
queries
get complex queries.

Distributed
No Yes Yes
Databases

SQL, NoSQL, or NewSQL – Which is the Best Solution to

Big Data?
SQL complies with ACID properties and does well with vertical scalability,
while NoSQL offers its horizontal scaling and provides for BASE properties.
However, NoSQL does not play by the ACID rules necessary to maintain a
reliable and consistent database. Fast-paced enterprises and organizations
generate terabytes of transactional data daily while working in an OLTP system.
NewSQL is the ideal choice.

NewSQL improves upon SQL by providing horizontal scalability while

maintaining ACID properties. This facilitates working with Big Data by
implementing concurrency. It also does well with ACID compliance. Thus,
NewSQL seems to have found the sweet spot between speed, scalability,
consistency, and availability. Even though it is still nascent, NewSQL ticks all
the right boxes to be an ideal database for Big Data and OLTP applications.
You can also explore the difference between Virlet and Kubevirt in this blog.

Types of Databases:

What Is a Database
A database is a systematically organized collection of data stored electronically, designed to
make it easier to store, access, manage and analyze information, including words, numbers,
images, videos, and files.

Types of databases include relational databases, NoSQL databases, object-oriented databases,

and graph databases. Relational databases use structured tables, while NoSQL supports
unstructured data. Object-oriented databases store data as objects, and graph databases
manage relationships using nodes and edges.

A “database” can also refer to a set of connected data accessed through a Database
Management System (DBMS), which is a software that allows users to interact with one or
more databases.

Because of the close relationship between them, the term “database” is often used casually to
refer to both a database and the DBMS used to manipulate it.

Database vs. Database Management System (DBMS)

A Database and a Database Management System (DBMS) are closely related terms, but they
serve different purposes:

 A database is a structured set of data. The data can be structured or unstructured

and stored in various formats like tables, documents, and key-value pairs. It could be
anything from a simple shopping list to a picture gallery or the vast amount of
information in a corporate network.
 A DBMS (Database Management System) is software used to interact with a
database. It provides an interface for users or applications to manipulate data, making
the handling of large amounts of data more efficient and less error-prone. A DBMS
oversees core administrative tasks such as data storage, retrieval, security,
 concurrency control, backup and recovery, and query processing. Examples include
Oracle Database, MySQL, Microsoft SQL Server, and MongoDB.

In simpler terms, a database is like a container that holds the information, while a DBMS is a
tool used to organize and manage the contents within that container.

To make things a little bit more complicated, the term “database” is often used informally to
denote the DBMS, the database system, or even an application connected to the database.

Moving forward, we will refer to the Database Management System (DBMS) as the
“database,” while the actual data stored will be referred to simply as “data.”

Understanding Database Structure

A database structure simply refers to how data is organized within a database. Here are the
key components:

 Tables: The fundamental blocks of a database structure are the tables, which consist
of rows and columns.
 Rows and columns: A crucial aspect of the database structure is the rows and
columns. Rows show independent columns, whereas columns show attributes of those
records.
 Primary key: A unique identifier for each record in a table.
 Foreign key: A column that highlights a relationship between tables by referring to a
primary key in another table.

This structure ensures proper storage, management, and retrieval of your data, resulting in
smoother business operations.

Introduction to Database Types

Databases can be classified into two primary types: Relational and NoSQL Databases.
NoSQL is then further divided into four types: Document-oriented, Key-Value, Wide-
Column, and Graph databases.

Relational
Attribute NoSQL Databases
Databases

Structured, tabular
Data
schema, fixed data Schema-less, accommodates a variety of data types.
Structure
types.

Scalability Typically scaled Designed for horizontal scaling (more servers).

vertically (more
 Relational
Attribute NoSQL Databases
Databases

CPU, RAM, SSD).

Flexibility Limited flexibility. High flexibility.

Adhere to ACID
ACID properties for
Some sacrifice ACID compliance for performance and scalability.
Properties reliable
transactions.

Financial systems,
Use Cases CRM, e-commerce Social media, IoT, real-time analytics, big data applications.
applications.

*It’s worth noting that some databases, like MongoDB, can fall into multiple categories as
they support different data models. Additionally, the list provided is not exhaustive, as there
are many other databases available in each category.

Let’s take a deeper look into each database type, exploring its unique characteristics, real-
world applications, and practical considerations.

What are the different types of databases?

The main types of databases include hierarchical, network, object-oriented, relational, and
NoSQL databases. Each type is designed for specific use cases, from managing structured
data to handling large-scale, unstructured data.

Hierarchical Database: Arranges data in a tree structure with parent-child relationships,

making it ideal for use in scenarios where there are clear hierarchies, like organizational
models, file systems, or hierarchical workflows.

Network Database: Uses a graph-like structure to represent many-to-many relationships,

often deployed in more complex environments that require multiple tangent points, like social
networks or complex data systems like airline reservation platforms.

Object-Oriented Database: Stores data as objects, similar to object-oriented programming,

and is ideal for applications with complex data models, such as multimedia or financial
systems.
Relational Database: Structures data in tables with rows and columns, adhering to ACID
properties for reliable transactions. Commonly used in applications like CRM, financial
systems, and e-commerce.

NoSQL Database: Offers flexible, schema-less structures for handling large volumes of
unstructured or semi-structured data. Examples include document-oriented, key-value, wide-
column, and graph databases, often used in big data and real-time applications.

Generally, databases are broadly categorized into relational (SQL) and non-relational
(NoSQL) systems, with organizations often choosing one or combining both, based on their
data and operational needs.

Relational Databases (also known as SQL Databases)

A relational database (RDB) is a method of organizing data into tables, rows, and columns to
show relationships between data points. This structure makes it straightforward to access,
create, read, modify, and delete data using a querying language—such as SQL.

Each table, also known as a relation, has rows (records) and columns (fields), where each row
represents an entity, and each column represents an attribute of that entity.

You can establish relationships between entities through primary and foreign keys by
guaranteeing data integrity and enabling complex queries.

Strengths of Relational Databases:

 The structured, table-like schema is easy to understand.

 Follow ACID (Atomicity, Consistency, Isolation, Durability) properties which makes
them reliable.
 SQL language is standardized, widely used, and applicable to a variety of database
management systems.

Weaknesses of Relational Databases:

 Can be difficult to scale out on multiple servers (horizontal scaling).

 Each table requires a predefined schema which means all data inserted into the table
must follow the same structure. This is not optimal when dealing with complex data
structures.

Common Use Cases:

 Widely used in the financial industry thanks to the ACID properties that ensure data
reliability in financial transactions and support complex financial analysis.
 In healthcare, relational databases are used to manage structured data such as patient
records, medical histories, and test results, aiding healthcare workflows.
 Used to manage structured data such as customer, order, product, and payment data in
e-commerce.
NoSQL Databases

NoSQL databases were developed as an alternative to traditional SQL databases, NoSQL

databases are especially useful when working with large or fast-moving data that may not fit
neatly into a table.

NoSQL databases use various data models for accessing and managing data. These databases
are optimized for applications needing flexible data models, handling large volumes of data,
and achieving low latency.

They accomplish this by relaxing some of the data consistency restrictions found in relational
databases, making them ideal for dynamic, high-performance applications that require
scalability and speed.

Instead of tables, NoSQL databases use more flexible data models, such as key-value pairs,
documents, or graphs. They offer scalability and flexibility, making them suitable for
handling large amounts of unstructured or semi-structured data. Examples include MongoDB,
CouchBase, Cassandra, and Redis.

Strengths of NoSQL Databases:

 Flexible and scalable, ideal for handling large amounts of unstructured or semi-
structured data.
 Scale horizontally across multiple servers, making them highly scalable and fault-
tolerant.
 Well-suited for use cases such as social media, e-commerce, and big data analytics.

Weaknesses of NoSQL Databases:

 Lack of standardized query language: each type of NoSQL database has its own
method for querying data, which can make these systems more challenging to learn,
integrate, and communicate with.
 Not well-suited for complex transactions or querying relationships between data.

Common Use Cases:

 Social media platforms employ NoSQL databases to store and manage user-generated
content, facilitating efficient storage and processing of diverse content types.
 In e-commerce, NoSQL databases are crucial for handling product recommendations
and customer analytics, enabling personalized recommendations and improved
customer experiences.
 NoSQL databases excel in big data analytics, handling high-velocity and high-volume
data streams and extracting valuable insights from rapidly changing data sources.
When should I use a NoSQL database over a relational database?
A NoSQL database is a better choice than a relational database when you’re dealing with
large amounts of unstructured, semi-structured or rapidly changing data that does not neatly
fit into rows and columns.

NoSQL databases excel in use cases requiring high scalability, horizontal distribution across
nodes, and low-latency performance, such as real-time analytics, content management, and
IoT applications. They are particularly suited for scenarios involving schema flexibility, like
storing JSON documents or graph-based data, or when working with key-value pairs for
caching and session management.

Additionally, NoSQL is better suited to scenarios where eventual consistency is acceptable

over strict ACID compliance, enabling high throughput for systems handling massive,
distributed workloads, such as social media platforms or recommendation engines.

Scenario When to Use NoSQL When to Use Relational Databases

NoSQL databases like

Handling large- MongoDB or Cassandra are
scale, ideal for managing Relational databases struggle with unstructured
unstructured unstructured or semi- data due to rigid schemas.
data structured data, such as JSON
files or multimedia.

Use NoSQL when your data

Dynamic or Relational databases are better when the data
model changes frequently or
evolving data model is stable and well-defined from the
requires flexible schema
models outset.
definitions.

Real-time NoSQL is suited for real-time

analytics or processing in applications like Relational databases may lag in high-velocity
high-velocity IoT, social media, or e- scenarios due to their strict ACID compliance.
data commerce analytics.

Avoid NoSQL if your

Complex application relies heavily on Relational databases excel at managing complex
relationships joins or complex relationships with structured data and foreign
and joins relationships; graph databases keys.
are an exception.

Horizontal NoSQL databases are Relational databases typically scale vertically,

Scenario When to Use NoSQL When to Use Relational Databases

designed for horizontal

scalability scaling across multiple
which can become expensive and less efficient.
requirements servers, making them ideal for
distributed systems.

Strict NoSQL databases often trade

Use relational databases for applications like
transactional strict consistency for
banking or financial systems that require ACID
integrity scalability, suitable for
properties.
(ACID) eventual consistency models.

NoSQL databases offer multi-

Support for
model support, such as Relational databases are less flexible for multi-
multi-model
combining document, key- model use cases.
use cases
value, and graph models.

NoSQL databases like key-

Data archival or
value stores are better for Relational databases are overkill for simple key-
simple read
caching or systems with high- value or archival needs.
operations
read, low-write requirements.

Applications
NoSQL may fall short due to
requiring Relational databases are ideal for complex
limited or specialized query
complex queries requiring SQL, joins, and aggregations.
capabilities.
querying

NoSQL supports rapid

Short Relational databases require upfront schema
development and iteration due
development design, slowing development in dynamic
to its flexible schema and
cycles environments.
scalability.

System
Not ideal unless specific use Relational databases are the best choice for
handling
cases demand alternative structured, tabular data with predictable
structured,
models. relationships.
tabular data
 Relational vs NoSQL Databases
SQL (Structured Query Language) and NoSQL (Not Only SQL) databases are fundamentally
different in how they are designed, how they store data, and how they are used.

 SQL databases use a structured, tabular schema, adhere to ACID (Atomicity,

Consistency, Isolation, and Durability) properties, and are optimized for complex
queries and relationships, but they typically scale vertically, which can have
limitations and higher costs.
 NoSQL databases offer flexibility with a schema-less structure and various data
types, scale horizontally for handling large data volumes, and provide simpler query
languages, but they may sacrifice ACID compliance for performance and are not
uniformly designed for handling complex relationships.

However, these distinctions are not always clear-cut, and the choice of database ultimately
depends on your specific use case and requirements.

Relational Databases NoSQL Databases Vector Databases

Structured, tabular
Data Schema-less accommodates
schema, fixed data Vector data
Structure a variety of data types.
types.

Typically scaled
Designed for horizontal
Scalability vertically (more CPU, Highly scalable
scaling (more servers).
RAM, SSD).

Flexibility Limited flexibility. High flexibility. Limited flexibility

Some sacrifice ACID

Adhere to ACID
ACID compliance for performance
properties for reliable N/A
Properties and scalability, others offer
transactions.
tunable consistency.

Often uses simpler query

Query Uses SQL for defining
languages, may provide APIs Large-scale similarity search
Language and manipulating data.
for data manipulation.

Relations Can handle relations Some can handle relations N/A

between different data effectively (like graph
points effectively. databases), but not all are
 designed to do so.

Ideal for applications

requiring complex Used for large data sets, real-
Recommendation systems, AI
queries, and multi-row time applications, and high
Use Cases products, Anomaly and fraud
transactions: financial write volume systems: social
detection
systems, e-commerce media, big data applications.
applications.

Types of NoSQL Databases

There are several types of NoSQL databases, including document-oriented databases, key-
value stores, wide-column stores, and graph databases, each designed to serve different needs
and use cases. Let’s take a look at the four most common types of NoSQL databases and
when to use them.

Graph Databases

Graph databases, a type of NoSQL database, are designed to handle complex relationships
between data points, making them ideal for use cases where relationships are as important as
the data itself, like in social networks or recommendation systems. They excel at representing
and querying graph-like structures, using nodes to represent entities and edges to represent
relationships. Popular examples include Neo4j, OrientDB, and ArangoDB.

Strength of Graph Databases:

 Excel in storing relationships and finding patterns in interconnected data.

 Ideal for use cases where relationships are crucial.

Weaknesses of Graph Databases:

 Limited efficiency for non-relationship-based queries.

 Learning curve and specialized skills.
 Not the best choice for use cases that primarily involve simple data storage or
retrieval without complex relationships.

Common Use Cases:

 Graph databases are used to track relationships between users on social networks
(friends, followers, etc.), model interactions (likes, shares, comments), and deliver
 personalized content and recommendations. They are especially good at the “friend-
of-a-friend” types of queries that are common in social networks.
 In e-commerce, they are used for handling product recommendations and customer
analytics.
 In logistics, they are used for tracking shipments and optimizing routes. They can
handle the complex, multi-point routing requirements common in logistics and
delivery systems.

Document-Oriented Databases

Category within NoSQL databases, characterized by its schema-less design, where each
document (data record) can have a different structure and is typically stored in a format like
JSON or XML. This structure is highly flexible and adaptable to complex data structures.
Examples include MongoDB and CouchDB.

Strengths of Document-Oriented Databases

 Flexible and dynamic, able to handle unstructured and semi-structured data.

 Don’t enforce a rigid schema, allowing documents to evolve and accommodate
changes.
 Can handle simple relationships through embedded documents or references.
 Widely used in various applications without a specific industry focus.

Weaknesses of Document-Oriented Databases

 Limited efficiency for complex relationship queries.

 Lack of standardization and interoperability between different document-oriented
databases.
 It may not be the best choice for applications that heavily rely on complex
relationships or require extensive relational querying capabilities.
 It is challenging to maintain data consistency and integrity without a predefined
schema.
 It may not be suitable for applications with extensive relational querying
requirements.