Welcome to AIKTC

Module 3|

Prof. Nusrat Jahan,

Department of Computer Engineering.
AIKTC – Anjuman-I-Islam’s Kalsekar Technical Campus.
 NoSQL(Syllabus)
• Introduction to NoSQL, NoSQL Business Drivers
• NoSQL Data Architecture Patterns: Key-value stores, Graph stores, Column
family (Bigtable)stores, Document stores, Variations of NoSQL architectural
patterns, NoSQL Case Study
• NoSQL solution for big data, Understanding the types of big data problems;
Analyzing big data with a shared-nothing architecture; Choosing distribution
models: master-slave versus peer-to-peer; NoSQL systems to handle big data
problems.

A.I. Kalsekar Technical Campus, New Panvel

 NoSQL Overview
• Non-relational database

• Doesn’t have a pre-defined schema

• Doesn’t store data in tables

• Generally used to store big data and real time data

• Follows CAP theorem(Any 2 at a time)

A.I. Kalsekar Technical Campus, New Panvel

 What is NoSQL?
• NoSQL stands for "Not Only SQL".
• NoSQL is a non-relational database system designed to store and retrieve large volumes
of unstructured, semi-structured, or structured data with high performance and
flexibility.
• NoSQL databases use flexible schemas and support horizontal scaling, making them ideal
for big data, real-time web apps, and distributed systems..
• It avoids joins, and is easy to scale.
• NoSQL databases are designed for:
High scalability
Flexible schema (structure)
Handling big data and real-time web apps
• For example, companies like Twitter, Facebook and Google collect terabytes of user data
every single day.
A.I. Kalsekar Technical Campus, New Panvel
Types of NoSQL Databases:

Type Description Example

Stores data as JSON or
Document MongoDB, CouchDB
BSON documents
Data is stored as key-value
Key-Value Redis, DynamoDB
pairs
Stores data in columns
Column-Family Cassandra, HBase
instead of rows
Stores data as nodes and
Graph Neo4j, ArangoDB
edges (relationships)

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
Why NoSQL?

Feature Relational DB (SQL) NoSQL

Schema Fixed (tables, columns) Flexible (schema-less)
Scalability Vertical (scale-up) Horizontal (scale-out)
Documents, key-value,
Data Storage Format Rows and tables
etc.
Unstructured or semi-
Best For Structured data
structured data
MongoDB, Cassandra,
Examples MySQL, PostgreSQL
Redis

A.I. Kalsekar Technical Campus, New Panvel

Why NoSQL?
• The concept of NoSQL databases became popular with Internet giants like
Google, Facebook, Amazon, etc. who deal with huge volumes of data. The
system response time becomes slow when you use RDBMS for massive volumes
of data.
• To resolve this problem, we could “scale up” our systems by upgrading our
existing hardware. This process is expensive.

• The alternative for this issue is to distribute database load on multiple hosts
whenever the load increases. This method is known as “scaling out.”

A.I. Kalsekar Technical Campus, New Panvel

NoSQL database is non-relational, so it scales out better than relational databases as they
are designed with web applications in mind.
A.I. Kalsekar Technical Campus, New Panvel
NoSQL vs. SQL: What's the difference?
Parameters SQL NoSQL
Types of DB Relational DB Non-Relational DB
Schema Predefined schema Dynamic Schema
DB Category Table based DB 4 types of DB (document, key-
value, wide-column, and
graph)
Scalability Vertically Scalable Horizontally scalable
Language SQL UQL
Online Processing Online transaction processing Online Analytical Processing
Base properties ACID properties Follows CAP(consistency,
availability, partition tolerance)

A.I. Kalsekar Technical Campus, New Panvel

 Brief History of NoSQL Databases
• 1998- Carlo Strozzi use the term NoSQL for his lightweight, open-source relational database

• 2000- Graph database Neo4j is launched

• 2004- Google BigTable is launched

• 2005- CouchDB is launched

• 2007- The research paper on Amazon Dynamo is released

• 2008- Facebooks open sources the Cassandra project

• 2009- The term NoSQL was reintroduced

A.I. Kalsekar Technical Campus, New Panvel

 When should NoSQL be used:
• When a huge amount of data needs to be stored and retrieved.

• The relationship between the data you store is not that important

• The data changes over time and is not structured.

• Support of Constraints and Joins is not required at the database level

• The data is growing continuously and you need to scale the database regularly to handle
the data means requirements for scale-out architecture

• Storage of structured ,semi-structured and unstructured data.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
What is the CAP Theorem?
• CAP theorem is also called brewer’s theorem.
• CAP theorem states that in networked shared-data systems or distributed systems, we
can only achieve at most two out of three guarantees for a
database: Consistency, Availability and Partition Tolerance.
• A distributed system is a network that stores data on more than one node (physical or
virtual machines) at the same time.
1. Consistency
2. Availability
3. Partition Tolerance

A.I. Kalsekar Technical Campus, New Panvel

Consistency:
The data should remain consistent even after the execution of an operation.
For example, after updating the order status, all the clients should be able to see the
same data.

Consistency: means that all clients see the same data at the same time, no matter
which node they connect to in a distributed system. To achieve consistency, whenever
data is written to one node, it must be instantly forwarded or replicated to all the other
nodes in the system before the write is deemed successful.

A.I. Kalsekar Technical Campus, New Panvel

Availability:
The database should always be available and responsive. It should not have any
downtime.

Availability: means that every non-failing node returns a response for all read and write requests in
a reasonable amount of time, even if one or more nodes are down.

Another way to state this — all working nodes in the distributed system return a valid response for
any request, without failing or exception.

A.I. Kalsekar Technical Campus, New Panvel

Partition Tolerance:
Partition Tolerance means that the system should continue to function even if the
communication among the servers is not stable. For example, the servers can be
partitioned into multiple groups which may not communicate with each other. Here, if
part of the database is unavailable, other parts are always unaffected.

Partition Tolerance: means that the system continues to operate despite arbitrary
message loss or failure of part of the system. In other words, even if there is a network
outage in the data center and some of the computers are unreachable, still the system
continues to perform.

A.I. Kalsekar Technical Campus, New Panvel

The CAP theorem categorizes systems into three categories:

A.I. Kalsekar Technical Campus, New Panvel

CP (Consistent and Partition Tolerant)
database
• CP (Consistent and Partition Tolerant) database: A CP database delivers consistency
and partition tolerance at the expense of availability. When a partition occurs between any
two nodes, the system has to shut down the non-consistent node (i.e., make it unavailable)
until the partition is resolved.
• Partition refers to a communication break between nodes within a distributed system.
Meaning, if a node cannot receive any messages from another node in the system, there is
a partition between the two nodes. Partition could have been because of network failure,
server crash, or any other reason.

A.I. Kalsekar Technical Campus, New Panvel

AP (Available and Partition Tolerant)
database
• AP (Available and Partition Tolerant) database: An AP database delivers availability
and partition tolerance at the expense of consistency. When a partition occurs, all nodes
remain available but those at the wrong end of a partition might return an older version of
data than others. When the partition is resolved, the AP databases typically resync the
nodes to repair all inconsistencies in the system.

A.I. Kalsekar Technical Campus, New Panvel

CA (Consistent and Available) database
• CA (Consistent and Available) database: A CA delivers consistency and
availability in the absence of any network partition. Often a single node’s DB
servers are categorized as CA systems. Single node DB servers do not need to deal
with partition tolerance and are thus considered CA systems.

• System designers must take into consideration the CAP theorem while designing
or choosing distributed storages as one needs to be sacrificed from C and A for
others.

A.I. Kalsekar Technical Campus, New Panvel

Some Example
• Blog website (A)
• Multiplayer Online games (A)
• Stock trading Platforms (C)
• Video streaming sites (A)
• Ticket booking system (C)
• Video Chat Applications (A)
• Bank (C)

A.I. Kalsekar Technical Campus, New Panvel

NoSQL Business Drivers

A.I. Kalsekar Technical Campus, New Panvel

 NoSQL Business Drivers

• Need to scale up
• Need for faster processing • Quick data generation
• Organizations shifted from Volume • Now a days bursts in web
serial to parallel processing Velocity
traffic
• Slow response time
Business • Costly to deal
Drivers

Agility Variability
• Diverse data types
• Quick and easy access and feeding • Change in meaning or
of data shape of data
• RDBMS finds it difficult if data • Availability
contains nests or many subgroups

A.I. Kalsekar Technical Campus, New Panvel

 NoSQL Business Drivers contd…

• In this figure, we see how the business drivers volume, velocity, variability, and
agility apply pressure to the single CPU system, resulting in the cracks.

• Volume and velocity refer to the ability to handle large datasets that arrive
quickly.

• Variability refers to how diverse data types don’t fit into structured tables,

• and agility refers to how quickly an organization responds to business change.

A.I. Kalsekar Technical Campus, New Panvel

 Types of NoSQL databases
• NoSQL Databases are mainly categorized into four types. Every
category has its unique attributes and limitations. None of the below-
specified database is better to solve all the problems. Users should select
the database based on their product needs.
• Four major types of NoSQL databases are:
1. document databases
2. key-value databases,
3. wide-column stores
4. graph databases.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
 NoSQL Data Architecture Patterns
• Architecture Pattern is a logical way of categorizing data that will be stored on the
Database. NoSQL is a type of database which helps to perform operations on big data
and store it in a valid format. It is widely used because of its flexibility and a wide
variety of services.

• Architecture Patterns of NoSQL: The data is stored in NoSQL in any of the following
four data architecture patterns.
1. Key-Value Store Database
2. Column Store Database
3. Document Database
4. Graph Database

A.I. Kalsekar Technical Campus, New Panvel

 Key-Value Store Database

• This model is one of the most basic models of NoSQL databases.

• The data is stored in form of Key-Value Pairs.
• The key is usually a sequence of strings, integers, characters or any other data types.
• The value is linked or co-related to the key.
• The key-value pair store data as a hash table where each key is unique. And value can be
of any type (JSON, BLOB(Binary Large Object), strings, etc).
• This type of pattern is usually used in shopping websites or e-commerce applications.

A.I. Kalsekar Technical Campus, New Panvel

Key -Value Pair Based
• Data is stored in key/value pairs. It is designed in such a way to handle
lots of data and heavy load.

Redis, Dynamo, Riak are some NoSQL examples of key-value store Databases. They are all based on
Amazon’s Dynamo paper.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
Key-Value Store Database contd..
Advantages:
• Can handle large amounts of data and heavy load,
• Easy retrieval of data by keys.
Limitations:
• Complex queries may attempt to involve multiple key-value pairs
which may delay performance.
• Data can be involving many-to-many relationships which may
collide.
Examples:
• DynamoDB
• Berkeley DB

A.I. Kalsekar Technical Campus, New Panvel

Column Store Database:
• Rather than storing data in relational tuples, the data is stored in individual
cells which are further grouped into columns.
• Column-oriented databases work only on columns. They store large amounts
of data into columns together.
• Format and titles of the columns can diverge from one row to other.
• Every column is treated separately.
• each individual column may contain multiple other columns like traditional
databases.
• Basically, columns are mode of storage in this type.

A.I. Kalsekar Technical Campus, New Panvel

Column-based
• Column-oriented databases work on columns and are based on BigTable paper by
Google.

➢ Column-based NoSQL databases are widely used to manage data warehouses, business
intelligence, CRM, Library card catalogs,
➢ HBase, Cassandra, Hypertable are examples of NOSQL column based database.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
Column Store Database contd…
• Advantages:
• Data is readily available
• Queries like SUM, AVERAGE, COUNT can be easily performed on columns.
• Examples:
• HBase
• Bigtable by Google
• Cassandra

A.I. Kalsekar Technical Campus, New Panvel

 Document Database
• The document database store data in the form of key-value pairs but here, the
values are called as Documents.
• Document can be any complex data structure.
• Document can be a form of text, arrays, strings, JSON, XML or any such format.
• Documents can be nested too.
• It is very effective as most of the data created is usually in form of JSONs and is
unstructured.
• The document type is mostly used for blogging platforms, real-time analytics
& e-commerce applications.
• It should not use for complex transactions which require multiple operations or
queries against varying aggregate structures.
• Amazon SimpleDB, CouchDB, MongoDB, Lotus Notes are popular
Document originated DBMS systems.

A.I. Kalsekar Technical Campus, New Panvel

Document-Oriented contd….
• Document-Oriented NoSQL DB stores and retrieves data as a key
value pair but the value part is stored as a document.
• The document is stored in JSON or XML formats. The value is
understood by the DB and can be queried.

Relational Vs. Document

A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
 Document Database contd…
Advantages:
• This type of format is very useful for semi-structured data.
• Storage retrieval and managing of documents is easy.
Limitations:
• Handling multiple documents is challenging
• Aggregation operations may not work accurately.
Examples:
• MongoDB
• CouchDB

A.I. Kalsekar Technical Campus, New Panvel

Graph Databases
• This architecture pattern deals with the storage and management of data in graphs.

• Graphs are basically structures that depict connections between two or more objects in some data.

• The objects or entities are called as nodes and are joined together by relationships called
Edges. Each edge has a unique identifier. Each node serves as a point of contact for the graph.

• This pattern is very commonly used in social networks where there are a large number of entities
and each entity has one or many characteristics which are connected by edges.

• The relational database pattern has tables that are loosely connected, whereas graphs are often
very strong and rigid in nature.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
 Graph-Based contd….
• A graph type database stores entities as well the relations amongst those entities. The entity is stored as
a node with the relationship as edges.
• An edge gives a relationship between nodes. Every node and edge has a unique identifier.

A.I. Kalsekar Technical Campus, New Panvel

Graph Databases contd..
• Advantages:
• Fastest traversal because of direct connections.
• Spatial data can be easily handled.
• Limitations:
Wrong connections may lead to infinite loops.
• Examples:
• Graph base database mostly used for social networks, logistics, spatial data.
• Neo4J,
• Infinite Graph,
• OrientDB,
• FlockDB( Used by Twitter)

A.I. Kalsekar Technical Campus, New Panvel

Example

Figure – Graph model format of NoSQL Databases

A.I. Kalsekar Technical Campus, New Panvel

NoSQL solution for big data

• A big data class problem is any business problem that’s so large that it can’t be
easily managed using a single processor.
• Big data problems force you to move away from a single-processor environment
toward the more complex world of distributed computing.
• Though great for solving big data problems, distributed computing environments
come with their own set of challenges .

A.I. Kalsekar Technical Campus, New Panvel

These are challenges for one or many databases.

A.I. Kalsekar Technical Campus, New Panvel

 Here are some typical big data use cases
• Bulk image processing
• Public web page data
• Remote sensor data
• Event log data
• Mobile phone data
• Social media data
• Game data
Big data problems like event log data and game data do need to store their data
directly into structures that can be queried and analyzed, so they will need
different NoSQL solutions.

A.I. Kalsekar Technical Campus, New Panvel

Understanding the types of big data problems

• There are many types of big data problems, each requiring a different combination
of NoSQL systems. After you’ve categorized your data and determined its type,
you’ll find there are different solutions. How you build your own big data
classification system might be different from this example, but the process of
differentiating data types should be similar.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
 Read-mostly
• Read-mostly data is the most common classification. It includes data that’s created
once and rarely altered.
• This type of data is typically found in data warehouse applications but is also
identified as a set of non-RDBMS items like images or video, event-logging data,
published documents, or graph data.
• Event data includes things like retail sales events, hits on a website, system logging
data, or real-time sensor data.

A.I. Kalsekar Technical Campus, New Panvel

 Log events
• When operational events occur in your enterprise, you can record it in a log file and include a
timestamp so you know when the event occurred. Log events may be a web page click or an out-of-
memory warning on a disk drive. In the past, the cost and amount of event data produced were so
large that many organizations opted not to gather or analyze it.

• Today, NoSQL systems are changing companies’ thoughts on the value of log data as the cost to
store and analyze it is more affordable. The ability to cost-effectively gather and store log events
from all computers in your enterprise has lead to BI operational intelligence systems.

• Operational intelligence goes beyond analyzing trends in your web traffic or retail transactions. It
can integrate information from network monitoring systems so you can detect problems before they
impact your customers. Cost-effective NoSQL systems can be part of good operations management
solutions

A.I. Kalsekar Technical Campus, New Panvel

 Full-text documents
• This category of data includes any document that contains natural-language text like
the English language. An important aspect of document stores is that you can query the
entire contents of your office document in the same way you would query rows in your
SQL system.
• This means that you can create new reports that combine traditional data in RDBMSs
as well as the data within your office documents. For example, you could create a
single query that extracted all the authors of titles of PowerPoint slides that contained
the keywords NoSQL or big data. The result of this list of authors could then be filtered
with a list of titles in the HR database to show which people had the title of Data
Architect or Solution Architect.
• This is a good example of how organizations are trying to tap into the hidden skills
that already exist within an organization for training and mentorship. Integrating
documents into what can be queried is opening new doors in knowledge management
and efficient staff utilization.

A.I. Kalsekar Technical Campus, New Panvel

 Analyzing big data with a shared-nothing
architecture
There are three ways that resources can be shared between computer systems:
• Shared RAM,
• Shared disk,
• and Shared-nothing.
Of the three alternatives, a shared-nothing architecture is most cost effective in
terms of cost per processor when you’re using commodity hardware.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
Three ways to share resources.

The left panel shows a shared RAM architecture, where many CPUs access a single
shared RAM over a high-speed bus. This system is ideal for large graph traversal.

The middle panel shows a shared disk system, where processors have independent
RAM but share disk using a storage area network (SAN).

The right panel shows an architecture used in big data solutions: cache-friendly,
using low-cost commodity hardware, and a shared-nothing architecture

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
A.I. Kalsekar Technical Campus, New Panvel
• Of the architectural data patterns row store, key-value store, graph store, document store, and
Bigtable store, only two (key-value store and document store) lend themselves to cache-friendliness.
• Bigtable stores scale well on shared-nothing architectures because their row-column identifiers are
similar to key-value stores.
• But row stores and graph stores aren’t cache-friendly since they don’t allow a large BLOB to be
referenced by a short key that can be stored in the cache.
• For graph traversals to be fast, the entire graph should be in main memory. This is why graph stores
work most efficiently when you have enough RAM to hold the graph. If you can’t keep your graph
in RAM, graph stores will try to swap the data to disk, which will decrease graph query
performance by a factor of 1,000. The only way to combat the problem is to move to a shared-
memory architecture, where multiple threads all access a large RAM structure without the graph
data moving outside of the shared RAM.
• The rule of thumb is if you have over a terabyte of highly connected graph data and you need real-
time analysis of this graph, you should be looking for an alternative to a shared-nothing architecture.
• A single CPU with 64 GB of RAM won’t be sufficient to hold your graph in RAM. Even if you
work hard to only load the necessary data elements into RAM, your links may traverse other nodes
that need to be swapped in from disk. This will make your graph queries slow.

A.I. Kalsekar Technical Campus, New Panvel

Choosing distribution models: master-slave
versus peer-to-peer
• From a distribution perspective, there are two main models:
• master-slave and peer-to-peer.
• Distribution models determine the responsibility for processing data when a
request is made.
• Understanding the pros and cons of each distribution model is important when
you’re looking at a potential big data solution. Peer-to-peer models may be more
resilient to failure than master-slave models. Some master-slave distribution
models have single points of failure that might impact your system availability, so
you might need to take special care when configuring these systems.

A.I. Kalsekar Technical Campus, New Panvel

A.I. Kalsekar Technical Campus, New Panvel
• In the master-slave model, one node is in charge (master). When there’s no single node with a
special role in taking charge, you have a peer-to-peer distribution model.

• The panel on the left illustrates a master-slave configuration where all incoming database requests
(reads or writes) are sent to a single master node and redistributed from there. The master node is
called the NameNode in Hadoop. This node keeps a database of all the other nodes in the cluster and
the rules for distributing requests to each node.

• The panel on the right shows how the peer-to-peer model stores all the information about the cluster
on each node in the cluster. If any node crashes, the other nodes can take over and processing can
continue.

A.I. Kalsekar Technical Campus, New Panvel

 Master-slave model
• With a master-slave distribution model, the role of managing the cluster is done on a
single master node.
• This node can run on specialized hardware such as RAID drives to lower the
probability that it crashes.
• The cluster can also be configured with a standby master that’s continually updated
from the master node.
• The challenge with this option is that it’s difficult to test the standby master without
jeopardizing the health of the cluster. Failure of the standby master to take over
from the master node is a real concern for high-availability operations.

A.I. Kalsekar Technical Campus, New Panvel

 Peer-to-peer model
• Peer-to-peer systems distribute the responsibility of the master to each node in the
cluster.
• In this situation, testing is much easier since you can remove any node in the cluster
and the other nodes will continue to function.
• The disadvantage of peer-to-peer networks is that there’s an increased complexity
and communication overhead that must occur for all nodes to be kept up to date with
the cluster status.

A.I. Kalsekar Technical Campus, New Panvel

 Four ways that NoSQL systems handle big data problems

[Link] queries to the data, not data to the queries

[Link] hash rings to evenly distribute data on a cluster
[Link] replication to scale reads
[Link] the database distribute queries evenly to data nodes

A.I. Kalsekar Technical Campus, New Panvel

Moving queries to the data, not data to the
queries

• With the exception of large graph databases, most NoSQL systems use commodity
processors that each hold a subset of the data on their local shared-nothing drives.
• When a client wants to send a general query to all nodes that hold data, it’s more
efficient to send the query to each node than it is to transfer large datasets to a
central processor.
• This may seem obvious, but it’s amazing how many traditional databases still
can’t distribute queries and aggregate query results.
• The entire data is kept inside hub/node in document form which means just the
query and result are needed to move over the network, thus keeping big data’s
queries quick.

A.I. Kalsekar Technical Campus, New Panvel

Using hash rings to evenly distribute data on a
cluster

A.I. Kalsekar Technical Campus, New Panvel

Using replication to scale reads

A.I. Kalsekar Technical Campus, New Panvel

Letting the database distribute queries evenly to
data nodes

A.I. Kalsekar Technical Campus, New Panvel

MU PYQ
1. Describe characteristics/ Features of a NoSQL database.
2. What is CAP theorem? How it is applicable to NoSQL systems?
3. Difference b/w RDBMS and NoSQL database.
4. What are the different architectural patterns in NoSQL? Explain graph data
store and column family store patterns with relevant examples.
5. Explain in detail key-value store NoSQL architectural pattern. Identify two
applications that can use this pattern.
6. How NoSQL data architecture patterns varies as you move from a single
processor to multiple processors?
7. Explain NoSQL system to handle big data problems.

A.I. Kalsekar Technical Campus, New Panvel