Module - NoSQL
Module - NoSQL
• Big Data:
• Collect
• Store
• Organize
• Analyze
• Share
• Scalability Data growth outruns
• Data format the ability to manage
it so we need
• Manageability scalable solutions
©RBJ
VESIT
NoSQL Business Drivers
Real Time
Velocity
Speed &
Comple
xity
Volume
RDBMS Variability
Schema
Definition
Agility Complex
queries
©RBJ
VESIT
NoSQL Name
• “SQL” = Traditional Relational DBMS
• Recognition over past decade or so
• Not every data management/analysis problem
is best solved using a traditional relational
DBMS
• “NoSQL” = “No SQL” =
• Not using traditional relational DBMS
• “No SQL” ≠ Don’t use SQL language
• “NoSQL” = ? “Not Only SQL” => NOSQL
A common misconception is that NoSQL databases or
non-relational databases don’t store relationship data
well.
NoSQL databases can store relationship data—they just
store it differently than relational databases do.
©RBJ
VESIT
NoSQL (Not Only SQL) Databases:
Features
• Simple and Easy API (if SQL, then only its very
restricted variant)
©RBJ
VESIT
• Three properties of a
distributed system
CAP Theorem (sharing data)
• Consistency: Guarantees
all storage and their
replicated nodes have the
same data at the same
time.
C
• Availability: Every reads
and writes are guaranteed
P to receive a success or
failure response.
• Partition-tolerance:
A System properties
(consistency and/or
availability) hold even
when network failures
prevent some machines
from communicating with
others (system continue to
operate).
©RBJ
VESIT
EXAMPLE: Distributed System
• You are asked to design a distributed cluster of 4
data nodes. Replication factor is 2 i.e. any data
written in cluster must be written on 2 nodes; so
when one goes down – second can serve the data.
• In distributed system, two things may happen
anytime i.e. node failure (hard disk crash) or
network failure (connection between two nodes go
down)
S1 v v S2
Client
©RBJ
VESIT
WRITE
S1 v v S2 S1 v1 v S2 S1 v1 v S2
Write
v1 Done
Client Client Client
S1 v v S2 S1 v v S2
v
Read
v Client Client
READ
©RBJ
VESIT
WRITE
v v
S1 v v S S1 v S2 S1 1 v S2
1
2
Write
v1 Don
Client Client e Client
A consistent system is one in which all reads are guaranteed to incorporate the
previous writes. In a consistent system, after an update operation is performed
by a writer, it is seen by all the readers.
S1 v v S2 S1 v v S2
1
Read v
Client Client
Inconsistent READ
©RBJ
System(AP)
VESIT
WRITE
v v v1
S1 v v S2 S1 v S2 S1 1 v S2
1
Done
Write
v1 Clie
Client Client
nt
v v v v v v
S1 1 1 S2 S1 1 1 S2 S1 1 1 S2
Done Read v1
©RBJ
VESIT
WRITE
v v
S1 v v S2 S1 v S2 S1 1 v S2
1
Write
v1 Don
Client Client e Client
S1 v S1
v S2 v v S2
1
Re v
Client ad Client
Inconsistent but
available System READ
©RBJ
VESIT
Partition Tolerance
This condition states that the system does not fail, regardless of if
messages are dropped or delayed between nodes in a system
(both nodes are up, but can’t communicate). Partition tolerance
refers to the ability of the system to continue performing its
operations in the event of network partitions. Network partitions
can occur when two (or more) sets of nodes are unable to connect
to each other.
S1 v v S S1 v v S
2 2
v
Read Clie Clie
v nt nt
READ
©RBJ
VESIT
CP [Consistency/Partition
Tolerance] Systems
• In distributed system, at the time of
reading the data, consistency is
determined by a voting kind of
mechanism, where all nodes who have
copy of data mutually agree that they
have “same copy” of requested data.
©RBJ
VESIT
CP [Consistency/Partition
Tolerance] Systems
• Now let’s assume that our requested data is
present in two nodes S1 and S2. Client tries to read
the data; and our CP system is partition tolerant as
well, so an expected network failure occurred and
S2 is detected as DOWN. Now system cannot
determine that S1’s data copy is latest or not; it
may be stale as well. So system decides to send an
ERROR event to client. Here system chose to prefer
data consistency over data availability.
• Similarly, at time of writing the data if replication
factor is 2, then system may reject write requests
until it finds two healthy nodes to write data fully in
consistent manner.
©RBJ
VESIT
AP [Availability/Partition
Tolerance] Systems
• What if in previous scenario, system
instead of sending ERROR (in case S2 is
down); it sends the data received from S1.
• Well client gets the data, but was it latest
data copy stored in system in past?? You
cannot decide.
• System chooses availability over
consistency. These are AP systems.
©RBJ
VESIT
CA [Consistency/Availability]
Systems
• In a distributed environment, we cannot avoid “P” of
CAP.
• So we have to choose between CP or AP systems.
• If we desire to have a consistent and available
system, then we must forget about partition
tolerance and it’s possible only in non-distributed
systems such as oracle and SQL server.
©RBJ
VESIT
In summary:
©RBJ
VESIT
Brewer’s CAP Theorem
C
There is risk of
Network some data becoming
problem unavailable.
may stop the EX: Mongo DB,
system Hbase, BigTable
Ex: RDBMS
A P
Clients may read
inconsistent data, we get
back old data. Ex:
Cassandra©RBJ
VESIT
CAP Theorem : Consistency
2 types of consistency: guarantees all
storage and their replicated nodes have
the same data at the same time.
1. Strong consistency – ACID (Atomicity,
Consistency, Isolation, Durability)
2. Weak consistency – BASE (Basically
Available, Soft-state , Eventual
consistency)
©RBJ
VESIT
BASE Transactions
• Basically Available: There will be a response to any
request. But the response could still be failure to
obtain the requested data or the data may be in an
inconsistent or changing state, much like waiting for
a cheque to clear in your bank account.
Basically Available indicates that the
system does guarantee availability, in terms of the
CAP theorem
• Soft State: The state of the system could change over
time, so even during times without input there may
be changes going on due to eventual consistency,
thus the state of the system is always soft.
• Eventually Consistent : The system will eventually
become consistent once it stops receiving input.
©RBJ
VESIT
BASE Transactions
• A BASE system gives up on consistency.
• Weak consistency meaning stale data is
okay.
• Availability has more priority.
• Best effort
• Approximate answers are okay.
• Aggressive (Optimistic)
• Simpler and Faster
©RBJ
VESIT
NoSQL Database Categories
• Key value stores – They are the simplest NoSQL databases.
Every single item in the database is stored as an attribute,
name or key together with its value.
©RBJ
VESIT
Key Value Pair Based
• This feature (No data type) is
especially useful when the data
type changes or we need to
support two or more data types
for the same attribute.
• Imagine for example a network
of sensors where some of them
returns integer value, other
logical state or enumeration or
even a string. There is no
problem with this in key-value
database.
©RBJ
VESIT
Key Value Pair Based
©RBJ
VESIT
• Key-Value stores can be used as collections,
dictionaries, associative arrays etc.
• Key-Value stores follow the 'Availability' and
'Partition' aspects of CAP theorem.
• Main features: Simplicity, Speed, Scalability.
• There are three operations performed on a
key-value store: put, get, and delete.
- put adds a new key-value pair to the table or
updates a value if this key is already present.
- get returns the value for a given key if it exists.
- delete removes a key and its value from the
table if it exists.
©RBJ
VESIT
Key value type uses hash table with unique key and pointer to a
particular item of data.
Bucket is logical group of keys.
©RBJ
VESIT
EX: Key Value
Pair based
©RBJ
VESIT
EX: Key Value Pair based
©RBJ
VESIT
ID First Name Last name
1 Joseph Faizer
2 Name
Table
3
Ecommerce
[Link] has one of the largest ecommerce operations, shopping all hours
of the day all around world takes place, so the site has to be up 24*7. Developers
can store unlimited amount of data by creating a data base table and saves it at
multiple servers specified at the customer and also replicates at multiple
available zones. The cart data and session data are stored in the key value store
and final completed order is saved in RDBMs. ©RBJ
VESIT
Who uses key value store ?
• Key-value data stores are ideal for storing user
profiles, blog comments, product
recommendations, and session information.
• Twitter uses Redis to deliver your Twitter
timeline
• Pinterest uses Redis to store lists of users,
followers, unfollowers, boards, and more
• Coinbase uses Redis to enforce rate limits and
guarantee correctness of Bitcoin transactions
• Quora uses Memcached to cache results from
slower, persistent databases
©RBJ
VESIT
Main Features of Key Value store
• Simple, Scalable
• Automated Storage scaling
• Distributed Horizontally
• Built in fault Tolerance
• Flexible
• Efficient Indexing
• Secure
• Resource Consumption monitoring
• No queries on values: No queries can be
performed on the values.
• As volume of data increases it will be difficult to
maintain unique value as keys
• Can’t model more complex data structure such as
objects
©RBJ
VESIT
When not to use : Key value store
• Relationships among data
• Multi-key Transactions (Want to know
information with reference to two keys)
• Query by Value
• Operations by Sets: since operations are limited to
one key at a time, there is no way to operate upon
value/multiple keys at the same time.
©RBJ
VESIT
▪ Every time you look something up in a row-oriented
database, every row is scanned, regardless of which columns
you require.
▪ Let’s say you only want a list of birthdays in September. The
database will scan the table from top to bottom and left to right,
as shown in figure, eventually returning the list of
birthdays.(Too much time !).
Id Dept Id Name DoJ DoB Hobbies
1 1 AAA 11-8-2011 11/8/1980 Playing
2 2 BBB 4-11-1999 9/5/1971 Singing
3 1 CCC 23-8-1997 30/5/1971 Cooking & Music
4 3 DDD . . -
5 3 EEE . . -
• Column database
• Column family database
• Column oriented database
• Wide column store database
• Wide column store
• Columnar database
• Columnar store
©RBJ
VESIT
Some videos for Column Types of
data bases
• [Link]
VHQ
• [Link]
3c470b6
• [Link]
d02f99c
• [Link]
bM
©RBJ
VESIT
©RBJ
VESIT
Column-based
• Column-Oriented databases are most similar to traditional
row based databases since both are usually relational.
• Column-oriented databases work on columns, simply store
data in columns instead of rows and are based on BigTable
paper by Google.
• Every column is treated separately. Values of single column
databases are stored contiguously.
• This allows for much faster querying and processing of data
(billions of records per second, searching by column is faster)
• They deliver high performance on aggregation queries like
SUM, COUNT, AVG, MIN etc. as the data is readily available
in a column. However inserts and updates are generally
slower in column databases.
©RBJ
VESIT
Every entity (person) is divided over multiple tables.
Every column is treated separately.
Values of single column (Ex: Name ) databases are stored contiguously with
their Ids..
Id Dept Id Name DoJ DoB Hobbies
1 1 AAA 11-8-2011 11/8/1980 Playing
2 2 BBB 4-11-1999 9/5/1971 Singing
3 1 CCC 23-8-1997 30/5/1971 Cooking & Music
4 3 DDD . . -
5 3 EEE . . -
Id 1 2 3 4 5
Dept Id 1 2 1 3 3
Id 1 2 3 4 5
Name AAA BBB CCC DDD EEE
Id 1 2 3 4 5
DoJ 11-8-2011 4-11-1999 23-8-1997
Id 1 2 3 4 5
DoB 11/8/1980 9/5/1971 30/5/1971
Id 1 2 3 4 5
Hobbies Playing Singing Cooking & Music
©RBJ
VESIT
In practice, columnar databases often store data in column families. In this
case you would probably not store the street, city, zip in completely
separate columns, but instead in a column family where they are stored
together.
©RBJ
VESIT
• Fast Read Operations : While a relational database
stores data in rows and reads data row by row, a
column store is organized as a set of columns. This
means that when you want to run analytics on a small
number of columns, you can read those columns
directly without consuming memory with the
unwanted data. Columns are often of the same type
and benefit from more efficient compression, making
reads even faster.
©RBJ
VESIT
• Reporting : Column store databases are generally
used when reporting is more important than data
modification.
• Column-based NoSQL databases are widely used to
manage data warehouses, business intelligence,
CRM, Library card catalogs (where speed is critical).
• Unfortunately there is no free lunch, which means
that while columnar databases are great for
analytics, the way in which they write data makes it
very difficult for them to be strongly consistent as
writes of all the columns require multiple write
events on disk. Relational databases don't suffer
from this problem as row data is written
contiguously to disk.
©RBJ
VESIT
Column type of data bases are usually much
slower for write operations, so they tend to be
used in OLAP while row based is used in OLTP
situations. Otherwise columnar databases are
very similar to traditional databases.
[Link]
M
Check how read & write operations are
performed.
©RBJ
VESIT
When to use Column data base ?
• When should you use a row-oriented database
and when should you use a column-oriented
database?
• In a column-oriented database it’s easy to add
another column because none of the existing
columns are affected by it. But adding an entire
record requires adapting all tables.
• This makes the row-oriented database preferable
over the column-oriented database for online
transaction processing (OLTP) because this
implies adding or changing records constantly.
©RBJ
VESIT
Examples of Column Databases
HBase,
Cassandra,
Hypertable,
Druid,
Google’s big table,
Vertica,
Accumlo
©RBJ
VESIT
Use Cases
• Column stores offer very high performance and a highly
scalable architecture. Because they’re fast to load and
query, they’ve been popular among companies and
organizations dealing with big data, IoT, and user
recommendation and personalization engines.
• Spotify uses Cassandra to store user profile attributes and
metadata about artists, songs, etc. for better
personalization
• Facebook initially built its revamped Messages on top of
HBase, but is now also used for other Facebook services
like the Nearby Friends feature and search indexing
• Outbrain uses Cassandra to serve over 190 billion
personalized content recommendations each month
©RBJ
VESIT
Primary key-foreign key relationship
It defines a one-to-many relationship between two tables in
a relational database.
A foreign key is a column or a set of columns in one table
that references the primary key columns in another table.
The primary key is defined as a column (or set of columns)
where each value is unique and identifies a single row of
the table.
©RBJ
VESIT
Primary key-foreign key relationships
• Primary key is used to
identify data uniquely
therefore two rows can’t
have the same primary
key. It can’t be null.
• On the other hand,
foreign key is used to
maintain relationship
between two tables.
• Primary of a table act as
forgein key in the other
table. Foreign key in a
table enforce Referential
Integrity constraint. It
can be more than one in
the table.
©RBJ
VESIT
Benefits of Column Stores
• Compression. Column stores are very efficient at data
compression and/or partitioning.
• Aggregation queries. Due to their structure, columnar
databases perform particularly well with aggregation
queries (such as SUM, COUNT, AVG, etc).
• Scalability. Columnar databases are very scalable. They
are well suited to massively parallel processing (MPP),
which involves having data spread across a large cluster
of machines – often thousands of machines.
• Fast to load and query. Columnar stores can be loaded
extremely fast. A billion row table could be loaded within
a few seconds. You can start querying and analysing
almost immediately.
©RBJ
VESIT
Document-Oriented:
• Document-Oriented NoSQL DB stores and retrieves
data as a key value pair but the value part is stored as a
document. The value can be any sort of value, array, or
even another document.
• The document is stored in JSON or XML formats.
• They're typically schemaless, so each document can
contain any data that you want them to have and you
can change it on the fly.
• The value is understood by the DB and can be queried.
• The document type is mostly used for Content
Management System, blogging platforms, real-time
analytics & e-commerce applications.
©RBJ
VESIT
In this diagram on your left you can see we have rows and
columns, and in the right, we have a document database
which has a similar structure to JSON.
©RBJ
VESIT
• It should not use for complex transactions which
require multiple operations or queries against
varying aggregate structures.
• Amazon SimpleDB, CouchDB, MongoDB, Riak,
Lotus Notes, MongoDB, are popular Document
originated DBMS systems.
• Key-Value and Document databases are very
similar to each other. The only real difference is
document databases store their values in a more
structured way.
• A key-value database may simply store blobs of
data for each key with little knowledge of what is in
each blob.
• A document database is still basically a key-value
store, but it knows information about the format of
each value. They may be XML, JSON, RDF, etc.
This allows the database to more easily query
information found in each document.
©RBJ
VESIT
• Query Mechanism tools for NoSQL
• The most common data retrieval mechanism is
the REST-based retrieval of a value based on
its key/ID with GET resource
• Document store Database offers more difficult
queries as they understand the value in a key-
value pair.
• For example, CouchDB allows defining views
with MapReduce
©RBJ
VESIT
Videos on Document type data
base
[Link]
7aa995
⚫ [Link]
0ec27536
⚫ [Link]
⚫ [Link]
©RBJ
VESIT
Graph-Based
• A graph database stores data in a graph.
• It is capable of elegantly representing any
kind of data in a highly accessible way.
• A graph type database stores entities as
well the relations amongst those entities.
• The entity is stored as a node such as a
student or business with the relationship
as edges. An edge gives a relationship
between nodes. Every node and edge has a
unique identifier.
©RBJ
VESIT
Graph-Based
• An edge always has a start node, end
node, type, and direction, and an edge
can describe parent-child relationships,
actions, ownership, and the like.
• Each node knows its adjacent nodes.
• There is no limit to the number and kind
of relationships a node can have.
• Compared to a relational database where
tables are loosely connected, a Graph
database is a multi-relational in nature.
©RBJ
VESIT
Graph-Based
• Traversing relationship is fast as they
are already captured into the DB, and
there is no need to calculate them.
• A graph in a graph database can be
traversed along specific edge types or
across the entire graph.
• In graph databases, traversing the joins
or relationships is very fast because the
relationships between nodes are not
calculated at query times but are
persisted in the database.
©RBJ
VESIT
Graph-Based
• Graph databases have advantages for use
cases such as social networking, logistics,
spatial data, recommendation engines, and
fraud detection, when you need to create
relationships between data and quickly
query these relationships.
• Neo4J, Infinite Graph, OrientDB, FlockDB
are some popular graph-based databases.
• As the number of nodes increases, the cost
of a local step (or hop) remains the same.
©RBJ
VESIT
Comparison between the classic
relational model and the graph model :
Relational model Graph Model
Rows Vertices
Join Edges
©RBJ
VESIT
• The graph shows an
example of a social
network graph.
• Given the people
(nodes) and their
relationships (edges),
• You can find out who
the "friends of
friends" of a
particular person
are—for example, the
friends of Howard's
friends.
©RBJ
VESIT
Bob’s information is stored in the form of graph data
base, like address, recent order, follower, reviews
posted by him and many more..
©RBJ
VESIT
Driver Car Relationship using graph data base: 6
relationships can be obtained
©RBJ
VESIT
How information is indexed
? In a triple store, the first
field in the database holds
the URI for the subject, the
second field holds the URI
for the predicate and the
third field holds a URI for
the object.
Rows
s p o
1 Bob Married to Julie
2 Bob Listens to Rock
While there are a number of music
different strategies that graph 3 Bob drives BMW
databases may use for storing
4 Bob Works for IBM
triples, most use an index that
abbreviates the three primary 5 Bob Brother of Steve
fields to 6 Julie Sisterinlaw Steve
{s, p, o}. to
7 Julie Listens to Rock
©RBJ Music
VESIT
How information in a graph
database is queried ?
• Each triple in a graph database only gets
stored once in the index. Just like
relational databases, it's a simple process
to do a straight lookup query in a graph
database.
• If the query is for what information is
known about Bob, the indexer
programming only needs to search rows 1-
5 of the database.
• The real power and speed of a graph
database comes from indexing
combinations of triples.
©RBJ
VESIT
Examples
⚫ If the query is for who Bob is married to, the
indexer will look for the predicate : married To
in rows 1-3 and then retrieve the matching
object. (Bob is married to Julie.)
⚫ If the query is to identify everyone who listens
to the same kind of music as Bob, the indexer
will first ask { :Bob :listensTo ?o } and identify
:RockMusic as the object.
⚫ In the second query, the results will return
:RockMusic in rows 2 and 7. The subject in row
2 is Bob himself, so whoever is the subject in
row 7 will be the other person who listens to
rock music. (It turns out to be Julie, Bob's wife.)
©RBJ
VESIT
Use case 1
Fraud detection
• Graph databases are capable of sophisticated
fraud prevention. With graph databases, you can
use relationships to process financial and
purchase transactions in near-real time.
• With fast graph queries, you are able to detect
that, for example, a potential purchaser is using
the same email address and credit card as
included in a known fraud case.
• Graph databases can also help you easily detect
relationship patterns such as multiple people
associated with a personal email address, or
multiple people sharing the same IP address but
residing in different physical addresses.
©RBJ
VESIT
Use Case 2
Recommendation engines
• Graph databases are a good choice for
recommendation applications.
• With graph databases, you can store in a graph
relationships between information categories such
as customer interests, friends, and purchase history.
• You can use a highly available graph database to
make product recommendations to a user based on
which products are purchased by others who follow
the same sport and have similar purchase history.
• Or, you can identify people who have a friend in
common but don’t yet know each other, and then
make a friendship recommendation.
©RBJ
VESIT
Blogs/ video on Graph store data
bases
⚫ [Link]
databases-are-the-future/
⚫ [Link]
databases-whats-the-big-deal-ec310b1bc0ed
⚫ [Link]
ction-to-graph-databases/
⚫ [Link]
AMfdg&feature=emb_logo
©RBJ
VESIT
NoSQL Database Categories
©RBJ
VESIT
Examples of NOSQL Data Stores
Key Value Store
• Redis, Amazon Dynamo DB, Azure Table
Storage (ATS)
• Riak, Memcached, Berkeley DB
Column Based Store
• Cassandra, Google Big table, Hbase,
Accumlo, Hyper Table,
Document Store
• Mongo DB, Couch Base, Couch DB,
Orient DB
Graph Store
• Neo4j, AllegroGraph, Teradata Aster,
FlockDB ©RBJ
VESIT
When to use which store ?
• Key Value Store : When processing a constant stream of
small reads and writes, session information, user profiles,
preferences, shopping cart data
• Document Types : For natural data modeling, Programmer
friendly, Rapid Development, content management system,
blogging platform, web analytics, real time analytics,
ecommerce applications
• Columnar Type: For massive write loads High availability
Multiple data centers , content management systems,
blogging platforms,
• Graph Type : For Graph Algorithms and many to many
relationships, for connected data such as social networks,
spatial data, routing information for goods and money,
recommendation engines
• Select the best data store for a given data type , Use case
and purpose !
©RBJ
VESIT