Big Data Analytics

Dr. Ranjan Bala Jain

Professor
Dept. of EXTC
Vivekanand Education Society’s Institute of
Technology, Mumbai
Instead of worrying about
what you cannot control,
Shift your energy to
what you can create.

― Roy T. Bennett

©RBJ VESIT
 Module 3
• Introduction to NoSQL
• NoSQL Business Drivers
• NoSQL Case Studies
• Understand the basic of NoSQL
• Four major types of NoSQL Database
• Using NoSQL to manage Big Data

©RBJ VESIT
 • A database as a big
Data Base container wherein all
the information about
a website, or an
application is stored
in a structured format
like tables, hierarchy.

Manage !
For Easy
Access & update !
©RBJ VESIT
 Database Components
The major components of the Database are:
• Hardware : This consists of a set of physical electronic
devices such as I/O devices, storage devices and many
more. It also provides an interface between computers
and real-world systems.
• Software: This is the set of programs that are used to
control and manage the overall Database. It also
includes the DBMS software itself. The Operating
System, the network software being used to share the
data among the users, the application programs used to
access data in the DBMS.
• Data: Database Management System collects, stores,
processes, and accesses data. The Database holds both
the actual or operational data and the metadata.
©RBJ VESIT
 DBMS
• DBMS or Database
Management System is
a software application
used to access, create,
and manage databases.

• A DBMS consists of a
group of commands to
manipulate the
database and acts as
an interface between
the end-users and the Database Management Systems
also aims to facilitate an overview
database.
of the databases
©RBJ VESIT
• Database Management Systems allows users to
do the following:
• Define Data – Allows the users to create, modify
and delete the definitions which define the
organization of the database.
• Update Data – Provides access to the users to
insert, modify and delete data from the database.
• Retrieve Data – Allows the users to retrieve data
from the database based on the requirement.
• Administration of users – Registers the users
and monitors their action, enforces data security,
maintains data integrity, monitors performance
and deals with concurrency control.

©RBJ VESIT
©RBJ VESIT
Centralized Database
• It is the type of database that
stores data at a centralized
database system.
• It comforts the users to access
the stored data from different
locations through several
applications.
• These applications contain the
authentication process to let
users access data securely.
• An example: Central Library •Not easy to update,
that carries a central database •Response time for
of each library in a fetching,
college/university. •Data loss in case of
server failure !
©RBJ VESIT
Distributed Database
• Data is distributed among different
database systems of an organization.
• These database systems are
connected via communication links.
Such links help the end-users to
access the data easily.
⚫ Modular development is possible in a
distributed database, i.e., the system
can be expanded by including new
computers and connecting them to
the distributed system.
⚫ One server failure will not affect the
Operating system
entire data set. Application process
⚫ Examples Apache Cassandra, Hardware devices
HBase, Ignite, etc.
©RBJ VESIT
 Relational Database
• This database is based on the relational data
model, which stores data in the form of rows
(tuple) and columns (attributes), and together
forms a table(relation).
• A relational database uses SQL for storing,
manipulating, as well as maintaining the data.
E.F. Codd invented the database in 1970.
• Each table in the database carries a key that
makes the data unique from others.
• Examples of Relational databases are MySQL,
Microsoft SQL Server, Oracle, IBM DB2 etc.

©RBJ VESIT
KNOWING IS NOT ENOUGH;
WE MUST APPLY.
WISHING IS NOT ENOUGH;
WE MUST DO.”
– JOHANN WOLFGANG VON GOETHE

©RBJ VESIT
Properties of Relational Database

©RBJ VESIT
Properties of Relational Database
ACID
• A means Atomicity: This ensures the data
operation will complete either with success or
with failure. It follows the 'all or nothing'
strategy. For example, a transaction will either
be committed or will abort.
• C means Consistency: If we perform any
operation over the data, its value before and
after the operation should be preserved. For
example, the account balance before and after
the transaction should be correct, i.e., it should
remain conserved.
©RBJ VESIT
Properties of Relational Database
• I means Isolation: There can be concurrent
users for accessing data at the same time from
the database. Thus, isolation between the data
should remain isolated. For example, when
multiple transactions occur at the same time,
one transaction effects should not be visible to
the other transactions in the database.
• D means Durability: It ensures that once it
completes the operation and commits the
data, data changes should remain permanent.

©RBJ VESIT
 Cloud Database
• A type of database
where data is stored in a
virtual environment and
executes over the cloud
computing platform.
• This data is accessible
over the Internet.
• Cloud databases are also
called a database as
service (DBaaS)

©RBJ VESIT
 Cloud Database
• It provides users with
various cloud computing
services (SaaS, PaaS,
IaaS, etc.) for accessing
the database.
• There are numerous cloud
platforms, but the best
options are: Amazon Web
Services(AWS), Microsoft
Azure, Kamatera,
PhonixNAP, ScienceSoft,
Oracle database cloud
services, Google Cloud
SQL, etc.
©RBJ VESIT
 Object-oriented Databases
• The type of database that uses the object-
based data model approach for storing data in
the database system.
• The data is represented and stored as objects
which are similar to the objects used in the
object-oriented programming language.
• Objects are the real-world entity, and types are
the collection of objects.
• It is an alternative implementation to that of the
relational model.
• properties: Objects, Classes, Inheritance,
Polymorphism, Encapsulation
©RBJ VESIT
Hierarchical Databases
• It is the type of database that
stores data in the form of
parent-children relationship
nodes.
• Here, it organizes data in a
tree-like structure.
• Data gets stored in the form
of records that are connected
via links.
• Each child record in the tree
will contain only one parent.
On the other hand, each
parent record can have
multiple child records.

©RBJ VESIT
 Network Databases
• It is the database that typically follows the
network data model.
• Here, the representation of data is in the
form of nodes connected via links
between them.
• Unlike the hierarchical database, it allows
each record to have multiple children and
parent nodes to form a generalized graph
structure.

©RBJ VESIT
• Personal Database: Collecting and storing data on
the user's system defines a Personal Database. This
database is basically designed for a single user.
• Operational Database: The type of database
which creates and updates the database in real-
time. It is basically designed for executing and
handling the daily data operations in several
businesses. For example, An organization uses
operational databases for managing per day
transactions.
• Enterprise Database: Large organizations or
enterprises use this database for managing a
massive amount of data. It helps organizations to
increase and improve their efficiency. Such a
database allows simultaneous access to users.
©RBJ VESIT
Quiz
The property of DBMS which ensures the
execution of all operations in a transaction
or none of the operation is classified as,
⮚ Atomicity
⮚ Consistency
⮚ Isolation
⮚ Durability

Atomicity

©RBJ VESIT
Quiz
In DBMS, the executing process or executing
program which considers the updating or
reading of a records stored in data base is
called as,
⮚ Conceptualization
⮚ Implementation
⮚ Execution
⮚ Transaction
Transaction

©RBJ VESIT
Quiz
Which of the following statements is false ?
⮚ DBMS is a software, manages the interaction
between end users and database.
⮚ HTML isn’t involved in Database Management
System.
⮚ A schema is a logical structure of data.
⮚ A relational database consists of a collection of
Tables. (Fields are the column of the relation or
tables. Records are each row in a relation.
Keys are the constraints in a relation)
⮚ None of the above.

©RBJ VESIT
Quiz
⚫ Isolation of the transactions is
achieved by
⮚ avoiding simultaneous transactions.
⮚ storing updates permanently.
⮚ preventing system failure.
⮚ concurrency control.

©RBJ VESIT
 Quiz
Database transaction durability ensures
that in the event of a system failure
⮚ a transaction is not lost once it has been
committed.
⮚ a transaction is completed uninterrupted.
⮚ a transaction is revered before its is
executed.
⮚ a transaction is saved before the failure
occurs.

©RBJ VESIT
Quiz
If an transaction is performed in a database
and committed, the changes are taken to
the previous state of transaction by
a) Flashback
b) Rollback
c) Both Flashback and Rollback
d) Cannot be done
Cannot be done

©RBJ VESIT
Quiz
If the state of the database no longer reflects
a real state of the world that the database
is supposed to capture, then such a state is
called
a) Consistent state
b) Parallel state
c) Atomic state
d) Inconsistent state
Inconsistent state

©RBJ VESIT
Quiz
⚫ Which of the following has “all-or-none”
property?
a) Atomicity
b) Durability
c) Isolation
d) All of the mentioned

⚫Atomicity

©RBJ VESIT
Quiz
Consider money is transferred from
(1)account-A to account-B and (2) account-
B to account-A. Which of the following form
a transaction?
Only 1
Only 2
Both 1 and 2 individually
Either 1 or 2

Both 1 and 2 individually

©RBJ VESIT
Quiz
__________ states that only valid data will be
written to the database.
a) Consistency
b) Atomicity
c) Durability
d) Isolation
a) Consistency

©RBJ VESIT
SQL
• SQL stands for Structured Query Language.
• It is used for storing and managing data in
relational database management system (RDBMS).
• It is a standard language for Relational Database
System. It enables a user to create, read, update
and delete relational databases and tables.
• All the RDBMS like MySQL, Informix, Oracle, MS
Access and SQL Server use SQL as their standard
database language.
• SQL allows users to query the database in a
number of ways, using English-like statements.

©RBJ VESIT
 Advantages of SQL
High speed: Using the SQL queries, the user can quickly
and efficiently retrieve a large amount of records from
a database.
No coding needed: In the standard SQL, it is very easy
to manage the database system. It doesn't require a
substantial amount of code to manage the database
system.
Portability: SQL can be used in laptop, PCs, server and
even some mobile phones.
Interactive language: SQL is a domain language used to
communicate with the database. It is also used to
receive answers to the complex questions in seconds.
Multiple data view: Using the SQL language, the users
can make different views of the database
©RBJ VESIT structure.
Humongous data storage is not possible with RDBMS, which stores
data in fixed schema and big data is having variety of data.
The system response time becomes slow when you use RDBMS for
massive volumes of data.
©RBJ VESIT
©RBJ VESIT
“Scale up" our systems by Distribute database load on
upgrading our existing multiple hosts whenever the
hardware. This process is load increases. This method is
expensive. known as “Scaling out."

©RBJ VESIT
 Why NoSQL ?
• Big Data:
• Collect
• Store
• Organize
• Analyze
• Share
Data growth outruns
• Scalability
the ability to
• Data format manage it so we
• Manageability need scalable
solutions
©RBJ VESIT
NoSQL Business Drivers
Real Time
Velocity

Speed &
Complex
ity
Volume
RDBMS Variability
Schema
Definition

Agility Complex
queries

©RBJ VESIT
Desirable Features of DBMS
• 24*7 data availability
• Location Transparency
• Schema less data Model
• Modern day Transaction analysis
• Architecture that suits big data
• Analytics & Business intelligence

©RBJ VESIT
Be Thankful for what you have,
You will end up having more…
If you concentrate on what you
don’t have,
You will never ever have
enough…
-Oprah Winfrey

©RBJ VESIT
 NoSQL Name
• “SQL” = Traditional Relational DBMS
• Recognition over past decade or so
• Not every data management/analysis problem is
best solved using a traditional relational DBMS
• “NoSQL” = “No SQL” =
• Not using traditional relational DBMS
• “No SQL” ≠ Don’t use SQL language
• “NoSQL” = ? “Not Only SQL” => NOSQL
A common misconception is that NoSQL databases or
non-relational databases don’t store relationship data
well.
NoSQL databases can store relationship data—they just
store it differently than relational databases do.
©RBJ VESIT
NoSQL (Not Only SQL) Databases:
Features

Mostly open source, Non-relational, Next

Generation data base
Schema Free: Do not require sort of
definition of the schema of the data, no
requirement to design the tables and
pushing the data to it, free from joins
Scaling : Horizontal Scaling, Implemented
on commodity hardware which has
separate RAM and disk
Distributed architecture and works on
multiple processors to give high
performance
Replication support.
©RBJ VESIT
 NoSQL (Not Only SQL) Databases:
Features

• Relax one or more of the ACID

properties(Atomicity, Consistency,
Isolation, Durability) but guarantees
BASE.

• Simple and Easy API (if SQL, then only its

very restricted variant)

• Do not fully support relational

features
• No join operations (except within
partitions),
• No referential integrity constraints across
partitions. ©RBJ VESIT
 • Three properties of a
distributed system
CAP Theorem (sharing data)
• Consistency: Guarantees
all storage and their
replicated nodes have the
same data at the same
time.
C • Availability: Every reads
and writes are guaranteed
P to receive a success or
failure response.
• Partition-tolerance:
A
System properties
(consistency and/or
availability) hold even
when network failures
prevent some machines
from communicating with
others (system continue to
operate).
©RBJ VESIT
EXAMPLE: Distributed System
• You are asked to design a distributed cluster of
4 data nodes. Replication factor is 2 i.e. any
data written in cluster must be written on 2
nodes; so when one goes down – second can
serve the data.
• In distributed system, two things may happen
anytime i.e. node failure (hard disk crash) or
network failure (connection between two
nodes go down) S v v S2
1

Client
©RBJ VESIT
 WRITE

v v1
S1 v v S2 S1 v S2 S1 v S2
1

Write
v1 Done
Client Client Client

S v v S2 S v v S2
1 1
v
Read
v Client Client

READ©RBJ VESIT
 WRITE

v v
S1 v v S S1 v S2 S1 1 v S2
1
2
Write
v1 Don
Client Client e Client

A consistent system is one in which all reads are guaranteed to incorporate the
previous writes. In a consistent system, after an update operation is performed
by a writer, it is seen by all the readers.

S v S
v S2 v v S2
1 1 1

Read v

Client Client

Inconsistent READ©RBJ VESIT

System(AP)
 WRITE

v v v1
S1 v v S2 S1 1 v S2 S1 1 v S2

Done
Write
v1 Clie
Client Client
nt

v v v v v v
S1 1 1 S2 S1 1 1 S2 S1 1 1 S2

Done Read v1

Client Client Client

Guarantees all storage and their

Consistent System replicated nodes have the same data
©RBJ
at the same VESIT
time.
 Available System
• In an available system, if our client sends a
request to a server and the server has not
crashed, then the server must eventually
respond to the client. The server is not allowed
to ignore the client's requests.
• Availability refers to the ability of the system to
respond to all the queries without being
unavailable.
• A distributed data system is called available
when it can continue to perform its operations
even in the event of failure of some of the
nodes.
©RBJ VESIT
 WRITE

v v
S1 v v S2 S1 1 v S2 S1 1 v S2

Write
v1 Don
Client Client e Client

S v S
v S2 v v S2
1 1 1

Re v

Client ad Client

Inconsistent but
available System READ©RBJ VESIT
 Partition Tolerance
This condition states that the system does not fail, regardless
of if messages are dropped or delayed between nodes in a
system (both nodes are up, but can’t communicate).
Partition tolerance refers to the ability of the system to
continue performing its operations in the event of network
partitions. Network partitions can occur when two (or more)
sets of nodes are unable to connect to each other.

S v v S S v v S
1 2 1 2
v
Read Clie Clie
v nt nt

READ©RBJ VESIT
CP [Consistency/Partition Tolerance] Systems

• In distributed system, at the time of

reading the data, consistency is
determined by a voting kind of
mechanism, where all nodes who
have copy of data mutually agree
that they have “same copy” of
requested data.

©RBJ VESIT
 CP [Consistency/Partition
Tolerance] Systems
• Now let’s assume that our requested data is present in
two nodes S1 and S2. Client tries to read the data; and
our CP system is partition tolerant as well, so an
expected network failure occurred and S2 is detected
as DOWN. Now system cannot determine that S1’s
data copy is latest or not; it may be stale as well. So
system decides to send an ERROR event to client.
Here system chose to prefer data consistency over
data availability.
• Similarly, at time of writing the data if replication
factor is 2, then system may reject write requests until
it finds two healthy nodes to write data fully in
consistent manner.
©RBJ VESIT
AP [Availability/Partition Tolerance] Systems

• What if in previous scenario, system

instead of sending ERROR (in case S2
is down); it sends the data received
from S1.
• Well client gets the data, but was it
latest data copy stored in system in
past?? You cannot decide.
• System chooses availability over
consistency. These are AP systems.
©RBJ VESIT
 CA [Consistency/Availability] Systems
• In a distributed environment, we cannot avoid “P” of
CAP.
• So we have to choose between CP or AP systems.
• If we desire to have a consistent and available system,
then we must forget about partition tolerance and it’s
possible only in non-distributed systems such as oracle
and SQL server.

For any system sharing data, it is “impossible” to

guarantee simultaneously all of these three
properties (C,A,P)
You can have at most two of these three properties
for any shared-data system.

©RBJ VESIT
In summary:

The CAP theorem states that the system can

either favor consistency and partition
tolerance over availability, or favor
availability and partition tolerance over
consistency.

©RBJ VESIT
• Let us take the example of NoSQL databases such
as Amazon DynamoDB and Cassandra.
• These databases prefer consistency and partition
tolerance over availability. Such systems are said
to be “eventually consistent” as all the writes are
eventually (not immediately) seen by all the nodes.
• In an eventually consistent system, clients can
experience an inconsistent state of the system
while the updates are being propagated to all the
nodes and the system has not yet reached a
steady state.
• In the event of network partitions, all the nodes
may not have the most recent updates and may
return inconsistent or out-dated information. When
the network partitions are resolved all the nodes
eventually see the updates.
©RBJ VESIT
 In summary:
⚫HBase, in contrast to DynamoDB and
Cassandra, prefers consistency and partition
tolerance over availability. By adopting strong
consistency, HBase ensures that updates are
immediately available to all clients. In the
event of network partitions, the system can
become unavailable to ensure consistency.

©RBJ VESIT
Lets have a quick ReCAP
• Consistency means that data is the same
across the cluster, so you can read or write
from/to any node and get the same data.
• True
• False
• True !!!

©RBJ VESIT
 Quiz
Which of the following is false ?
• Availability means the ability to access the
cluster even if a node in the cluster goes down.
• High Availability means the application will be
available, without interruption.
• Using multiple nodes with clustering is a
common way to achieve higher level of
availability in web applications.
• Available system will always give consistent
results
Available system will always give consistent
results
©RBJ VESIT
Quiz
• Partition tolerance means that the cluster
continues to function even if there is a
"partition" (communication break) between two
nodes (both nodes are up, but can't
communicate).
• If you have both availability (the cluster
doesn't go down) and partition tolerance then
you can't guarantee that all nodes will always
have all the data.
• Both true
• None of the above.
• Both true
©RBJ VESIT
Quiz
• The CAP Theorem states that in the
presence of a network partition, one
has to choose between consistency
and availability.
• True
• False

©RBJ VESIT
Brewer’s CAP Theorem
C

There is risk of some

Network data becoming
problem may unavailable.
stop the EX: Mongo DB,
system Hbase, BigTable
Ex: RDBMS

A P
Clients may read
inconsistent data, we get
back old data. Ex:
Cassandra©RBJ VESIT
©RBJ VESIT
CAP Theorem : Consistency
2 types of consistency: guarantees all
storage and their replicated nodes have
the same data at the same time.
1. Strong consistency – ACID (Atomicity,
Consistency, Isolation, Durability)
2. Weak consistency – BASE (Basically
Available, Soft-state , Eventual
consistency)

©RBJ VESIT
 BASE Transactions
• Basically Available: There will be a response to any
request. But the response could still be failure to
obtain the requested data or the data may be in an
inconsistent or changing state, much like waiting for a
cheque to clear in your bank account.
Basically Available indicates that the
system does guarantee availability, in terms of the
CAP theorem
• Soft State: The state of the system could change over
time, so even during times without input there may
be changes going on due to eventual consistency,
thus the state of the system is always soft.
• Eventually Consistent : The system will eventually
become consistent once it stops receiving input.
©RBJ VESIT
 BASE Transactions
• A BASE system gives up on consistency.
• Weak consistency meaning stale data is
okay.
• Availability has more priority.
• Best effort
• Approximate answers are okay.
• Aggressive (Optimistic)
• Simpler and Faster

©RBJ VESIT
NoSQL Database Categories
• Key value stores – They are the simplest NoSQL databases.
Every single item in the database is stored as an attribute,
name or key together with its value.

• Wide column stores/ Big Tables - Wide column stores such

as Cassandra and HBase are optimized for queries over large
datasets, and store columns of data together, instead of rows.

• Document Database – It pairs each key with a complex data

structure known as document. It can contain many different
key value pairs, or key array pairs or even nested documents

• Graph store – They are used to store information about

networks, such as social connections. Graph stores include
Neo4J and Hyper Graph DB.
©RBJ VESIT
NoSQL Database Categories

Key-value: DynamoDB, Voldermort, Scalaris , Riak, Tokyo cabinet,

Redis server
Column-based: BigTable, Cassandra, Hbase, Hyper Table
Document-based: MongoDB, CouchDB, Orient DB, Raven DB
Graph-based: Neo4J, InfoGrid, Infinite©RBJ
Graph, Flock DB
VESIT
 Key Value Pair Based
• It is one of the most basic types of NoSQL databases, Based
on Amazon’s Dynamo paper.
• It is designed in such a way to handle lots of data and heavy
load.
• A very simple data model which resembles dictionary.
• A key-value store is a pattern for storing data. The pattern is
simply a collection of key-value pairs. Data is stored in
key/value pairs, and store data as a hash table where each
key is unique.
• A key may be strings, hashes, lists, sets, sorted sets and
values are stored against these keys.
• Values in key-value pairs have no type. They are, generally
BLOBs (Binary Large Objects), so we can put everything we
want. It’s up to the application to determine what type of data
is being used, such as an integer, string, JSON, XML file, or
even binary data like image.
©RBJ VESIT
Key Value Pair Based
• This feature (No data type) is
especially useful when the data
type changes or we need to
support two or more data types
for the same attribute.
• Imagine for example a network of
sensors where some of them
returns integer value, other
logical state or enumeration or
even a string. There is no
problem with this in key-value
database.

For example a key-value pair might

consist of a key like "Name" that is
associated with a value like "Robin".

©RBJ VESIT
Key Value Pair
Based

©RBJ VESIT
• Key-Value stores can be used as collections,
dictionaries, associative arrays etc.
• Key-Value stores follow the 'Availability' and
'Partition' aspects of CAP theorem.
• Main features: Simplicity, Speed, Scalability.
• There are three operations performed on a
key-value store: put, get, and delete.
- put adds a new key-value pair to the table or
updates a value if this key is already present.
- get returns the value for a given key if it exists.
- delete removes a key and its value from the
table if it exists.

©RBJ VESIT
Key value type uses hash table with unique key and pointer to a
particular item of data.
Bucket is logical group of keys.
©RBJ VESIT
EX: Key Value Pair
based

©RBJ VESIT
EX: Key Value Pair based

©RBJ VESIT
 ID First Last name
Name
1 Joseph Faizer Name
2 Table
3
ID Contact Type Phone
Id
1
Contact
2 1 Home (123) 456- Table
7890
3 1 Mobile (949) 123-
ID Contact Description 4567
Street City Stat Post
4 Id e al
1
2
3 1 Business
4 1 Home

Requirement of Join Address

©RBJ VESIT
operation table
 Key value store: Java program
{ id: "1",
firstName: "Joseph",
lastName: "Frazier",
phone: [ { type: "home", phone: "(123) 456-7890" },
{ type: "mobile", phone: "(949) 123-4567" } ],
addresses: [ { description: "Home", street: "1234
Something Lane", city: "Los Angeles", state: "CA",
postal: "90001" },
{ description: "Business", street: "321 Main St", city:
"Los Angeles", state: "CA", postal: "90001" }
]
}

©RBJ VESIT
©RBJ VESIT
 E-commerce shopping cart uses key value store

Ecommerce

Cart data Session

data
Final
Order
Key
Key
value
value
store
store
RDBMS

[Link] has one of the largest ecommerce operations, shopping all hours of
the day all around world takes place, so the site has to be up 24*7. Developers
can store unlimited amount of data by creating a data base table and saves it at
multiple servers specified at the customer and also replicates at multiple
available zones. The cart data and session data are stored in the key value store
and final completed order is saved in RDBMs.©RBJ VESIT
Use cases
• Session management at high scale
• User preference and profile stores: Almost every
user has a unique userID as well as preferences
such as language, color, timezone, which
products the user has access to, and so on.
• Product recommendations; latest items viewed on a
retailer website drive future customer product
recommendations
• Ad servicing; customer shopping habits result in
customized ads, coupons, etc. for each customer in
real-time
• Can effectively work as a cache for heavily
accessed(For ex, discounts offered, during
festivals) but rarely updated data
©RBJ VESIT
Who uses key value store ?
• Key-value data stores are ideal for storing
user profiles, blog comments, product
recommendations, and session information.
• Twitter uses Redis to deliver
your Twitter timeline
• Pinterest uses Redis to store lists of users,
followers, unfollowers, boards, and more
• Coinbase uses Redis to enforce rate limits and
guarantee correctness of Bitcoin transactions
• Quora uses Memcached to cache results from
slower, persistent databases
©RBJ VESIT
Main Features of Key Value store
• Simple, Scalable
• Automated Storage scaling
• Distributed Horizontally
• Built in fault Tolerance
• Flexible
• Efficient Indexing
• Secure
• Resource Consumption monitoring
• No queries on values: No queries can be performed
on the values.
• As volume of data increases it will be difficult to
maintain unique value as keys
• Can’t model more complex data structure such as
objects
©RBJ VESIT
When not to use : Key value store
• Relationships among data
• Multi-key Transactions (Want to know information
with reference to two keys)
• Query by Value
• Operations by Sets: since operations are limited to
one key at a time, there is no way to operate upon
value/multiple keys at the same time.

©RBJ VESIT
Quiz Time
• Is key-value store a database or a pattern?
• Pattern

• What type of value can a key-value pair

have? String, Array, Objects, Binary, All of
these !
• All of these !

©RBJ VESIT
▪ Every time you look something up in a row-oriented database,
every row is scanned, regardless of which columns you require.
▪ Let’s say you only want a list of birthdays in September. The
database will scan the table from top to bottom and left to right,
as shown in figure, eventually returning the list of birthdays.(Too
much time !).

Id Dept Id Name DoJ DoB Hobbies

1 1 AAA 11-8- 11/8/1980 Playing
2011
2 2 BBB 4-11- 9/5/1971 Singing
1999
3 1 CCC 23-8- 30/5/1971 Cooking &
1997 Music
4 3 DDD . . -
Indexing
5 the
3 data onEEE
certain . columns. can significantly
- improve
lookup speed, but indexing every column brings extra overhead
and the database is still scanning all the columns.

©RBJ VESIT
Hence Column Type data bases !!!
Column-based
A column store database is a type of database that
stores data using a column oriented model.
A column store database can also be referred to as a:

• Column database
• Column family database
• Column oriented database
• Wide column store database
• Wide column store
• Columnar database
• Columnar store

©RBJ VESIT
 Some videos for Column Types of
data bases
• [Link]
Q
• [Link]
c470b6
• [Link]
d02f99c
• [Link]
M

©RBJ VESIT
©RBJ VESIT
Column-based
• Column-Oriented databases are most similar to traditional
row based databases since both are usually relational.
• Column-oriented databases work on columns, simply store
data in columns instead of rows and are based on BigTable
paper by Google.
• Every column is treated separately. Values of single column
databases are stored contiguously.
• This allows for much faster querying and processing of data
(billions of records per second, searching by column is
faster)
• They deliver high performance on aggregation queries like
SUM, COUNT, AVG, MIN etc. as the data is readily available
in a column. However inserts and updates are generally
slower in column databases.

©RBJ VESIT
Every entity (person) is divided over multiple tables.
Every column is treated separately.
Values of single column (Ex: Name ) databases are stored contiguously with their
Ids..
Id Dept Name DoJ DoB Hobbies
Id
1 1 AAA 11-8- 11/8/1980 Playing
2011
2 2 BBB 4-11- 9/5/1971 Singing
1999
3 1 CCC 23-8- 30/5/1971 Cooking &
1997 Music
4Id 31 2
DDD . 3 . 4 5 -
Dept Id 1 2 1 3 3
5Id 31 EEE
2 . 3 . 4 5 -
Name AAA BBB CCC DDD EEE
Id 1 2 3 4 5
DoJ 11-8-2011 4-11- 23-8-1997
1999
Id 1 2 3 4 5
DoB 11/8/1980 9/5/1971 30/5/1971
Id 1 2 3 4 5
Hobbies Playing Singing Cooking & ©RBJ VESIT
Music
In practice, columnar databases often store data in column families.
In this case you would probably not store the street, city, zip in
completely separate columns, but instead in a column family where
they are stored together.

©RBJ VESIT
• Fast Read Operations : While a relational database
stores data in rows and reads data row by row, a
column store is organized as a set of columns. This
means that when you want to run analytics on a small
number of columns, you can read those columns
directly without consuming memory with the
unwanted data. Columns are often of the same type
and benefit from more efficient compression, making
reads even faster.

• Analytics operations: The column-oriented database

shines when performing analytics. It is easy to
aggregate quickely the value of a given column ( For
ex: adding up the total sales for the year, search
about the list of employees who joined in a particular
year, whose birthdays falling in a particular month
and so on.). ©RBJ VESIT
• Reporting : Column store databases are generally
used when reporting is more important than data
modification.
• Column-based NoSQL databases are widely used
to manage data warehouses, business
intelligence, CRM, Library card catalogs (where
speed is critical).
• Unfortunately there is no free lunch, which means
that while columnar databases are great for
analytics, the way in which they write data makes
it very difficult for them to be strongly consistent
as writes of all the columns require multiple write
events on disk. Relational databases don't suffer
from this problem as row data is written
contiguously to disk. ©RBJ VESIT
⚫Column type of data bases are usually
much slower for write operations, so they
tend to be used in OLAP while row based is
used in OLTP situations. Otherwise
columnar databases are very similar to
traditional databases.
⚫[Link]
SDLbM
⚫Check how read & write operations are
performed.

©RBJ VESIT
When to use Column data base ?
• When should you use a row-oriented database
and when should you use a column-oriented
database?
• In a column-oriented database it’s easy to add
another column because none of the existing
columns are affected by it. But adding an entire
record requires adapting all tables.
• This makes the row-oriented database
preferable over the column-oriented database
for online transaction processing (OLTP)
because this implies adding or changing
records constantly.
©RBJ VESIT
Examples of Column Databases
HBase,
Cassandra,
Hypertable,
Druid,
Google’s big table,
Vertica,
Accumlo

©RBJ VESIT
 Use Cases
• Column stores offer
very high performance and a highly scalable architectu
re
. Because they’re fast to load and query, they’ve been
popular among companies and organizations dealing
with big data, IoT, and user recommendation and
personalization engines.
• Spotify uses Cassandra to store user profile attributes
and metadata about artists, songs, etc. for better
personalization
• Facebook initially built its revamped Messages on top
of HBase, but is now also used for other Facebook
services like the Nearby Friends feature and search
indexing
• Outbrain uses Cassandra to serve over 190 billion
©RBJ VESIT
personalized content recommendations each month
Primary key-foreign key relationship
It defines a one-to-many relationship between two
tables in a relational database.
A foreign key is a column or a set of columns in one
table that references the primary key columns in
another table.
The primary key is defined as a column (or set of
columns) where each value is unique and identifies a
single row of the table.

©RBJ VESIT
Primary key-foreign key relationships
• Primary key is used to
identify data uniquely
therefore two rows can’t
have the same primary
key. It can’t be null.
• On the other hand,
foreign key is used to
maintain relationship
between two tables.
• Primary of a table act as
forgein key in the other
table. Foreign key in a
table enforce Referential
Integrity constraint. It can
be more than one in the
table.

©RBJ VESIT
Benefits of Column Stores
• Compression. Column stores are very efficient at data
compression and/or partitioning.
• Aggregation queries. Due to their structure, columnar
databases perform particularly well with aggregation
queries (such as SUM, COUNT, AVG, etc).
• Scalability. Columnar databases are very scalable.
They are well suited to massively parallel processing (
MPP), which involves having data spread across a large
cluster of machines – often thousands of machines.
• Fast to load and query. Columnar stores can be
loaded extremely fast. A billion row table could be
loaded within a few seconds. You can start querying and
analysing almost immediately.
©RBJ VESIT
©RBJ VESIT
Document-Oriented:
• Document-Oriented NoSQL DB stores and retrieves
data as a key value pair but the value part is stored
as a document. The value can be any sort of value,
array, or even another document.
• The document is stored in JSON or XML formats.
• They're typically schemaless, so each document can
contain any data that you want them to have and you
can change it on the fly.
• The value is understood by the DB and can be
queried.
• The document type is mostly used for Content
Management System, blogging platforms, real-time
analytics & e-commerce applications.
©RBJ VESIT
In this diagram on your left you can see we have rows
and columns, and in the right, we have a document
database which has a similar structure to JSON.

Now for the relational database, you have to know

what columns you have and so on. However, for a
document database, you have data store like JSON
object. You do not require to define which make it
flexible.

©RBJ VESIT
• It should not use for complex transactions which require
multiple operations or queries against varying aggregate
structures.
• Amazon SimpleDB, CouchDB, MongoDB, Riak, Lotus
Notes, MongoDB, are popular Document originated
DBMS systems.
• Key-Value and Document databases are very similar to
each other. The only real difference is document
databases store their values in a more structured way.
• A key-value database may simply store blobs of data for
each key with little knowledge of what is in each blob.
• A document database is still basically a key-value store,
but it knows information about the format of each value.
They may be XML, JSON, RDF, etc. This allows the
database to more easily query information found in each
document.
©RBJ VESIT
• Query Mechanism tools for NoSQL
• The most common data retrieval
mechanism is the REST-based retrieval of a
value based on its key/ID with GET
resource
• Document store Database offers more
difficult queries as they understand the
value in a key-value pair.
• For example, CouchDB allows defining
views with MapReduce

©RBJ VESIT
 Videos on Document type data
base
[Link]
5f4f46c48cf97f3f847aa995
⚫[Link]
ec27536
⚫[Link]
⚫[Link]

©RBJ VESIT
Quiz
MongoDB is a
• Key Value data base model
• Document type
• Graph store
• Key value
Document type !

©RBJ VESIT
Quiz
⚫Which of the following is not a wide-column
store?
a) Cassandra
b) hbase
c) MongoDB
d) Google big Table
⚫ Mongo dB

©RBJ VESIT
Quiz
⚫Most NoSQL databases support automatic
__________ meaning that you have high
availability and disaster recovery.
a) processing
b) scalability
c) replication
d) all of the mentioned
c) replication

©RBJ VESIT
 Quiz
• Which of the following is not a NoSQL
database?
a) Hbase
b) Anazon DB
c) Cassandra
d) Microsft SQL
• Microsft SQL

©RBJ VESIT
Quiz
Which of the following data bases lack in
formal structure and hence searching is not
possible ?
• Column Family
• Document store
• Graph store
• Key value pair
• Column Family & Key value pair

©RBJ VESIT
Quiz: Identify the data base store
• In these data bases the item is searched by the
path like
• Employee [id =34] / Address/
street/Building/text()
• Any item within these data bases can be
searched, because the values are stored in
subsection
• These data bases use the JSON format for storing,
retrieving, and managing data in tree structure.
• In these data bases the fields are key and value,
the values are stored in subsection
• MongoDB, Couch base & Couch DB

©RBJ VESIT
Graph-Based
• A graph database stores data in a graph.
• It is capable of elegantly representing any
kind of data in a highly accessible way.
• A graph type database stores entities as
well the relations amongst those entities.
• The entity is stored as a node such as a
student or business with the relationship as
edges. An edge gives a relationship
between nodes. Every node and edge has a
unique identifier.

©RBJ VESIT
 Graph-Based
• An edge always has a start node, end
node, type, and direction, and an edge
can describe parent-child relationships,
actions, ownership, and the like.
• Each node knows its adjacent nodes.
• There is no limit to the number and kind of
relationships a node can have.
• Compared to a relational database where
tables are loosely connected, a Graph
database is a multi-relational in nature.
©RBJ VESIT
 Graph-Based
• Traversing relationship is fast as they are
already captured into the DB, and there is
no need to calculate them.
• A graph in a graph database can be
traversed along specific edge types or
across the entire graph.
• In graph databases, traversing the joins or
relationships is very fast because the
relationships between nodes are not
calculated at query times but are persisted
in the database.
©RBJ VESIT
 Graph-Based
• Graph databases have advantages for use
cases such as social networking, logistics,
spatial data, recommendation engines,
and fraud detection, when you need to
create relationships between data and
quickly query these relationships.
• Neo4J, Infinite Graph, OrientDB, FlockDB
are some popular graph-based databases.
• As the number of nodes increases, the
cost of a local step (or hop) remains the
same.
©RBJ VESIT
Comparison between the classic
relational model and the graph model :

Relational model Graph Model

Tables Vertices and Edges

set
Rows Vertices

Columns Key value pairs

Join Edges

©RBJ VESIT
• The graph shows an
example of a social
network graph.
• Given the people
(nodes) and their
relationships (edges),
• You can find out who
the "friends of
friends" of a particular
person are—for
example, the friends
of Howard's friends.

©RBJ VESIT
Bob’s information is stored in the form of graph
data base, like address, recent order, follower,
reviews posted by him and many more..

©RBJ VESIT
Driver Car Relationship using graph data base: 6
relationships can be obtained

©RBJ VESIT
 How information is indexed ?
In a triple store, the first field
in the database holds the URI
for the subject, the second
field holds the URI for the
predicate and the third field
holds a URI for the object.

Row
s s p o
1 Bob Married to Julie
2 Bob Listens to Rock
While there are a number of music
different strategies that graph 3 Bob drives BMW
databases may use for storing
4 Bob Works for IBM
triples, most use an index that
abbreviates the three primary 5 Bob Brother of Steve
fields to 6 Julie Sisterinlaw Steve
{s, p, o}. to
7 Julie Listens to Rock
©RBJ VESIT Music
How information in a graph
database is queried ?
• Each triple in a graph database only gets
stored once in the index. Just like relational
databases, it's a simple process to do a
straight lookup query in a graph database.
• If the query is for what information is
known about Bob, the indexer
programming only needs to search rows 1-
5 of the database.
• The real power and speed of a graph
database comes from indexing
combinations of triples.
©RBJ VESIT
 Examples
⚫ If the query is for who Bob is married to, the indexer
will look for the predicate : married To in rows 1-3
and then retrieve the matching object. (Bob is
married to Julie.)
⚫ If the query is to identify everyone who listens to the
same kind of music as Bob, the indexer will first ask
{ :Bob :listensTo ?o } and identify :RockMusic as the
object.
⚫ In the second query, the results will
return :RockMusic in rows 2 and 7. The subject in
row 2 is Bob himself, so whoever is the subject in
row 7 will be the other person who listens to rock
music. (It turns out to be Julie, Bob's wife.)
©RBJ VESIT
Use case 1
Fraud detection
• Graph databases are capable of sophisticated fraud
prevention. With graph databases, you can use
relationships to process financial and purchase
transactions in near-real time.
• With fast graph queries, you are able to detect that,
for example, a potential purchaser is using the same
email address and credit card as included in a known
fraud case.
• Graph databases can also help you easily detect
relationship patterns such as multiple people
associated with a personal email address, or multiple
people sharing the same IP address but residing in
different physical addresses.
©RBJ VESIT
Use Case 2
Recommendation engines
• Graph databases are a good choice for
recommendation applications.
• With graph databases, you can store in a graph
relationships between information categories such as
customer interests, friends, and purchase history.
• You can use a highly available graph database to
make product recommendations to a user based on
which products are purchased by others who follow
the same sport and have similar purchase history.
• Or, you can identify people who have a friend in
common but don’t yet know each other, and then
make a friendship recommendation.

©RBJ VESIT
 Blogs/ video on Graph store data
bases
⚫[Link]
-are-the-future/
⚫[Link]
ases-whats-the-big-deal-ec310b1bc0ed
⚫[Link]
on-to-graph-databases/
⚫[Link]
v=IuwAAfAMfdg&feature=emb_logo

©RBJ VESIT
NoSQL Database Categories

©RBJ VESIT
Examples of NOSQL Data Stores
Key Value Store
• Redis, Amazon Dynamo DB, Azure Table
Storage (ATS)
• Riak, Memcached, Berkeley DB

Column Based Store

• Cassandra, Google Big table,
Hbase, Accumlo, Hyper Table,

Document Store
• Mongo DB, Couch Base, Couch DB,
Orient DB

Graph Store
• Neo4j, AllegroGraph, Teradata
Aster, FlockDB©RBJ VESIT
 When to use which store ?
• Key Value Store : When processing a constant stream of
small reads and writes, session information, user profiles,
preferences, shopping cart data
• Document Types : For natural data modeling, Programmer
friendly, Rapid Development, content management system,
blogging platform, web analytics, real time analytics,
ecommerce applications
• Columnar Type: For massive write loads High availability
Multiple data centers , content management systems,
blogging platforms,
• Graph Type : For Graph Algorithms and many to many
relationships, for connected data such as social networks,
spatial data, routing information for goods and money,
recommendation engines
• Select the best data store for a given data type , Use
case and purpose !
©RBJ VESIT
QUIZ 1
Identify the data base store:
⚫ When processing a constant stream of
small reads and writes, session
information, user profiles, preferences,
shopping cart data

©RBJ VESIT
 Quiz 2
Identify the data base store:

⚫ For natural data modeling, Programmer

friendly, Rapid Development, content
management system, blogging platform,
web analytics, real time analytics,
ecommerce applications

©RBJ VESIT
Quiz 3
Identify the data base store:

⚫ For massive write loads , High

availability, Multiple data centers,
content management systems, blogging
platforms,

©RBJ VESIT
Quiz 4
Identify the data base store:

⚫ One to many, Many to many

relationships, for connected data such
as social networks, spatial data, routing
information for goods and money,
recommendation engines

©RBJ VESIT
©RBJ VESIT
Understand the big data problem:
Categorize the data and determine its type:
Read Read-Write
• Data is read once & rarely • When write happens,
changed master will record this
⮚ Related to warehouse change data to its binary
applications,
log. The slave copies the
⮚ item sets (Images),
change data from master.
⮚ event logging (clickstreams,
sales events, real time sensor • Most of the times after a
data: write operation, there is a
⮚ log file records all the events read of the same data set.
along with their time stamp for
each transaction, • Related to Transaction data
⮚ can be an error/ warning or a ⮚ Making a purchase (session
normal event like a click on a management)
web page, Users browse the
website for articles, posts, view ⮚ When adding a comment
products ⮚ Messages on a forum
⮚ Documents or graph data
©RBJ VESIT
What is Big data NoSQL solution
• Recommendation • Decide the type of Data:
• User Profile  Only Read
Management  Read-Write
• Real Time Data • Distributed Computing
Handling Architecture:
• Content management  Shared RAM,
• Catalog management  Shared Disk,
 Shared nothing
• 360 Degree Customer
Review • Data Distribution Model:
 Sharding: Horizontal &
• Mobile Application
Vertical
• Internet of Things
 Replication: Peer to peer &
• Fraud detection master Slave
©RBJ VESIT
CPU Registers are high speed registers in CPU serve as working memory for
instruction and temporary storage of data.
Cache memory, that is positioned logically between the CPU registers and main
memory. storage capacity of a cache is less than of main memory, but with an
access time of one to three cycles, the cache is much faster than main memory
because some or all of it can reside on the same IC as the CPU. For high
performance computers, caches are essential components.
Primary memory/main memory is the computer memory that is directly
accessible by CPU. It is comprised of DRAM and provides the actual working space to
the processor. It holds the data and instructions that the processor is currently
working on.
Secondary Memory / Mass Storage: The contents of the secondary memory first
get transferred to the primary memory and then are accessed by the processor, this
is because the processor does not directly interact with the secondary memory.
©RBJ VESIT
Distributed Computing
architecture: Each architecture works with
different type of data

©RBJ VESIT
 Shared RAM
• In this architecture many CPUs, access a
single shared RAM, over a high speed bus.
• Each CPU has its own cache for faster access.
• This architecture is useful for large
computations and graph store, for graph
traversal entire graph should be in main
memory.
• Key value store and document store are
cache friendly as they are indexed by a short
key.

©RBJ VESIT
Shared Disk Architecture
• This Architecture is an architecture which is used
in distributed computing in which the nodes share
same disk devices but each node has its own
private memory.
• The disks have active nodes which share memory
in case of any failures.
• In this architecture the disks are accessible from
all the cluster nodes.
• This architecture has quick adaptability to the
changing workloads. It uses robust optimization
techniques.

©RBJ VESIT
Shared Disk Architecture
• Shared Disk has two advantages over
Shared memory. Firstly, each processor
has its own memory, the memory bus is
not a bottleneck; secondly, the system
offers a simple way to provide a degree of
fault tolerance.

©RBJ VESIT
Shared Nothing Architecture
• Shared nothing architecture is an
architecture which is used in distributed
computing in which each node is
independent and different nodes are
interconnected by a network.
• Every node is made of a processor, main
memory and disk.
• The main motive of this architecture is to
remove the contention among nodes. Here
the Nodes do not share memory or storage.

©RBJ VESIT
Shared Nothing Architecture
• The disks have individual nodes which
cannot be shared.
• It works effectively on high volume and
read write environment.
• Systems based on this architecture are are
also commonly known as
MPP (Massively Parallel Processing)
systems.
• In order to achieve a good workload
distribution MPP systems have to use a
hash algorithm to distribute (partition) data
evenly across available©RBJ
CPUVESIT
cores.
Aggregate type of data bases
• Key value, Document type and column family
• An aggregate is a collection of data that we
interact with as unit.
• These units of data or aggregates form the
boundaries and can be easily stored over
cluster or any machine and can be retrieved
with all the related data.
• These work best, when most data interaction
is done with the same aggregate, for ex when
there is a need to get an order and all its
details, its better to store order as an
aggregate.
©RBJ VESIT
 Choosing Distribution Models
• Aggregate oriented data bases make
distribution of data easier, since the
distribution mechanism has to move the
aggregate and not have to worry about the
related data as all the related data is
contained in the aggregate.
• Styles : Sharding & Replication
• A system may use either or both techniques
• Ex: Riak database shards the data and also
replicates
©RBJ VESIT
Sharding
• Database sharding is the process of making
partitions of data in a database or search
engine, such that the data is divided into
various smaller distinct chunks, or shards.
• Each shard could be a table, a Postgres
schema, or a different physical database held
on a separate database server instance.
• To shard your data, you need to decide a key,
called a sharding key, to partition your data on.
• The shard key is either an indexed field or
indexed compound fields that exist in every
document in the collection.©RBJ VESIT
Some data within the database remains present in all shards
(vertical sharding),
Some data appear only in single shards (horizontal sharding).

©RBJ VESIT
Replication
• Copy of data across multiple server. ( So as
to make data available )
• It has two forms : Master Slave & Peer to
Peer replication
• Master Slave Replication: One node is
authorized one, who handle write operation
and all slaves synchronize with the master
and handle reads
• Reduces conflicts during updates

©RBJ VESIT
 Replication
• SPoF : It can be avoided by using RAID drives
or by using a standby master that is
continuously updated by the master node.
• Hbase uses master slave model
• Redundant Array of Independent/Inexpensive
Disks (RAID) is a technology that allows
storing data across multiple hard drives.
• The purpose of RAID is to achieve data
redundancy to reduce data loss and, in a lot of
cases, improve performance.

©RBJ VESIT
©RBJ VESIT
Peer to Peer Replication: This allows write to any node, the
nodes coordinate between themselves to synchronise their copies of
data.

• All writes handling is not

on a single node.
• No SpoF
• Each node takes the
responsibility of master
so if one node gets fail
other nodes can continue
to function.
• All nodes have to be kept
up to date with the
cluster status which
increases communication
overhead.
• Cassandra uses peer to
peer model
©RBJ VESIT
 Which one ?
For Batch processing jobs :
• Master / slave ( Hbase) distribution model

High availability Jobs :

• Peer to Peer distribution model(Cassandra)

©RBJ VESIT
Using NoSQL to manage Big Data

Moving Data to Queries

Use of hash rings to

distribute the loads

Replicate the scale read

Allows the data base to
distribute queries evenly to
data nodes
©RBJ VESIT
 Manage data better by moving
Queries to the data

Nosql Systems uses

1. Fragmented data (Sharding/ Replication),
Commodity hardware, Shared Nothing
architecture.
2. Moving the Query to each©RBJ
dataVESIT
node.
 Indexing
Access
• Candidates name to perform a search, iterate the
elements until we find the desired one, o(n)
• Candidate ID: We could place all members in an
array, with their indexes matching their IDs This
would allow us to access each member directly,
with no search at all, complexity 0(1)
• What if IDs were big, non-sequential or random
numbers? Or, if searching by ID were not
acceptable, and we needed to search by name (or
some other field) instead? : hash functions

©RBJ VESIT
• What is “hashing” : hash roughly means
“chop and mix”
• A hash function is a function that maps one
piece of data—typically describing some
kind of object, often of arbitrary size—to
another piece of data, typically an integer,
known as hash code, or simply hash.
• Hash functions have many uses and for
each one, different properties may be
desired.
• Applications: Indexing, Encryption &
Decryption, data integrity check, Password
Storage and many more.
©RBJ VESIT
A hash function is a mathematical function that converts a numerical
input value/ string into another compressed numerical value. The input
to the hash function is of arbitrary length but output is always of fixed
length. For instance, Some hash function designed to hash strings, with
an output range of 0 .. 100, may map the string Hello to, say, the
number 57, Hasta la vista, baby to the number 33, and any other
possible string to some number within that range.

Hello 57 Values returned by a

hash function are
Hasta la vista, baby 33 called
message digest or
simply hash values.

Message Hash
(M) Hash Value (h)
Arbitrary Fixed
Function Length
Length
(H)

©RBJ VESIT
Password Storage

©RBJ VESIT
Data Integrity Check

©RBJ VESIT
• First, a good hash function generally has a wide output
range (typically, the whole range of a 32 or 64-bit
integer), so building an array for all possible indices
would be either impractical or plain impossible, and a
waste of memory.
• To overcome that, we can have a reasonably sized
array (say, just twice the number of elements we
expect to store) and perform a modulo operation on
the hash to get the array index.
• So, the index would be index = hash(object/data) mod
N, where N is the size of the array.
• Second, object/data hashes will not be unique (unless
we’re working with a fixed dataset and a custom-built
perfect hash function). There will be collisions (further
increased by the modulo operation), and therefore a
simple direct index access won’t work.
©RBJ VESIT
• To solve this problem,
a bucket to each array
index which holds all the
objects sharing a given
index.
• Index/address of bucket
= hash(object/data/key)
mod N
• The hash table or hash
map is used as
a dictionary, and that’s
the way some high-level
languages implement
objects or associative
arrays.
When the hash Tables are large, memory
limitation ? Partition !! ©RBJ VESIT
 Quiz
State True statements :Replication provides
• Improved Read performance in browsing
websites
• Database node availability at all time
• Scale out solution to distribute the data evenly
and queries across all available nodes.
• Consistent read, if a read operation occurs from
replication node before the update happens.
• Consistent Read, if a read operation occurs from
the same node where client has written.

©RBJ VESIT
Quiz
Which of the following data bases is not
aggregate type?
• Column Family
• Document store
• Graph store
• Key value pair
⚫Graph store

©RBJ VESIT
Quiz
Which of the following data bases use
shared RAM architecture(ideal for large
computation)
• Column Family
• Document store
• Graph store
• Key value pair
⚫Graph Store

©RBJ VESIT
Quiz
Which of the following data bases are cache
friendly ?
• Column Family
• Document store
• Graph store
• Key value pair
⚫Column Family & Key value pair (They
are references by short key, so it can
be stored in the cache)

©RBJ VESIT
Quiz
Identify the data base store
• These data bases are used in social
networks & rule based engines.
• These databases handle the relationship
between objects/data and store, process
and query efficiently.
• In these data bases the three fields are,
nodes, Relationship and properties
• Neo4j is an example of this data bases.

©RBJ VESIT
Scaling Out: Distributed Hashing
Hash(Key) modulo 4 is used to generate the index of nodes

Key Value Node

Index This distribution
A125 Data 1
scheme is simple,
intuitive, and works
A240 Data 1 1 A
fine. That is, until the
A500 Data 1 number of nodes
B277 Data 2 changes.
B440 Data 2
2 B
If one node
C050 Data 3
C
fails or
C200 Data 3 3 becomes
C602 Data 3
C850 Data 4
unavailable or
D
C900 Data 4 4 more nodes
are added ??
©RBJ VESIT
 Note that all key locations
The Rehashing Problem :

changed, not only the ones

Hash(Key) modulo 3 is used to generate the index of nodes

• keys need to be redistributed

to include the new from node C. If Node C fails
nodes/exclude the faulty Key Value Node
one. This is true for any Index
distribution scheme.
A125 Data 2
• But the problem with our
A240 Data 1
simple modulo distribution is
that when the number of A500 Data 2
nodes changes, most hashes B277 Data 3
modulo N will change. B440 Data 2
• So most keys will need to be C050 Data 1
moved to a different node.
C200 Data 2
So, even if a single node is
removed or added, all keys C602 Data 3
will likely need to be C850 Data 3
rehashed into a different C900 Data 1
node. ©RBJ VESIT
 Miss of Querries !!!
• So, all of a sudden if a node fails, the keys
won’t be found because they won’t yet be
present at their new location.
• So, most queries will result in misses, and
the original data will likely need retrieving
again from the source to be rehashed,
thus placing a heavy load on the origin
node(s) (typically a database).
• This may very well degrade performance
severely and possibly crash the node
servers.
©RBJ VESIT
Consistent hashing !
So, how can this problem be solved? We
need a distribution scheme that
does not depend directly on the number of
nodes, so that, when adding or removing
nodes, the number of keys that need to be
relocated is minimized.
• One such scheme—a clever, yet
surprisingly simple one—is called consistent
hashing, and was first described by Karger
et al. at MIT in an academic paper from
1997
©RBJ VESIT
 Consistent hashing
• Consistent Hashing is a distributed
hashing scheme that operates
independently of the number of nodes,
servers or objects in a distributed hash
table by assigning them a position on
an abstract circle, or hash ring.
• This allows servers/nodes and objects
to scale without affecting the overall
system.

©RBJ VESIT
 Consistent hashing
• Imagine we map the hash output
range onto the edge of a circle.
• That means that the minimum possible
hash value, zero, would correspond to an
angle of zero.

correspond to an angle of 2𝝅 radians (or

• The maximum possible value would

360 degrees) and

• All other hash values would linearly fit
somewhere in between.
©RBJ VESIT
• So, take a key, compute
its hash, and find out
where it lies on the Key Valu Hash Angles
e
circle’s edge. A125 Data 3450 10
A240 Data 8356 310
B440 A500 Data 4589 55
A24
A500 A500
B277 Data 3678 25
C050 B27
A24 7 B440 Data 6756 90
0
A500 A12 C05 Data 7845 120
C602 5 0
0 A24 C20 Data 8000 210
A24 A500 0
A500 C850 C60 Data 5678 176
A24
C200 2
A24 A500
0
A500 A240 C85 Data 8967 340
0
C900 0
0
C90 VESIT
©RBJ Data 7900 270
0
0
 Key Valu Hash Angles
• Similarly find the hash e
of nodes A, B & C and A125 Data 3450 10
find out where they lie A240 Data 8356 310
on the circle’s edge. A500 Data 4589 55
B277 Data 3678 25
B440 B440 Data 6756 90
A24
A A500
A500 C05 Data 7845 120
C050
B27 0
A24
0
A500 7 C20 Data 8000 210
A12 0
C602
5
C60 Data 5678 176
0 A24
C 2
A24 A500
A500 C850 C85 Data 8967 340
A24 0
C200
A24 A500
0 C90 Data 7900 270
B A500 A240
0 0
C900
0 A 4890 110
0 ©RBJ VESIT
B 8100 220
Using Consistent Hashing Data on a cluster :
Since we have the keys for both the objects and
the nodes on the same circle, we may define a
simple rule to associate the former with the
latter:
• Each object key will belong in the node whose
key is closest, in a counterclockwise direction
(or clockwise, depending on the conventions
used).
• In other words, to find out which node to ask
for a given key, we need to locate the key on
the circle and move in the ascending angle
direction until we find a node.

©RBJ VESIT
 Key Valu Has Angl label
• So, if node B fails, all e h es
its keys will be
A125 Data 345 10 A
assigned to C without 0
touching other keys. B277 Data 367 25 A
8
B44
A24 0
A500 Data 458 55 A
A A500 9
A500
C050 B440 Data 675 90 A
A24 B27 6
0
A500 7
A 489 110
C602 A125
0
0 D A24 C050 Data 784 120 B
C
A24 A500 5
A500 C850 C602 Data 567 176 B
A24
C200 8
A24 A500
0
B A500 A240 C200 Data 800 210 B
0
C900 0
0
B
©RBJ VESIT 810 220
0
0
Quiz
Which of the following ways that help NoSQL
system in handling big data problems ?
• Query is moved to node for processing instead of
transferring the huge data on a single processor.
• Extract the table, serialize it, transmit over
network, assemble over server and execute the
query
• Only Query and results are moved over the
networks.
• Query is moved to node for processing instead of
transferring the huge data on a single processor &
Only Query and results are moved over the
networks.
©RBJ VESIT
Quiz
Which of the following ways that help NoSQL
system in handling network problem or node
failure ?
• Replication.
• Consistent hashing
• Node failure identification by hashing the
value’s key modulo N, N is number of nodes.
• Renumber all nodes and saving key value
again

⚫ Replication &Consistent hashing

©RBJ VESIT
Quiz
Which of the following is /are not true in NOSQL data bases ?
• Velocity, Variety, Volume and Agility are the business
drivers of NoSQL.
• It provides ACID properties.
• It is Schema less & free of join operations.
• It uses distributed architecture, works on multiple nodes
to give high performance .
• They are vertically scalable based on data loads.
• They manage data bases better by moving queries to data
and not data to queries.
• They use consistent hashing to distribute the data/ queries
evenly in a cluster.
It provides ACID properties & They are vertically scalable
based on data loads.

©RBJ VESIT
कोई ना दे हमें खुश रहने की दुआ
तो भी कोई बात नहीं
वैसे भी हम खुशियां रखते नहीं
बांट दिया करते हैं!
जैसे एक मोमबत्ती से हजारों
मोमबत्तियां
जलाने से भी मोमबत्ती जा जीवन
कम नहीं
होता, ©RBJ VESIT
THANK YOU

©RBJ VESIT