24.

1 Introduction to NoSQL Systems

Introduction
NoSQL Databases are database systems designed to manage:
 Very large volumes of data
 Unstructured and semi-structured data
 Distributed storage environments
 High-speed web and cloud applications
Traditional relational SQL databases are sometimes unsuitable for modern Internet-scale
applications because of:
 Fixed schemas
 Complex joins
 Scalability limitations
 High overhead of advanced services
NoSQL systems provide:
 Flexibility
 Scalability
 High performance
 Distributed storage support

24.1.1 Emergence of NoSQL Systems

Need for NoSQL Systems
Many organizations manage applications containing:
 Millions of users
 Billions of records
 Huge multimedia files
 Continuous data generation
Traditional relational databases became insufficient for such applications.

Example 1: E-Mail Applications

Applications such as:
 Google Gmail
 Yahoo Mail
store:
 Millions of user accounts
 Thousands of emails per user

Problems with Traditional SQL Systems

1. Too Many Unnecessary Services
Relational databases provide:
 Complex query processing
 Concurrency control
 Strict transactions
Many large-scale web applications do not require all these features.

2. Rigid Schema Structure

Relational databases require:
 Fixed schemas
 Structured tables
Modern applications often store:
  Text
 Images
 Videos
 JSON documents
 Dynamic user data
Such data is difficult to fit into relational tables.

Example 2: Social Media Applications

Applications such as:
Facebook
manage:
 User profiles
 Friend relationships
 Posts
 Images
 Videos
 Comments
 Likes
These systems require:
 Massive storage
 Fast retrieval
 Distributed processing
Traditional databases are not ideal for handling these requirements efficiently.

Characteristics Required for Modern Data

Storage
Modern applications require databases that support:
Requirement Description
Scalability Ability to grow with increasing data
Flexibility Support for unstructured data
High Availability Continuous operation
Distributed Storage Data stored across multiple servers
Fast Processing Quick read/write operations
Fault Tolerance Recovery from failures

Development of NoSQL Systems

Many companies developed specialized NoSQL databases to solve their large-scale data
problems.

1. Google BigTable
BigTable was developed by Google.
Used in:
 Gmail
 Google Maps
 Web indexing

Characteristics
  Distributed storage
 High scalability
 Fast data access

Related Open Source System

Apache HBase is based on similar concepts.

Category
BigTable belongs to:
Column-Based NoSQL Systems
Also called:
 Wide Column Stores
 Column Family Stores

2. Amazon DynamoDB
Amazon DynamoDB was developed by Amazon.

Category
Key-Value Data Stores
Also known as:
 Key-Tuple Stores
 Key-Object Stores

Features
 Extremely fast lookups
 Distributed architecture
 Cloud integration

3. Cassandra
Apache Cassandra was originally developed by Facebook.
Now it is open source.

Features
 Combines:
o Key-value concepts
o Column-based concepts
 High scalability
 Fault tolerance

4. Document-Based NoSQL Systems

Examples:
 MongoDB
 CouchDB

Characteristics
 Store data as documents
 Usually use JSON-like structures
 Flexible schemas
Advantages
 Easy handling of semi-structured data
 Suitable for web applications
 Flexible data representation

5. Graph-Based NoSQL Systems

Examples:
 Neo4j
 GraphBase

Purpose
Designed to manage:
 Networks
 Relationships
 Social connections

Applications
 Social networks
 Recommendation systems
 Fraud detection

6. Multi-Model NoSQL Systems

Example:
 OrientDB

Characteristics
Combines features from:
 Document databases
 Graph databases
 Key-value systems

7. Object-Oriented and XML Databases

Database systems based on:
 Object models
 Native XML models
can also be considered NoSQL systems.
However:
 They may lack high scalability
 They may not support advanced replication like modern NoSQL systems

Major Categories of NoSQL Databases

Category Description Examples
Key-Value Stores Data stored as key-value pairs DynamoDB
Column-Based Stores Data stored in columns BigTable, HBase
Document Stores Data stored as documents MongoDB, CouchDB
Graph Databases Data stored as nodes and edges Neo4j
 Category Description Examples
Multi-Model Databases Combination of multiple models OrientDB

Advantages of NoSQL Systems

Scalability
Can handle huge amounts of data.
Flexible Schema
Supports dynamic and unstructured data.
Distributed Architecture
Data can be stored across multiple servers.
High Performance
Fast read and write operations.
Fault Tolerance
System continues working even during failures.

Disadvantages of NoSQL Systems

Limited Standardization
Different systems use different models and query languages.
Reduced ACID Support
Some systems relax transaction guarantees.
Complex Management
Distributed systems are difficult to manage.
Limited Complex Query Support
Joins and complex relational operations may be difficult.

Conclusion
NoSQL systems emerged because modern applications required:
 Massive data storage
 Distributed processing
 Flexible data models
 High scalability
Companies like Google, Amazon, and Facebook developed specialized NoSQL databases to
handle Internet-scale applications efficiently. Today, NoSQL databases are widely used in
cloud computing, social media, big data analytics, and real-time web applications.

24.1.2 Characteristics of NoSQL Systems

Introduction
NoSQL Databases systems differ from traditional SQL databases in terms of:
 Distributed system architecture
 Scalability
 Replication
 Data models
 Query processing
NoSQL systems focus mainly on:
  High availability
 Scalability
 High performance
 Flexible data storage
rather than strict relational modeling and complex SQL querying.

Categories of NoSQL Characteristics

The characteristics of NoSQL systems can be divided into two categories:
Category Description
Distributed System Characteristics Scalability, replication, availability, sharding
Data Model & Query Characteristics Flexible schema, APIs, versioning

A. NoSQL Characteristics Related to

Distributed Systems
1. Scalability
Definition
Scalability refers to the ability of a system to grow as data volume increases.

Types of Scalability
1. Horizontal Scalability
Adding more nodes (servers) to the distributed system.
2. Vertical Scalability
Increasing storage or processing power of existing servers.

NoSQL Systems Mainly Use

Horizontal Scalability
because:
 Data volume continuously increases
 New nodes can be added dynamically
 System can continue operating without shutdown

Advantages of Horizontal Scalability

 Better load distribution
 Supports massive datasets
 Continuous system operation
 Cost-effective expansion

Example
If one server becomes overloaded:
 Additional servers are added
 Data is redistributed among servers
2. Availability, Replication, and Eventual
Consistency
Availability
Definition
Availability means the system continues functioning even when some nodes fail.

Replication
Definition
Replication means storing copies of data on multiple nodes.

Advantages of Replication
High Availability
If one node fails, another copy is available.
Improved Read Performance
Read requests can be processed from replicated nodes.
Fault Tolerance
Data loss is prevented.

Problem with Replication

Write operations become slower because:
 Every copy must be updated.

Eventual Consistency
Traditional SQL systems require:
Serializable Consistency
where all copies are always identical immediately.
Many NoSQL systems instead use:

Eventual Consistency
meaning:
 Updates propagate gradually.
 All replicas eventually become consistent.

Benefits of Eventual Consistency

 Faster performance
 Better scalability
 Higher availability

Limitation
Temporary inconsistencies may occur between replicas.

3. Replication Models
Two major replication models are used in NoSQL systems.

3.1 Master-Slave Replication

Definition
One node acts as:
 Master → handles writes
 Slave → stores replicated copies

Working
Write Operations
 Applied first to master.
 Later propagated to slaves.
Read Operations
Can be configured in two ways:
Method Description
Reads from Master Only Latest data guaranteed
Reads from Slaves Faster reads but may return old data

Advantages
 Simpler management
 Better write control

Disadvantages
 Master failure affects writes
 Slaves may contain outdated data

3.2 Master-Master Replication

Definition
All replicas can:
 Read data
 Write data

Advantages
 Higher availability
 Better performance
 No single point of failure

Disadvantages
Different nodes may temporarily store different values.
Conflicts occur when:
 Multiple users update the same item simultaneously.

Reconciliation
Conflict resolution methods are required to:
 Detect conflicts
 Merge updates
 Maintain consistency

4. Sharding of Files
Definition
Sharding means dividing a large file into smaller partitions called:

Shards
Also known as:

Horizontal Partitioning
Need for Sharding
Large applications may contain:
 Millions of records
 Thousands of concurrent users
Storing all records in one node causes:
 Performance bottlenecks
 Heavy load

Advantages of Sharding
Load Balancing
Data access workload is distributed.
Better Performance
Each node handles fewer records.
Scalability
Additional shards can be added easily.

Combination of Sharding and Replication

Technique Purpose
Sharding Load balancing
Replication High availability
Together they improve overall system performance.

5. High-Performance Data Access

NoSQL systems must quickly locate records among millions of objects.
Two major techniques are used:
1. Hashing
2. Range Partitioning

5.1 Hashing
Definition
A hash function is applied to object keys.
h(K)h(K)h(K)
where:
 K = object key
 h(K) = storage location

Advantages
 Very fast data retrieval
 Uniform data distribution
Disadvantages
Range queries become difficult.

5.2 Range Partitioning

Definition
Objects are stored according to ranges of key values.
Example:
Key Range Location
1 – 1000 Node 1
1001 – 2000 Node 2

Advantages
Efficient for range queries.
Example:
 Retrieve students with IDs between 100 and 500.

Disadvantages
Data may become unevenly distributed.

B. NoSQL Characteristics Related to Data

Models and Query Languages
1. No Requirement of Schema
Traditional relational databases require:
 Fixed schemas
 Predefined tables
NoSQL systems often support:

Schema-less Data Storage

Advantages
Flexibility
Different records can have different structures.
Semi-Structured Data Support
Supports:
 JSON
 XML
 Dynamic documents

Common Data Formats

JSON
(JavaScript Object Notation)
XML
(Extensible Markup Language)
Limitation
Since schemas are absent:
 Constraints must be enforced in application programs.

2. Less Powerful Query Languages

Many NoSQL applications mainly retrieve:
 Single objects
 Using object keys
Hence, complex SQL functionality is unnecessary.

APIs in NoSQL Systems

NoSQL systems usually provide APIs for operations.

CRUD Operations
Operation Meaning
Create Insert data
Read Retrieve data
Update Modify data
Delete Remove data

SCRUD Operations
Some systems include:
Search (Find)
along with CRUD.

Query Language Characteristics

NoSQL query languages:
 Simpler than SQL
 Limited join support
 Optimized for performance

Limitation
Joins are often implemented manually in application programs.

3. Versioning
Some NoSQL systems store:

Multiple Versions of Data

along with timestamps.

Advantages
 Historical tracking
 Recovery of old values
 Better concurrency management

Example
A document may store:
 Current version
 Previous versions
 Update timestamps

Comparison: SQL vs NoSQL

Characteristics
Feature SQL Systems NoSQL Systems
Schema Fixed Flexible
Scalability Vertical Horizontal
Consistency Strong consistency Eventual consistency
Query Language Powerful SQL Limited APIs/queries
Replication Optional Core feature
Sharding Less common Common
Joins Supported Often unsupported
Data Model Structured Semi-structured/unstructured

Advantages of NoSQL Characteristics

High Scalability
Supports Internet-scale applications.
High Availability
System remains operational during failures.
Better Performance
Optimized for large-scale reads/writes.
Flexible Data Storage
Supports dynamic structures.
Distributed Processing
Handles huge datasets efficiently.

Disadvantages
Weak Consistency
Temporary inconsistencies may occur.
Limited Querying
Complex joins are difficult.
Complex Conflict Resolution
Especially in master-master replication.
Application-Level Constraints
Validation handled by programs.

Conclusion
NoSQL Databases systems are designed for:
 Large-scale distributed applications
  Flexible data storage
 High availability
 Horizontal scalability
They achieve high performance through:
 Replication
 Sharding
 Eventual consistency
 Schema-less design
These features make NoSQL systems suitable for modern web applications, cloud computing,
social media platforms, and big data environments.

24.1.3 Categories of NoSQL Systems

Introduction
NoSQL Databases are classified into different categories based on:
 Data storage model
 Data access methods
 Query processing techniques
 Distributed architecture
The most common classification divides NoSQL systems into four major categories:
1. Document-based systems
2. Key-value stores
3. Column-based systems
4. Graph-based systems
Additional categories also exist for systems that combine multiple features.

Major Categories of NoSQL Systems

Category Main Idea
Document-Based Systems Store data as documents
Key-Value Stores Store data as key-value pairs
Column-Based Systems Store data in column families
Graph-Based Systems Store data as graphs

1. Document-Based NoSQL Systems

Definition
Document-based systems store data in the form of:

Documents
usually using formats such as:
 JSON
 BSON
 XML

Characteristics
 Flexible schema
 Semi-structured data storage
 Easy handling of nested data
 Fast retrieval using document IDs
Access Methods
Documents can be accessed using:
 Document ID
 Secondary indexes

Data Format Example

{
"StudentID": 101,
"Name": "Rahul",
"Department": "CSE"
}

Advantages
Flexible Structure
Different documents can have different fields.
Easy Data Representation
Suitable for real-world objects.
High Scalability
Supports distributed storage.

Applications
 Content management systems
 E-commerce websites
 User profiles
 Social media applications

Examples
 MongoDB
 CouchDB

2. NoSQL Key-Value Stores

Definition
Key-value stores maintain data in the form:
Key Value
Unique Key Associated Data

Characteristics
 Simple data model
 Very fast access
 Optimized for lookups

Working
The system retrieves data directly using the key.
Example:
Key Value
1001 Customer Record
A25 Product Information
Value May Contain
 Record
 Object
 Document
 Complex data structure

Advantages
Extremely Fast Retrieval
Ideal for real-time applications.
High Performance
Efficient read/write operations.
Scalability
Suitable for distributed systems.

Applications
 Shopping carts
 Session management
 Caching systems
 Real-time recommendations

Examples
 Amazon DynamoDB
 Redis

3. Column-Based or Wide Column NoSQL

Systems
Definition
These systems organize data into:

Column Families
instead of storing entire rows together.

Concept
Traditional relational databases store data row-wise.
Column-based systems store:
 Related columns together.
This is a form of:

Vertical Partitioning
Characteristics
 High compression efficiency
 Fast column retrieval
 Efficient analytics processing
 Supports versioning

Column Family Example

Row Key Name Age Department
101 Rahul 20 CSE
Columns may be stored separately:
 Name Family
 Academic Family
 Personal Family

Versioning
These systems often maintain:
 Multiple versions of data
 Timestamps for each version

Advantages
Efficient for Big Data Analytics
Only required columns are read.
High Scalability
Supports distributed storage.
Faster Aggregation
Good for analytical queries.

Applications
 Big data processing
 Data warehousing
 Real-time analytics

Examples
 BigTable
 Apache HBase
 Apache Cassandra

4. Graph-Based NoSQL Systems

Definition
Graph databases represent data using:
 Nodes
 Edges
 Relationships

Characteristics
 Relationship-oriented storage
 Efficient graph traversal
 Path-based querying

Components of Graph Databases

Component Meaning
Node Entity
Edge Relationship
Path Traversal between nodes
Example
Students connected through friendships:
Rahul → Friend → Anil

Advantages
Efficient Relationship Handling
Excellent for connected data.
Fast Traversal
Quickly navigates relationships.
Flexible Data Modeling
Supports complex networks.

Applications
 Social networks
 Recommendation systems
 Fraud detection
 Network analysis

Examples
 Neo4j
 GraphBase

Additional Categories of NoSQL Systems

5. Hybrid NoSQL Systems
Definition
Hybrid systems combine features from:
 Document systems
 Graph systems
 Key-value stores
 Column stores

Advantages
 Flexible architecture
 Multiple data models
 Supports diverse applications

Example
 OrientDB

6. Object Databases
Definition
Object databases store data as:
 Objects
 Classes
  Methods
based on object-oriented concepts.

Characteristics
 Object identity
 Encapsulation
 Inheritance

Applications
 CAD systems
 Multimedia systems
 Engineering applications

7. XML Databases
Definition
XML databases store data in:

XML Format
Characteristics
 Hierarchical structure
 Self-describing data
 Flexible representation

Applications
 Web services
 Document management
 Data exchange systems

Keyword-Based Search Engines as NoSQL

Systems
Search engines also manage:
 Huge volumes of data
 Fast retrieval mechanisms
Hence, they can be considered:
 Large-scale NoSQL data stores

Examples
 Google Search
 Microsoft Bing

Comparison of NoSQL Categories

Category Storage Model Best For Examples
Document Store Documents Semi-structured data MongoDB
Key-Value Store Key-value pairs Fast lookups DynamoDB
Column Store Column families Big data analytics HBase
 Category Storage Model Best For Examples
Graph Database Nodes & edges Relationship data Neo4j

Advantages of NoSQL Categorization

Specialized Solutions
Each category addresses specific application needs.
Better Performance
Optimized for particular workloads.
Scalability
Handles large-scale distributed data.
Flexibility
Supports different data structures.

Disadvantages
Lack of Standardization
Different systems use different models.
Learning Complexity
Each category has unique concepts.
Limited Cross-Compatibility
Migration between systems may be difficult.

Conclusion
NoSQL Databases are classified into multiple categories based on how they store and access
data. The four primary categories are:
 Document-based systems
 Key-value stores
 Column-based systems
 Graph-based systems
Additional categories such as hybrid, object-oriented, and XML databases further extend
NoSQL capabilities. These systems are widely used in big data, cloud computing, analytics,
social networking, and distributed web applications because of their scalability, flexibility,
and high performance.

24.2 The CAP Theorem

Introduction
CAP Theorem is an important principle in distributed database systems and NoSQL
Databases. It explains the trade-offs among:
 Consistency
 Availability
 Partition Tolerance
in distributed systems that use data replication.
The CAP theorem helps database designers decide which properties are most important for
their applications.
Background: Replication in Distributed
Databases
In distributed databases:
 Data is stored on multiple nodes.
 Copies of the same data item may exist at different locations.
This improves:
 Availability
 Fault tolerance
 Performance
However, replication introduces consistency challenges.

Problem with Replicated Data

Suppose:
 Data item X exists on Node A and Node B.
 Transaction T1 updates X on Node A.
 Transaction T2 updates X on Node B.
Now:
 Two different versions of X exist.
 Transactions reading X may obtain different values.
This creates:

Inconsistency Among Replicas

ACID Properties in Distributed Systems
Traditional SQL systems attempt to maintain:
ACID Property Meaning
Atomicity All or nothing execution
Consistency Database remains valid
Isolation Concurrent transactions behave serially
Durability Permanent storage after commit

Difficulty in Distributed Systems

When replication exists:
 Concurrency control becomes complex.
 Maintaining serializability causes high overhead.
 System performance decreases.

Strong Consistency vs Weak Consistency

Distributed systems may use different levels of consistency.
Type Description
Strong Consistency All copies always identical
Weak Consistency Temporary differences allowed

Strong Consistency
Ensures:
 Every user sees the latest data.
  All replicas remain synchronized.
However:
 High communication overhead
 Reduced performance
 Slower writes

Weak Consistency
Allows:
 Temporary inconsistency between replicas.
 Better performance and availability.
Many NoSQL systems prefer weak consistency.

The CAP Theorem

The CAP theorem states that a distributed system cannot simultaneously guarantee all three
properties:
1. Consistency (C)
2. Availability (A)
3. Partition Tolerance (P)

CAP Properties
1. Consistency (C)
Definition
All nodes see the same data at the same time.
After an update:
 Every replica reflects the same value.

Example
If a bank balance changes:
 All users must immediately see the updated balance.

Note About Consistency

The meaning of consistency in CAP differs from consistency in ACID.

Consistency in CAP
Refers to:
 Uniformity among replicated copies.

Consistency in ACID
Refers to:
 Preservation of database integrity constraints.

2. Availability (A)
Definition
Every request receives a response.
The system:
  Continues operating
 Even if some nodes fail

Characteristics
 Read/write requests are processed.
 Users receive successful responses whenever possible.

Example
A social media site remains accessible even if some servers fail.

3. Partition Tolerance (P)

Definition
The system continues functioning even when network failures partition nodes into separate
groups.

Network Partition
A partition occurs when:
 Nodes cannot communicate with each other.
Example:
 Network cable failure
 Communication breakdown

Partitioned System Example

Partition 1 Partition 2
Node A Node C
Node B Node D
Nodes in different partitions cannot communicate.

Main Statement of CAP Theorem

A distributed system with replication cannot guarantee all three simultaneously:
Property Meaning
C Consistency
A Availability
P Partition Tolerance
At most:

Any Two Properties Can Be Guaranteed

CAP Combinations
1. CA Systems (Consistency + Availability)
Characteristics
 Data remains consistent.
 System remains available.
Limitation
 Cannot tolerate network partitions.

Suitable For
Traditional centralized SQL systems.

2. CP Systems (Consistency + Partition

Tolerance)
Characteristics
 Strong consistency maintained.
 Works despite network failures.

Limitation
 Some requests may be rejected.
 Availability decreases during partitions.

Suitable For
Applications requiring accurate data:
 Banking systems
 Financial applications

3. AP Systems (Availability + Partition

Tolerance)
Characteristics
 System always available.
 Continues operating during partitions.

Limitation
 Temporary inconsistencies may occur.

Suitable For
Most NoSQL systems:
 Social media
 E-commerce
 Web applications

Eventual Consistency
Many NoSQL systems use:

Eventual Consistency
instead of strong consistency.

Definition
Replicas may temporarily differ, but eventually all copies become consistent.
Characteristics
 Faster performance
 Better scalability
 Higher availability

Example
A social media post may appear on some servers immediately and on others after a short
delay.

Why NoSQL Systems Prefer Eventual

Consistency
NoSQL systems prioritize:
 Availability
 Partition tolerance
 Performance
because modern web applications require:
 Continuous operation
 Fast response times
 Large-scale scalability

CAP Theorem and NoSQL Systems

Most NoSQL Databases choose:

AP (Availability + Partition Tolerance)

with weaker consistency models.

Examples
System CAP Preference
MongoDB AP / eventual consistency
Apache Cassandra AP
Apache HBase CP

Advantages of CAP-Based Design

Better Scalability
Supports distributed growth.
High Availability
System remains operational.
Improved Fault Tolerance
Handles network failures.
Better Performance
Reduces synchronization overhead.

Disadvantages
Temporary Inconsistency
Users may see different values.
Complex Design
Conflict resolution becomes difficult.
Reduced Transaction Guarantees
Strong ACID properties may not be fully supported.

Comparison: SQL vs NoSQL Based on CAP

Feature SQL Systems NoSQL Systems
Priority Consistency Availability & scalability
Consistency Model Strong consistency Eventual consistency
Scalability Limited High
Partition Tolerance Lower Higher
Performance Moderate High

Examples of CAP Trade-Offs

Application Preferred CAP Choice
Banking CP
Airline Reservation CP
Social Media AP
Online Shopping Cart AP
Messaging Systems AP

Importance of CAP Theorem

The CAP theorem helps designers:
 Understand distributed system trade-offs
 Choose suitable consistency models
 Design scalable architectures
 Balance performance and reliability

Conclusion
CAP Theorem explains that a distributed replicated system cannot simultaneously achieve:
 Consistency
 Availability
 Partition tolerance
Designers must choose the most suitable combination depending on application requirements.
Most NoSQL Databases prioritize:
 Availability
 Partition tolerance
and therefore use weaker consistency models such as eventual consistency to achieve high scalability
and performance in distributed environments.

24.3 Document-Based NoSQL Systems and

MongoDB
Introduction
NoSQL Databases include a category known as:

Document-Based NoSQL Systems

These systems store data in the form of:
 Documents
 Collections of documents
They are also called:

Document Stores
Characteristics of Document-Based Systems

Flexible schema

Self-describing data

Semi-structured storage

Easy scalability

Fast retrieval
Unlike relational databases:
 No fixed schema is required.
 Documents in the same collection may have different structures.

Document Structure
Documents resemble:
 Complex objects
 XML documents
 JSON objects
However:
 No predefined schema is mandatory.
 New attributes can be added dynamically.

Self-Describing Data
Each document contains:
 Data values
 Attribute names
Hence, the database can automatically identify:
 Fields
 Data types
 Structure

JSON Format
A commonly used document format is:

JSON
(JavaScript Object Notation)

Example of JSON Document

{
"StudentID": 101,
"Name": "Rahul",
 "Department": "CSE"
}

Examples of Document-Based NoSQL

Systems
 MongoDB
 CouchDB

24.3.1 MongoDB Data Model

Introduction to MongoDB
MongoDB is one of the most popular document-oriented NoSQL databases.
It stores data in:

BSON Format
(Binary JSON)

BSON
Definition
BSON is:
 A binary representation of JSON
 More storage efficient
 Supports additional data types

Collections in MongoDB
Documents are grouped into:

Collections
A collection is similar to a table in relational databases.

Creating Collections
MongoDB uses:
[Link]()

Example
[Link]("project",
{ capped : true, size : 1310720, max : 500 })

Explanation
Parameter Meaning
project Collection name
capped Fixed-size collection
size Maximum storage size
max Maximum number of documents

Another Collection Example

[Link]("worker",
{ capped : true, size : 5242880, max : 2000 })

ObjectId in MongoDB
Every document contains a unique field:

_id
Characteristics
 Automatically indexed
 Unique for each document
 Similar to primary key

ObjectId Generation
MongoDB can:
 Automatically generate ObjectIds
OR
 Accept user-defined ObjectIds

System-Generated ObjectId Components

Component Size
Timestamp 4 bytes
Node ID 3 bytes
Process ID 2 bytes
Counter 3 bytes

Schema-less Nature of MongoDB

Collections do not require fixed schemas.
Different documents may contain:
 Different fields
 Different structures

Document Design Approaches

MongoDB supports:
1. Denormalized design
2. Embedded references
3. Normalized design

1. Denormalized Design
Embedded Subdocuments
Workers are stored directly inside the project document.

Example
{
"_id": "P1",
"Pname": "ProductX",
"Workers": [
 {
"Ename": "John Smith",
"Hours": 32.5
}
]
}

Advantages
 Faster retrieval
 No joins needed
 Good read performance

Disadvantages
 Data redundancy
 Update complexity

2. Embedded Array of References

Instead of embedding full worker documents:
 Only worker IDs are stored.

Example
{
"_id": "P1",
"WorkerIds": ["W1", "W2"]
}
Separate worker documents:
{
"_id": "W1",
"Ename": "John Smith"
}

Advantages
 Reduced redundancy
 Better modularity

Disadvantages
 Additional lookups required

3. Normalized Design
Project and worker documents are stored separately.

Example
Project document:
{
"_id": "P1",
"Pname": "ProductX"
}
Worker document:
{
"_id": "W1",
"ProjectId": "P1"
}
Advantages
 Reduced duplication
 Better consistency

Disadvantages
 More complex retrieval
 Requires application-level joins

MongoDB Insert Operation

Documents are inserted using:
[Link]()

Example
[Link](
{
_id: "P1",
Pname: "ProductX"
})

24.3.2 MongoDB CRUD Operations

CRUD stands for:
Operation Meaning
Create Insert data
Read Retrieve data
Update Modify data
Delete Remove data

1. Create Operation
Insert
[Link](document)

2. Read Operation
Find
[Link](condition)

Example
[Link]({Ename:"John Smith"})

3. Update Operation
Updates documents using:
[Link]()
with:
 Selection condition
 $set clause
Example
[Link](
{_id:"W1"},
{$set:{Hours:40}})

4. Delete Operation
MongoDB uses:
[Link](condition)

Example
[Link]({_id:"W1"})

24.3.3 MongoDB Distributed System

Characteristics
Atomicity and Transactions
Most MongoDB operations are atomic for:
 Single documents
For multidocument transactions:
 Two-Phase Commit Protocol is used.

Replication in MongoDB
MongoDB uses:

Replica Sets
to maintain multiple copies of data.

Components of Replica Set

Component Function
Primary Handles writes
Secondary Stores replicas
Arbiter Participates in elections

Primary Node
 Stores main copy
 Processes all writes

Secondary Nodes
 Maintain replicated copies
 Can process reads

Arbiter
 Does not store data
 Helps elect new primary
Replica Set Requirements
 Minimum 3 participants
 Total members should be odd
Example:
 1 Primary
 1 Secondary
 1 Arbiter

Read and Write Operations

Writes
 Always processed by primary.
Reads
Can be configured:
 Primary only
 Secondary replicas
 Any replica

Read Preference
Mode Advantage Limitation
Primary Latest data Lower read scalability
Secondary Faster reads Possible outdated data

Sharding in MongoDB
Definition
Sharding divides collections into:

Shards
Also called:

Horizontal Partitioning
Purpose of Sharding
 Improve performance
 Support large datasets
 Enable horizontal scalability
 Achieve load balancing

Shard Key
MongoDB partitions documents using:

Shard Key
Requirements of Shard Key
Must:
 1. Exist in every document
2. Have an index

Partitioning Methods
MongoDB supports:
1. Range partitioning
2. Hash partitioning

1. Range Partitioning
Documents grouped by key ranges.
Example:
Range Shard
1–1000 Shard 1
1001–2000 Shard 2

Advantages
 Efficient range queries

Disadvantages
 Uneven data distribution

2. Hash Partitioning
A hash function is applied:
h(K)h(K)h(K)
where:
 K = shard key

Advantages
 Uniform distribution
 Better load balancing

Disadvantages
 Inefficient range queries

Query Router
MongoDB uses a:

Query Router
to:
 Track shard locations
 Route queries to correct shards

Query Routing Process

Situation Action
Shard known Query sent to specific shard
 Situation Action
Shard unknown Query broadcast to all shards

Replication vs Sharding
Feature Replication Sharding
Purpose Availability Scalability
Method Copy data Partition data
Benefit Fault tolerance Load balancing

Advantages of MongoDB
Flexible Schema
Supports dynamic documents.
Horizontal Scalability
Easy addition of nodes.
High Availability
Replica sets provide fault tolerance.
High Performance
Fast document retrieval.
Distributed Processing
Supports large-scale applications.

Disadvantages of MongoDB
Weak Consistency
Eventual consistency may occur.
Complex Distributed Management
Replication and sharding increase complexity.
Data Redundancy
Denormalized designs duplicate data.

Applications of MongoDB
 Social media
 Real-time analytics
 E-commerce
 Content management
 Big data applications

Conclusion
MongoDB is a powerful document-based NoSQL Databases system that stores data as
flexible BSON documents. It supports:
 Schema-less design
 CRUD operations
 Replication
 Sharding
  Horizontal scalability

24.4 NoSQL Key-Value Stores

Introduction
NoSQL Databases include an important category called:

Key-Value Stores
These systems focus on:
 High performance
 Scalability
 Availability
 Fast data access

Basic Concept of Key-Value Stores

A key-value store maintains data as:

(Key, Value) Pairs

Components
Component Description
Key Unique identifier
Value Actual data item

Key
The key:
 Uniquely identifies data
 Is used for fast retrieval
Example:
 StudentID
 ProductID
 UserID

Value
The value:
 Contains actual data
 May be structured or unstructured
Possible formats:
 String
 Array of bytes
 JSON document
 Object
 Record

Main Characteristic
Every value must be associated with:
 A unique key
Retrieval using the key must be:
 Extremely fast
Features of Key-Value Stores
High Performance
Fast key-based access.
Horizontal Scalability
New nodes can be added easily.
High Availability
Replication ensures data availability.
Simplicity
Simple data model and operations.
Distributed Storage
Data stored across multiple nodes.

Data Types Supported

Key-value stores can manage:
Data Type Description
Structured Relational-like records
Semi-structured JSON, XML
Unstructured Binary data, files

Query Language
Most key-value stores:
 Do not use SQL
 Provide APIs and operations instead
Common operations:
 Get
 Put
 Delete

Examples of Key-Value Stores

 Amazon DynamoDB
 Project Voldemort
 Redis
 Apache Cassandra
 Oracle NoSQL Database

24.4.1 DynamoDB Overview

Introduction
Amazon DynamoDB is a distributed key-value NoSQL database provided through:
Amazon cloud services.
It is part of:
 AWS (Amazon Web Services)

DynamoDB Data Model

DynamoDB uses:
 Tables
 Items
 Attributes

Table
A table:
 Stores a collection of items
 Does not require a fixed schema

Item
An item is:
 A self-describing record/object
 Similar to a row in relational databases

Attribute
An attribute:
 Represents a field/value pair
Example:
{
"StudentID": 101,
"Name": "Anil",
"Department": "CSE"
}

JSON Support
DynamoDB allows:
 JSON input format
The system converts JSON into:
 Internal storage format

Primary Key in DynamoDB

When creating a table:
 A primary key must be specified.
The primary key:
 Rapidly locates items
 Acts as the key in the key-value model

Types of Primary Keys

DynamoDB supports:
1. Hash Type Primary Key
2. Hash and Range Type Primary Key

1. Hash Type Primary Key

Uses:
A single attribute
The system creates:
 Hash index on the attribute
Example
StudentID

Characteristics
 Fast lookup
 Items not stored in order

2. Hash and Range Type Primary Key

Uses:
 Two attributes
Attribute Purpose
A Hashing
B Ordering

Example
Attribute Role
ItemID Hash key
Timestamp Range key

Advantages
 Supports multiple versions
 Allows ordering
 Enables secondary indexes

DynamoDB Distributed Characteristics

Amazon DynamoDB supports:
 Replication
 Sharding
 Horizontal scalability
through distributed storage techniques.

24.4.2 Voldemort Key-Value Distributed

Data Store
Introduction
Project Voldemort is:
 Open-source
 Inspired by DynamoDB
 Used by LinkedIn

Main Goals of Voldemort

 High performance
 Horizontal scalability
  Replication
 Fast read/write response time

Basic Operations in Voldemort

A store contains:
 Collection of (key, value) pairs
Assume store name:
s

1. Put Operation
[Link](k, v)

Purpose
Inserts:
 Key k
 Value v

2. Get Operation
v = [Link](k)

Purpose
Retrieves:
 Value associated with key k

3. Delete Operation
[Link](k)

Purpose
Deletes:
 Item with key k

Storage Format
Internally:
 Keys and values stored as arrays of bytes

High-Level Data Formats

Values can also be stored using:
 JSON
 Serialized objects

Serializer Class
A serializer:
 Converts user data into byte arrays
 Converts byte arrays back into user format
Consistent Hashing
Project Voldemort uses:

Consistent Hashing
for:
 Data distribution
 Replication
 Scalability

Hash Function
A hash function is applied:
h(k)h(k)h(k)
where:
 k = key

Hash Value Range

Usually:
0≤h(k)≤2n−10 \leq h(k) \leq 2^n - 10≤h(k)≤2n−1

Ring Structure
Consistent hashing visualizes:
 Hash values
 Nodes
on a circular ring.

Node Placement
Nodes are placed:
 Pseudorandomly on the ring
Each node may appear:
 Multiple times
depending on capacity.

Data Placement Rule

A key-value pair:
 Is stored on the node immediately following its hash value clockwise.

Advantages of Consistent Hashing

Horizontal Scalability
Easy addition of new nodes.
Minimal Data Movement
Only some data redistributed when adding nodes.
Load Balancing
Data distributed across nodes.
Fault Tolerance
Failed node data reassigned automatically.
Adding a New Node
When node D is added:
 Only nearby ranges move to D.
 Entire database redistribution is unnecessary.

Replication in Voldemort
Replication is achieved by:
 Storing replicas on successive nodes clockwise.

Sharding in Voldemort
Different key-value pairs stored on different nodes.
This provides:

Horizontal Partitioning (Sharding)

Handling Node Failures
If a node fails:
 Neighboring nodes take over its data.

Consistency and Versioning

Project Voldemort allows:
 Concurrent writes
This may temporarily create:
 Multiple versions of the same value.

Vector Clocks
Each write operation receives:
 A vector clock value
to track versions.

Read Repair
During reads:
 Different replicas compared
 Conflicts resolved
This technique is called:

Read Repair
Conflict Resolution
If system cannot automatically reconcile versions:
 Multiple versions returned to application
 Application resolves conflict

24.4.3 Other Key-Value Stores

Oracle NoSQL Database
Oracle NoSQL Database:
 Developed by Oracle Corporation
 Based on key-value concepts

Redis
Redis is:
 In-memory key-value store
 Extremely fast

Features of Redis
Main Memory Caching
Stores data in RAM.
High Performance
Very low latency.
Replication
Supports master-slave replication.
Persistence
Backs up data to disk.

Apache Cassandra
Apache Cassandra:
 Hybrid NoSQL system
 Combines:
o Key-value features
o Column-based features
Originally developed by:
 Facebook

Features of Cassandra
 Distributed architecture
 High scalability
 High availability
 Fault tolerance

Advantages of Key-Value Stores

Extremely Fast Access
Key-based retrieval is very efficient.
Simple Design
Easy to implement and scale.
High Scalability
Supports large distributed systems.
High Availability
Replication improves reliability.
Flexible Data Storage
Supports many data formats.

Disadvantages of Key-Value Stores

Limited Query Support
Complex queries difficult.
No Join Operations
Relationships handled in application code.
Weak Consistency
Some systems use eventual consistency.
Limited Structure
Data validation may be difficult.

Applications of Key-Value Stores

 Shopping carts
 Session management
 User profiles
 Caching systems
 Real-time applications
 Distributed web applications

Comparison: Relational DBMS vs Key-

Value Store
Feature Relational DBMS Key-Value Store
Schema Fixed Flexible
Query Language SQL API operations
Joins Supported Not supported
Scalability Moderate High
Performance Moderate Very high
Data Model Tables Key-value pairs

Conclusion
NoSQL Databases key-value stores provide:
 High performance
 Scalability
 Fast key-based access
 Distributed storage
Systems such as Amazon DynamoDB, Project Voldemort, and Redis are widely used for
large-scale distributed applications where fast retrieval and availability are more important
than complex query processing.

24.5 Column-Based or Wide Column

NoSQL Systems – Lecture Notes
Introduction
 Another category of NoSQL Databases is:

Column-Based / Wide Column Systems

 Designed for:
o Big data storage
o High performance
o Distributed processing

Examples
 Google Bigtable
 Apache HBase
 Apache Cassandra

Storage Systems Used

System Storage Used
BigTable GFS (Google File System)
HBase HDFS (Hadoop Distributed File System)
HBase Amazon S3 also supported

Important Feature
Column-based systems use:
 Multidimensional keys
Key consists of:
 Table name
 Row key
 Column family
 Column qualifier
 Timestamp

24.5.1 HBase Data Model

HBase Concepts
HBase organizes data using:
 Namespaces
 Tables
 Rows
 Column Families
 Column Qualifiers
 Columns
 Cells
 Versions

Tables and Rows

Table
 Stores data
 Has a table name
Row
 Self-describing
 Identified by unique row key

Row Key
 Must be lexicographically ordered
 Used for fast retrieval

Column Families
 Group related columns together
 Defined during table creation
 Cannot be changed later
Example:
create 'EMPLOYEE', 'Name', 'Address', 'Details'

Column Qualifiers
 Defined dynamically during insertion
 Makes data self-describing

Column
A column is:

ColumnFamily : ColumnQualifier
Example:
Name:Fname
Details:Job

Data Insertion Example

put 'EMPLOYEE', 'row1', 'Name:Fname', 'John'
put 'EMPLOYEE', 'row1', 'Details:Job', 'Engineer'

Versions and Timestamps

HBase stores:
 Multiple versions of data
Each version has:
 Timestamp
Timestamp:
 Represents creation time

Timestamp System
HBase uses:
 January 1, 1970 UTC as zero time
Timestamp stored in:
 Milliseconds

Cells
A cell stores:
 Basic data item
Cell key contains:
 Table
 RowID
 Column Family
 Column Qualifier
 Timestamp

Namespace
Namespace:
 Collection of tables
Similar to:
 Database in relational DBMS

24.5.2 HBase CRUD Operations

Main CRUD Operations
Operation Purpose
create Create table
put Insert data
get Retrieve single row
scan Retrieve all rows

Create Table
create 'EMPLOYEE', 'Name', 'Address', 'Details'

Insert Data
put 'EMPLOYEE', 'row1', 'Name:Fname', 'John'

Retrieve Single Row

get 'EMPLOYEE', 'row1'

Retrieve All Rows

scan 'EMPLOYEE'

Important Point
HBase provides:
 Only low-level CRUD operations
Complex operations like:
 Joins
must be handled by:
 Application programs
24.5.3 HBase Storage and Distributed
Concepts
Regions
 Tables divided into regions
 Each region stores range of row keys

Stores
 Each column family stored separately
 Stored inside regions

Region Servers
 Regions assigned to region servers
 Region servers act as storage nodes

Master Server
Responsible for:
1. Monitoring region servers
2. Splitting tables into regions
3. Assigning regions

Zookeeper
Apache ZooKeeper used for:
 Coordination
 Synchronization
 Replication
 Node management

HDFS
Hadoop Distributed File System used for:
 Distributed file storage

HBase Architecture
HBase built on:
 HDFS
 Zookeeper

Advantages of Column-Based Systems

1. High scalability
2. Fast data access
3. Efficient storage
4. Supports big data
5. Version management
Disadvantages
1. No complex query support
2. Joins not supported directly
3. Complex application logic needed

Applications
 Gmail
 Big data analytics
 Cloud computing
 Distributed storage systems

Important Exam Points

 HBase is a column-based NoSQL system
 Uses column families and qualifiers
 Stores multiple versions using timestamps
 CRUD operations:
o create
o put
o get
o scan
 HBase built on:
o HDFS
o Zookeeper
 Regions used for distributed storage

24.6 NoSQL Graph Databases and Neo4j –

Important Lecture Notes
Introduction to Graph Databases
 Graph databases are a type of NoSQL database.
 Data is represented as:
o Nodes (Vertices)
o Relationships (Edges)
 Both nodes and relationships can store data.
Example
 Person → works in → Department
 Employee → works on → Project

Neo4j Overview
 Neo4j is a popular graph database.
 Open-source system.
 Implemented in Java.
 Used for applications involving connected data.

24.6.1 Neo4j Data Model

1. Nodes
 Nodes represent entities.
 Similar to entities in ER model.
Examples
 EMPLOYEE
 DEPARTMENT
 PROJECT
 LOCATION
Example
CREATE (e1: EMPLOYEE {Empid:'1', Fname:'John'})

2. Labels
 Labels identify node types.
 Nodes with same label belong to same group.
Example
EMPLOYEE
DEPARTMENT
PROJECT
Multiple Labels
A node can have multiple labels.
Example:
PERSON:EMPLOYEE:MANAGER

3. Properties
 Properties store data values.
 Written inside { }.
Example
{Name:'John', Salary:'50000'}

4. Relationships
 Relationships connect nodes.
 Relationships are directed.
Example
(e1)-[:WorksFor]->(d1)
Meaning:
Employee works for department.

5. Relationship Types
Examples:
 WorksFor
 Manager
 LocatedIn
 WorksOn

6. Relationship Properties
Relationships can also store data.
Example
(e1)-[:WorksOn {Hours:'32.5'}]->(p1)
Here:
 Hours is relationship property.
7. Paths
 Path = traversal through graph.
 Used in graph queries.
Example
Employee → WorksOn → Project

8. Optional Schema
 Neo4j does not require schema.
 Schema can be added optionally.
Supports:
 Constraints
 Indexes

9. Indexing
 Neo4j creates unique internal node identifiers.
 Indexes improve search speed.
Examples
 Empid index for EMPLOYEE
 Dno index for DEPARTMENT
 Pno index for PROJECT

24.6.2 Cypher Query Language

Cypher
 Query language of Neo4j.
 Used to:
o Create data
o Retrieve data
o Update data
o Delete data

Important Cypher Clauses

Clause Purpose
CREATE Create nodes/relationships
MATCH Find pattern
WHERE Specify condition
RETURN Display output
ORDER BY Sort output
LIMIT Restrict number of results
DELETE Remove data
SET Update property
REMOVE Remove property

Creating Nodes
Example
CREATE (e1: EMPLOYEE {Empid:'1', Fname:'John'})
Creating Relationships
Example
CREATE (e1)-[:WorksFor]->(d1)

Example Queries
1. Find Department Locations
MATCH (d:DEPARTMENT {Dno:'5'})-[:LocatedIn]->(loc)
RETURN [Link], [Link]

2. Find Projects of Employee

MATCH (e:EMPLOYEE {Empid:'2'})-[w:WorksOn]->(p)
RETURN [Link], [Link], [Link]

3. Employees Working on Project

MATCH (e)-[w:WorksOn]->(p:PROJECT {Pno:2})
RETURN [Link], [Link], [Link]

4. Sorting Results
ORDER BY [Link]

5. Limiting Results
LIMIT 10

6. Using WHERE Clause

WHERE numOfprojs > 2

7. Updating Property
MATCH (e:EMPLOYEE {Empid:'2'})
SET [Link]='Engineer'

24.6.3 Neo4j Features and Distributed

Characteristics
1. Enterprise and Community Editions
Both Editions Support
 Graph database model
 Cypher query language
 ACID properties
Enterprise Edition Additional Features
 Caching
 Clustering
 Locking
 Better performance

2. Graph Visualization
 Neo4j can display data graphically.
  Useful for:
o Query visualization
o Relationship analysis

3. Master-Slave Replication
 One master node controls updates.
 Slave nodes contain replicated data.
Advantages:
 High availability
 Fault tolerance

4. Caching
 Main memory cache improves speed.

5. Logical Logs
 Logs used for recovery after failures.

Advantages of Graph Databases

 Efficient relationship handling
 Fast graph traversal
 Flexible schema
 Suitable for connected data

Applications of Neo4j
 Social networks
 Recommendation systems
 Fraud detection
 Network analysis
 Knowledge graphs

Important Exam Points

Neo4j Important Concepts
 Nodes
 Relationships
 Labels
 Properties
 Paths
 Cypher queries
Important Cypher Commands
 CREATE
 MATCH
 RETURN
 WHERE
 SET
 DELETE
Neo4j Features
 Graph visualization
 Replication
 Caching
 ACID support