NoSQL Basics
What is NoSQL?
NoSQL stands for “Not Only SQL.”
It refers to a wide class of database systems that store and manage data in non-
relational ways.
Unlike traditional relational databases (RDBMS), NoSQL databases don’t use tables
with fixed schemas or SQL-based queries.
They are designed to handle large-scale data storage and high user loads
efficiently.
Key Characteristics:
1. Schema-free: No fixed schema; fields can vary from document to document.
2. Distributed: Designed for distributed data across many servers.
3. Scalable: Horizontally scalable (by adding more machines).
4. High Performance: Faster data access for big data workloads.
5. Flexible Data Model: Can handle structured, semi-structured, or unstructured data.
Why NoSQL?
To handle Big Data, real-time web apps, IoT data, social media, etc.
RDBMS struggles with scalability and flexibility under such loads.
Examples of NoSQL Databases:
Type Examples
Document-based MongoDB, CouchDB
Key-Value Redis, Riak, DynamoDB
Columnar Cassandra, HBase
Graph-based Neo4j, OrientDB
Storage Architecture
Components of NoSQL Storage:
1. Data Nodes: Store the actual data.
2. Coordinator Nodes: Manage queries and distribute tasks.
3. Replication Nodes: Maintain data copies for fault tolerance.
4. Shards: Subsets of data distributed across servers.
Types of Storage Architectures:
1. In-memory Storage: Data stored in RAM for fast access (e.g., Redis).
2. Disk-based Storage: Data stored on disk; slower but durable (e.g., Cassandra).
3. Hybrid Storage: Combination of both (e.g., MongoDB uses memory-mapped files).
Data Partitioning:
Horizontal Partitioning (Sharding): Splits rows of data across multiple servers.
Ensures scalability and performance for large datasets.
Operations in NoSQL
Basic Operations:
Create: Insert data into the database.
Read: Retrieve data.
Update: Modify existing data.
Delete: Remove data.
CRUD Operations Example (MongoDB):
[Link]({name: "Prafull", age: 21})
[Link]({name: "Prafull"})
[Link]({name: "Prafull"}, {$set: {age: 22}})
[Link]({name: "Prafull"})
Query Model
NoSQL databases do not use SQL queries.
Instead, they use custom APIs or query languages.
Query Models by Type:
1. Key-Value Model: Access data using keys (like a dictionary).
2. Document Model: Query through attributes using JSON/BSON.
3. Columnar Model: Query columns instead of rows.
4. Graph Model: Query using graph traversal algorithms (e.g., Cypher in Neo4j).
Modifying Data Stores and Managing Evolution
Schema Evolution:
Since NoSQL is schema-less, schema changes (like adding new fields) are easy.
Applications handle structure changes dynamically.
Managing Data Evolution:
Versioning: Maintain versions of documents.
Migration Tools: Used to move and update existing data formats.
Backups and Snapshots: For data safety during updates.
Indexing and Ordering Data Sets
Indexing:
Indexes improve query performance.
Common index types:
o Single Field Index
o Compound Index
o Geospatial Index
o Full-text Index
Ordering:
Data can be sorted by keys or attributes.
Some NoSQL databases maintain natural order (like Sorted Sets in Redis).
Managing Transactions and Data Integrity
Transactions:
A transaction is a unit of work that must be Atomic, Consistent, Isolated, and
Durable (ACID).
NoSQL often trades strict ACID compliance for BASE properties:
o Basically Available
o Soft state
o Eventually consistent
Integrity Management:
Use replication and version control.
Application-level constraints instead of database-level constraints.
Using NoSQL in the Cloud
Cloud environments are ideal for NoSQL databases because of their scalability.
Cloud providers like AWS DynamoDB, Google Cloud Firestore, and Azure
Cosmos DB offer managed NoSQL services.
Benefits:
Auto-scaling
High availability
Global replication
Reduced management cost
Scalable Parallel Processing with MapReduce
MapReduce Overview:
Framework for processing large datasets across distributed systems.
Developed by Google.
Phases:
1. Map Phase: Processes and transforms data into key-value pairs.
2. Reduce Phase: Aggregates and summarizes results.
Example:
Word Count problem:
Map: Emit (word, 1)
Reduce: Sum counts for each word.
Used in systems like Hadoop and MongoDB MapReduce.
Analyzing Big Data with Hive
Apache Hive:
Data warehouse tool built on Hadoop.
Allows querying large datasets using HiveQL (similar to SQL).
Features:
Supports structured data analysis.
Converts queries into MapReduce jobs.
Ideal for batch processing of big data.
Surveying Database Internals
Internal Components:
1. Storage Engine
2. Query Processor
3. Transaction Manager
4. Replication Controller
5. Cache Layer
Focus:
Data consistency
Fault tolerance
Performance optimization
Replication and synchronization mechanisms
NoSQL Data Models
Aggregate Models
Aggregates are collections of related data treated as a single unit.
Simplifies data management and retrieval.
Each aggregate is self-contained, similar to an object in object-oriented design.
Document Data Model
Stores data as documents (JSON, BSON, XML).
Flexible schema.
Commonly used in MongoDB, CouchDB.
Example:
{
"student_id": 101,
"name": "Prafull Sharma",
"subjects": ["DBMS", "NoSQL", "AI"]
}
Advantages:
Easy mapping to application objects.
Rich queries.
Dynamic schema evolution.
Key-Value Data Model
Simplest NoSQL type.
Data stored as key-value pairs, like a dictionary.
Example:
Key: 101
Value: {"name":"Prafull","course":"NoSQL"}
Use Case:
Caching, session management, quick lookups.
Example Systems: Redis, Riak, DynamoDB.
Columnar Data Model
Data stored by columns instead of rows.
Excellent for analytical queries and aggregation.
Each column family stores related columns together.
Example Systems:
Apache Cassandra, HBase.
Benefits:
Efficient for queries on specific columns.
Great compression and scalability.
Graph-Based Data Model
Represents data as nodes (entities) and edges (relationships).
Used to model social networks, recommendation systems, etc.
Example:
(Prafull) -[FRIEND]-> (Ravi)
Query Language:
Cypher (Neo4j)
Gremlin
NoSQL Ways to Handle Big Data Problems
1. Sharding (Horizontal Partitioning)
2. Replication (for fault tolerance and read scalability)
3. Caching (in-memory data for fast access)
4. Eventual Consistency (to improve performance)
5. Parallel Processing (e.g., MapReduce)
Moving Queries to Data (Not Data to Query)
Instead of moving large data sets across the network, NoSQL executes queries where
the data resides.
Reduces network latency and cost.
Common in Hadoop, Cassandra, Elasticsearch.
Hash Rings to Distribute Data on Clusters
Consistent Hashing: A method to distribute data evenly across nodes.
Ensures load balancing and easy node addition/removal.
Example:
Each node is assigned a position on a hash ring.
Data items are placed on the node corresponding to their hash value.
Replication to Scale Reads
Replication means maintaining multiple copies of the same data on different servers.
Increases read scalability and fault tolerance.
Types:
Master-Slave Replication
Peer-to-Peer Replication
Database Distributed Queries to Data Nodes
In distributed databases, queries are split and executed on multiple data nodes.
Results are combined and sent back to the client.
Improves performance and parallelism.
Summary Table: NoSQL Data Models
Model Structure Example Systems Key Use Case
Key-Value Key-Value Pairs Redis, DynamoDB Caching, session data
MongoDB,
Document JSON/BSON Documents Content management
CouchDB
Columns grouped by
Columnar Cassandra, HBase Analytical workloads
family
Social networks,
Graph Nodes and Edges Neo4j, OrientDB
recommendations