04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

S Q L V S N O S Q L — PA R T 1

The Big Picture & SQL Deep Dive

Why database choice matters, what SQL databases are, how they work internally, ACID,
JOINs, normalisation, and when SQL is the right choice

Section 01 — The Big Picture: Why Database Choice

Matters
What is a database, what is a DBMS, and the three pillars that decide SQL vs NoSQL

1 What Is a Database?

DEFINITION

A database is an organised collection of structured data stored and managed electronically.

Think of it as a giant, highly organised digital filing cabinet where data can be stored, retrieved,
updated, and deleted in a controlled, efficient manner.

Every app you use — Instagram, WhatsApp, your banking app — relies on a database to store
its data.

2 What Is a DBMS?

DEFINITION

A Database Management System (DBMS) is the software layer that manages the database. It
is the middle-man between your application code and the raw data. It handles security,
querying, indexing, backups, and ensures data integrity.

Example: MySQL (the DBMS) manages a MySQL database. The two main families of DBMS are
Relational (SQL) and Non-Relational (NoSQL).

[Link] 1/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

LIBRARY ANALOGY

A database is like a library — it holds all the books (data).

The DBMS is the librarian — they know where everything is, let authorised people access
books, keep the catalog updated, and ensure books are returned in the right order.

HOW YOUR APP TALKS TO DATA

DBMS
YOUR APPLICATION queries reads/writes DATABASE
(MySQL, PostgreSQL, MongoDB)
([Link], Python, Java) (Actual data on disk)
Security · Querying · Indexing
Sends SQL or API queries Backups · Integrity · Concurrency Tables, Documents, Files

3 The Core Question Every Interview Asks

THE QUESTION

"Would you use SQL or NoSQL here?" — This is one of the most common system design
interview questions. Choosing wrong means you'll either struggle to scale past a few million
users (wrong NoSQL choice) or spend months debugging data inconsistencies that corrupt
your business logic (wrong SQL choice).

The answer always depends on three pillars:

THREE PILLARS OF DATABASE CHOICE

SQL or NoSQL?

① CONSISTENCY MODEL ② SCALABILITY NEED ③ DATA STRUCTURE

Does every read need to see the Well-structured with clear
latest write immediately? 10,000 users or 10 million? relationships? → SQL
10 GB or 10 TB of data?
Or can it tolerate slightly stale Flexible, unstructured, or
data for a few seconds? Vertical vs horizontal scaling? rapidly evolving? → NoSQL

Section 02 — SQL Databases: Full Deep Dive

[Link] 2/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

What SQL is, how relational databases work internally, ACID, JOINs, normalisation, and when
to choose SQL

1 What Is SQL?

DEFINITION

SQL stands for Structured Query Language. It is both the language used to interact with
relational databases and the informal name for relational databases themselves.

SQL databases organise data into tables (like spreadsheets) with rows and columns. Each
table represents one type of entity (Users, Orders, Products), and relationships between tables
are defined using primary and foreign keys.

EXCEL ANALOGY

A SQL database is like a well-organised Excel workbook with multiple sheets. Each sheet has
predefined column headers. Every row must follow the same structure. You can cross-
reference data between sheets using a shared ID column.

SQL DATABASE — TABLES, ROWS, COLUMNS & RELATIONSHIPS

USERS TABLE ORDERS TABLE

id (PK) name email city FOREIGN KEY id (PK) user_id (FK) total status
orders.user_id → [Link]
1 Amit amit@.. Mumbai 101 1 5000 done
2 Priya priya@.. Delhi 102 3 2500 pending
3 Sara sara@.. BLR 103 1 8000 done

Each row = one user. Each column = one field.

HOW RELATIONSHIPS WORK

• PRIMARY KEY (PK): Unique identifier for each row. [Link] = 1, 2, 3 — no duplicates.
• FOREIGN KEY (FK): Links to PK in another table. orders.user_id references [Link].
• This lets you JOIN: "Show me all orders for Amit" → match user_id=1 across both tables.

2 Core RDBMS Concepts You Must Know

Tables, Primary Keys & Foreign Keys

[Link] 3/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

CORE CONCEPTS — TABLES, ROWS, COLUMNS

TABLE = The fundamental building block.

Has rows (records) and columns (fields).
Example: 'Users' table with columns: id, name, email, created_at

PRIMARY KEY (PK) = Unique identifier for every row.

No two rows can share the same PK. Guarantees uniqueness.
Example: user_id = 1, 2, 3...

FOREIGN KEY (FK) = A column that references the PK of another table.

This is how RELATIONSHIPS are enforced.
Example: orders.user_id references [Link]

INDEX = Data structure that speeds up retrieval (B+ Tree).

Without it: scan every row. With it: jump directly.
(You already mastered this in the Indexing chapters!)

Normalisation — Eliminating Redundancy

WHAT IS NORMALISATION?

The process of organising tables to eliminate data redundancy and ensure data integrity. You
break one big table into multiple smaller, related tables so that each piece of data is stored
exactly once.

WHY NORMALISE? — BEFORE AND AFTER

BEFORE NORMALISATION (one big denormalised table):

┌────┬───────┬────────────┬───────────────┬───────────┐
│ id │ name │ email │ order_total │ product │
├────┼───────┼────────────┼───────────────┼───────────┤
│ 1 │ Amit │ amit@.. │ 5000 │ Laptop │
│ 1 │ Amit │ amit@.. │ 8000 │ Phone │ ← Amit's data
│ 2 │ Priya │ priya@.. │ 2500 │ Headphone │ DUPLICATED!
└────┴───────┴────────────┴───────────────┴───────────┘
Problem: Amit's name and email are stored TWICE. If email changes,
you must update EVERY row — miss one and data is inconsistent.

AFTER NORMALISATION (split into related tables):

Users: Orders:

[Link] 4/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

┌────┬───────┬─────────┐ ┌─────┬─────────┬───────┬───────────┐
│ id │ name │ email │ │ id │ user_id │ total │ product │
├────┼───────┼─────────┤ ├─────┼─────────┼───────┼───────────┤
│ 1 │ Amit │ amit@.. │ │ 101 │ 1 │ 5000 │ Laptop │
│ 2 │ Priya │ priya@..│ │ 102 │ 2 │ 2500 │ Headphone │
└────┴───────┴─────────┘ │ 103 │ 1 │ 8000 │ Phone │
└─────┴─────────┴───────┴───────────┘
Amit's data stored ONCE. Orders reference via FK (user_id=1).
Change email? Update ONE row. Always consistent.

JOINs — SQL's Superpower

A JOIN is an SQL operation that combines rows from two or more tables based on a related
column. This is what makes relational databases powerful — you split data across tables
(normalisation) and then recombine it on the fly with JOINs.

THE FOUR TYPES OF JOINS

SELECT [Link], [Link], [Link]

FROM users u
JOIN orders o ON [Link] = o.user_id;
-- Result: Amit|5000|Laptop, Amit|8000|Phone, Priya|2500|Headphone

INNER JOIN: Only rows that have a match in BOTH tables.

Users with orders + their orders. Users without orders excluded.

LEFT JOIN: ALL rows from left table + matching rows from right.
ALL users + their orders (if any). Users without orders show NULL.

RIGHT JOIN: ALL rows from right table + matching rows from left.
ALL orders + their user info. Orphan orders show NULL user.

FULL JOIN: ALL rows from BOTH tables. Non-matching show NULL.
Every user and every order, matched where possible.

FOUR TYPES OF JOINS — VISUAL

A B

INNER JOIN LEFT JOIN RIGHT JOIN FULL JOIN

Only matching rows All left + matching right All right + matching left All from both tables

[Link] 5/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

3 ACID — The Guarantee SQL Databases Provide

WHAT IS ACID?

ACID is a set of four properties that guarantee database transactions are processed reliably.
This is the cornerstone of SQL databases and the reason banks, payment systems, and
anything involving money use SQL.

ACID — THE FOUR GUARANTEES

A = ATOMICITY
"All or nothing." A transaction either completes fully or not at all.
If transferring ₹5000 from Account A to B:
Step 1: Debit A by 5000
Step 2: Credit B by 5000
If Step 2 fails, Step 1 is ROLLED BACK. Money doesn't vanish.

C = CONSISTENCY
"Always valid state." The database moves from one valid state to another.
Constraints (foreign keys, unique, NOT NULL) are always enforced.
You can never have an order pointing to a user_id that doesn't exist.

I = ISOLATION
"Transactions don't interfere." Multiple transactions running at
the same time don't see each other's partial changes.
User A transferring money and User B checking balance simultaneously
— B sees either the state BEFORE or AFTER, never a half-done state.

D = DURABILITY
"Once committed, it's permanent." After a transaction commits,
the data survives even if the server crashes or loses power.
The database writes to disk (write-ahead log) before confirming.

ACID — WHY BANKS USE SQL

A C I D
Atomicity Consistency Isolation Durability
All or nothing. Always valid state. No interference. Once committed,
Debit+Credit both Constraints always Concurrent txns don't it's permanent. Survives
happen, or neither. enforced. No bad data. see partial changes. crashes & power loss.

[Link] 6/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

BANK TRANSFER ANALOGY

You transfer ₹5000 from your account to a friend. Atomicity ensures the money is both
debited from you AND credited to your friend — or neither happens. Consistency ensures total
money in the system doesn't change. Isolation ensures if someone checks your balance mid-
transfer, they see the old or new balance, not a weird in-between. Durability ensures once the
transfer is confirmed, it stays even if the bank's server crashes.

4 The SQL Query Language — What You Can Do

SQL is divided into four sub-languages, each for a different type of operation:

TYPE FULL NAME COMMANDS PURPOSE

DDL Data Definition CREATE , ALTER , DROP Define and modify database
Language structure (tables, columns)

DML Data Manipulation SELECT , INSERT , Read and write actual data
Language UPDATE , DELETE

DCL Data Control GRANT , REVOKE Manage permissions and access

Language control

TCL Transaction Control COMMIT , ROLLBACK , Manage transactions (ACID

Language SAVEPOINT guarantees)

QUICK EXAMPLES OF EACH SUB-LANGUAGE

DDL — Define structure:

CREATE TABLE users (id INT PRIMARY KEY, name VARCHAR(100));
ALTER TABLE users ADD COLUMN email VARCHAR(255);
DROP TABLE users;

DML — Read/write data:

SELECT * FROM users WHERE city = 'Mumbai';
INSERT INTO users (name, email) VALUES ('Amit', 'amit@[Link]');
UPDATE users SET email = 'new@[Link]' WHERE id = 1;
DELETE FROM users WHERE id = 5;

DCL — Permissions:
[Link] 7/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

GRANT SELECT ON users TO readonly_user;

REVOKE DELETE ON users FROM junior_dev;

TCL — Transactions:
BEGIN;
UPDATE accounts SET balance = balance - 5000 WHERE id = 1;
UPDATE accounts SET balance = balance + 5000 WHERE id = 2;
COMMIT; -- both succeed together (atomicity!)
-- or ROLLBACK; if something goes wrong

5 Popular SQL Databases

DATABASE LICENCE BEST FOR WHO USES IT

PostgreSQL Open Complex queries, extensibility, Instagram, Apple, Twitch

Source JSON support

MySQL Open Web apps, LAMP stack, e- Facebook (early), WordPress,

Source commerce Airbnb

SQL Server Commercial Enterprise Windows systems, BI Stack Overflow, Dell

Oracle DB Commercial Mission-critical enterprise systems Banks, telecom, government

SQLite Open Embedded, mobile, local storage Android apps, browsers,

Source Firefox

6 SQL Scaling — Vertical vs Horizontal

THE SCALING CHALLENGE

SQL databases traditionally scale vertically — you make the single server bigger (more RAM,
faster CPU, bigger disk). This works until you hit hardware limits. Horizontal scaling (adding
more servers) is possible but complex for SQL because JOINs across servers are hard and
maintaining ACID across distributed nodes is expensive.

[Link] 8/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

VERTICAL VS HORIZONTAL SCALING

VERTICAL SCALING ("Scale Up"):

One server → make it BIGGER
4 GB RAM → 32 GB RAM → 256 GB RAM → 1 TB RAM
✓ Simple — no code changes
✓ ACID stays simple (one machine)
✗ Has a ceiling — can't buy infinite hardware
✗ Single point of failure
✗ Expensive (doubling RAM = more than double cost)

HORIZONTAL SCALING ("Scale Out"):

One server → add MORE servers
1 server → 5 servers → 50 servers → 500 servers
✓ No ceiling — keep adding machines
✓ Redundancy (one fails, others handle it)
✗ Complex for SQL — JOINs across servers?
✗ ACID across distributed nodes = very hard
✗ Need sharding, replication, consensus protocols

SQL databases CAN scale horizontally (sharding, read replicas),

but it's much harder than with NoSQL databases that were BUILT for it.

VERTICAL SCALING vs HORIZONTAL SCALING

VERTICAL (Scale Up) HORIZONTAL (Scale Out)

↑ Server Srv 1 Srv 2 Srv 3 Srv 4

BIGGER Server
More RAM, CPU, Disk
Srv 5 Srv 6 →keep adding!

7 SQL Advantages & Disadvantages

✅ SQL ADVANTAGES ❌ SQL DISADVANTAGES

→ Strong ACID transaction guarantees → Vertical scaling hits hardware limits
→ Rich JOINs & aggregation query power → Horizontal sharding is complex to
→ Enforced schema prevents bad data entry implement

[Link] 9/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

→ 40+ years of tools, ecosystem, → Schema changes need careful migrations

documentation → Not ideal for unstructured data (logs,
→ Easy to find skilled developers JSON)
→ Excellent for complex reporting & BI → Can bottleneck at very high write loads
→ Foreign key constraints ensure referential → JOINs across huge tables are slow
integrity → Fixed schema slows rapid prototyping
→ Well-understood backup & recovery → Poor fit for hierarchical or graph data
strategies

8 When to Choose SQL — Decision Framework

CHOOSE SQL WHEN

1. Your data has clear relationships (users have orders, orders have items)
2. You need ACID transactions (money transfers, inventory, bookings)
3. Your schema is well-defined and stable (not changing every week)
4. You need complex queries — JOINs, aggregations, GROUP BY, subqueries
5. Data integrity is more important than raw write speed
6. You need reporting and analytics on structured data

CLASSIC SQL USE CASES

Banking & Financial Systems → ACID is non-negotiable. Money can't vanish.

E-Commerce → Products, orders, payments — clear relationships.
Healthcare → Patient records need strict integrity.
ERP / CRM Systems → Complex business rules, many entity relationships.
Inventory Management → Stock counts must be exact (no overselling).
Booking Systems (Hotels/Flights) → Double-booking = disaster. Need isolation.
Reporting & BI Dashboards → Complex JOINs and aggregations across tables.

📋 SQL Deep Dive — Quick Revision

[Link] 10/11
04/04/2026, 12:57 SQL vs NoSQL — Part 1: The Big Picture & SQL Deep Dive

WHAT IS SQL?
Structured Query Language. Data in tables (rows + columns). Relationships via primary/foreign
keys. Schema enforced — every row must match the structure.

CORE CONCEPTS
Table = rows + columns. PK = unique row ID. FK = link to another table's PK. Index = fast lookup
(B+ Tree). Normalisation = eliminate redundancy.

JOINS — SQL'S SUPERPOWER

INNER = matching rows only. LEFT = all left + matching right. RIGHT = all right + matching left.
FULL = everything from both.

ACID GUARANTEES
Atomicity (all or nothing) · Consistency (always valid) · Isolation (no interference) · Durability
(permanent after commit)

SQL SUB-LANGUAGES
DDL = CREATE/ALTER/DROP (structure). DML = SELECT/INSERT/UPDATE/DELETE (data). DCL =
GRANT/REVOKE (permissions). TCL = COMMIT/ROLLBACK (transactions).

SCALING
Vertical (scale up) = bigger server. Simple but hits ceiling.
Horizontal (scale out) = more servers. Possible but complex for SQL (JOINs across nodes,
distributed ACID).

WHEN TO CHOOSE SQL

Clear relationships · ACID needed · Stable schema · Complex queries/JOINs · Data integrity
critical · Reporting/BI

POPULAR DATABASES
PostgreSQL (complex queries, JSON). MySQL (web apps, LAMP). SQL Server (enterprise/BI).
Oracle (mission-critical). SQLite (embedded/mobile).

[Link] 11/11
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

S Q L V S N O S Q L — PA R T 2

NoSQL Deep Dive & ACID vs BASE

All 6 types of NoSQL databases explained, when to use each, eventual consistency, CAP
theorem, and the ACID vs BASE battle

Section 03 — NoSQL Databases: Full Deep Dive

6 types of NoSQL, what each stores, how they work internally, and when to pick which

1 What Is NoSQL?

DEFINITION

NoSQL stands for "Not Only SQL". It is a broad family of databases that don't use the
relational table model. Instead of one single design (tables + SQL), NoSQL offers multiple
data models, each optimised for a specific type of data or workload.

The common thread: they sacrifice some consistency guarantees in exchange for horizontal
scalability, schema flexibility, and high availability.

STORAGE TOOLS ANALOGY

If SQL is one type of organised filing cabinet (rigid but reliable), NoSQL is a collection of very
different storage tools — a backpack, a whiteboard, a spreadsheet, a corkboard — each
perfect for a specific purpose. You pick the tool that fits your data, not force your data into one
shape.

2 The 6 Types of NoSQL Databases

INTERVIEW CRITICAL

[Link] 1/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

You must know all 6 types, what they store, and when to use them. Each type is designed for a
fundamentally different kind of data. Interviewers will ask: "What type of NoSQL would you use
for X?" — and you need to pick the right one with reasoning.

THE 6 TYPES OF NoSQL DATABASES

① KEY-VALUE ④ GRAPH
Giant hash map. GET/SET. Nodes + edges. Relationships.
Redis, DynamoDB Neo4j, Neptune

NoSQL
② DOCUMENT 6 Types
⑤ IN-MEMORY
JSON docs, flexible schema. RAM-first. Microsecond latency.
MongoDB, Firestore Redis, Memcached

③ WIDE-COLUMN ⑥ TIME-SERIES
Flexible cols per row. Massive writes. Timestamped data. Metrics.
Cassandra, HBase InfluxDB, Prometheus

TYPE 1

Key-Value Store — The Simplest NoSQL (Giant Hash Table)

GET key, SET key value, DELETE key — nothing more, nothing less

WHAT IS IT?

Stores data as simple key → value pairs. The key is a unique identifier. The value can
be anything: a string, number, JSON blob, or binary data. The database doesn't care
what's inside the value — it just stores and retrieves it by key. Operations are extremely
fast: GET , SET , DELETE .

KEY-VALUE STORE — HOW IT LOOKS

KEY VALUE
─────────────────────────────────────────────────
"user:1001:session" → "abc123xyz789"
"user:1001:cart" → {"items": [{"id": 42, "qty": 2}]}
"rate_limit:ip:1.2.3" → "47"
"cache:product:555" → "{name: 'Laptop', price: 75000}"
"leaderboard:game1" → [{"user": "amit", "score": 9500}, ...]

Operations:

[Link] 2/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

SET "user:1001:session" "abc123xyz789" → store

GET "user:1001:session" → retrieve → "abc123xyz789"
DELETE "user:1001:session" → remove
TTL "user:1001:session" 3600 → auto-expire in 1 hour

Use Cases: Session storage, user preferences, shopping carts, rate limiting, real-time
leaderboards, caching results of expensive queries.
Examples: Redis (most popular), Amazon DynamoDB, Azure Cosmos DB, Memcached.

WHY IT'S SO FAST

It's essentially a giant hash table. hash(key) → memory location → value. O(1) lookups.
Redis stores everything in RAM — reads take microseconds, not milliseconds. This is
why it's used as a caching layer in front of slower databases.

TYPE 2

Document Store — JSON Documents, Flexible & Powerful

Self-contained documents with nested data — no schema required

WHAT IS IT?

Stores data as self-contained documents — usually JSON or BSON format. Each

document can have different fields (schema-less). Documents can be nested (e.g., an
Order document containing an array of Product sub-documents). Ideal when your data
is naturally hierarchical.

DOCUMENT STORE — A USER PROFILE DOCUMENT (MONGODB STYLE)

// In SQL: this would need 3 tables (users, addresses, preferences)

// In MongoDB: ONE document holds everything

{
"_id": "user_1001",
"name": "Amit Sharma",
"email": "amit@[Link]",
"age": 28,
"address": { ← nested object
"city": "Mumbai",
"state": "Maharashtra",
"pin": "400001"

[Link] 3/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

},
"orders": [ ← nested array
{"id": "ord_501", "total": 5000, "status": "delivered"},
{"id": "ord_502", "total": 2500, "status": "pending"}
],
"preferences": { ← flexible fields
"theme": "dark",
"notifications": true
},
"tags": ["premium", "early_adopter"] ← arrays
}

No JOIN needed — everything for this user is in ONE document.

Different users can have DIFFERENT fields (schema-less).

Use Cases: User profiles, product catalogs, blog posts with comments, content
management systems, e-commerce catalogs, configuration storage.
Examples: MongoDB (most popular), CouchDB, Amazon DocumentDB, Firestore
(Firebase).

SQL VS DOCUMENT — THE TRADE-OFF

SQL: User data split across users, addresses, orders tables → needs JOINs → data
always consistent → but slower for "get everything about this user."

Document: Everything for a user in ONE document → no JOINs → blazing fast reads →
but if you update the address format, you must update each document individually.

TYPE 3

Wide-Column Store — Tables but with Flexible Columns per Row

Massive writes, time-series, billions of events — built for distribution

WHAT IS IT?

Similar to SQL tables but each row can have completely different columns. Data is
stored in column families rather than rows. Optimised for writing massive amounts of
data and for time-series queries ("give me all records between time A and time B").
Built for distribution across many nodes.

WIDE-COLUMN VS SQL TABLE

[Link] 4/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

SQL TABLE (every row has SAME columns):

┌─────────┬───────┬────────┬───────┬────────┐
│ user_id │ name │ email │ phone │ city │
├─────────┼───────┼────────┼───────┼────────┤
│ 1 │ Amit │ a@.. │ 9876 │ Mumbai │
│ 2 │ Priya │ p@.. │ 8765 │ Delhi │ ← every row, same columns
│ 3 │ Sara │ s@.. │ 7654 │ BLR │
└─────────┴───────┴────────┴───────┴────────┘

WIDE-COLUMN (each row can have DIFFERENT columns):

Row Key │ Columns (vary per row!)
──────────┼──────────────────────────────────────────
user:1 │ name="Amit" email="a@.." phone="9876" city="Mumbai"
user:2 │ name="Priya" email="p@.." age="25" ← no phone, has age
user:3 │ name="Sara" twitter="@sara" ← only 2 columns!

Each row is identified by a ROW KEY.

Columns are grouped into COLUMN FAMILIES.
Rows don't need to share the same columns.

Use Cases: Time-series data, IoT sensor logs, activity feeds, messaging platforms,
telemetry, analytics, write-heavy workloads with billions of events.
Examples: Apache Cassandra (most popular), HBase, Google Bigtable (the original),
Azure Table Storage.

WHY CASSANDRA HANDLES MASSIVE WRITES

Cassandra uses a Log-Structured Merge Tree (LSM Tree) — it writes sequentially to

an in-memory table, then flushes to disk. No random I/O on writes. This lets it handle
millions of writes per second across distributed nodes. Instagram uses Cassandra to
store billions of user activity events.

TYPE 4

Graph Database — Nodes, Edges, and Relationships

When relationships ARE the data — social networks, fraud detection, recommendations

WHAT IS IT?

Data is stored as nodes (entities) and edges (relationships between entities). Ideal
when the relationships between data are just as important as the data itself.

[Link] 5/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

Traversing millions of connections is extremely fast — impossible in SQL without many

expensive JOIN operations.

GRAPH DATABASE — SOCIAL NETWORK EXAMPLE

[Amit] ──FOLLOWS──→ [Priya]

│ │
FOLLOWS FOLLOWS
↓ ↓
[Sara] ──FOLLOWS──→ [Rahul]
│
LIKES
↓
[Post #42]

NODES = Amit, Priya, Sara, Rahul, Post #42 (entities)

EDGES = FOLLOWS, LIKES (relationships)

Query: "Find all friends-of-friends of Amit"

SQL: Multiple self-JOINs on a huge table → SLOW
Graph: Traverse 2 edges from Amit → INSTANT

Query: "Find everyone who follows someone that Amit follows"

SQL: Complex subquery with JOINs → gets exponentially slower
Graph: Traverse edges → stays fast regardless of data size

Use Cases: Social networks (friend-of-friend), fraud detection (suspicious patterns),

recommendation engines, knowledge graphs, network topology, access control
systems.
Examples: Neo4j (most popular), Amazon Neptune, ArangoDB, TigerGraph.

WHY NOT JUST USE SQL FOR RELATIONSHIPS?

SQL handles relationships via JOINs. For "friends of friends," you need a self-JOIN. For
"friends of friends of friends," you need another JOIN. Each level of depth adds
another JOIN — and SQL JOINs get exponentially slower. Graph databases store
relationships as first-class citizens — traversing 5 levels deep is just as fast as
traversing 1 level. That's the fundamental difference.

[Link] 6/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE
TYPE 5

In-Memory Database — RAM-First, Microsecond Latency

Eliminates disk I/O entirely — orders of magnitude faster

WHAT IS IT?

Data is stored primarily in RAM (memory) rather than on disk. This eliminates disk I/O,
making reads and writes orders of magnitude faster. Trade-off: data is volatile (lost on
power failure) unless persistence mechanisms are enabled. Used as a speed layer in
front of other databases.

IN-MEMORY AS A CACHING LAYER

Without cache:
App → Database (disk) → 5-50ms per read

With in-memory cache (Redis):

App → Redis (RAM) → 0.1ms per read ← 50-500x FASTER
↓ (cache miss)
Database (disk) → fill cache → next time: from RAM

TYPICAL PATTERN:
1. App checks Redis first (cache hit? → return instantly)
2. Cache miss → query actual database
3. Store result in Redis with TTL (e.g., expire in 5 minutes)
4. Next request for same data → served from RAM

This is called CACHE-ASIDE or READ-THROUGH pattern.

Use Cases: Caching (most common), session management, real-time leaderboards,

pub/sub messaging, rate limiting, temporary data that changes rapidly.
Examples: Redis (also a key-value store), Memcached, Apache Ignite, Hazelcast, SAP
HANA.

PERSISTENCE IN REDIS

Redis supports two persistence modes so data isn't lost on restart:

RDB (snapshotting): Periodic point-in-time snapshots to disk.
AOF (Append-Only File): Logs every write command to disk. More durable but slightly

[Link] 7/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

slower.
Most production Redis setups use both for safety.

TYPE 6

Time-Series Database — Optimised for Timestamped Data

Millions of data points per second, chronological queries, automatic downsampling

WHAT IS IT?

Specialised database for storing data points indexed by time. Data is naturally ordered
chronologically. Highly optimised for ingesting millions of events per second, and for
queries like "average CPU over the last 5 minutes" or "all readings between 9am and
10am."

TIME-SERIES DATA — WHAT IT LOOKS LIKE

METRIC: server_cpu_usage
┌─────────────────────┬───────────┬──────────┐
│ timestamp │ server_id │ cpu_% │
├─────────────────────┼───────────┼──────────┤
│ 2024-03-15 09:00:01 │ srv-01 │ 45.2 │
│ 2024-03-15 09:00:01 │ srv-02 │ 72.8 │
│ 2024-03-15 09:00:02 │ srv-01 │ 47.1 │
│ 2024-03-15 09:00:02 │ srv-02 │ 71.5 │
│ 2024-03-15 09:00:03 │ srv-01 │ 52.3 │
│ ...millions more... │ │ │
└─────────────────────┴───────────┴──────────┘

SPECIAL FEATURES:
• Automatic downsampling: keep per-second data for 1 day,
per-minute data for 1 month, per-hour data for 1 year
• Built-in aggregation: AVG, MIN, MAX, PERCENTILE over time windows
• Compression: timestamps compress extremely well (sequential)
• Retention policies: auto-delete data older than X days

Use Cases: DevOps monitoring (CPU, memory metrics), IoT sensor data, financial tick
data, application performance monitoring (APM), smart grid data, weather data.

[Link] 8/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

Examples: InfluxDB, Prometheus (+ Grafana), TimescaleDB, Amazon Timestream,

Graphite.

3 All 6 NoSQL Types — Quick Comparison

TOP
TYPE DATA MODEL BEST FOR
EXAMPLE

Key-Value key → value (any blob) Caching, sessions, rate limiting Redis

Document JSON/BSON documents User profiles, catalogs, CMS MongoDB

Wide- Row key + flexible IoT, time-series, massive writes Cassandra

Column columns

Graph Nodes + edges Social networks, fraud, Neo4j

recommendations

In-Memory RAM-first (any model) Caching layer, real-time data Redis

Time-Series Timestamped data points Monitoring, IoT metrics, APM InfluxDB

4 NoSQL Advantages & Disadvantages

✅ NOSQL ADVANTAGES ❌ NOSQL DISADVANTAGES

→ Horizontal scaling — add nodes easily → Eventual consistency — stale reads
→ Schema-less — iterate without migrations possible

→ Built for massive write throughput → No native JOINs — complex queries are
hard
→ High availability and fault tolerance
→ Limited multi-document transaction
→ Each type optimised for its use case
support
→ Multi-region distribution is built-in
→ No enforced schema = risky data quality

[Link] 9/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

→ Great for unstructured/semi-structured → Each type needs specialised knowledge

data → Less mature tooling and BI integration
→ Often faster for simple key-based lookups → Higher risk of data inconsistency bugs
→ Debugging distributed issues is complex

Section 04 — ACID vs BASE: The Consistency Battle

Two philosophies: correctness vs availability. Understanding both is essential for database
interviews.

1 ACID — The SQL Standard (Recap with Depth)

ACID is a set of four properties that guarantee database transactions are processed reliably
and that data is never left in a corrupted state, even during system failures. Every major SQL
database is ACID-compliant by default.

ACID — THE FOUR GUARANTEES (IN DEPTH)

A = ATOMICITY — "All or Nothing"

Every operation in a transaction completes fully or not at all.
Bank transfer: debit Account A AND credit Account B — both succeed
or both are rolled back. Money NEVER vanishes between accounts.

C = CONSISTENCY — "Always Valid"

Transaction must bring DB from one valid state to another.
Rule: balance >= 0. A transaction making balance negative is REJECTED.
Foreign key: can't have order pointing to non-existent user.

I = ISOLATION — "Don't Interfere"

Concurrent transactions execute as if sequential.
Two users buying the LAST concert ticket simultaneously:
Only one succeeds. The other sees "already sold" — not a partial state.

D = DURABILITY — "Never Forgotten"

Once committed, data survives crashes and power failures.

[Link] 10/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

Written to disk (WAL — Write-Ahead Log) before confirming.

Server crashes 1ms after commit? Data is NOT lost.

2 BASE — The NoSQL Philosophy

WHAT IS BASE?

BASE is the alternative philosophy adopted by most distributed NoSQL databases. It relaxes
ACID's strict consistency guarantees to achieve higher availability and easier horizontal
scaling.

BASE stands for: Basically Available, Soft State, Eventually Consistent.

BASE — THE THREE PROPERTIES EXPLAINED

BA = BASICALLY AVAILABLE
The system guarantees SOME response to every request,
even during partial failures. The response might be stale
or from a replica, but the system never goes completely down.

Example: During Amazon's Black Friday sale, the site stays UP

and accepts orders — even if some product counts are slightly off.

S = SOFT STATE
The system's state may change over time even WITHOUT new inputs,
as data propagates between nodes asynchronously.

Example: A social media post you just liked shows 100 likes to you,
but still shows 99 to someone in another region — the update
hasn't propagated to their node yet.

E = EVENTUALLY CONSISTENT
Given enough time with no new updates, ALL nodes will converge
to the same data. Consistency is guaranteed EVENTUALLY — not
immediately.

Example: After editing your Twitter profile, it appears updated

to you instantly, but a friend in another country might see the
OLD bio for a few seconds until the change propagates.

[Link] 11/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

EVENTUAL CONSISTENCY — HOW IT WORKS

T = 0s
Node A Node B Node C
Write arrives Reads from B or C return STALE data
✓ Updated ✗ Stale ✗ Stale

T = 2s
Node A Node B Node C
Propagating B is catching up, C still behind
✓ Updated ⟳ Syncing ✗ Stale

T = 5s
Node A Node B Node C
Converged! ALL nodes now have same data ✓
✓ Updated ✓ Updated ✓ Updated

This is "Eventually Consistent"

3 ACID vs BASE — Head-to-Head

DIMENSION ACID (SQL) BASE (NOSQL)

Consistency Strong — immediate, always Eventual — converges over time

Availability May sacrifice for consistency Prioritised above consistency

Transactions Multi-row, multi-table ACID Single-document or limited

Failure Handling Roll back entire transaction Resolve conflicts after the fact

Scalability Harder to scale horizontally Built for horizontal scale

Data Integrity Enforced by the database Enforced by application code

Latency Higher (coordination overhead) Lower (no sync coordination)

Use When Financial, inventory, medical Social, analytics, IoT, sessions

Example DBs MySQL, PostgreSQL, Oracle Cassandra, DynamoDB, MongoDB

ACID vs BASE — THE SPECTRUM

ACID Most systems use BASE

Strong consistency a MIX of both! Eventual consistency
Lower availability High availability
Vertical scaling Horizontal scaling
Banks, payments Social, IoT, analytics

KEY INTERVIEW INSIGHT — AMAZON'S TWO SYSTEMS

[Link] 12/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

Amazon's shopping cart uses BASE/eventual consistency intentionally. They would rather
show a slightly stale cart than fail to load the page.

But Amazon's payment processing uses ACID/strong consistency — you can NEVER have
eventual consistency for money. The payment must either go through completely or not at all.

Real systems use BOTH. The same company picks ACID for critical paths and BASE for
everything else. This is the nuanced answer interviewers want.

4 CAP Theorem — The Distributed Systems Trade-off

CAP THEOREM ( BREWER'S THEOREM )

In a distributed system, you can only guarantee two out of three properties at the same time:

C = Consistency — every read gets the most recent write

A = Availability — every request gets a response (not an error)
P = Partition Tolerance — the system works even if network between nodes breaks

Since network partitions always happen in distributed systems, you really choose between
CP (consistency + partition tolerance) or AP (availability + partition tolerance).

CAP THEOREM — PICK TWO (but P is mandatory)

C
Consistency

AP Systems CP Systems
Cassandra, DynamoDB, CouchDB MongoDB, HBase, Redis
Available but may serve stale data Consistent but may reject reads

A P
Availability Partition Tol.

CA = only works if no network issues (single node / not distributed)

CAP IN PRACTICE — WHICH DATABASES CHOOSE WHAT?

CP (Consistency + Partition Tolerance):

[Link] 13/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

→ System may become UNAVAILABLE during network partition

→ But data is always CONSISTENT when you can read it
→ Examples: MongoDB (default), HBase, Redis Cluster, Zookeeper

AP (Availability + Partition Tolerance):

→ System ALWAYS responds, even during network issues
→ But response might contain STALE data
→ Examples: Cassandra, DynamoDB, CouchDB, Riak

CA (Consistency + Availability):
→ Only possible on SINGLE node (no partitions)
→ Traditional single-server SQL databases (PostgreSQL, MySQL)
→ Not truly distributed — if that node fails, everything fails

REALITY: Most modern databases let you TUNE the trade-off.

Cassandra: adjustable consistency (ONE, QUORUM, ALL)
MongoDB: configurable read/write concern
DynamoDB: strongly consistent reads available (at higher cost)

📋 NoSQL & ACID vs BASE — Quick Revision

WHAT IS NOSQL?
"Not Only SQL." Multiple data models. Sacrifices consistency for scalability, flexibility,
availability.

6 NOSQL TYPES
1. Key-Value: hash map. GET/SET. Redis, DynamoDB. → sessions, cache, rate limiting
2. Document: JSON docs. MongoDB, Firestore. → user profiles, catalogs, CMS
3. Wide-Column: flexible cols/row. Cassandra, HBase. → IoT, time-series, massive writes
4. Graph: nodes + edges. Neo4j, Neptune. → social networks, fraud, recommendations
5. In-Memory: RAM-first. Redis, Memcached. → caching layer, real-time, leaderboards
6. Time-Series: timestamped. InfluxDB, Prometheus. → monitoring, IoT metrics, APM

ACID (SQL)
Atomicity (all or nothing) · Consistency (always valid) · Isolation (no interference) · Durability
(permanent). Used for: money, inventory, medical.

BASE (NOSQL)
Basically Available (always responds) · Soft state (may change without input) · Eventually
consistent (converges over time). Used for: social, analytics, IoT.
[Link] 14/15
04/04/2026, 12:57 SQL vs NoSQL — Part 2: NoSQL Deep Dive & ACID vs BASE

CAP THEOREM
Pick 2 of 3: Consistency, Availability, Partition tolerance. P is mandatory in distributed systems.
Choose CP (MongoDB, HBase) or AP (Cassandra, DynamoDB).

AMAZON'S APPROACH (INTERVIEW GOLD)

Shopping cart = BASE (show stale cart, never fail to load page).
Payment processing = ACID (money must never be inconsistent).
Real systems use BOTH. Pick per use case.

[Link] 15/15
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

S Q L V S N O S Q L — PA R T 3

Comparison, Sharding, Replication &

Advanced Topics
Complete side-by-side comparison, the golden decision framework, normalisation vs
denormalisation, sharding strategies, federation, replication vs mirroring

Section 06 — SQL vs NoSQL: Complete Side-by-Side

The master comparison table, the golden decision framework, and the correct interview
answer

1 The Complete Comparison Table

DIMENSION SQL NOSQL INTERVIEW TIP

Data Model Tables, rows, Documents, KV, Graph, Structured → SQL; Flexible →
columns Column NoSQL

Schema Fixed, enforced Dynamic, schema-less Stable domain → SQL;

upfront Evolving → NoSQL

Scalability Vertical (scale up) Horizontal (scale out) <10TB → SQL; >100TB →
NoSQL

Consistency Strong (ACID) Eventual (BASE) Financial → SQL; Social →

NoSQL

Transactions Full multi-table ACID Single-doc / limited Multi-step ops → SQL

Query Power Rich JOINs, Simple key lookups Ad-hoc analytics → SQL
aggregations

Relationships Foreign keys, JOINs Embedded / Complex relations → SQL

denormalised

[Link] 1/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

DIMENSION SQL NOSQL INTERVIEW TIP

Write Speed Moderate (ACID High (no coordination) High write volume → NoSQL
overhead)

Read Speed Fast with indexes + Fast for key-based Complex reads → SQL;
JOINs reads Simple → NoSQL

Fault Single point of failure Distributed, fault- High availability → NoSQL

Tolerance tolerant

Schema Migration needed Just start writing new Rapid iteration → NoSQL
Change fields

Best Use Banking, e- Social, IoT, analytics, Match to consistency need

Cases commerce, ERP cache

2 The Golden Decision Framework

CHOOSE SQL WHEN... CHOOSE NOSQL WHEN...

→ You need ACID transactions across → You need to scale beyond 100TB or 1M+
multiple records writes/second
→ Your data has complex relationships → Can tolerate eventual consistency for your
requiring JOINs domain
→ Dataset fits under ~10TB and write load < → Schema evolves rapidly or varies per
100K writes/sec record
→ You need ad-hoc analytics and complex → Primary queries are simple key-based
reporting lookups
→ Domain is well-defined and schema is → High availability across regions is critical
stable → Data is naturally document-shaped or
→ Team has strong SQL expertise graph-shaped
→ Compliance or audit requirements mandate → Append-only data: logs, events, sensor
strong consistency readings

[Link] 2/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

SQL OR NoSQL? — DECISION FLOWCHART

YES Need ACID

→ SQL transactions?

NO

YES Complex JOINs

→ SQL
& relationships?

NO

NO Scale > 100TB or YES

→ SQL → NoSQL
1M+ writes/sec?

Most large systems use BOTH!

SQL for transactions + NoSQL for caching/analytics/scale

THE CORRECT INTERVIEW ANSWER

"Which is better — SQL or NoSQL?"

The answer is ALWAYS: "It depends on the consistency requirements, expected scale, team
expertise, and nature of the data." Show nuance. Most large systems use BOTH — SQL for
transactional paths, NoSQL for scale, caching, and analytics.

Section 07 — Advanced Topics

Normalisation vs Denormalisation, Sharding strategies, Database Federation

1 Normalisation vs Denormalisation

Two opposite database design strategies. Understanding when to use each is a classic
interview question.

Normalisation — Eliminate Redundancy

WHAT IS IT?
[Link] 3/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

Normalisation organises tables to reduce data duplication. The same piece of information is
stored only once, and other tables reference it. There are formal levels called Normal Forms.

NORMAL
RULE EXAMPLE
FORM

1NF Atomic values only. No repeating groups or Split "phone1, phone2" into separate
arrays. rows

2NF Meets 1NF + no partial dependency on Customer name depends on

composite key customer_id alone

3NF Meets 2NF + no transitive dependency (non- City shouldn't depend on zip_code in
key → non-key) Orders table

Denormalisation — Optimise for Read Speed

WHAT IS IT?

Denormalisation intentionally adds redundancy by combining tables or pre-computing JOINs.

This speeds up reads (no expensive JOINs at query time) at the cost of slower writes and
potential data inconsistency if updates are missed.

NORMALISED (SQL-STYLE) DENORMALISED (NOSQL-STYLE)

→ Data stored once, referenced by FK → Data duplicated for fast access

→ Writes are fast (update one place) → Writes slower (update many copies)
→ Reads are slower (JOIN needed) → Reads are lightning fast (no JOIN)
→ Best for OLTP: banking, inventory → Best for OLAP / high read workloads
→ Less storage space used → More storage space used
→ Data always consistent → Risk of inconsistency if updates missed

NORMALISED VS DENORMALISED — VISUAL EXAMPLE

NORMALISED (3 tables, no duplication):

Users: {id:1, name:"Amit", email:"amit@.."}
Orders: {id:101, user_id:1, total:5000} ← references Users via FK
Products: {id:50, name:"Laptop", price:5000}

[Link] 4/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

To show "Amit bought Laptop for 5000" → JOIN all 3 tables.

Change Amit's name? Update ONE row in Users. Done.

DENORMALISED (1 document, data duplicated):

{
order_id: 101,
user_name: "Amit", ← duplicated from Users
user_email: "amit@..", ← duplicated from Users
product_name: "Laptop", ← duplicated from Products
total: 5000
}

To show "Amit bought Laptop for 5000" → read ONE document. No JOIN!
Change Amit's name? Must update EVERY document that has his name.

NORMALISED vs DENORMALISED — THE TRADE-OFF

NORMALISED DENORMALISED
Users Orders Products One Big Document / Table

Data stored ONCE Data DUPLICATED

JOINs needed to combine No JOINs — read one doc
✓ Consistent ✗ Slower reads ✓ Fast reads ✗ Risk of inconsistency

2 Sharding — Horizontal Scaling for SQL

WHAT IS SHARDING?

Sharding splits a large table across multiple database servers. Each server holds a subset
(shard) of the data. This is how SQL databases can scale horizontally despite not being built
for it natively.

SHARDING — SPLITTING DATA ACROSS SERVERS

BEFORE SHARDING (single server, hitting limits):

┌────────────────────────────────────┐
│ Server 1: ALL 100 million users │ ← too much for one server
└────────────────────────────────────┘

AFTER SHARDING (split across 4 servers):

[Link] 5/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

┌──────────────────┐ ┌──────────────────┐
│ Shard 1: Users │ │ Shard 2: Users │
│ A-F (25M users) │ │ G-M (25M users) │
└──────────────────┘ └──────────────────┘
┌──────────────────┐ ┌──────────────────┐
│ Shard 3: Users │ │ Shard 4: Users │
│ N-S (25M users) │ │ T-Z (25M users) │
└──────────────────┘ └──────────────────┘

Each shard is a separate database server.

Application needs to know which shard to query.

Three Sharding Strategies

STRATEGY HOW IT WORKS PROS CONS

Range- Shard by value ranges. Users Simple. Range Hot shards if distribution is
Based A-M → Server 1; N-Z → queries stay on one uneven (more users start
Server 2 shard. with "S" than "X")

Hash- hash(shard_key) % Even distribution. Range queries span ALL

Based num_shards → determines No hot shards. shards. Adding shards =
shard reshuffling data.

Directory- A lookup table maps each Most flexible. Can Lookup table itself can be
Based key → specific shard move individual bottleneck + single point of
keys. failure.

THREE SHARDING STRATEGIES

RANGE-BASED HASH-BASED DIRECTORY-BASED

A-M → Shard 1 N-Z → Shard 2 hash(key) % N → shard lookup_table[key] → shard

✓ Simple logic ✓ Even distribution ✓ Most flexible

✓ Range queries local ✓ No hot shards ✓ Can move individual keys
✗ Hot shards risk ✗ Range queries = all shards ✗ Lookup table = bottleneck
✗ Uneven distribution ✗ Adding shards = reshuffle ✗ Single point of failure

THE DOWNSIDE OF SHARDING

Cross-shard JOINs are very slow or impossible — you can't easily JOIN data that lives on
different servers.
ACID transactions across shards require expensive two-phase commit (2PC) protocol — all
shards must agree before committing.

[Link] 6/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

This is why sharding is a last resort for SQL databases — try vertical scaling, read replicas,
and caching first.

3 Database Federation

WHAT IS FEDERATION?

Federation (also called functional partitioning) splits different databases by function — a

Users DB, an Orders DB, a Products DB — each on separate servers. A federation layer sits on
top and provides a unified query interface.

DATABASE FEDERATION — HOW IT WORKS

┌─────────────────────┐
│ FEDERATION LAYER │ ← routes queries to correct DB
│ (unified endpoint) │
└──────┬──────┬───────┘
│ │ │
┌────────────┤ │ ├────────────┐
↓ ↓ ↓ ↓ ↓
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Users DB │ │ Orders DB │ │Products DB │
│ (PostgreSQL) │ │ (MySQL) │ │ (MongoDB) │
└──────────────┘ └──────────────┘ └──────────────┘

BENEFITS:
• Transparency: App queries one endpoint, federation routes to correct DB
• Heterogeneity: Each DB can be a different type!
• Autonomy: Each database is self-contained, updated independently
• Natural fit for MICROSERVICES (each service has its own DB)

TRADE-OFFS:
• Cross-database JOINs are slow
• Global transactions across federated DBs are complex
• Added complexity in the federation layer

[Link] 7/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

Section 08 — Data Replication vs Data Mirroring

Copies of data for safety, availability, and performance — knowing the difference is a strong
interview signal

1 Data Replication

WHAT IS REPLICATION?

Copying data from one location (the primary/source) to one or more other locations
(replicas/secondaries). Replication can be synchronous or asynchronous, and creates
multiple copies across different servers or geographic locations.

MASS EMAIL ANALOGY

Replication is like sending a mass email. You write one email and it gets delivered to 10 people.
Each recipient gets their own copy. The delivery can be instant (sync) or delayed by a few
seconds (async). You control who gets updates and when.

Three Types of Replication

TYPE HOW IT WORKS TRADE-OFF EXAMPLE

Synchronous Primary waits for ALL Zero data loss Google Spanner, PG
replicas to confirm before synchronous standby
ACK to client

Asynchronous Primary confirms Fast but risk of data MySQL read replicas,
immediately, replicates in loss if primary crashes MongoDB replica sets
background

Semi- Primary waits for at least Middle ground — MySQL semi-sync

Synchronous ONE replica to confirm reduced risk, plugin (used at
reasonable speed Facebook)

[Link] 8/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

THREE REPLICATION MODES

SYNCHRONOUS ASYNCHRONOUS SEMI-SYNCHRONOUS

Primary Primary Primary

Replica 1 ✓ Replica 2 ✓ Replica 1 ⟳ Replica 2 ⟳ Replica 1 ✓ Replica 2 ⟳

Waits for ALL to ACK Confirms IMMEDIATELY Waits for at least ONE
✓ Zero data loss ✓ Fast writes ✓ Reduced risk

✗ Slower writes ✗ Risk of data loss ~ Moderate speed

Google Spanner MySQL, MongoDB MySQL (Facebook)

Use Cases for Replication

5 KEY USE CASES FOR REPLICATION

1. READ SCALING
Add read replicas to distribute read traffic.
Primary handles ALL writes; replicas handle reads.

2. GEOGRAPHIC DISTRIBUTION
Replicate to data centres in Asia, Europe, US
for low-latency local reads.

3. DISASTER RECOVERY
If primary data centre fails, promote a replica to primary.

4. REPORTING & ANALYTICS

Run expensive analytics queries on a replica
without impacting production.

5. ZERO-DOWNTIME MIGRATIONS
Replicate to new server, then switch over without downtime.

2 Data Mirroring

WHAT IS MIRRORING?

Creating an exact, real-time replica of a database. Mirroring is almost always synchronous.

Every write to the primary is simultaneously written to the mirror. The mirror is an exact,

[Link] 9/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

current duplicate — not a slightly-behind copy. Used specifically for high availability and
instant failover.

SHADOW ANALOGY

Mirroring is like a live shadow. When a dancer moves, the shadow moves at exactly the same
instant. There is never a gap between the two. If the dancer "fails," the shadow can
immediately take over without any missed frames.

HOW DATABASE MIRRORING WORKS

ARCHITECTURE:
┌──────────────┐ ┌──────────────┐
│ PRINCIPAL │ ──────→ │ MIRROR │
│ (Primary) │ sync │ (Standby) │
│ Server │ write │ Server │
└──────────────┘ └──────────────┘
↑ ↑
│ ┌──────────────┘
│ │
┌─────────────────┐
│ WITNESS │ ← optional: automates failover decisions
│ (monitors both) │ if principal fails, witness + mirror
└─────────────────┘ vote to promote mirror to principal

FLOW:
1. Client sends write to Principal
2. Principal writes AND sends to Mirror simultaneously
3. Mirror applies transaction, sends ACK
4. ONLY after both have the data → client gets confirmation
5. If Principal fails → Witness detects → Mirror promoted → zero downtime

Use Cases for Mirroring

WHEN TO USE MIRRORING

• HIGH AVAILABILITY (HA): Instant automatic failover. Zero data loss.

• DISASTER RECOVERY: Mirror in separate building/city. Primary burns? Mirror takes over
• FINANCIAL TRADING: Every millisecond matters. Transactions cannot be lost.
• HEALTHCARE RECORDS: Patient data must NEVER be lost. Live duplicate always exists.

[Link] 10/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

3 Replication vs Mirroring — Head-to-Head

DIMENSION REPLICATION MIRRORING

Synchronisation Sync OR Async (flexible) Always synchronous (real-time)

Number of Copies Multiple (1 to many) Usually one mirror (1-to-1)

Primary Purpose Availability, load balancing, analytics High availability, instant failover

Data Lag Async: seconds of lag possible No lag — always current

Performance Async: minimal; Sync: write latency Higher write latency (wait for mirror
Impact ACK)

Flexibility Very flexible — many configurations More rigid — exact replication only

Failover Manual promotion usually Automatic failover (with witness)

Use Cases Read scaling, reporting, geo- HA systems, financial, critical data
distribution

Consistency Async replicas may serve stale Mirror always has same data
reads

Cost Lower per replica (async = cheap) Higher (synchronous = double write
cost)

INTERVIEW SUMMARY

Replication = copies for scale + backup (can be async). Flexible, many copies, may have lag.
Mirroring = always-on exact copy for zero-downtime failover (always sync). Rigid, 1-to-1, zero
lag.

Most enterprise systems use BOTH: mirroring for HA failover + async replication for read
scaling.

[Link] 11/12
04/04/2026, 12:57 SQL vs NoSQL — Part 3: Comparison, Sharding, Replication & Advanced Topics

📋 SQL vs NoSQL Part 3 — Quick Revision

SQL VS NOSQL DECISION
ACID needed? → SQL. Complex JOINs? → SQL. >100TB or 1M+ writes/sec? → NoSQL. Schema
evolving? → NoSQL. Most systems use BOTH.

NORMALISATION VS DENORMALISATION
Normalised: no duplication, FK references, JOINs needed, consistent. Best for OLTP (banking).
Denormalised: data duplicated, no JOINs, fast reads, risk of inconsistency. Best for OLAP /
reads.

SHARDING (HORIZONTAL SCALING FOR SQL)

Range: shard by value range (simple but hot shards). Hash: even distribution (no range queries).
Directory: lookup table (flexible but bottleneck).
Cross-shard JOINs are slow. Distributed ACID needs 2PC. Sharding = last resort — try vertical
scaling, replicas, caching first.

FEDERATION
Split by function: Users DB, Orders DB, Products DB on separate servers. Federation layer
routes queries. Natural fit for microservices. Cross-DB JOINs are complex.

REPLICATION
Sync: wait for all replicas → zero data loss, slower. Async: confirm immediately → fast, risk of
loss. Semi-sync: wait for ONE → middle ground.
Use cases: read scaling, geo-distribution, disaster recovery, analytics offloading.

MIRRORING
Always synchronous, 1-to-1, zero lag. Principal + Mirror + Witness. Instant automatic failover.
Used for HA, financial, healthcare.

REPLICATION VS MIRRORING
Replication = flexible, many copies, can be async. For scale + backup.
Mirroring = rigid, exact copy, always sync. For zero-downtime failover.
Enterprise systems use both: mirroring for HA + async replication for read scaling.

[Link] 12/12
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

S Q L V S N O S Q L — PA R T 4

Real-World Case Studies

Instagram, Discord, Twitter, Uber — how real companies made database decisions under
pressure, and what you can learn for interviews

WHY CASE STUDIES MATTER IN INTERVIEWS

Interviewers don't just want to know SQL vs NoSQL theory — they want to see that you
understand how these decisions play out in real systems at real scale. Referencing actual
company decisions shows depth and credibility. These four case studies cover the four most
common architectural patterns.

CASE STUDY 1

Instagram — PostgreSQL + Cassandra

The gold-standard "polyglot persistence" example — using BOTH SQL and NoSQL, each for what it
does best

The Problem
Instagram launched with PostgreSQL and scaled to 300 million daily active users. Two very
different data needs emerged:

DATA NEEDING STRONG CONSISTENCY DATA NEEDING MASSIVE THROUGHPUT

→ User profiles and account data → Activity feeds (your home feed)
→ Follower/following relationships → Likes (millions per second globally)
→ Photo metadata (which photo belongs to → Notifications
whom) → Eventual consistency is fine here
→ You can NEVER show a photo in the wrong → Seeing 99 likes vs 100 for 2 seconds? No
account problem
→ Follower counts must be accurate → Needs millions of writes/second

[Link] 1/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

→ Needs ACID transactions + complex

queries

What They Did — The Architecture

INSTAGRAM'S DATABASE ARCHITECTURE

Instagram Application Layer

ACID required Scale required

PostgreSQL (SQL) Apache Cassandra (NoSQL)

What it stores: What it stores:

• User profiles & account data • Activity feeds (home feed)
• Follower/following relationships • Likes (millions per second)
• Photo metadata • Notifications
Why PostgreSQL: Why Cassandra:
• ACID transactions (critical data) • Massive write throughput
• Complex JOINs for relationships • Horizontal scaling (100s of nodes)
• Sharded by user_id for locality • Simple key-based lookups
Strong consistency guaranteed Eventual consistency acceptable

Why This Design Works — Deep Analysis

INSTAGRAM'S DATA SPLIT — WHY EACH DATABASE WAS CHOSEN

PostgreSQL for Users/Relationships/Photos:

─────────────────────────────────────────────
1. ACID: "Show Amit's photo on Priya's profile" — MUST be accurate.
Wrong photo on wrong account = critical bug.
2. JOINs: "Get all followers of user X who also follow user Y"
→ requires multi-table JOIN on follower relationships.
3. Sharding by user_id: Each user's data stays on ONE shard.
All of Amit's data (profile + photos + followers) on same server.
ACID works because all related data is local to one shard.

Cassandra for Feeds/Likes/Notifications:

─────────────────────────────────────────────
1. WRITE VOLUME: When a celebrity posts, millions of followers'
feeds need updating. 1 post → millions of feed writes.
PostgreSQL can't handle this write volume.
2. EVENTUAL CONSISTENCY: If your feed shows a post 2 seconds
late, nobody notices. If a like count shows 999 instead of
1000 for 3 seconds — no problem.
3. SIMPLE ACCESS: "Get feed for user X" = single key lookup.
No JOINs needed. Cassandra excels at this.
[Link] 2/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

4. HORIZONTAL SCALE: Add Cassandra nodes as users grow.

No resharding complexity like PostgreSQL.

INTERVIEW TAKEAWAY

Instagram is the gold-standard polyglot persistence example. The lesson: don't force one
database to do everything. Use SQL where consistency matters and NoSQL where scale
matters. When interviewers ask "Would you use SQL or NoSQL?", the best answer is often: "I'd
use BOTH — here's how I'd split the data."

CASE STUDY 2

Discord — MongoDB → Cassandra

A NoSQL-to-NoSQL migration — proving that even within NoSQL, choosing the wrong TYPE is costly

The Problem
Discord initially chose MongoDB for message storage — attracted by its flexible JSON schema
and easy development setup. But as they scaled past 100 million messages per channel,
serious problems emerged:

WHY MONGODB FAILED FOR MESSAGES WHAT MESSAGES ACTUALLY NEED

→ Read latency degraded badly at scale → Append-only (never updated after posting)
→ MongoDB's storage engine wasn't → Always queried by time range ("messages
optimised for time-series data in channel X between Y and Z")
→ Messages are always queried by TIME → Predictable low latency at massive scale
range, not by flexible fields → Simple key-based access pattern
→ Sharding became operationally nightmarish → Write-optimised for high throughput
→ Hot partitions on popular channels

What They Did — Migration to Cassandra

[Link] 3/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

DISCORD'S DATABASE MIGRATION

BEFORE: MongoDB AFTER: Apache Cassandra

Flexible JSON docs Write-optimised LSM Tree engine
migrated to
NOT optimised for time-series Built-in time-series capabilities
Read latency grew with scale Predictable latency at massive scale
Sharding was a nightmare Partition: (channel_id, bucket)
❌ Wrong NoSQL type! ✅ Right NoSQL type!

DISCORD'S CASSANDRA DATA MODEL FOR MESSAGES

PARTITION KEY: (channel_id, bucket)

→ All messages in a channel land in the same partition
→ "bucket" splits very active channels into time-based chunks
→ Prevents any single partition from growing too large

CLUSTERING KEY: message_id (contains timestamp)

→ Messages within a partition are sorted chronologically
→ "Get messages from 9am to 10am" = sequential read within partition
→ No scanning, no filtering — data is PRE-SORTED

QUERY PATTERN:
SELECT * FROM messages
WHERE channel_id = 'abc123'
AND bucket = '2024-03-15'
AND message_id > [timestamp_for_9am]
AND message_id < [timestamp_for_10am];
→ Reads EXACTLY the messages needed, nothing more.

WHY CASSANDRA FITS:

• Messages are APPEND-ONLY (never updated after posting)
• Queries are ALWAYS time-range based
• Cassandra's LSM Tree = sequential writes to disk (fast!)
• No JOINs needed — just "give me messages for this channel"

THE KEY LESSON

NoSQL is not one thing. MongoDB's document flexibility was NOT the right NoSQL type for
time-series messages. Cassandra's write-optimised wide-column model was. The lesson:
don't just choose "NoSQL" — choose the right TYPE of NoSQL for your specific access
pattern.

[Link] 4/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies
CASE STUDY 3

Twitter (X) — MySQL → Manhattan (Custom NoSQL)

Building a custom distributed key-value store because no existing solution fit the scale

The Problem
Twitter started with Ruby on Rails and MySQL for all data, including tweets. As Twitter grew
from thousands to hundreds of millions of users, MySQL couldn't handle the volume:

WHY MYSQL COULDN'T HANDLE TWEETS AT SCALE

TWEET CHARACTERISTICS:
• 300K+ tweets posted per hour
• Tweets are IMMUTABLE (never edited after posting — at that time)
• Accessed primarily by tweet_id or user_id
• Need enormous write throughput + read throughput
• Timeline generation requires reading tweets from 100s of followed users

THE "FAIL WHALE":

Twitter's infamous downtime error page was largely caused by
DATABASE BOTTLENECKS. MySQL on a single server simply couldn't:
→ Ingest 300K+ tweets/hour
→ Generate timelines for millions of users simultaneously
→ Handle the read amplification of "fan-out on read"

WHY MySQL STRUGGLED:

1. VERTICAL SCALING LIMIT — single server maxed out
2. WRITE CONTENTION — all tweets go to one table, one server
3. TIMELINE QUERY — "get latest tweets from 500 people I follow"
= 500 separate queries or a massive JOIN on a huge table
4. SHARDING COMPLEXITY — sharding MySQL breaks JOINs

What They Did — The Architecture

[Link] 5/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

TWITTER'S SPLIT ARCHITECTURE

Twitter Application

MySQL (kept for ACID data) Manhattan (Custom NoSQL KV Store)

What stays in MySQL: What moved to Manhattan:

• User accounts & authentication • Tweets (immutable, key-based access)
• User settings & preferences • Timelines (pre-computed feeds)
• Follow relationships • Social graph data
Why MySQL stays: Why custom-built:
• ACID for authentication (critical) • Distributed key-value store
• Username uniqueness must be enforced • 300K+ tweets/hour throughput
Strong consistency non-negotiable Eventual consistency acceptable

TWITTER'S TECHNOLOGY STACK (ADDITIONAL COMPONENTS)

ADDITIONAL SYSTEMS TWITTER BUILT:

Apache Kafka:
→ Event streaming between services
→ When a tweet is posted, Kafka distributes the event to:
- Timeline service (add to followers' feeds)
- Notification service (notify mentioned users)
- Search index service (make tweet searchable)
- Analytics pipeline (count impressions)

Redis:
→ Caching layer for timelines
→ "Get timeline for user X" → check Redis first (O(1))
→ Cache miss → query Manhattan → store in Redis

TIMELINE GENERATION (Fan-out):

Celebrity posts a tweet
→ Celebrity has 50 million followers
→ Fan-out-on-write: pre-compute and push to each follower's timeline
→ For celebrities: fan-out-on-read instead (query at read time)
→ Hybrid approach balances write cost vs read latency

INTERVIEW TAKEAWAY

Twitter kept ACID where it mattered (accounts, authentication — you can NEVER have two
users with the same username) and adopted NoSQL for high-volume, read-optimised data.
The fan-out problem (1 tweet → millions of timeline updates) is a classic system design
question. Know the trade-off between fan-out-on-write vs fan-out-on-read.

[Link] 6/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

CASE STUDY 4

Uber — MySQL + Schemaless (Custom NoSQL on SQL)

The most creative solution — getting NoSQL benefits WITHOUT leaving MySQL

The Problem
Uber had a unique constraint that the other companies didn't face:

UBER'S CONFLICTING REQUIREMENTS

THEY NEEDED NoSQL BECAUSE:

→ Billions of trips with EVOLVING schema
(new fields added constantly: surge pricing, pool trips, Uber Eats)
→ Schema changes on billions of rows in SQL = migration nightmare
→ Massive scale (millions of trips/day)

BUT THEY ALSO NEEDED SQL BECAUSE:

→ MySQL's PROVEN operational reliability
→ ACID transactions for trip data (billing, payments)
→ Existing backup tooling and monitoring
→ Team expertise was in MySQL
→ Running a SEPARATE NoSQL cluster adds:
- Operational complexity
- New expertise requirements
- Additional infrastructure cost

THE DILEMMA: How do you get NoSQL's flexibility

WITHOUT leaving MySQL's reliability?

What They Did — Schemaless: NoSQL on Top of MySQL

[Link] 7/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

UBER'S "SCHEMALESS" — NoSQL LAYER ON TOP OF MySQL

Uber Application

"Schemaless" — NoSQL Abstraction Layer

Accepts JSON documents · Key-based lookups · Schema-free writes

stores as blob

MySQL (underneath — actual storage)

JSON docs stored in MySQL's BLOB columns · Sharded by entity_id
Retains: ACID transactions · Backup tools · Operational maturity

HOW SCHEMALESS WORKS — DEEP DIVE

THE CLEVER TRICK:

1. APPLICATION writes JSON documents to "Schemaless" API:

{
"trip_id": "trip_abc123",
"rider": "user_456",
"driver": "driver_789",
"fare": 350,
"surge_multiplier": 1.5, ← new field! no migration needed
"pool_riders": [...] ← another new field! just add it
}

2. SCHEMALESS LAYER receives the JSON and stores it in MySQL:

INSERT INTO trips (entity_id, blob_data, created_at)
VALUES ('trip_abc123', '{"rider":"user_456",...}', NOW());

The JSON goes into a BLOB column — MySQL doesn't care about
the schema of the JSON. You can add any field anytime.

3. ACCESS PATTERN is key-based:

SELECT blob_data FROM trips WHERE entity_id = 'trip_abc123';
→ Returns the full JSON document.
→ No JOINs needed (all trip data in one document).

4. SHARDING by entity_id:
Each trip's data stays on ONE MySQL shard.
ACID transactions work because data is local to one shard.

WHAT YOU GET:

✓ NoSQL schema flexibility (add fields without migration)
✓ NoSQL key-based access pattern (fast lookups)
✓ MySQL ACID guarantees (billing/payments are safe)

[Link] 8/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

✓ MySQL operational tooling (backups, monitoring, team expertise)

✓ No need to learn/operate a separate NoSQL system

WHY THIS IS BRILLIANT

Uber realised their access pattern was always key-based ("get trip by trip_id"). They never
needed SQL's JOINs for trip data. So the lack of JOIN power wasn't a constraint. By storing
JSON in MySQL blob columns, they got NoSQL's flexibility while keeping MySQL's battle-
tested reliability. This shows that you can achieve NoSQL benefits on SQL if your access
patterns allow it.

INTERVIEW TAKEAWAY

Uber's approach teaches a critical lesson: always match the solution to the actual query
patterns, not the hype. They didn't need a fancy NoSQL cluster — they needed schema
flexibility + key lookups + ACID. MySQL with a JSON blob layer gave them all three. This is the
kind of nuanced thinking interviewers love.

5 All 4 Case Studies — Comparison Table

COMPANY PATTERN SQL FOR NOSQL FOR KEY LESSON

Instagram SQL + NoSQL Users, relationships, Feeds, likes, Use BOTH — each
side-by-side photos (ACID) notifications (scale) for what it does best

Discord NoSQL → — MongoDB → Right TYPE of

different Cassandra for NoSQL matters
NoSQL messages hugely

Twitter SQL + custom Accounts, auth Tweets, timelines ACID where it

NoSQL built (ACID) (Manhattan KV) matters, NoSQL for
volume

Uber NoSQL layer MySQL underneath Schemaless JSON Match solution to

ON TOP of for ACID + ops layer on top access patterns, not
SQL hype

[Link] 9/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

FOUR ARCHITECTURAL PATTERNS FOR SQL + NoSQL

① POLYGLOT ② MIGRATION ③ CUSTOM BUILD ④ NoSQL ON SQL

(Instagram) (Discord) (Twitter) (Uber)

SQL NoSQL NoSQL A → NoSQL B SQL + Custom KV

NoSQL API layer

Side by side Wrong type → right type Build your own when MySQL underneath

Different data to Same family, different nothing off-the-shelf

different databases data model fits the scale NoSQL flexibility
+ SQL reliability
Most common pattern Type selection matters! Only at massive scale
Best of both worlds

📋 Real-World Case Studies — Quick Revision

INSTAGRAM (POLYGLOT PERSISTENCE)
PostgreSQL for users, relationships, photos (ACID needed). Cassandra for feeds, likes,
notifications (scale needed). PG sharded by user_id. Gold-standard example of using BOTH.

DISCORD (NOSQL TYPE MIGRATION)

MongoDB → Cassandra for messages. MongoDB's document model was wrong for time-series
messages. Cassandra's LSM Tree + wide-column model = perfect fit. Partition key:
(channel_id, bucket) . Lesson: right TYPE of NoSQL matters.

TWITTER (SQL + CUSTOM NOSQL)

MySQL kept for accounts/auth (ACID). Manhattan (custom KV store) built for tweets/timelines.
Plus Kafka for event streaming, Redis for caching. Lesson: ACID where it matters, NoSQL for
volume.

UBER (NOSQL LAYER ON SQL)

Schemaless = NoSQL API layer on top of MySQL. JSON in blob columns. Schema flexibility +
ACID + MySQL operational maturity. Lesson: match solution to access patterns, not hype.

FOUR PATTERNS TO KNOW

① Polyglot (SQL + NoSQL side by side) — most common
② Migration (wrong NoSQL → right NoSQL) — type matters
③ Custom Build (build your own at extreme scale)
④ NoSQL on SQL (abstraction layer over MySQL) — creative hybrid
UNIVERSAL LESSON

[Link] 10/11
04/04/2026, 13:01 SQL vs NoSQL — Part 4: Real-World Case Studies

No company uses just one database at scale. They all use SQL for critical data + NoSQL for
scale. The interview answer is always: "It depends — and real systems use both."

[Link] 11/11
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

N O S Q L D E E P D I V E — D O C U M E N T S TO R E

MongoDB — Complete Deep Dive

From zero to expert: what it is, how it works internally, querying, scaling, trade-offs, and
real-world examples with architecture decisions

1 What Is MongoDB? (Starting From Zero)

DEFINITION

MongoDB is a document-oriented NoSQL database. Instead of storing data in tables with

rows and columns (like SQL), MongoDB stores data as JSON-like documents (technically
BSON — Binary JSON). Each document is a self-contained unit that can have any structure —
no two documents need to look the same.

THE FILING CABINET ANALOGY

Imagine SQL as a spreadsheet — every row has the same columns, strictly enforced. Now
imagine MongoDB as a folder of papers. Each paper (document) can have different
information on it. One paper has name + email + phone. Another has name + address + age.
No one forces them to match. You just put whatever you need on each paper and file it in a
folder (collection).

SQL vs MongoDB — Terminology Mapping

SQL CONCEPT MONGODB EQUIVALENT EXAMPLE

Database Database myapp_db

Table Collection users collection

Row Document One user's data (JSON object)

Column Field name , email , age

Primary Key _id field Auto-generated ObjectId

[Link] 1/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

SQL CONCEPT MONGODB EQUIVALENT EXAMPLE

JOIN Embedding or $lookup Nest related data inside document

Schema (enforced) Schema (optional) Can validate, but not required

SQL TABLE vs MONGODB COLLECTION

SQL TABLE — Fixed Schema MONGODB COLLECTION — Flexible

id name email age Document 1:
{ name:"Amit", email:"amit@..", age:28,
1 Amit amit@.. 28
address:{city:"Mumbai"}, phone:"9876" }
2 Priya priya@.. 25
3 Sara sara@.. 30 Document 2:
{ name:"Priya", email:"priya@..", age:25,
Every row MUST have same columns tags:["premium"], preferences:{dark:true} }
Can't add "phone" to just one row
Schema change = ALTER TABLE migration Document 3:
{ name:"Sara", age:30, bio:"Developer" }
Related data → separate table + JOIN
Each document can have DIFFERENT fields!
Addresses, orders = separate tables
Related data EMBEDDED inside document

2 How MongoDB Stores Data — Documents & BSON

What Is a Document?
A document is a JSON-like object that holds all the data for one entity. Unlike SQL where you
split data across multiple tables and JOIN them, MongoDB encourages you to embed related
data inside one document.

A COMPLETE MONGODB DOCUMENT — USER PROFILE

{
"_id": ObjectId("507f1f77bcf86cd799439011"), // auto-generated unique ID
"name": "Amit Sharma",
"email": "amit@[Link]",
"age": 28,
"created_at": ISODate("2024-03-15T10:30:00Z"),

// NESTED OBJECT — in SQL this would be a separate "addresses" table

"address": {
"street": "MG Road",
"city": "Mumbai",
[Link] 2/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

"state": "Maharashtra",
"pin": "400001"
},

// ARRAY OF OBJECTS — in SQL this would be a separate "orders" table

"orders": [
{ "order_id": "ORD-001", "total": 5000, "status": "delivered" },
{ "order_id": "ORD-002", "total": 2500, "status": "pending" }
],

// FLEXIBLE FIELDS — other documents may not have these

"preferences": { "theme": "dark", "notifications": true },
"tags": ["premium", "early_adopter"],
"referral_code": "AMT2024"
}

WHY EMBEDDING IS POWERFUL

In SQL, to show "Amit's profile with his address and orders," you'd need to JOIN 3 tables
(users + addresses + orders). That's 3 separate disk lookups and a computation to merge
them.

In MongoDB, ALL of Amit's data is in ONE document. One read from disk → you have
everything. This is why MongoDB is often faster for "get everything about this entity" queries.

What Is BSON?

BSON = BINARY JSON

MongoDB doesn't actually store raw JSON text. It converts JSON to BSON (Binary JSON) — a
binary-encoded format that's more efficient to store and faster to parse. BSON also supports
additional data types that JSON doesn't have: Date , ObjectId , Decimal128 , Binary ,
RegExp .

You write JSON in your application code, but MongoDB stores BSON on disk. This is
transparent — you never need to think about the conversion.

The _id Field — MongoDB's Primary Key

OBJECTID — HOW MONGODB AUTO-GENERATES UNIQUE IDS

[Link] 3/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

Every document MUST have an "_id" field. If you don't provide one,
MongoDB auto-generates an ObjectId:

ObjectId("507f1f77bcf86cd799439011")
├────┤├──┤├──┤├────────┤
│ │ │ │
│ │ │Counter (3 bytes) — unique within same second
└─
│ │ └────── Process ID (2 bytes, since v3.4: random)
│ └─────────── Machine ID (3 bytes, since v3.4: random)
└───────────────── Timestamp (4 bytes) — seconds since epoch

WHY ObjectId is clever:

• Contains a TIMESTAMP — you can extract when a document was created
• Globally unique — no coordination needed between servers
• Roughly SORTED by creation time — helps with index performance
• 12 bytes — smaller than a UUID (16 bytes)

3 Internal Architecture — How MongoDB Works Under the Hood

WiredTiger Storage Engine

WHAT IS A STORAGE ENGINE?

The storage engine is the component that manages how data is stored on disk and read into
memory. It's the lowest layer — everything else (queries, indexes, replication) sits on top of it.
Since MongoDB 3.2, the default storage engine is WiredTiger.

WIREDTIGER — KEY FEATURES

1. DOCUMENT-LEVEL LOCKING
In older MongoDB (MMAPv1 engine): collection-level lock.
One writer blocks ALL other writers on the same collection.

WiredTiger: DOCUMENT-LEVEL concurrency control.

Two users can update two DIFFERENT documents simultaneously.
Much higher write throughput for concurrent workloads.

2. COMPRESSION
WiredTiger compresses data on disk using:
[Link] 4/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

• Snappy (default): fast compression, moderate ratio

• Zlib: better ratio, slower
• Zstd: best balance (MongoDB 4.2+)
Typically 50-70% space savings vs uncompressed.

3. WRITE-AHEAD LOG (JOURNAL)

Every write is first logged to a JOURNAL (write-ahead log).
If the server crashes, the journal replays to recover data.
This gives DURABILITY (the "D" in ACID).

4. IN-MEMORY CACHE
WiredTiger maintains a cache in RAM (default: 50% of RAM).
Hot data stays in memory → reads don't hit disk.
Eviction policies manage what stays and what gets flushed.

5. B-TREE INDEXES
Indexes are stored as B-Tree structures (similar to B+ Tree).
Default index on _id is created automatically.
Additional indexes must be created manually.

MONGODB INTERNAL ARCHITECTURE

Your Application ([Link], Python, Java)

MongoDB Driver (converts queries to wire protocol)

MongoDB Query Engine

Query Parser → Optimizer → Execution Plan → Cursor

WiredTiger Storage Engine

Cache (RAM) | B-Tree Indexes | Compression | Journal (WAL) | Concurrency Control

Disk — BSON data files + Index files + Journal logs

How a Write Works — Step by Step

WRITE PATH — WHAT HAPPENS WHEN YOU INSERT A DOCUMENT

[Link]({ name: "Amit", email: "amit@[Link]", age: 28 })

Step 1: DRIVER converts the document to BSON format

Step 2: BSON sent to MongoDB server over wire protocol
Step 3: MongoDB VALIDATES the document (optional schema validation)
Step 4: Write goes to WIREDTIGER IN-MEMORY CACHE

[Link] 5/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

Step 5: Write is logged to JOURNAL (write-ahead log on disk)

→ At this point, write is DURABLE (survives crash)
Step 6: Periodically, WiredTiger CHECKPOINTS — flushes dirty
pages from cache to the data files on disk
Step 7: If the document has INDEXED fields, the index B-Trees
are also updated (in memory, then flushed)

WRITE CONCERN controls how much durability you want:

{ w: 0 } → "Fire and forget" — don't even wait for ACK
{ w: 1 } → Wait for PRIMARY to confirm (default)
{ w: "majority" }→ Wait for MAJORITY of replica set to confirm
(strongest guarantee, highest latency)

How a Read Works — Step by Step

READ PATH — WHAT HAPPENS WHEN YOU QUERY

[Link]({ email: "amit@[Link]" })

Step 1: Query engine PARSES the query

Step 2: OPTIMIZER checks available indexes
→ Index on email exists? Use it (index scan)
→ No index? Full COLLECTION SCAN (read every document)
Step 3: If using index: traverse B-Tree → find matching _id(s)
→ Fetch full document from cache (or disk if not cached)
Step 4: If collection scan: read every document, filter in memory
Step 5: Return matching documents as a CURSOR to the driver
Step 6: Driver converts BSON back to JSON/native objects

READ PREFERENCE controls WHERE you read from:

primary → Read from primary only (strongest consistency)
primaryPreferred → Primary if available, else secondary
secondary → Read from secondary only (may be stale)
secondaryPreferred → Secondary if available, else primary
nearest → Whichever node has lowest network latency

4 Querying MongoDB — CRUD Operations

MongoDB uses its own query language (not SQL). Here are the core operations:

[Link] 6/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

CREATE — Inserting Documents

INSERT OPERATIONS

// Insert ONE document

[Link]({
name: "Amit",
email: "amit@[Link]",
age: 28,
address: { city: "Mumbai", state: "Maharashtra" }
})

// Insert MANY documents at once (batch insert — much faster)

[Link]([
{ name: "Priya", email: "priya@[Link]", age: 25 },
{ name: "Sara", email: "sara@[Link]", age: 30, bio: "Developer" },
{ name: "Rahul", email: "rahul@[Link]", age: 22, tags: ["student"] }
])
// Note: Sara has "bio", Rahul has "tags" — different fields! No problem.

READ — Finding Documents

READ OPERATIONS — FROM SIMPLE TO COMPLEX

// Find ALL documents in a collection

[Link]()

// Find with a FILTER (like SQL WHERE)

[Link]({ city: "Mumbai" }) // WHERE city = 'Mumbai'
[Link]({ age: { $gt: 25 } }) // WHERE age > 25
[Link]({ age: { $gte: 20, $lte: 30 } }) // WHERE age BETWEEN 20 AND 30
[Link]({ name: { $in: ["Amit", "Sara"] } }) // WHERE name IN ('Amit','Sara')

// Find with AND (multiple conditions)

[Link]({ city: "Mumbai", age: { $gt: 25 } }) // city='Mumbai' AND age>25

// Find with OR
[Link]({ $or: [{ city: "Mumbai" }, { age: { $gt: 30 } }] })

// Find ONE document (returns first match)

[Link]({ email: "amit@[Link]" })

// PROJECTION — select specific fields (like SQL SELECT name, email)

[Link] 7/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

[Link]({ city: "Mumbai" }, { name: 1, email: 1, _id: 0 })

// SORT + LIMIT + SKIP (pagination)

[Link]().sort({ age: -1 }).limit(10).skip(20) // Page 3, 10 per page

// COUNT
[Link]({ city: "Mumbai" })

// QUERY NESTED FIELDS (dot notation)

[Link]({ "[Link]": "Mumbai" }) // query inside nested object

// QUERY ARRAY ELEMENTS

[Link]({ tags: "premium" }) // any doc where tags contains "premium"
[Link]({ "[Link]": "pending" }) // nested array object field

UPDATE — Modifying Documents

UPDATE OPERATIONS

// Update ONE document

[Link](
{ email: "amit@[Link]" }, // filter (which document?)
{ $set: { age: 29, "[Link]": "Pune" } } // update (what to change?)
)

// Update MANY documents at once

[Link](
{ city: "Mumbai" }, // all Mumbai users
{ $set: { region: "West" } } // add a new field to all of them
)

// INCREMENT a number
[Link](
{ _id: ObjectId("507f1f77bcf86cd799439011") },
{ $inc: { age: 1 } } // age = age + 1
)

// PUSH to an array
[Link](
{ email: "amit@[Link]" },
{ $push: { tags: "verified" } } // add "verified" to tags array
)

// UPSERT — update if exists, insert if not

[Link](

[Link] 8/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

{ email: "new@[Link]" },
{ $set: { name: "New User", age: 20 } },
{ upsert: true } // creates document if not found
)

DELETE — Removing Documents

DELETE OPERATIONS

// Delete ONE document

[Link]({ email: "amit@[Link]" })

// Delete MANY documents

[Link]({ age: { $lt: 18 } }) // delete all users under 18

// Delete ALL documents in a collection (but keep the collection)

[Link]({})

// Drop entire collection (deletes collection + all indexes)

[Link]()

Aggregation Pipeline — MongoDB's Power Feature

WHAT IS THE AGGREGATION PIPELINE?

MongoDB's equivalent of SQL's GROUP BY , HAVING , JOIN , and subqueries — all in one.
Data flows through a series of stages, each transforming the data. Like a factory assembly line
where each station does one operation.

AGGREGATION PIPELINE — SQL EQUIVALENT EXAMPLES

// SQL: SELECT city, COUNT(*) as count FROM users GROUP BY city ORDER BY count DESC
[Link]([
{ $group: { _id: "$city", count: { $sum: 1 } } },
{ $sort: { count: -1 } }
])

// SQL: SELECT city, AVG(age) FROM users WHERE age > 20 GROUP BY city HAVING COUNT(*) >
[Link]([
{ $match: { age: { $gt: 20 } } }, // Stage 1: WHERE age > 20
{ $group: { // Stage 2: GROUP BY city
[Link] 9/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

_id: "$city",
avgAge: { $avg: "$age" },
count: { $sum: 1 }
}},
{ $match: { count: { $gt: 5 } } }, // Stage 3: HAVING count > 5
{ $sort: { avgAge: -1 } } // Stage 4: ORDER BY avgAge DESC
])

// $lookup = MongoDB's LEFT JOIN

[Link]([
{ $lookup: {
from: "users", // join with users collection
localField: "user_id", // orders.user_id
foreignField: "_id", // matches users._id
as: "user_info" // output field name
}}
])

PIPELINE STAGES (most common):

$match → Filter documents (WHERE)
$group → Group and aggregate (GROUP BY)
$sort → Sort results (ORDER BY)
$project → Select/rename fields (SELECT)
$limit → Limit results (LIMIT)
$skip → Skip results (OFFSET)
$unwind → Flatten arrays (one doc per array element)
$lookup → Join with another collection (LEFT JOIN)
$addFields → Add computed fields

5 Indexing in MongoDB

Just like SQL databases, MongoDB uses B-Tree indexes to speed up queries. Without indexes,
MongoDB must scan every document in a collection (collection scan). With indexes, it jumps
directly to matching documents.

MONGODB INDEX TYPES AND HOW TO CREATE THEM

// DEFAULT: _id index (created automatically on every collection)

// Every collection always has an index on _id.

// SINGLE FIELD INDEX

[Link] 10/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

[Link]({ email: 1 }) // 1 = ascending, -1 = descending

// COMPOUND INDEX (composite — same left-prefix rule as SQL!)

[Link]({ city: 1, age: -1 }) // city ascending, age descending

// UNIQUE INDEX (prevents duplicates)

[Link]({ email: 1 }, { unique: true })

// TEXT INDEX (for full-text search)

[Link]({ title: "text", body: "text" })
[Link]({ $text: { $search: "mongodb tutorial" } })

// TTL INDEX (auto-delete documents after a time period)

[Link]({ createdAt: 1 }, { expireAfterSeconds: 3600 })
// Documents auto-deleted 1 hour after createdAt — perfect for sessions!

// PARTIAL INDEX (index only a subset of documents)

[Link](
{ created_at: 1 },
{ partialFilterExpression: { status: "pending" } }
)

// MULTIKEY INDEX (index arrays — MongoDB-specific!)

[Link]({ tags: 1 })
// Automatically indexes EACH element of the tags array
// [Link]({ tags: "premium" }) → uses this index

// CHECK WHAT INDEXES EXIST

[Link]()

// EXPLAIN — see if query uses index

[Link]({ email: "amit@[Link]" }).explain("executionStats")

MONGODB INDEXING GOTCHAS

1. Same left-prefix rule as SQL: compound index {a:1, b:1, c:1} works for queries on a ,
a+b , a+b+c — NOT b alone.
2. Each index slows down writes (same trade-off as SQL).
3. MongoDB has a 64 index limit per collection.
4. Multikey indexes on arrays can get very large — index each array element.
5. TTL indexes are amazing for session data, cache entries, and logs — auto-cleanup!

[Link] 11/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

6 Scaling MongoDB — Replica Sets & Sharding

Replica Sets — High Availability & Read Scaling

WHAT IS A REPLICA SET?

A replica set is a group of MongoDB servers that maintain the same data. One server is the
primary (handles all writes), and the others are secondaries (receive copies of the data). If the
primary fails, a secondary is automatically elected as the new primary — automatic failover.

MONGODB REPLICA SET — HIGH AVAILABILITY

Application

PRIMARY
All WRITES go here

SECONDARY 1 SECONDARY 2
Async replication from Primary Async replication from Primary
Can serve reads (read preference) Automatic failover if Primary dies

REPLICA SET — KEY BEHAVIORS

AUTOMATIC FAILOVER:
1. Primary goes down (crash, network issue)
2. Secondaries detect primary is unresponsive (heartbeat timeout)
3. Secondaries hold an ELECTION
4. One secondary is PROMOTED to new primary
5. Application driver automatically reconnects to new primary
6. Total failover time: typically 10-30 seconds

READ SCALING:
Route read queries to secondaries to spread the load.
But reads from secondaries may return SLIGHTLY STALE data
(async replication lag — usually milliseconds to seconds).

MINIMUM SETUP: 3 members recommended

→ 1 Primary + 2 Secondaries (can survive 1 failure)
→ Or: 1 Primary + 1 Secondary + 1 Arbiter (votes but holds no data)

Sharding — Horizontal Scaling for Massive Data

[Link] 12/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

WHAT IS SHARDING IN MONGODB?

When your data is too large for one server (or write throughput exceeds one server's
capacity), you shard — split data across multiple servers. MongoDB has built-in sharding,
unlike SQL databases where sharding is an afterthought.

MONGODB SHARDED CLUSTER ARCHITECTURE

Application

mongos (Query Router) Config Servers

Routes queries to correct shard(s) Metadata: which shard has what

Shard 1 (Replica Set) Shard 2 (Replica Set) Shard 3 (Replica Set)

Primary Sec 1 Sec 2 Primary Sec 1 Sec 2 Primary Sec 1 Sec 2

Users A-H Users I-P Users Q-Z

(33M documents) (33M documents) (34M documents)
Each shard IS a replica set!

SHARDING COMPONENTS AND SHARD KEY SELECTION

THREE COMPONENTS OF A SHARDED CLUSTER:

1. mongos (Router)
→ Your app connects to mongos (not directly to shards)
→ mongos knows which shard has which data
→ Routes queries to the correct shard(s)
→ Multiple mongos instances for load balancing

2. Config Servers (Replica Set)

→ Store metadata: which chunks live on which shard
→ Store the shard key ranges
→ Must be a 3-member replica set for durability

3. Shards (each is a Replica Set)

→ Each shard holds a SUBSET of the data
→ Each shard is itself a replica set (for HA)
→ Data is split into CHUNKS based on shard key

SHARD KEY SELECTION — THE MOST CRITICAL DECISION:

The shard key determines HOW data is distributed across shards.

[Link] 13/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

Bad shard key = hot spots, uneven distribution, terrible performance.

GOOD shard keys:

• High cardinality (many distinct values) → even distribution
• Write distribution (writes spread across shards, not all to one)
• Query isolation (most queries target one shard, not all)
• Example: user_id → each user's data on one shard, queries by user = one shard

BAD shard keys:

• Low cardinality (e.g., status: "active"/"inactive") → 2 chunks max
• Monotonically increasing (e.g., _id, timestamp) → all writes go to
ONE shard (the one with the highest range). Creates HOT SHARD.
• Example: created_at → all new data goes to latest shard

STRATEGIES:
• Hashed shard key: hash(user_id) → even write distribution
But range queries on user_id now span all shards.
• Range shard key: user_id ranges → range queries are efficient
But risk of hot shards if distribution is uneven.
• Compound shard key: { region: 1, user_id: 1 }
→ Queries for one region go to one shard group

7 Read-Heavy vs Write-Heavy — How MongoDB Handles Both

READ-HEAVY WORKLOAD WRITE-HEAVY WORKLOAD

→ Strategy: Add read replicas (secondaries) → Strategy: Shard to distribute writes

→ Route reads to secondaries via read → Use hashed shard key for even write
preference distribution
→ Create proper indexes for hot queries → Minimize indexes (each slows writes)
→ Use covering indexes (all fields in index) → Use bulk inserts (insertMany) for batching
→ WiredTiger cache keeps hot data in RAM → Lower write concern for non-critical data
→ Consider embedding related data (avoid → WiredTiger journal handles burst writes
$lookup) → Example: IoT data, event logs, analytics
→ Example: Product catalog, user profiles → Real-world: CERN uses MongoDB for
→ Real-world: eBay uses MongoDB for particle physics data
product catalog reads

[Link] 14/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

READ-HEAVY VS WRITE-HEAVY — ARCHITECTURE PATTERNS

READ-HEAVY PATTERN (e-commerce product pages):

┌──────────┐ reads ┌───────────┐
│ App │ ──────────────→│ Secondary │ (read replica)
│ │ reads ├───────────┤
│ │ ──────────────→│ Secondary │ (read replica)
│ │ writes ├───────────┤
│ │ ──────────────→│ Primary │ (all writes here)
└──────────┘ └───────────┘
Reads spread across 3 servers. Writes go to 1.
Ratio: 90% reads / 10% writes → this works great.

WRITE-HEAVY PATTERN (IoT sensor data):

┌──────────┐ write ┌──────────────────┐
│ App │ ──────────→│ mongos (router) │
└──────────┘ └─────┬──────┬──────┘
│ │ │
┌────┘ │ └────┐
↓ ↓ ↓
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │
│ writes A │ │ writes B │ │ writes C │
└──────────┘ └──────────┘ └──────────┘
Writes distributed across 3 shards.
Each shard handles 1/3 of write volume.
Hashed shard key ensures even distribution.

8 Challenges & Trade-offs

✅ MONGODB STRENGTHS ❌ MONGODB WEAKNESSES

→ Schema flexibility — add fields without → No native JOINs — $lookup is slow at scale
migration → Data duplication — embedded data can go
→ Embedded documents — one read gets stale
everything → Limited transactions — multi-doc ACID
→ Built-in sharding — horizontal scale native since 4.0 but slower than SQL

[Link] 15/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

→ Replica sets — automatic failover → No enforced schema by default — bad

→ Rich query language — aggregation data can sneak in
pipeline → Memory hungry — WiredTiger wants lots
→ Multikey indexes — index array elements of RAM

→ TTL indexes — auto-expire documents → Shard key is immutable — can't change

after creation
→ Geo-spatial queries — location-based
search → Not ideal for highly relational data —
graphs, complex JOINs
→ Change streams — real-time event
notifications → Write amplification — indexes + journal +
replication
→ Developer experience — JSON = natural
for web devs → Eventual consistency on secondary reads
→ Size limit — 16MB per document

THE 16MB DOCUMENT LIMIT

A single MongoDB document cannot exceed 16 MB. This means you can't embed unlimited
data inside one document. If an array (like comments on a post) can grow unboundedly,
eventually the document hits 16MB and writes fail. For unbounded arrays, use a separate
collection with a reference — don't embed.

EMBEDDING VS REFERENCING — THE CORE DESIGN DECISION

EMBED when: data is always accessed together, data is bounded (won't grow forever), one-to-
few relationship (user has 1-3 addresses).

REFERENCE (separate collection) when: data is accessed independently, data can grow
unboundedly (user has millions of comments), many-to-many relationship (students ↔
courses).

9 When to Use MongoDB — And When NOT To

✅ USE MONGODB WHEN ❌ DON'T USE MONGODB WHEN

→ Data is document-shaped (user profiles, → You need complex JOINs across many
catalogs) entities

[Link] 16/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

→ Schema evolves rapidly (startups, → ACID transactions are critical across

prototypes) multiple docs
→ You need horizontal scaling built-in → Data is highly relational (many-to-many
→ Primary access = key-based lookups everywhere)

→ Data has nested structures (JSON-like) → You need strong consistency on every
read
→ Real-time analytics with aggregation
pipeline → Financial/banking data (use SQL + ACID)

→ Content management systems → Graph traversals (use Neo4j)

→ Mobile/web apps with JSON APIs → Time-series metrics at extreme scale (use
Cassandra/InfluxDB)
→ IoT data with varying sensor schemas
→ Team has zero NoSQL experience

10 Real-World Companies Using MongoDB — And Why

WHAT THEY USE MONGODB

COMPANY WHY MONGODB
FOR

eBay Product catalog, search Flexible product schemas (electronics vs clothing have
suggestions different fields). Billions of listings.

Forbes Content management Articles have varying structures (text, video, galleries).
system Schema flexibility is critical.

Adobe User data platform, Handles 100B+ events/day. Schema-less events with
analytics varying properties.

Uber Geospatial data, trip MongoDB's geospatial indexes for finding nearby drivers.
matching Location data changes constantly.

Coinbase Cryptocurrency portfolio Rapidly evolving data models as new cryptocurrencies and
data features are added.

EA Player profiles, game state Each game has different data structures. Player state
Games varies widely between games.

Toyota Connected vehicle data IoT sensor data from vehicles with varying sensor
configurations per model.

[Link] 17/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

THE PATTERN ACROSS ALL THESE COMPANIES

Notice the common thread: every company chose MongoDB because their data was naturally
document-shaped with varying fields. A product listing for a laptop has "RAM, CPU, GPU"
fields. A product listing for a shirt has "size, color, material" fields. In SQL, you'd need either a
massive table with hundreds of nullable columns or complex EAV (Entity-Attribute-Value)
patterns. In MongoDB, each document simply has the fields it needs.

11 Interview Questions & Model Answers

Q1: "WHEN WOULD YOU CHOOSE MONGODB OVER POSTGRESQL?"

When my data is naturally document-shaped with nested structures (user profiles, product
catalogs), when the schema evolves frequently (startup, rapid iteration), when I need built-in
horizontal sharding for scale, and when my primary access pattern is key-based lookups rather
than complex multi-table JOINs. I'd choose PostgreSQL when I need ACID transactions across
multiple entities, complex relational queries, or the data has many-to-many relationships.

Q2: "HOW DOES MONGODB ACHIEVE HIGH AVAILABILITY?"

Through replica sets — a group of MongoDB instances maintaining the same data. One
primary handles all writes, secondaries replicate asynchronously. If the primary fails, an
automatic election promotes a secondary within 10-30 seconds. The driver automatically
reconnects. For read scaling, you can route reads to secondaries using read preferences.

Q3: "HOW WOULD YOU HANDLE A USE CASE WHERE A DOCUMENT KEEPS GROWING ( E.G.,
COMMENTS ON A POST)?"

I would NOT embed comments inside the post document because it can grow unboundedly
and hit the 16MB document limit. Instead, I'd use a separate comments collection with a
reference: each comment has a post_id field. I'd create an index on { post_id: 1,
created_at: -1 } for efficient retrieval. This is the "reference" pattern vs the "embed"
pattern — use embedding for bounded, always-accessed-together data, and referencing for
unbounded or independently-accessed data.

Q4: "WHAT MAKES A GOOD SHARD KEY IN MONGODB?"

A good shard key has high cardinality (many distinct values for even distribution), even write
distribution (writes don't all go to one shard), and query isolation (most queries can be routed
[Link] 18/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

to a single shard). Example: user_id is often a good shard key. A bad shard key is something
monotonically increasing like created_at because all new writes go to the latest shard,
creating a hot spot.

Q5: "DOES MONGODB SUPPORT TRANSACTIONS?"

Yes, since MongoDB 4.0 (2018), it supports multi-document ACID transactions across replica
sets, and since 4.2, across sharded clusters. However, transactions are slower than SQL
transactions and should be used sparingly. MongoDB's philosophy is to design your data
model so that most operations affect a single document (which is always atomic) and only use
multi-document transactions when absolutely necessary.

Q6: "EMBEDDING VS REFERENCING — HOW DO YOU DECIDE?"

Embed when: the data is always read together, the relationship is 1-to-few (user has 2-3
addresses), the embedded data is bounded and won't grow forever.
Reference when: the data is accessed independently, the relationship is 1-to-many or many-
to-many, the array could grow unboundedly (comments, orders), or the referenced data
changes frequently and you don't want to update it everywhere.

📋 MongoDB — Complete Quick Revision

WHAT IS MONGODB?
Document-oriented NoSQL. Stores JSON-like documents (BSON) in collections. Schema-less
— each document can have different fields. No tables, no rows, no JOINs by default.

CORE CONCEPTS
Database → Collection (table) → Document (row) → Field (column). _id = auto-generated
ObjectId (primary key). Embed related data inside documents.

STORAGE ENGINE
WiredTiger: document-level locking, compression (Snappy/Zstd), journal (WAL for durability),
in-memory cache (50% RAM), B-Tree indexes.

QUERIES
find() , insertOne/Many() , updateOne/Many() , deleteOne/Many() . Aggregation pipeline:
$match → $group → $sort → $project → $lookup (JOIN).

[Link] 19/20
04/04/2026, 13:14 MongoDB — Complete Deep Dive from Zero to Expert

INDEXES
B-Tree indexes. Single, compound (left-prefix rule!), unique, text, TTL (auto-expire), partial,
multikey (arrays). 64 index limit per collection.

SCALING
Replica Sets: 1 primary + N secondaries. Automatic failover. Read scaling via secondaries.
Async replication.
Sharding: Built-in. mongos router → config servers → shards (each a replica set). Shard key =
most critical decision. Hashed (even writes) vs ranged (efficient range queries).

READ VS WRITE HEAVY

Read-heavy: add secondaries, proper indexes, embedding, caching. Write-heavy: shard with
hashed key, fewer indexes, bulk inserts, lower write concern.

EMBED VS REFERENCE
Embed: always accessed together, bounded, 1-to-few. Reference: independent access,
unbounded growth, many-to-many. 16MB doc limit!

WHEN TO USE
Use: document-shaped data, evolving schema, horizontal scale, key-based lookups, nested
data, CMS, catalogs, mobile apps.
Don't use: complex JOINs, multi-doc ACID critical, highly relational, financial/banking, graph
traversals.

COMPANIES
eBay (catalogs), Forbes (CMS), Adobe (analytics), Uber (geospatial), Coinbase (crypto), EA
Games (player state), Toyota (IoT).

[Link] 20/20
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

N O S Q L D E E P D I V E — W I D E - C O L U M N S TO R E

Apache Cassandra — Complete Deep Dive

From zero to expert: the write-optimised distributed database — LSM Trees, SSTables,
partitioning, tunable consistency, and why the biggest companies use it for billions of
events

1 What Is Apache Cassandra? (Starting From Zero)

DEFINITION

Apache Cassandra is a distributed, wide-column NoSQL database designed for massive

write throughput, high availability, and linear horizontal scaling. It was originally built at
Facebook to power their inbox search feature and later open-sourced. It's now used by Apple,
Netflix, Instagram, Discord, and Uber to handle billions of events per day.

THE POSTAL SYSTEM ANALOGY

Imagine a postal system with hundreds of post offices across a country. When you send a
letter (write), it goes to the nearest post office — no single central office handles everything.
Each post office stores letters for its local area. If one post office burns down, the others still
work. Letters are eventually delivered to all relevant offices (eventual consistency). No single
post office is the "boss" — they're all equal (peer-to-peer). That's Cassandra — no single
master, every node is equal, writes go to the nearest available node.

Why Cassandra Exists — The Problem It Solves

THE PROBLEM TRADITIONAL DATABASES CAN'T SOLVE

SCENARIO: Instagram Activity Feed

→ 2 billion users
→ When a celebrity posts, millions of followers' feeds need updating
→ 1 post = millions of write operations
→ Writes must be fast (users expect instant feed updates)
→ System must NEVER go down (even during server failures)
→ Data spans multiple continents (US, Europe, Asia)

[Link] 1/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

WHY SQL CAN'T HANDLE THIS:

→ Single master = bottleneck for writes
→ Vertical scaling has a ceiling
→ Multi-region ACID = extremely expensive and slow
→ One master fails = entire system down until failover

WHY MONGODB CAN'T HANDLE THIS WELL:

→ Primary handles ALL writes = single-node write bottleneck
→ Sharding helps but primary per shard is still a bottleneck
→ Not optimised for extreme write throughput

WHY CASSANDRA WAS BUILT:

→ NO single master — every node accepts writes (peer-to-peer)
→ Writes are the CHEAPEST operation (LSM Tree architecture)
→ Linear scaling — add nodes, get proportional throughput
→ Multi-datacenter replication built into the core
→ Designed to NEVER go down

Cassandra vs MongoDB vs SQL — Quick Positioning

ASPECT SQL (POSTGRESQL) MONGODB CASSANDRA

Architecture Single master Primary + Peer-to-peer (no master!)

secondaries

Write Speed Moderate Fast Extremely fast (LSM Tree)

Read Speed Fast (B+ Tree) Fast (B-Tree) Moderate (multi-level reads)

Consistency Strong (ACID) Configurable Tunable (ONE → QUORUM →

ALL)

Query Power Rich JOINs, SQL Aggregation pipeline Limited (no JOINs, restricted
WHERE)

Horizontal Complex (sharding) Built-in sharding Native, linear, effortless

Scale

Availability Single point of failure Automatic failover Always available (no master)

Best For Complex queries, Flexible docs, Massive writes, time-series, IoT
ACID catalogs

[Link] 2/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

2 Cassandra's Data Model — Partitions, Rows & Columns

THE MOST IMPORTANT CONCEPT

Cassandra's data model looks like SQL tables on the surface, but underneath it works
completely differently. The partition key determines which node stores the data. The
clustering key determines the sort order within a partition. Getting this right is the single most
critical design decision.

CASSANDRA TERMINOLOGY MAPPED TO SQL

SQL Concept Cassandra Equivalent Key Difference

──────────────────────────────────────────────────────────────────
Database Keyspace Also defines replication strategy
Table Table Looks similar but storage is different
Row Partition (+ rows inside) Grouped by partition key
Primary Key Partition Key + Clustering Key Determines data placement + sort
Column Column Each row can have different columns
Index Secondary Index / SAI Limited compared to SQL indexes
JOIN DOES NOT EXIST Denormalise instead!

Partition Key & Clustering Key — The Heart of Cassandra

UNDERSTANDING PARTITION KEY AND CLUSTERING KEY

CREATE TABLE messages (

channel_id UUID, -- PARTITION KEY: determines which node
message_id TIMEUUID, -- CLUSTERING KEY: sort order within partition
author TEXT,
content TEXT,
created_at TIMESTAMP,
PRIMARY KEY (channel_id, message_id)
);
-- ↑ ↑
-- partition key clustering key

WHAT THE PARTITION KEY DOES:

→ hash(channel_id) = determines which NODE stores this data
→ All messages for channel "abc" land on the SAME node
→ This is why reads for one channel are fast — all data is local

[Link] 3/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

WHAT THE CLUSTERING KEY DOES:

→ Within a partition, rows are SORTED by message_id
→ message_id is a TIMEUUID (contains timestamp)
→ So messages within a channel are automatically sorted by time
→ Range queries within a partition are extremely efficient:
"Get messages from channel X between 9am and 10am"

RESULT:
Partition: channel_id = "gaming-chat"
┌──────────────────────────────────────────────────────┐
│ message_id (sorted!) │ author │ content │
├──────────────────────┼─────────┼─────────────────────┤
│ 2024-03-15T09:00:01 │ Amit │ "Hello everyone!" │
│ 2024-03-15T09:00:05 │ Sara │ "Hey Amit!" │
│ 2024-03-15T09:01:12 │ Rahul │ "Good morning" │
│ 2024-03-15T09:03:45 │ Amit │ "How's it going?" │
│ ...millions more... │ │ │
└──────────────────────┴─────────┴─────────────────────┘
All sorted by time within this partition → range reads are FAST

CRITICAL RULE: PARTITION SIZE

All data with the same partition key lives on the same node. If a partition grows too large
(100MB+), it becomes a hot spot — one node handles disproportionate traffic while others are
idle. This is why Discord uses (channel_id, bucket) — the bucket splits very active
channels into time-based chunks so no single partition gets too big.

Compound Partition Keys & Composite Keys

DIFFERENT KEY DESIGNS FOR DIFFERENT USE CASES

-- SIMPLE PARTITION KEY (most common)

PRIMARY KEY (user_id)
→ All data for one user on one node.
→ Good when queries always filter by user_id.

-- COMPOUND PARTITION KEY (split large partitions)

PRIMARY KEY ((channel_id, bucket), message_id)
→ Data split by channel AND bucket (e.g., daily bucket)
→ Prevents single channel from creating a mega-partition
→ Discord uses this exact pattern!

-- COMPOSITE KEY (partition + multiple clustering)

[Link] 4/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

PRIMARY KEY (user_id, year, month)

→ Partition by user_id, sorted by year then month
→ "Get all records for user X in 2024" = efficient range query

3 Internal Architecture — Why Writes Are So Fast

THE CORE INNOVATION: LSM TREE

Cassandra uses a Log-Structured Merge Tree (LSM Tree) instead of a B+ Tree. This is the
fundamental reason Cassandra writes are so fast. In a B+ Tree (SQL/MongoDB), writes require
finding the right position on disk and updating in-place — this means random disk I/O. In an
LSM Tree, writes go to an in-memory table first, then get flushed sequentially to disk.
Sequential writes are 100x faster than random writes.

The Write Path — Step by Step

CASSANDRA WRITE PATH — LSM TREE ARCHITECTURE

Client Write Request

① COMMIT LOG (on disk) + ② MEMTABLE (in RAM)

Append-only sequential write — FAST. Durability guarantee. In-memory sorted data structure. Fastest possible write.

Write ACK sent to client here! ✅

No disk seek needed — that's why writes are fast
when full

WHY THIS IS FAST:

③ SSTABLE (on disk)
1. Commit log = sequential append (no seek) Sorted String Table — immutable, sequential write to disk.
2. Memtable = RAM write (microseconds)
3. Client gets ACK after step 1+2
4. SSTable flush happens in background
5. No random disk I/O at write time! ④ COMPACTION (background process)
Merges multiple SSTables into fewer, larger ones.
Result: millions of writes/second
Removes deleted data (tombstones) and duplicates. Reclaims space.
on commodity hardware

WRITE PATH — DETAILED STEP BY STEP

Step 1: COMMIT LOG (Append-Only, On Disk)

→ Write is appended to the end of the commit log file
→ This is a SEQUENTIAL write (fast — no disk seeking)
→ Purpose: DURABILITY. If server crashes, replay the log to recover.
→ Think of it like a journal — log every operation for safety.

[Link] 5/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

Step 2: MEMTABLE (In-Memory Sorted Structure)

→ Simultaneously, the write goes into the Memtable (in RAM)
→ Memtable is a sorted data structure (like a Red-Black Tree)
→ Writes to RAM = microseconds (no disk I/O!)
→ ACK is sent to the client RIGHT HERE.
The client doesn't wait for the data to reach disk files.
This is WHY Cassandra writes are so fast.

Step 3: SSTABLE FLUSH (Memtable → Disk)

→ When Memtable reaches a size threshold (e.g., 64MB),
it is FLUSHED to disk as an SSTable (Sorted String Table)
→ SSTable is IMMUTABLE — once written, never modified
→ Write is sequential (dump entire sorted table at once)
→ Old commit log entries for this data can now be deleted

Step 4: COMPACTION (Background Merge)

→ Over time, many SSTables accumulate on disk
→ Compaction merges multiple SSTables into fewer, larger ones
→ Removes tombstones (deleted data markers)
→ Removes duplicate/overwritten values (keeps latest)
→ Reclaims disk space
→ Strategies: Size-Tiered (default), Leveled, Time-Window

The Read Path — Why Reads Are Slower

READ PATH — STEP BY STEP

Query: SELECT * FROM messages WHERE channel_id = 'abc' AND message_id = 'xyz';

Step 1: Check MEMTABLE (RAM)

→ Is the data in the current in-memory table?
→ If YES → return immediately (fastest possible read)

Step 2: Check BLOOM FILTERS (probabilistic check)

→ For each SSTable on disk, Cassandra has a Bloom Filter
→ Bloom Filter says: "This SSTable DEFINITELY doesn't have the key"
or "This SSTable MIGHT have the key"
→ Skips SSTables that definitely don't have the data
→ Avoids reading unnecessary files (huge optimisation)

Step 3: Check KEY CACHE (RAM)

→ Caches the byte offset of partition keys in SSTables
→ If key is cached: jump directly to the right position on disk
→ Avoids scanning the SSTable index

[Link] 6/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

Step 4: Check PARTITION INDEX → PARTITION SUMMARY

→ If not in key cache, use the on-disk index to find the partition
→ Partition Summary (in RAM) narrows down the search range
→ Partition Index (on disk) gives exact byte offset

Step 5: Read from SSTABLE on disk

→ Use the byte offset to read the actual data
→ May need to read from MULTIPLE SSTables
(data could be spread across several — before compaction)

Step 6: MERGE results from all sources

→ Combine data from Memtable + multiple SSTables
→ Use timestamps to keep the LATEST version of each column
→ Return the final, merged result to the client

WHY READS ARE SLOWER THAN WRITES:

→ Writes: 2 steps (commit log + memtable) → done
→ Reads: up to 5+ steps, may hit multiple SSTables on disk
→ Compaction reduces the number of SSTables (improves reads)
→ Bloom Filters prevent unnecessary SSTable reads
→ Key cache and row cache further accelerate hot data

B+ TREE VS LSM TREE — THE FUNDAMENTAL DIFFERENCE

B+ Tree (SQL/MongoDB): Reads are fast (sorted tree, one lookup). Writes are slower (find
position, update in-place, random I/O). Read-optimised.

LSM Tree (Cassandra): Writes are fast (append to log + RAM, sequential I/O). Reads are
slower (check memtable + multiple SSTables). Write-optimised.

This is the fundamental trade-off. Cassandra chose to optimise for writes because its target
use cases (IoT, feeds, messaging) are write-heavy.

4 Distributed Architecture — No Master, Everyone Is Equal

PEER-TO-PEER ARCHITECTURE

Unlike SQL (single master) or MongoDB (primary + secondaries), Cassandra uses a peer-to-
peer architecture. Every node is equal — there is no master, no primary, no leader. Any node

[Link] 7/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

can accept reads and writes. If one node goes down, the others continue serving traffic
seamlessly. No failover election needed because there's nothing to fail over FROM.

CASSANDRA'S RING ARCHITECTURE — CONSISTENT HASHING

Token: 0-24

Node A
Node F is DOWN
Other nodes handle its
data via replication → no downtime!
DOWN! ✗ Token: 25-49
Node F Node B

Write: user_id = "amit"

hash("amit") = 37
→ Token range 25-49
→ Goes to Node B

Node E Node C
Token: 100-124 Token: 50-74

Node D

Token: 75-99

CONSISTENT HASHING — HOW DATA IS DISTRIBUTED

THE TOKEN RING:

→ Cassandra uses CONSISTENT HASHING to distribute data
→ The entire key space is arranged in a RING (0 to 2^63)
→ Each node is assigned a TOKEN RANGE on the ring
→ When you write data, Cassandra hashes the partition key:
hash(partition_key) → token number → which node's range?

EXAMPLE:
Node A: tokens 0-24 Node D: tokens 75-99
Node B: tokens 25-49 Node E: tokens 100-124
Node C: tokens 50-74 Node F: tokens 125-149

Write user "amit": hash("amit") = 37 → falls in 25-49 → Node B

Write user "priya": hash("priya") = 88 → falls in 75-99 → Node D

ADDING A NEW NODE:

Add Node G with tokens 30-39
→ Node B gives up tokens 30-39 to Node G
→ Only data in token range 30-39 needs to move
→ Other nodes are NOT affected (minimal data movement)
→ This is WHY scaling is LINEAR and easy

Replication — How Data Is Copied for Safety

[Link] 8/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

REPLICATION FACTOR AND HOW COPIES ARE DISTRIBUTED

REPLICATION FACTOR (RF) = number of copies of each piece of data

RF = 3 (most common production setting):

→ Each piece of data is stored on 3 DIFFERENT nodes
→ If one node dies, 2 copies still exist
→ If two nodes die, 1 copy still exists

HOW REPLICAS ARE PLACED:

Write: hash("amit") = 37 → primary = Node B (tokens 25-49)
→ Replica 1: Node B (primary)
→ Replica 2: Node C (next node clockwise on ring)
→ Replica 3: Node D (next node after C)

3 copies on 3 different nodes:

┌──────────────────────────────────┐
│ Node B: amit's data (primary) │
│ Node C: amit's data (replica) │
│ Node D: amit's data (replica) │
└──────────────────────────────────┘

MULTI-DATACENTER REPLICATION:
CREATE KEYSPACE myapp WITH replication = {
'class': 'NetworkTopologyStrategy',
'us-east': 3, -- 3 copies in US East
'eu-west': 3, -- 3 copies in EU West
'ap-south': 3 -- 3 copies in Asia
};
→ Data automatically replicated across 3 continents
→ Local reads are fast (nearby datacenter)
→ If entire US datacenter goes down, EU and Asia still work

5 Tunable Consistency — Cassandra's Superpower

WHAT IS TUNABLE CONSISTENCY?

Unlike SQL (always strong) or most NoSQL (always eventual), Cassandra lets you choose per-
query how consistent you want the response to be. You control the trade-off between
consistency and latency/availability for each individual read and write operation.

[Link] 9/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

CONSISTENCY LEVELS — FROM WEAKEST TO STRONGEST

With Replication Factor = 3 (3 copies on 3 nodes):

WRITE Consistency Levels:

ONE → Write to 1 node, ACK immediately.
Fastest. Risk: if that node dies before replicating, data lost.
QUORUM → Write to majority (2 of 3 nodes), then ACK.
Good balance. Survives 1 node failure.
ALL → Write to ALL 3 nodes, then ACK.
Slowest. Strongest guarantee. If any node is down, write FAILS.

READ Consistency Levels:

ONE → Read from 1 node (nearest). Fastest. May return stale data.
QUORUM → Read from majority (2 of 3). Compare responses, return latest.
Good balance. Higher latency than ONE.
ALL → Read from ALL 3 nodes. Compare all. Strongest but slowest.

THE MAGIC FORMULA for strong consistency:

W + R > RF → STRONG CONSISTENCY

Where: W = write consistency, R = read consistency, RF = replication factor

Examples (RF = 3):

W=QUORUM(2) + R=QUORUM(2) = 4 > 3 → STRONG ✓ (most common!)
W=ALL(3) + R=ONE(1) = 4 > 3 → STRONG ✓
W=ONE(1) + R=ONE(1) = 2 < 3 → EVENTUAL ✗ (fastest but may read stale)

TUNABLE CONSISTENCY — PER QUERY CONTROL (RF=3)

CL = ONE CL = QUORUM CL = ALL

✓ ACK wait wait ✓ ACK ✓ ACK async ✓ ACK ✓ ACK ✓ ACK

Fastest ⚡ Balanced ⚖️ Strongest but Slowest 🐢

1 node confirms Majority (2 of 3) confirms ALL 3 nodes must confirm
May read stale data Strong if W+R > RF 1 node down = operation FAILS
Use for: analytics, logs Use for: most production Use for: rarely (sacrifices HA)

6 Querying Cassandra — CQL (Cassandra Query Language)

[Link] 10/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

CQL looks like SQL but has strict constraints on what you can query. This is by design —
Cassandra trades query flexibility for performance.

CQL CRUD OPERATIONS

-- CREATE KEYSPACE (like a database)

CREATE KEYSPACE myapp
WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3};

USE myapp;

-- CREATE TABLE
CREATE TABLE messages (
channel_id UUID,
message_id TIMEUUID,
author TEXT,
content TEXT,
created_at TIMESTAMP,
PRIMARY KEY (channel_id, message_id)
) WITH CLUSTERING ORDER BY (message_id DESC); -- newest first

-- INSERT
INSERT INTO messages (channel_id, message_id, author, content, created_at)
VALUES (uuid(), now(), 'Amit', 'Hello world!', toTimestamp(now()));

-- READ (must include partition key!)

SELECT * FROM messages WHERE channel_id = some-uuid;

-- READ with clustering key range (time-range query)

SELECT * FROM messages
WHERE channel_id = some-uuid
AND message_id > minTimeuuid('2024-03-15 09:00:00')
AND message_id < minTimeuuid('2024-03-15 10:00:00');

-- UPDATE (actually an UPSERT — insert if not exists)

UPDATE messages SET content = 'Edited message'
WHERE channel_id = some-uuid AND message_id = some-timeuuid;

-- DELETE
DELETE FROM messages
WHERE channel_id = some-uuid AND message_id = some-timeuuid;

-- DELETE with TTL (auto-delete after 7 days)

INSERT INTO messages (channel_id, message_id, author, content)
VALUES (uuid(), now(), 'Amit', 'Temporary message')
USING TTL 604800; -- 604800 seconds = 7 days

[Link] 11/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

CQL RESTRICTIONS — WHAT YOU CANNOT DO

1. No JOINs. Period. Denormalise your data instead.

2. WHERE clause must include the partition key. You can't do SELECT * FROM messages
WHERE author = 'Amit' without a secondary index — Cassandra doesn't know which node
has it.
3. No GROUP BY on arbitrary columns. Aggregations are limited.
4. No subqueries. Each query is a simple key-based lookup.
5. Range queries only on clustering key, not partition key.

The rule: Model your tables based on your queries, NOT based on your entities. In SQL you
model entities then figure out queries. In Cassandra you start with queries and design tables to
serve them.

QUERY-FIRST DATA MODELLING

In Cassandra, you create one table per query pattern. If you have 5 different ways to query the
same data, you create 5 tables — each optimised for one query. This means data duplication
is normal and expected. The trade-off: more disk space + write amplification, but reads are
always fast.

QUERY-FIRST MODELLING — EXAMPLE

QUERIES WE NEED:
Q1: Get all messages in a channel (sorted by time)
Q2: Get all messages by a specific user (across all channels)
Q3: Get message count per channel

TABLES WE CREATE (one per query!):

-- Table for Q1: messages by channel

CREATE TABLE messages_by_channel (
channel_id UUID,
message_id TIMEUUID,
author TEXT, content TEXT,
PRIMARY KEY (channel_id, message_id)
);

-- Table for Q2: messages by user (SAME DATA, different partition key)
CREATE TABLE messages_by_user (
user_id UUID,
message_id TIMEUUID,
channel_id UUID, content TEXT,

[Link] 12/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

PRIMARY KEY (user_id, message_id)

);

-- Table for Q3: message counts (pre-computed counter)

CREATE TABLE channel_message_counts (
channel_id UUID PRIMARY KEY,
message_count COUNTER
);

EVERY INSERT writes to ALL relevant tables.

Data is duplicated but each query is served by ONE table with ONE lookup.

7 Scaling Cassandra — Linear and Effortless

HOW CASSANDRA SCALES LINEARLY

LINEAR SCALING — The Dream:

6 nodes = 60,000 writes/second

12 nodes = 120,000 writes/second (just doubled capacity!)
24 nodes = 240,000 writes/second (doubled again!)
100 nodes = 1,000,000 writes/second

Why is it linear?
→ No master = no bottleneck
→ Each node handles its own partition range
→ Adding a node = existing nodes give it some token ranges
→ Data automatically rebalances in the background
→ No downtime during scaling!

HOW TO ADD A NODE:

1. Start new Cassandra process on new server
2. Tell it to join the existing cluster
3. Cassandra automatically assigns token ranges
4. Data starts streaming from existing nodes to new node
5. Once streaming is complete, new node is fully operational
6. No application changes needed!

COMPARED TO SQL:
SQL: Add a server → manually set up sharding, update routing
logic, migrate data, test, pray → days/weeks of work

[Link] 13/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

Cassandra: Add a server → join cluster → automatic → minutes

8 Read-Heavy vs Write-Heavy Workloads

WRITE-HEAVY (CASSANDRA'S SWEET READ-HEAVY (CASSANDRA WORKS, NOT

SPOT) OPTIMAL)

→ LSM Tree = writes go to RAM then → Reads may hit multiple SSTables (pre-
sequential disk compaction)
→ No locks, no contention on writes → No JOINs — must denormalise for each
→ Peer-to-peer = writes distributed across all query
nodes → Key cache + row cache help for hot data
→ Millions of writes/sec on commodity → Compaction improves read performance
hardware over time
→ Examples: IoT sensors, activity feeds, logs, → For read-heavy with complex queries →
messaging, metrics, event sourcing use MongoDB or SQL
→ Real-world: Instagram (billions of → For read-heavy with simple key lookups →
likes/day), Discord (billions of messages) Cassandra works

9 Challenges & Trade-offs

✅ CASSANDRA STRENGTHS ❌ CASSANDRA WEAKNESSES

→ Extreme write throughput (LSM Tree) → No JOINs — must denormalise everything
→ Linear horizontal scaling — add nodes, → Limited query flexibility — must include
done partition key
→ No single point of failure (peer-to-peer) → Reads slower than writes (multiple
→ Multi-datacenter replication built-in SSTable lookups)

→ Tunable consistency per query → Data modelling is hard — query-first, not

entity-first

[Link] 14/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

→ Always available — survives node/rack/DC → No ad-hoc queries — must plan all queries
failures upfront
→ Time-series optimised (clustering by time) → Tombstones — deletes create markers that
→ TTL support — auto-expire data slow reads

→ Proven at massive scale (Apple: 150K+ → Compaction — uses CPU and I/O in
nodes) background
→ Operational complexity — tuning, repair,
monitoring
→ Steep learning curve for developers from
SQL background

THE TOMBSTONE PROBLEM

When you DELETE data in Cassandra, it doesn't immediately remove it. Instead, it writes a
tombstone — a marker that says "this data is deleted." During compaction, tombstones are
eventually cleaned up. But until then, reads must scan through tombstones, which slows down
reads. If you delete a lot of data frequently, tombstones can accumulate and cripple read
performance. This is why Cassandra is best for append-heavy workloads where deletes are
rare.

10 When to Use Cassandra — And When NOT To

✅ USE CASSANDRA WHEN ❌ DON'T USE CASSANDRA WHEN

→ Write-heavy workload (logs, IoT, feeds, → You need complex JOINs or ad-hoc
messaging) queries
→ You need massive horizontal scale → ACID transactions across multiple
→ High availability is non-negotiable (zero partitions
downtime) → Data is highly relational (many-to-many)
→ Data is time-series or append-only → Small dataset (<10GB — overkill for this)
→ Multi-region deployment needed → You need flexible querying (SQL or
→ Queries are simple key-based lookups MongoDB better)

→ Can tolerate eventual consistency (or use → Heavy updates/deletes (tombstone

QUORUM) problem)

[Link] 15/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

→ Data model is known upfront (query-first → Team has no distributed systems

design) experience
→ Read-heavy with complex aggregations

11 Real-World Companies Using Cassandra — And Why

WHAT THEY USE

COMPANY SCALE WHY CASSANDRA
CASSANDRA FOR

Apple iCloud, Siri, Maps, iTunes 150,000+ Largest known deployment. Multi-DC
nodes replication for global availability.

Netflix User viewing history, 2,500+ nodes Write-heavy viewing events. Multi-
recommendations region for global streaming.

Instagram Activity feeds, likes, Billions Massive write throughput. Eventual

notifications events/day consistency fine for likes.

Discord Message storage Billions of Append-only messages. Time-range

messages queries. Partition: (channel, bucket).

Uber Driver/rider location, trip Millions Real-time location writes. Multi-DC for
data trips/day global operations.

Spotify User activity, playlists Hundreds of Write-heavy user events. Playlist data
nodes with flexible schema.

Twitter/X Timeline storage, analytics Massive scale Timeline fan-out writes. Append-only
tweets.

THE PATTERN ACROSS ALL COMPANIES

Every company using Cassandra has the same characteristics: write-heavy workload (feeds,
events, messages, metrics), time-series or append-only data, simple key-based access
patterns, and multi-region deployment. None of them use Cassandra for complex relational
queries or heavy JOINs — they pair it with SQL or MongoDB for those needs.

[Link] 16/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

12 Interview Questions & Model Answers

Q1: "WHY ARE WRITES SO FAST IN CASSANDRA?"

Because of the LSM Tree architecture. Writes go to an in-memory Memtable (microseconds)

and an append-only commit log (sequential I/O — fast). The client gets acknowledgment
immediately. Data is flushed to immutable SSTables on disk in the background. No random disk
I/O at write time — sequential writes are 100x faster than random.

Q2: "HOW DOES CASSANDRA ACHIEVE HIGH AVAILABILITY WITH NO MASTER?"

Peer-to-peer architecture with consistent hashing. Every node is equal — any node can
accept reads and writes. Data is replicated to multiple nodes (RF=3 typically). If one node dies,
others serve its data from replicas. No election, no failover delay. The ring topology with
consistent hashing distributes data evenly and makes adding/removing nodes seamless.

Q3: "HOW WOULD YOU ACHIEVE STRONG CONSISTENCY IN CASSANDRA?"

Use QUORUM for both reads and writes. With RF=3, QUORUM means 2 out of 3 nodes must
respond. Write QUORUM (2) + Read QUORUM (2) = 4 > RF (3), which satisfies the formula W
+ R > RF for strong consistency. This means any read will always see the latest write. The
trade-off is higher latency compared to consistency level ONE.

Q4: "WHAT MAKES A GOOD PARTITION KEY IN CASSANDRA?"

A good partition key has high cardinality (many distinct values for even distribution), avoids
hot partitions (no one value dominates), and matches query patterns (queries should include
the partition key). Example: user_id for user-specific data. Bad example: country (few
values = uneven distribution) or created_date (all today's writes go to one partition).

Q5: "WHY DID DISCORD MIGRATE FROM MONGODB TO CASSANDRA FOR MESSAGES?"

Messages are append-only (never updated), always queried by time range within a channel,
and need predictable low latency at massive scale. MongoDB's B-Tree storage wasn't
optimised for this pattern — read latency degraded at scale. Cassandra's LSM Tree handles
append-heavy writes better, and clustering keys provide natural time-ordering within
partitions. Partition key (channel_id, bucket) prevents any single partition from growing too
large.

Q6: "WHAT IS THE TOMBSTONE PROBLEM?"

[Link] 17/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

When you delete data in Cassandra, it doesn't immediately remove it — it writes a tombstone
marker. Tombstones are cleaned up during compaction, but until then, reads must scan
through them. If you delete a lot of data frequently, tombstones accumulate and slow down
reads significantly. Mitigation: use TTL for time-based auto-expiry instead of explicit deletes,
tune compaction strategy, avoid frequent delete-heavy patterns.

Q7: "WHEN WOULD YOU CHOOSE CASSANDRA OVER MONGODB?"

When I need extreme write throughput (millions/sec), multi-datacenter replication as a core

feature, always-on availability with no single point of failure, and my access pattern is time-
series or simple key-based lookups. MongoDB is better when I need flexible ad-hoc queries,
aggregation pipeline power, or my data is naturally document-shaped. Cassandra forces you to
know all query patterns upfront; MongoDB is more flexible for exploration.

📋 Apache Cassandra — Complete Quick Revision

WHAT IS CASSANDRA?
Distributed wide-column NoSQL. Peer-to-peer (no master). Write-optimised (LSM Tree). Built
for massive scale, high availability, multi-DC. Built at Facebook, used by Apple (150K+ nodes).

DATA MODEL
Keyspace → Table → Partitions → Rows. Partition Key = which node stores data. Clustering
Key = sort order within partition. Model tables per query, not per entity. Data duplication is
normal.

WRITE PATH (LSM TREE)

Commit Log (sequential append, durability) + Memtable (RAM, fast) → ACK to client → Flush to
SSTable (immutable, on disk) → Compaction (background merge). No random I/O at write time.

READ PATH
Memtable → Bloom Filters (skip irrelevant SSTables) → Key Cache → Partition Index → SSTable
→ Merge results. Reads check multiple SSTables (slower than writes).

DISTRIBUTION
Consistent hashing ring. hash(partition_key) → token → node. Replication Factor (RF=3): 3
copies on 3 nodes. Multi-DC: NetworkTopologyStrategy. Add nodes = linear scaling, no
downtime.

TUNABLE CONSISTENCY
[Link] 18/19
04/04/2026, 13:39 Apache Cassandra — Complete Deep Dive from Zero to Expert

ONE (fastest, may be stale) → QUORUM (majority, balanced) → ALL (strongest, slowest). Strong
consistency: W + R > RF . QUORUM + QUORUM with RF=3 = strong.

CQL
Looks like SQL but no JOINs, no arbitrary WHERE, must include partition key. CREATE TABLE,
INSERT, SELECT, UPDATE, DELETE, TTL. Aggregation limited.

TRADE-OFFS
Strengths: Extreme writes, linear scale, no SPOF, multi-DC, tunable consistency, time-series,
TTL.
Weaknesses: No JOINs, limited queries, reads slower, tombstone problem, data modelling is
hard, operational complexity, steep learning curve.

WHEN TO USE
Use: Write-heavy, time-series, IoT, feeds, messaging, metrics, multi-region, always-on. Don't
use: Complex JOINs, ad-hoc queries, heavy deletes, small data, relational data, read-heavy with
aggregations.

COMPANIES
Apple (150K nodes), Netflix, Instagram, Discord, Uber, Spotify, Twitter. All use it for write-heavy,
append-only, time-series data.

[Link] 19/19
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

N O S Q L D E E P D I V E — G R A P H DATA B A S E

Neo4j — Complete Deep Dive

From zero to expert: what graphs are, how Neo4j stores and traverses data, Cypher
queries, why JOINs can't compete, scaling, and real-world use cases

1 What Is Neo4j? (Starting From Zero)

DEFINITION

Neo4j is a graph database — it stores data as nodes (entities/things) and relationships

(connections between things). Unlike SQL tables or MongoDB documents, the connections
between data are first-class citizens — stored, indexed, and traversed natively. When your
questions are about how things are connected, graph databases are orders of magnitude
faster than anything else.

THE SOCIAL NETWORK ANALOGY

Imagine a whiteboard with sticky notes and strings. Each sticky note is a person (node). You
draw a string between two notes to show they're friends (relationship). The string itself can
have a label — "friends since 2020" or "works with." To find "friends of friends of Amit," you
just follow the strings — you don't need to look up anything in a table. That's how Neo4j works.
Following connections is instant because the connections are physically stored alongside the
data.

Why Graph Databases Exist — The Problem SQL Can't Solve Efficiently

THE PROBLEM: "FIND FRIENDS OF FRIENDS" IN SQL VS GRAPH

QUESTION: "Find all friends-of-friends of Amit"

IN SQL (relational database):

Table: friendships (user_id, friend_id)

-- Level 1: Amit's direct friends

SELECT friend_id FROM friendships WHERE user_id = 'Amit';
-- Returns: Priya, Sara
[Link] 1/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

-- Level 2: Friends of Amit's friends (friends of friends)

SELECT f2.friend_id
FROM friendships f1
JOIN friendships f2 ON f1.friend_id = f2.user_id
WHERE f1.user_id = 'Amit';
-- That's a SELF-JOIN on a potentially HUGE table

-- Level 3: Friends of friends of friends?

-- Another JOIN. And another. Each level = one more JOIN.
-- With 1 billion users: each JOIN scans the entire table.

PERFORMANCE:
Level 1: fast
Level 2: slow (1 JOIN on 1B rows)
Level 3: very slow (2 JOINs)
Level 4: unbearably slow (3 JOINs)
Level 5: times out or crashes

IN NEO4J (graph database):

MATCH (amit:Person {name: "Amit"})-[:FRIENDS*1..3]-(fof)
RETURN DISTINCT [Link];

-- This traverses 1 to 3 levels of FRIENDS relationships

-- Starting from Amit's node, follow the FRIENDS edges

PERFORMANCE:
Level 1: instant
Level 2: instant
Level 3: instant
Level 4: instant
Level 5: STILL instant!

WHY? Because Neo4j doesn't scan a table.

It starts at Amit's node and WALKS the connections.
The time depends on the number of connections traversed,
NOT on the total size of the database.
10 users or 10 billion users — same speed for local traversal.

[Link] 2/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

SQL JOINs vs GRAPH TRAVERSAL — Finding "Friends of Friends"

SQL APPROACH — Self-JOINs GRAPH APPROACH — Walk the Edges

friendships TABLE (1 billion rows) Priya Neha
user_id | friend_id
Amit → Priya, Amit → Sara, Sara → Rahul, ... Amit Rahul

Each level deeper = another JOIN on this HUGE table

Sara

Level 1 (direct friends): ~1ms Vikram

Level 2 (JOIN on 1B rows): ~10s Level 1: ~1ms

Level 3 (2 JOINs on 1B): ~minutes Level 2: ~2ms
Level 4 (3 JOINs on 1B): ~timeout! Level 3: ~3ms
Level 5: 💀 crashes Level 4: ~4ms
Level 5: ~5ms ✅
Gets EXPONENTIALLY slower Stays CONSTANT regardless
with each level of depth of total database size!

THE FUNDAMENTAL DIFFERENCE

SQL: Relationship traversal time depends on the total size of the table. 1B rows → every JOIN
scans parts of 1B rows. Gets exponentially slower with depth.

Neo4j: Relationship traversal time depends on the number of connections traversed, NOT the
total database size. 10 users or 10 billion users — traversing Amit's 50 friends takes the same
time. This is called index-free adjacency — the killer feature of graph databases.

2 Graph Concepts — Nodes, Relationships & Properties

Before diving into Neo4j specifics, you need to understand three building blocks of any graph:

THE THREE BUILDING BLOCKS OF A GRAPH

1. NODES (Entities / Things)

→ A person, a product, a city, a movie, an IP address
→ Each node can have LABELS (like types): Person, Product, City
→ Each node can have PROPERTIES (key-value pairs):
(:Person {name: "Amit", age: 28, city: "Mumbai"})

2. RELATIONSHIPS (Connections / Edges)

→ A connection between two nodes
→ Always has a DIRECTION: (Amit)-[:FOLLOWS]->(Priya)
→ Always has a TYPE: FOLLOWS, PURCHASED, LIVES_IN, KNOWS
→ Can have PROPERTIES too:

[Link] 3/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

(Amit)-[:FOLLOWS {since: "2022-01-15"}]->(Priya)

3. PROPERTIES (Key-Value Data)

→ Attached to both nodes AND relationships
→ Like columns in SQL, but flexible per node
→ name: "Amit", age: 28, weight: 0.8 (on a relationship)

VISUAL EXAMPLE:

(:Person {name:"Amit", age:28})

│
[:FOLLOWS {since:"2022"}]
│
▼
(:Person {name:"Priya", age:25})
│
[:PURCHASED {amount:5000}]
│
▼
(:Product {name:"Laptop", brand:"Dell"})

ANATOMY OF A GRAPH — Nodes, Relationships & Properties

since: "2022"
:Person
LEGEND
:FOLLOWS name: "Priya"
:Person age: 25
Node — an entity (person, product)
name: "Amit"
age: 28 Relationship — connection + direction
amount: 5000
Label — :Person, :Product (node type)
:PURCHASED
:LIVES_IN Properties — key:value on nodes/edges
:Product Type — :FOLLOWS, :PURCHASED (edge type)
name: "Laptop"
Relationships ALWAYS have direction
brand: "Dell"
:City
Mumbai

3 Internal Architecture — Index-Free Adjacency

THE KILLER FEATURE: INDEX-FREE ADJACENCY

In SQL, to find related records you look up an index, scan a table, and join. In Neo4j, each node
physically stores pointers to its adjacent nodes. To traverse from Amit to his friends, Neo4j
doesn't consult any index — it follows the direct pointer stored on Amit's node. This is called

[Link] 4/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

index-free adjacency — the reason graph traversals are O(1) per hop regardless of database
size.

HOW NEO4J STORES DATA ON DISK — THE PHYSICAL STRUCTURE

NEO4J STORAGE FILES:

1. NODE STORE ([Link])

Each node is a fixed-size record (15 bytes):
┌──────────────────────────────────────────────┐
│ in_use │ first_rel_id │ first_prop_id │ labels│
└──────────────────────────────────────────────┘
→ first_rel_id: pointer to FIRST relationship of this node
→ From there, relationships form a LINKED LIST
→ No index lookup needed to find relationships!

2. RELATIONSHIP STORE ([Link])

Each relationship is a fixed-size record (34 bytes):
┌───────────────────────────────────────────────────────────┐
│ start_node │ end_node │ type │ next_rel_start │ next_rel_end │
└───────────────────────────────────────────────────────────┘
→ Doubly linked list: each relationship points to
the NEXT relationship of the start node AND end node
→ Traversal: follow the linked list — O(1) per hop

3. PROPERTY STORE ([Link])

→ Key-value pairs stored separately
→ Nodes and relationships point to their first property
→ Properties form another linked list

4. LABEL STORE
→ Maps label names (:Person, :Product) to internal IDs

INDEX-FREE ADJACENCY IN ACTION:

"Find Amit's friends"

SQL approach:
1. Scan friendships table (could be billions of rows)
2. Filter where user_id = 'Amit'
3. Time: proportional to TABLE SIZE

Neo4j approach:
1. Go to Amit's node record (direct address, O(1))
2. Read first_rel_id → pointer to first relationship
[Link] 5/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

3. Follow linked list of relationships

4. Time: proportional to NUMBER OF AMIT'S FRIENDS
(not total database size!)

Amit has 50 friends in a database of 1 billion nodes?

→ SQL: scans parts of 1B row table
→ Neo4j: reads exactly 50 relationship records. Done.

INDEX-FREE ADJACENCY — Direct Pointers Between Nodes

REL#101 :FOLLOWS REL#102 :FOLLOWS REL#103 :LIKES

end→Priya, next→REL#102 end→Sara, next→REL#103 end→Post#42, next→NULL
NODE: Amit
↑ points to NEXT relationship
first_rel → REL#101
props → {name,age}
labels → [:Person]
Priya Sara Post#42

Each node points DIRECTLY to its relationships. No index scan needed.

Traversal = follow pointers. O(1) per hop. Doesn't matter if DB has 10 or 10B nodes.

Native Graph Storage vs Graph Layer on SQL

WHY "NATIVE" MATTERS

Some databases add a "graph layer" on top of SQL (like Amazon Neptune on SQL tables). They
store graph data in regular tables and simulate traversals with JOINs. Neo4j is a native graph
database — the storage engine itself is designed for nodes and relationships. This is why
Neo4j traversals are orders of magnitude faster than graph-on-SQL solutions for deep
traversals (3+ levels).

4 Cypher — Neo4j's Query Language

Cypher is Neo4j's declarative query language. It uses ASCII art to represent graph patterns —
you literally draw the pattern you're looking for.

CYPHER SYNTAX AT A GLANCE

Nodes: (variable:Label {prop: value}) — round brackets = node

Relationships: -[:TYPE {prop: value}]-> — arrows = relationship
Direction: --> (left to right), <-- (right to left), -- (either direction)

[Link] 6/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

CREATE — Creating Nodes and Relationships

CREATING DATA IN CYPHER

// Create a node
CREATE (:Person {name: "Amit", age: 28, city: "Mumbai"})

// Create multiple nodes

CREATE (:Person {name: "Priya", age: 25}),
(:Person {name: "Sara", age: 30}),
(:Product {name: "Laptop", price: 75000})

// Create a relationship between existing nodes

MATCH (a:Person {name: "Amit"}), (p:Person {name: "Priya"})
CREATE (a)-[:FOLLOWS {since: "2022-01-15"}]->(p)

// Create node + relationship together

CREATE (amit:Person {name: "Amit"})
-[:PURCHASED {amount: 75000, date: "2024-03-15"}]->
(:Product {name: "Laptop"})

MATCH — Finding Patterns (The Core of Cypher)

READING DATA — FROM SIMPLE TO COMPLEX

// Find a node by property

MATCH (p:Person {name: "Amit"})
RETURN p

// Find all people Amit follows

MATCH (a:Person {name: "Amit"})-[:FOLLOWS]->(friend)
RETURN [Link], [Link]

// Find friends OF friends (2 levels deep!)

MATCH (a:Person {name: "Amit"})-[:FOLLOWS]->()-[:FOLLOWS]->(fof)
WHERE [Link] <> "Amit" // exclude Amit himself
RETURN DISTINCT [Link]

// Find friends up to 3 levels deep (variable depth!)

MATCH (a:Person {name: "Amit"})-[:FOLLOWS*1..3]->(distant)
RETURN DISTINCT [Link]

// Find the SHORTEST PATH between two people

[Link] 7/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

MATCH path = shortestPath(

(a:Person {name: "Amit"})-[:FOLLOWS*]-(b:Person {name: "Rahul"})
)
RETURN path, length(path)

// Find what products Amit's friends purchased

MATCH (a:Person {name: "Amit"})-[:FOLLOWS]->(friend)
-[:PURCHASED]->(product:Product)
RETURN [Link], [Link], [Link]

// Count followers per person (aggregation)

MATCH (p:Person)<-[:FOLLOWS]-(follower)
RETURN [Link], COUNT(follower) AS follower_count
ORDER BY follower_count DESC

// Pattern: "People who follow someone who purchased a Laptop"

MATCH (buyer:Person)-[:PURCHASED]->(:Product {name: "Laptop"}),
(follower:Person)-[:FOLLOWS]->(buyer)
RETURN [Link] AS "Might want a laptop too"

UPDATE and DELETE

UPDATING AND DELETING

// Update a property
MATCH (a:Person {name: "Amit"})
SET [Link] = 29, [Link] = true

// Add a label
MATCH (a:Person {name: "Amit"})
SET a:PremiumUser

// Delete a relationship
MATCH (a:Person {name: "Amit"})-[r:FOLLOWS]->(p:Person {name: "Priya"})
DELETE r

// Delete a node (must delete all relationships first!)

MATCH (p:Person {name: "OldUser"})
DETACH DELETE p // DETACH removes all relationships, then deletes node

// Delete all data (careful!)

MATCH (n) DETACH DELETE n

Advanced Cypher Patterns

[Link] 8/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

POWERFUL GRAPH QUERIES — INTERVIEW AMMUNITION

// RECOMMENDATION: "People who bought X also bought Y"

MATCH (target:Person {name: "Amit"})-[:PURCHASED]->(p:Product),
(other:Person)-[:PURCHASED]->(p), // others who bought same products
(other)-[:PURCHASED]->(rec:Product) // what else did they buy?
WHERE NOT (target)-[:PURCHASED]->(rec) // that Amit hasn't bought yet
RETURN [Link], COUNT(*) AS score
ORDER BY score DESC LIMIT 5

// FRAUD DETECTION: "Find circular money transfers"

MATCH path = (a:Account)-[:TRANSFERRED*3..5]->(a) // cycle back to same account
WHERE ALL(r IN relationships(path) WHERE [Link] > 10000)
RETURN path

// SHORTEST PATH with conditions

MATCH path = shortestPath(
(start:City {name: "Mumbai"})-[:CONNECTED_TO*]-(end:City {name: "Delhi"})
)
WHERE ALL(r IN relationships(path) WHERE [Link] < 500)
RETURN [n IN nodes(path) | [Link]] AS route,
reduce(total = 0, r IN relationships(path) | total + [Link]) AS total_km

// PAGE RANK: find most influential person

CALL [Link]('myGraph')
YIELD nodeId, score
RETURN [Link](nodeId).name AS name, score
ORDER BY score DESC

5 Indexing in Neo4j

IMPORTANT DISTINCTION

Neo4j has two types of lookups: (1) Finding a starting node (WHERE name = "Amit") — this
uses indexes, just like SQL. (2) Traversing from that node (following relationships) — this uses
index-free adjacency (direct pointers), no index needed. You only need indexes for the initial
lookup to find where to start traversing.

NEO4J INDEX TYPES

[Link] 9/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

// B-TREE INDEX — for equality and range lookups

CREATE INDEX FOR (p:Person) ON ([Link])
CREATE INDEX FOR (p:Person) ON ([Link])

// COMPOSITE INDEX — multiple properties

CREATE INDEX FOR (p:Person) ON ([Link], [Link])

// UNIQUE CONSTRAINT (creates an index automatically)

CREATE CONSTRAINT FOR (p:Person) REQUIRE [Link] IS UNIQUE

// FULL-TEXT INDEX — for text search inside properties

CREATE FULLTEXT INDEX personNames FOR (p:Person) ON EACH [[Link], [Link]]

// EXISTENCE CONSTRAINT — property must exist

CREATE CONSTRAINT FOR (p:Person) REQUIRE [Link] IS NOT NULL

// POINT INDEX — for geospatial queries (nearby locations)

CREATE POINT INDEX FOR (l:Location) ON ([Link])

// Check existing indexes

SHOW INDEXES

// Explain query plan (like SQL EXPLAIN)

EXPLAIN MATCH (p:Person {name: "Amit"})-[:FOLLOWS]->(f) RETURN f
PROFILE MATCH (p:Person {name: "Amit"})-[:FOLLOWS]->(f) RETURN f

6 Scaling Neo4j — The Honest Reality

SCALING IS NEO4J'S BIGGEST WEAKNESS

Unlike Cassandra (linear horizontal scaling) or MongoDB (built-in sharding), Neo4j is harder to
scale horizontally. Graph data is inherently interconnected — if you split the graph across
servers, traversals that cross server boundaries become network hops instead of local pointer
follows, which destroys the performance advantage.

NEO4J SCALING OPTIONS

1. VERTICAL SCALING (Scale Up)

→ Add more RAM, faster CPU, bigger SSD to one server

[Link] 10/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

→ Neo4j loves RAM — entire graph in memory = blazing fast

→ Works until you hit hardware limits (~1TB RAM, ~34B nodes)
→ This is the PRIMARY scaling strategy for most Neo4j deployments

2. READ REPLICAS (Neo4j Causal Cluster)

→ One primary (READ + WRITE) + multiple read replicas (READ only)
→ Writes go to primary, async replication to replicas
→ Reads distributed across replicas
→ Good for read-heavy workloads
→ Provides fault tolerance (replica can be promoted)

3. FABRIC (Neo4j 4.0+ — Federated Queries)

→ Split the graph into multiple DATABASES
→ Each database on its own server
→ Fabric layer lets you query across databases
→ But: cross-database traversals are SLOWER (network hops)
→ Works when graph has natural "seams" to split on

4. SHARDING (Neo4j 5+ Composable Architecture)

→ Still evolving. Not as mature as Cassandra/MongoDB sharding.
→ Graph sharding is fundamentally hard:
cutting a graph = cutting relationships = slow cross-shard queries
→ Best when sub-graphs are relatively independent

WHY GRAPH SHARDING IS HARD:

Imagine Amit (Shard 1) follows Priya (Shard 2) who follows Sara (Shard 3).
A "friends of friends" query crosses 2 shard boundaries = 2 network hops.
The whole point of Neo4j is LOCAL traversal. Cross-shard = slow.

Compare: Cassandra's partition key ensures queries hit ONE node.

Neo4j's interconnected nature means queries can touch MANY nodes.

NEO4J CAUSAL CLUSTER — Read Scaling

Application
writes reads

PRIMARY async replication READ REPLICA 1 READ REPLICA 2

Reads + Writes Reads only Reads only
Full graph copy Full graph copy Full graph copy

[Link] 11/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

7 Read-Heavy vs Write-Heavy Workloads

READ-HEAVY (NEO4J'S SWEET SPOT) WRITE-HEAVY (NOT NEO4J'S

STRENGTH)
→ Index-free adjacency = traversals are
instant → All writes go to single primary node
→ Add read replicas for horizontal read → Creating relationships requires locking both
scaling nodes
→ Graph fits in RAM → all traversals in → No distributed writes (unlike Cassandra)
memory → Bulk imports are slow via Cypher (use
→ Complex multi-hop queries stay fast neo4j-admin import)
→ Examples: recommendation engines, fraud → For write-heavy: use Cassandra or
detection, knowledge graphs, social MongoDB, then batch-load into Neo4j for
queries analysis
→ Real-world: LinkedIn (who viewed your → Common pattern: write to Kafka → process
profile, connection suggestions) → bulk insert into Neo4j

8 Challenges & Trade-offs

✅ NEO4J STRENGTHS ❌ NEO4J WEAKNESSES

→ Blazing fast traversals (index-free → Horizontal scaling is hard (graph sharding
adjacency) = cutting connections)
→ Intuitive data model (whiteboard = → Write scalability limited (single primary)
database) → Not for bulk analytics (use
→ Cypher is readable and powerful Spark/Cassandra for that)
→ Variable-depth queries (1..N hops) are → Memory hungry — best when entire graph
trivial fits in RAM
→ Shortest path algorithms built-in → Commercial license for enterprise features
→ Graph Data Science library (PageRank, (clustering)
community detection, etc.) → Not for simple key-value lookups (use
→ ACID compliant (unlike most NoSQL!) Redis)

→ Schema optional but constraints available → Not for tabular data (use SQL)

[Link] 12/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

→ Visual browser for exploring data → Learning curve for Cypher if coming from
SQL
→ Aggregations not as powerful as SQL's
GROUP BY

9 When to Use Neo4j — And When NOT To

✅ USE NEO4J WHEN ❌ DON'T USE NEO4J WHEN

→ Relationships ARE the data (social → Data is tabular with simple relationships
network, org chart) (use SQL)
→ Need multi-hop traversals (friends of → Need massive write throughput (use
friends of friends) Cassandra)
→ Recommendation engines ("people who → Simple key-value lookups (use Redis)
bought X also bought Y") → Document storage with flexible schemas
→ Fraud detection (circular money transfers, (use MongoDB)
suspicious patterns) → Time-series metrics (use InfluxDB)
→ Knowledge graphs (connecting concepts, → Full-text search is the primary use case
entities, facts) (use Elasticsearch)
→ Network topology (IT infrastructure, → Dataset is too large for single server
dependencies) without natural graph seams
→ Access control (who has permission to → Workload is heavily write-dominant
what, via what role)
→ Shortest path problems (routing, logistics)

10 Real-World Companies Using Neo4j — And Why

[Link] 13/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

COMPANY WHAT THEY USE NEO4J FOR WHY NEO4J

LinkedIn Connection suggestions, Social graph traversal: "who knows who" at 3-4 levels
"People You May Know," deep. SQL JOINs would be impossibly slow at 900M+
profile viewers users.

eBay Shipping route optimisation, Finding optimal routes through delivery network graph.
delivery network Shortest path algorithms built into Neo4j.

Walmart Real-time recommendation "Customers who bought X also bought Y" requires
engine traversing purchase relationships across millions of
customers.

NASA Knowledge graph for space Connecting lessons learned, equipment, missions,
mission data failures across decades of data. Relationships between
data points are the value.

Panama Investigative journalism — Exposed hidden ownership networks. 11.5M

Papers ICIJ documents connected by shell companies, people, and
banks. Graph queries found connections invisible in
spreadsheets.

Airbnb Trust and safety, fraud Detecting fake reviews by finding circular patterns:
prevention user A reviews B, B reviews C, C reviews A. Graph
pattern matching is perfect for this.

UBS Bank Regulatory compliance, risk Tracing exposure: "If company X defaults, which
analysis portfolios are affected?" Requires traversing ownership
and investment chains.

THE PATTERN ACROSS ALL COMPANIES

Every company using Neo4j has one thing in common: the value is in the connections, not
just the data. LinkedIn doesn't just store users — it navigates the social graph. Panama Papers
didn't just store documents — it exposed hidden networks. Walmart doesn't just store products
— it traverses purchase patterns. If your app's core value comes from navigating
relationships between entities, Neo4j is the right choice.

11 Interview Questions & Model Answers

[Link] 14/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

Q1: "WHY WOULD YOU USE NEO4J INSTEAD OF SQL FOR RELATIONSHIP-HEAVY DATA?"

Because of index-free adjacency. In SQL, finding "friends of friends of friends" requires

multiple self-JOINs on a potentially billion-row table — each JOIN scans parts of the entire
table, and performance degrades exponentially with depth. In Neo4j, each node stores direct
pointers to its neighbors. Traversal is O(1) per hop regardless of total database size. A 3-level
traversal in Neo4j takes ~3ms whether the database has 1,000 or 1,000,000,000 nodes.

Q2: "WHAT IS INDEX-FREE ADJACENCY AND WHY DOES IT MATTER?"

Each node in Neo4j physically stores pointers to its relationships, and each relationship stores
pointers to its start/end nodes and the next relationship. To traverse from one node to its
neighbors, you follow these pointers directly — no index lookup, no table scan. This means
traversal time is proportional to the number of relationships traversed, not the total size of the
graph. It's the reason graph databases are orders of magnitude faster than SQL for
relationship-heavy queries.

Q3: "WHAT ARE THE SCALING LIMITATIONS OF NEO4J?"

Neo4j's biggest challenge is horizontal scaling. Graph data is inherently interconnected — if

you shard the graph across servers, traversals that cross shard boundaries become network
hops instead of local pointer follows, destroying the performance advantage. Neo4j primarily
scales vertically (more RAM, bigger machine) and adds read replicas for read scaling. For
write-heavy workloads at massive scale, I'd pair Neo4j with a write-optimised store like
Cassandra and batch-load data into Neo4j for graph analysis.

Q4: "HOW WOULD YOU BUILD A RECOMMENDATION ENGINE WITH NEO4J?"

Use collaborative filtering via graph traversal: (1) Start at the target user node. (2) Find
products they've purchased. (3) Find OTHER users who purchased the same products. (4) Find
what ELSE those users purchased. (5) Filter out products the target user already has. (6) Rank
by how many "similar users" purchased each recommendation. This is a natural graph
traversal — follow PURCHASED edges, then reverse-follow PURCHASED edges. In Cypher it's
about 5 lines. In SQL it would be multiple complex JOINs.

Q5: "HOW DOES NEO4J HANDLE TRANSACTIONS? IS IT ACID COMPLIANT?"

Yes — Neo4j is fully ACID compliant, which is unusual for NoSQL databases. Every read and
write operation happens within a transaction. Multiple operations can be grouped into a single
transaction that either fully commits or fully rolls back. This makes Neo4j reliable for use cases

[Link] 15/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

like financial fraud detection where data integrity is critical. However, transactions are limited
to a single database instance (not distributed transactions across a cluster).

Q6: "WHEN WOULD YOU NOT USE NEO4J?"

When the data is tabular with simple relationships (SQL is simpler and more mature), when I
need massive write throughput (Cassandra's peer-to-peer architecture handles this better),
when the queries are simple key-value lookups (Redis is faster), when the data is document-
shaped (MongoDB is more natural), or when the dataset is so large it can't fit on one server
and doesn't have natural graph boundaries for sharding.

📋 Neo4j — Complete Quick Revision

WHAT IS NEO4J?
Native graph database. Data stored as nodes (entities) + relationships (connections).
Connections are first-class citizens. ACID compliant. Built for traversing highly connected data.

CORE CONCEPTS
Nodes = entities (:Person, :Product). Relationships = connections (:FOLLOWS, :PURCHASED).
Both have labels/types and properties (key-value). Relationships always have direction.

KILLER FEATURE: INDEX-FREE ADJACENCY

Each node stores direct pointers to its relationships. No index scan for traversal. O(1) per hop
regardless of DB size. SQL: JOINs get slower with depth. Neo4j: stays constant. This is WHY
graph DBs exist.

CYPHER QUERY LANGUAGE

ASCII art patterns. (node)-[:REL]->(node) . MATCH = find patterns. CREATE = add data. WHERE
= filter. Variable-depth: -[:FOLLOWS*1..3]-> . shortestPath() built-in.

STORAGE
Fixed-size records: Node Store, Relationship Store, Property Store. Relationships form doubly
linked lists from each node. WiredTiger-like caching — best when graph fits in RAM.

SCALING
Vertical first (more RAM). Read replicas for read scaling (Causal Cluster). Fabric for federated
queries. Graph sharding is HARD — cutting relationships = network hops = slow. Write scaling is
limited (single primary).

[Link] 16/17
04/04/2026, 13:39 Neo4j — Complete Deep Dive from Zero to Expert

READ VS WRITE
Read-heavy = sweet spot. Traversals are instant. Add read replicas. Write-heavy = NOT ideal.
Single primary bottleneck. Pair with Cassandra/Kafka for writes, batch into Neo4j.

WHEN TO USE
Use: Social graphs, recommendations, fraud detection, knowledge graphs, shortest paths,
access control, network topology.
Don't use: Tabular data (SQL), massive writes (Cassandra), key-value (Redis), documents
(MongoDB), time-series (InfluxDB), data too large for one server.

COMPANIES
LinkedIn (social graph), eBay (routing), Walmart (recommendations), NASA (knowledge graph),
Panama Papers (investigative journalism), Airbnb (fraud), UBS (risk).

INTERVIEW GOLD
"Friends of friends" in SQL: self-JOINs → exponentially slower with depth. In Neo4j: follow
pointers → constant time per hop. 3 levels in 3ms whether DB has 1K or 1B nodes.

[Link] 17/17
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

N O S Q L D E E P D I V E — I N - M E M O RY DATA B A S E / K E Y-VA L U E S TO R E

Redis — Complete Deep Dive

From zero to expert: why RAM makes everything 100x faster, the 8 data structures,
persistence, caching patterns, pub/sub, clustering, and how every major company uses
Redis

1 What Is Redis? (Starting From Zero)

DEFINITION

Redis stands for REmote DIctionary Server. It is an in-memory data structure store that can be
used as a database, cache, message broker, and streaming engine. The core idea: all data
lives in RAM (not on disk), making every operation happen in microseconds instead of
milliseconds. It supports multiple data structures beyond simple key-value — strings, lists, sets,
sorted sets, hashes, streams, and more.

THE DESK VS FILING CABINET ANALOGY

Imagine you need a document. A disk-based database (MySQL, MongoDB) is like walking to a
filing cabinet in another room, opening the drawer, finding the folder, pulling the document, and
walking back — takes seconds. Redis is like having the document already on your desk, right
in front of you — you just look down and read it — takes a fraction of a second.

That's the difference between disk I/O (~5-10ms) and RAM access (~0.1μs). RAM is roughly
100,000x faster than disk.

Why Redis Exists — The Speed Problem

THE LATENCY GAP REDIS FILLS

LATENCY OF DIFFERENT STORAGE:

CPU L1 Cache: ~1 ns (nanoseconds)

CPU L2 Cache: ~4 ns
RAM Access: ~100 ns ← REDIS LIVES HERE

[Link] 1/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

SSD Read: ~100,000 ns (100 μs)

HDD Read: ~10,000,000 ns (10 ms)
Network Round Trip: ~500,000 ns (0.5 ms)

WHAT THIS MEANS IN PRACTICE:

MySQL query (cold, disk): 5-50 ms

MongoDB query (warm, cached): 1-10 ms
Redis query (always in RAM): 0.01-0.1 ms ← 100-1000x FASTER

For a page that makes 20 database calls:

MySQL: 20 × 10ms = 200ms loading time
Redis: 20 × 0.1ms = 2ms loading time

200ms → 2ms. That's the difference users FEEL.

WHERE REDIS SITS — THE CACHING LAYER

① Check cache first REDIS (RAM)

RESULT:
Your App 0.01-0.1ms per query
⚡ 95% of requests served here Without Redis: 200ms
([Link], Java,
HIT → instant! With Redis: 2ms
Python, etc.)
② MISS → query DB ③ Store result 100x faster! ⚡
20 DB calls in Redis + TTL
per page load
PostgreSQL / MongoDB
5-50ms per query (disk)
5% of requests reach here

2 Redis Is NOT Just Key-Value — 8 Data Structures

THIS IS WHAT MAKES REDIS SPECIAL

Most people think Redis is just SET key value / GET key . That's like saying a smartphone is
just a phone. Redis provides 8 rich data structures, each with atomic operations built in. This
is why Redis can replace entire application-level data logic — not just caching.

① Strings — The Simplest Type

STRINGS — BINARY-SAFE, UP TO 512MB

[Link] 2/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

SET user:1001:name "Amit Sharma" // store a string

GET user:1001:name // → "Amit Sharma"

SET counter 100 // store a number (as string)

INCR counter // → 101 (atomic increment!)
INCRBY counter 50 // → 151
DECR counter // → 150

SETNX lock:order:555 "processing" // SET if NOT EXISTS (distributed lock!)

SET session:abc123 "user_data" EX 3600 // set with 1-hour expiry

USE CASES: Caching, counters, rate limiters, distributed locks, session tokens

② Hashes — Mini Documents

HASHES — LIKE A MINI-OBJECT (FIELD-VALUE PAIRS INSIDE ONE KEY)

HSET user:1001 name "Amit" age 28 city "Mumbai" email "amit@.."

HGET user:1001 name // → "Amit"
HGETALL user:1001 // → {name:"Amit", age:"28", city:"Mumbai"...}
HINCRBY user:1001 age 1 // → age becomes 29 (atomic!)
HDEL user:1001 email // remove one field

// One key "user:1001" holds MULTIPLE fields — like a MongoDB document

// but in RAM. Perfect for user sessions and profiles.

USE CASES: User sessions, profiles, shopping carts, config settings

③ Lists — Ordered Sequences

LISTS — DOUBLY LINKED LIST (PUSH/POP FROM BOTH ENDS)

LPUSH notifications:user1 "New follower: Priya" // push to LEFT (front)

LPUSH notifications:user1 "New like on your post"
RPUSH notifications:user1 "Order shipped" // push to RIGHT (back)
LRANGE notifications:user1 0 9 // get first 10 items
LPOP notifications:user1 // pop from front
LLEN notifications:user1 // length

// Blocking operations (for message queues):

BLPOP queue:tasks 30 // block for up to 30 seconds waiting for new item
[Link] 3/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

USE CASES: Activity feeds, notification lists, message queues, recent items

④ Sets — Unique Unordered Collections

SETS — UNIQUE VALUES, SET OPERATIONS BUILT IN

SADD user:1001:followers "Priya" "Sara" "Rahul"

SADD user:1002:followers "Amit" "Sara" "Neha"

SMEMBERS user:1001:followers // → {"Priya", "Sara", "Rahul"}

SISMEMBER user:1001:followers "Priya" // → 1 (true) — O(1) lookup!
SCARD user:1001:followers // → 3 (count)

// SET OPERATIONS (extremely powerful):

SINTER user:1001:followers user:1002:followers // → {"Sara"} (mutual followers!)
SUNION user:1001:followers user:1002:followers // → all unique followers combined
SDIFF user:1001:followers user:1002:followers // → who follows 1001 but NOT 1002

USE CASES: Tags, unique visitors, mutual friends, online users, voting

⑤ Sorted Sets — Ranked Data (Leaderboards!)

SORTED SETS — UNIQUE VALUES WITH A SCORE FOR RANKING

ZADD leaderboard 9500 "Amit" 8700 "Priya" 9200 "Sara" 7800 "Rahul"

ZRANGE leaderboard 0 -1 WITHSCORES // all, lowest to highest score

ZREVRANGE leaderboard 0 2 WITHSCORES // top 3, highest first:
// → Amit:9500, Sara:9200, Priya:8700

ZSCORE leaderboard "Amit" // → 9500

ZRANK leaderboard "Amit" // → 3 (0-based rank from bottom)
ZREVRANK leaderboard "Amit" // → 0 (rank from TOP = #1!)
ZINCRBY leaderboard 300 "Priya" // Priya's score += 300 → 9000

ZRANGEBYSCORE leaderboard 9000 10000 // players with score 9000-10000

ZCOUNT leaderboard 9000 10000 // count of players in range

// All operations are O(log N) — fast even with millions of members!

[Link] 4/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

USE CASES: Leaderboards, priority queues, rate limiting windows,

trending topics, scheduled tasks, autocomplete

⑥ Bitmaps — Memory-Efficient Boolean Arrays

BITMAPS — 1 BIT PER FLAG, MASSIVELY COMPACT

// Track daily active users (1 bit per user)

SETBIT daily_active:2024-03-15 1001 1 // user 1001 was active today
SETBIT daily_active:2024-03-15 1002 1 // user 1002 was active too
GETBIT daily_active:2024-03-15 1001 // → 1 (was active)
BITCOUNT daily_active:2024-03-15 // → count of active users

// 100 million users → only ~12MB of RAM!

USE CASES: Daily active users, feature flags, online status, A/B test groups

⑦ HyperLogLog — Approximate Unique Counts

HYPERLOGLOG — COUNT UNIQUE ITEMS WITH ~0.81% ERROR USING ONLY 12KB

PFADD unique_visitors:page1 "user1" "user2" "user3" "user1" // user1 counted once

PFCOUNT unique_visitors:page1 // → 3 (approximate unique count)

// 1 billion unique visitors → uses only 12 KB of RAM!

// 0.81% standard error — close enough for analytics.

USE CASES: Unique page views, unique search queries, cardinality estimation

⑧ Streams — Append-Only Log (Event Streaming)

STREAMS — LIKE APACHE KAFKA BUT BUILT INTO REDIS

XADD events * user_id 1001 action "purchase" amount 5000

XADD events * user_id 1002 action "page_view" url "/products"

XLEN events // number of entries

XRANGE events - + // read all entries

[Link] 5/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

XRANGE events 1234567890-0 + // read from specific ID onward

// Consumer Groups (like Kafka consumer groups):

XGROUP CREATE events mygroup $ // create consumer group
XREADGROUP GROUP mygroup consumer1 COUNT 10 BLOCK 5000 STREAMS events >

USE CASES: Event sourcing, activity streams, real-time analytics, chat messages

3 Internal Architecture — How Redis Works Under the Hood

Single-Threaded Event Loop

REDIS IS SINGLE-THREADED (AND THAT'S A FEATURE!)

Redis processes all commands in a single thread using an event loop (like [Link]). This
sounds slow but is actually genius: no locks, no context switching, no race conditions. Since
everything is in RAM and operations are simple (O(1) or O(log N)), a single thread can process
100,000+ operations per second. Multithreading would add complexity without much benefit
because the bottleneck is network I/O, not CPU.

HOW REDIS PROCESSES COMMANDS — EVENT LOOP

CLIENT REQUESTS ARRIVE VIA NETWORK:

Client 1: GET user:1001:name
Client 2: SET counter 42
Client 3: LPUSH queue "task1"

REDIS EVENT LOOP (single thread):

┌─────────────────────────────────────┐
│ 1. Read all pending requests │
│ 2. Execute GET user:1001:name │ ← ~0.001ms (RAM lookup)
│ 3. Execute SET counter 42 │ ← ~0.001ms (RAM write)
│ 4. Execute LPUSH queue "task1" │ ← ~0.001ms (RAM list push)
│ 5. Send all responses │
│ 6. Loop back to step 1 │
└─────────────────────────────────────┘

WHY THIS WORKS:

→ Each operation takes ~1 MICROSECOND (it's just RAM access)
→ In 1 second: 1,000,000 microseconds ÷ 1μs per op = ~1M ops possible
[Link] 6/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

→ Real-world: ~100K-300K ops/sec (network overhead)

→ NO LOCKS needed (single thread = no concurrency issues)
→ ALL operations are ATOMIC by default (no partial updates)

SINCE Redis 6.0:

I/O threading added — network reads/writes happen on multiple threads
But command execution is STILL single-threaded (no locking!)

4 Persistence — Surviving Crashes (Data in RAM = Volatile?)

THE BIG QUESTION

If all data is in RAM, doesn't a power failure or crash lose everything? No — Redis offers two
persistence mechanisms to save data to disk. Most production deployments use both.

REDIS PERSISTENCE — TWO MODES

RDB (Snapshotting) AOF (Append-Only File)

Point-in-time snapshot of entire dataset Log every write command to disk
→ Saves complete dataset to disk periodically → Every SET, LPUSH, etc. appended to log file
→ "Save every 60s if ≥1000 keys changed" → fsync policies: always / every sec / never
→ Creates a compact binary .rdb file → On restart: replay all commands to rebuild
→ Great for backups and disaster recovery → AOF rewrite compacts the log periodically
✓ Fast restarts (load .rdb file) ✓ Minimal data loss (at most 1 second)
✓ Compact file size ✓ Human-readable log file
✗ Data loss: up to last snapshot gap ✗ Larger file than RDB
✗ If crash 30s after snapshot → 30s of data lost ✗ Slower restarts (replay every command)

PRODUCTION RECOMMENDATION: USE BOTH

RDB for fast backups and disaster recovery (hourly/daily snapshots).

AOF with appendfsync everysec for minimal data loss (at most 1 second of writes lost on
crash).
On restart: Redis uses AOF first (more complete), falls back to RDB if AOF is unavailable.

5 Caching Patterns — How Applications Use Redis

[Link] 7/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

4 CACHING PATTERNS EVERY DEVELOPER MUST KNOW

PATTERN 1: CACHE-ASIDE (Lazy Loading) — MOST COMMON

1. App checks Redis for data
2. Cache HIT → return directly (fast!)
3. Cache MISS → query database → store result in Redis with TTL
4. Next request → cache HIT

Pseudocode:
data = [Link]("user:1001")
if data is None:
data = [Link]("SELECT * FROM users WHERE id = 1001")
[Link]("user:1001", 3600, data) // cache for 1 hour
return data

✓ Only caches what's actually requested

✓ Cache failures don't break the app (just slower)
✗ First request always slow (cache miss)
✗ Data can become stale until TTL expires

PATTERN 2: WRITE-THROUGH
1. App writes to Redis AND database simultaneously
2. Every read is always from Redis (always fresh!)

✓ Cache is always up-to-date

✗ Every write is slower (write to two places)
✗ Caches data that might never be read

PATTERN 3: WRITE-BEHIND (Write-Back)

1. App writes to Redis ONLY
2. Redis asynchronously flushes to database in background

✓ Extremely fast writes

✗ Risk of data loss if Redis crashes before flush
✗ Complex to implement reliably

PATTERN 4: REFRESH-AHEAD
1. Proactively refresh cache BEFORE TTL expires
2. Background job refreshes frequently-accessed keys

✓ No cache misses for hot keys

✗ Wastes resources refreshing rarely-accessed keys

[Link] 8/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

Cache Eviction Policies

WHAT HAPPENS WHEN REDIS RUNS OUT OF MEMORY?

When maxmemory is reached, Redis must EVICT (remove) some keys.

You choose the eviction policy:

noeviction → Return error on writes (safest, no data loss)

allkeys-lru → Remove LEAST RECENTLY USED key (most common!)
allkeys-lfu → Remove LEAST FREQUENTLY USED key (Redis 4.0+)
volatile-lru → LRU but only among keys WITH an expiry set
volatile-lfu → LFU but only among keys with expiry
allkeys-random → Random eviction
volatile-random → Random among keys with expiry
volatile-ttl → Evict keys closest to expiring

MOST COMMON: allkeys-lru

→ "Remove whatever hasn't been used recently"
→ Good default for cache use cases

6 Beyond Caching — Redis as a Primary Data Store

10 USE CASES WHERE REDIS IS THE PRIMARY SYSTEM, NOT JUST A CACHE

1. SESSION STORE
Hash per session: HSET session:abc123 user_id 1001 role "admin"
Set TTL: EXPIRE session:abc123 3600
→ Faster than DB sessions. Shared across app servers.

2. RATE LIMITER
INCR rate:user:1001:minute → increment counter
EXPIRE rate:user:1001:minute 60
If count > 100: reject request (100 req/min limit)

3. REAL-TIME LEADERBOARD
Sorted Set: ZADD leaderboard 9500 "player1"
Top 10: ZREVRANGE leaderboard 0 9 WITHSCORES
→ Millions of players, instant ranking updates.

[Link] 9/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

4. DISTRIBUTED LOCK
SET lock:order:555 "server1" NX EX 30
→ NX = set only if not exists. EX = auto-expire in 30s.
→ Prevents two servers from processing the same order.

5. PUB/SUB MESSAGING
SUBSCRIBE channel:notifications
PUBLISH channel:notifications "New order received!"
→ Real-time events between services. Like a mini Kafka.

6. REAL-TIME ANALYTICS
Bitmaps for DAU: SETBIT active:2024-03-15 1001 1
HyperLogLog for unique visitors: PFADD visits "user1" "user2"
→ Billions of events, 12KB of memory.

7. GEOSPATIAL
GEOADD restaurants 72.8777 19.0760 "Pizza Place"
GEORADIUS restaurants 72.88 19.08 5 km COUNT 10
→ Find 10 nearest restaurants within 5km.

8. MESSAGE QUEUE
LPUSH queue:emails "send welcome email to user 1001"
BRPOP queue:emails 30 → worker blocks until new task arrives

9. AUTOCOMPLETE / TYPEAHEAD
Sorted Set with prefix matching.
User types "am" → ZRANGEBYLEX suggestions "[am" "[am\xff" LIMIT 0 10

10. FEATURE FLAGS

SET feature:dark_mode "enabled"
→ Instant feature toggle across all servers, no deployment needed.

Pub/Sub — Real-Time Messaging

REDIS PUB/SUB — HOW IT WORKS

PUBLISHER (Server A):

PUBLISH chat:room1 "Hello from Amit!"

SUBSCRIBERS (Server B, C, D — all listening):

SUBSCRIBE chat:room1
→ All subscribers INSTANTLY receive "Hello from Amit!"

Architecture:
Server A ──PUBLISH──→ [ REDIS ] ──NOTIFY──→ Server B

[Link] 10/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

──NOTIFY──→ Server C
──NOTIFY──→ Server D

LIMITATIONS:
→ Fire-and-forget: if subscriber is disconnected, message is LOST
→ No persistence: messages are NOT stored
→ For durable messaging: use Redis Streams instead of Pub/Sub
→ For production message queues: consider Kafka or RabbitMQ

7 Scaling Redis — Replication & Clustering

Replication — Read Scaling + High Availability

REDIS REPLICATION — MASTER + REPLICAS

┌──────────────┐
│ MASTER │ ← All writes go here
│ (read+write) │
└──────┬───────┘
│ async replication
┌────┴────┐
↓ ↓
┌──────────┐ ┌──────────┐
│ REPLICA 1│ │ REPLICA 2│ ← Read-only copies
│ (read) │ │ (read) │ Serve read traffic
└──────────┘ └──────────┘

→ Writes go to master, async replicated to replicas

→ Reads can be served by any replica (horizontal read scaling)
→ If master dies: Redis Sentinel auto-promotes a replica

Redis Sentinel — Automatic Failover

SENTINEL — THE WATCHDOG FOR REDIS HA

Redis Sentinel is a SEPARATE process that monitors Redis instances:

┌───────────┐ ┌───────────┐ ┌───────────┐

[Link] 11/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

│ Sentinel 1│ │ Sentinel 2│ │ Sentinel 3│ ← monitors all Redis nodes

└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└───────────────┼───────────────┘
│ monitors
↓
┌──────────────────┐
│ Master + Replicas│
└──────────────────┘

IF MASTER DIES:
1. Sentinels detect master is down (quorum vote)
2. Sentinels elect a replica to become new master
3. Other replicas reconfigure to follow new master
4. Clients automatically reconnect to new master
→ Failover: 10-30 seconds

Redis Cluster — Horizontal Scaling (Data Sharding)

REDIS CLUSTER — HASH SLOT SHARDING

16,384 hash slots distributed across master nodes. CRC16(key) % 16384 → slot → node.

Application

Master 1 (slots 0-5460) Master 2 Master 3 (slots 10923-16383)

(5461-10922)
Replica 1A Replica 1B Replica 3A Replica 3B
~⅓ of data
Keys: user:1001, session:abc Keys: cart:999, rate:ip:5
~⅓ of all data + replicas ~⅓ of all data
If Master 1 dies → Replica 1A promoted

REDIS CLUSTER — HOW IT WORKS

16,384 HASH SLOTS (fixed number):

→ Every key is mapped to a slot: CRC16(key) % 16384
→ Slots are distributed across master nodes
→ 3 masters: each handles ~5,461 slots

ADDING A NODE:
→ Add Master 4. Redistribute some slots to it.
→ Data for those slots migrates automatically.
→ Client libraries auto-discover new topology.

LIMITATIONS:
→ Multi-key operations only work if all keys are in SAME slot

[Link] 12/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

→ Use hash tags to force keys to same slot: {user:1001}:cart, {user:1001}:session

→ No cross-slot transactions (only within one slot)

8 Read-Heavy vs Write-Heavy — Redis Handles Both

READ-HEAVY (REDIS'S SWEET SPOT) WRITE-HEAVY (ALSO FAST!)

→ All reads from RAM → microsecond latency → All writes to RAM → equally fast
→ Add read replicas for horizontal read → Redis Cluster shards writes across masters
scaling → Counters: INCR is atomic single-threaded
→ Cache-aside pattern: 95%+ cache hit ratio = no locks
→ Sorted sets for leaderboards: O(log N) → Streams: append-only log for high-
reads throughput events
→ Examples: caching, session stores, → Examples: counters, rate limiting, metrics
leaderboards → Real-world: Instagram uses Redis counters
→ Real-world: Twitter uses Redis for timeline for likes
caching

9 Challenges & Trade-offs

✅ REDIS STRENGTHS ❌ REDIS WEAKNESSES

→ Blazing fast — sub-millisecond latency → RAM is expensive — 256GB RAM $$$ vs
→ 8 rich data structures — not just key-value 10TB SSD $$

→ Atomic operations — INCR, LPUSH, etc. → Dataset must fit in RAM — can't store 10TB
are thread-safe → No complex queries — no JOINs, no
→ TTL built in — auto-expire keys WHERE with multiple conditions

→ Pub/Sub + Streams — real-time messaging → No relations — purely key-based access

→ Lua scripting — atomic multi-step logic → Single-threaded — CPU-bound ops (large

server-side Lua scripts) block everything

[Link] 13/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

→ Geospatial support — nearby search built → Persistence is best-effort — not as

in durable as PostgreSQL
→ Cluster mode — horizontal scaling for large → Cluster limitations — multi-key ops across
datasets slots fail
→ Huge ecosystem — every language has a → No built-in search (unless using
Redis client RediSearch module)
→ Simple operations — O(1) or O(log N), → Data loss risk — async replication + crash
predictable latency = possible loss
→ Memory fragmentation over time with
many deletes

THE BIGGEST RISK: RAM COST

Storing 1TB of data in Redis requires ~1TB of RAM. At cloud prices, that's ~$10,000/month.
The same data on disk (SSD) costs ~$100/month. This is why Redis is typically used as a
caching layer (store only hot data in Redis, rest on disk) rather than a primary database for
large datasets. Use Redis for the hottest 5-10% of your data.

10 When to Use Redis — And When NOT To

✅ USE REDIS WHEN ❌ DON'T USE REDIS WHEN

→ Caching — reduce load on primary → Data is larger than available RAM
database → Need complex queries (JOINs,
→ Session storage — shared across app aggregations)
servers → Need strong ACID transactions across
→ Real-time leaderboards — sorted sets multiple keys
→ Rate limiting — INCR + TTL → Data is relational with many relationships
→ Distributed locks — SETNX + TTL → Need guaranteed durability (use
→ Real-time counters — page views, likes PostgreSQL)

→ Pub/Sub messaging — real-time → Need full-text search (use Elasticsearch)

notifications → Storing large blobs (files, images)
→ Geospatial queries — nearby search → Need long-term data storage (use disk-
→ Temporary data — OTPs, tokens, carts based DB)

[Link] 14/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

→ Need sub-millisecond latency

11 Real-World Companies Using Redis — And Why

WHAT THEY USE REDIS

COMPANY WHY REDIS
FOR

Twitter/X Timeline caching, 800M+ timelines cached. Sub-ms reads. Without Redis,
session store every page load would query the DB for 300+ tweets.

Instagram Like counters, feeds, Atomic INCR for like counts. Sorted sets for feed ranking.
suggestions 300M+ DAU need instant response.

GitHub Job queue, caching, Background job queue (Resque/Sidekiq uses Redis).
real-time features Cache for repo metadata. Real-time notifications.

Snapchat Rate limiting, real-time Rate limit API calls per user. Fast pub/sub for chat
messaging messages. Ephemeral data (TTL = natural fit).

Pinterest Session store, follower Sets for "who follows whom." Sorted sets for feed ranking.
lists, feeds Session data shared across servers.

Uber Geospatial driver GEORADIUS to find nearby drivers in real-time. Counters

matching, surge pricing for ride demand (surge calculation).

Airbnb Search result caching, Cache expensive search queries. Session store across
sessions microservices. Feature flags.

Slack Real-time presence, "Online" status indicators. Pub/sub for message delivery.
message broker Rate limiting for API.

Stack Caching (primary speed Entire site rendered from Redis cache. 50M+ page
Overflow layer) views/day served at sub-ms latency.

THE UNIVERSAL PATTERN

Every major tech company uses Redis — but almost never as the only database. Redis sits in
front of a primary database (PostgreSQL, MongoDB, Cassandra) as a speed layer. It caches
hot data, handles counters, manages sessions, and powers real-time features. The primary

[Link] 15/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

database handles durability and complex queries. This combination is the standard
architecture for any application serving more than a few thousand users.

12 Interview Questions & Model Answers

Q1: "WHY IS REDIS SO FAST?"

Three reasons: (1) All data is in RAM — RAM access is ~100,000x faster than disk. (2) Single-
threaded event loop — no locks, no context switching, no concurrency overhead. Every
operation is atomic by default. (3) Simple data structures with O(1) or O(log N) operations —
GET is O(1), sorted set operations are O(log N). The bottleneck is network I/O, not the database
itself.

Q2: "IF REDIS IS SINGLE-THREADED, HOW DOES IT HANDLE MILLIONS OF REQUESTS?"

Because each operation takes about 1 microsecond (it's just a RAM lookup). In one second,
there are 1,000,000 microseconds. Even with network overhead, a single Redis instance
handles 100,000-300,000 operations per second. For more throughput, you use Redis
Cluster to shard across multiple instances — each instance handles its share of requests in
parallel.

Q3: "HOW DO YOU HANDLE THE RISK OF DATA LOSS WITH REDIS?"

Use both persistence mechanisms: RDB snapshots for periodic backups + AOF (append-only
file) with appendfsync everysec for near-real-time durability. Additionally, use replication —
even if the master crashes, a replica has the data. For critical data that must never be lost, use
Redis as a cache only — the source of truth stays in PostgreSQL/MySQL, and Redis can be
rebuilt from the primary database.

Q4: "HOW WOULD YOU IMPLEMENT A RATE LIMITER WITH REDIS?"

Sliding window counter: For each user, create a key like rate:user:1001:minute . On each
request: INCR the counter (atomic!). If it's the first increment, EXPIRE the key in 60 seconds.
If the counter exceeds the limit (e.g., 100), reject the request. After 60 seconds, the key auto-
expires and the counter resets. This is O(1) per request and handles millions of users with
minimal memory.

[Link] 16/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

Q5: "HOW WOULD YOU IMPLEMENT A DISTRIBUTED LOCK WITH REDIS?"

Use SET lock:resource NX EX 30 . NX means "set only if not exists" (atomic check-and-set).
EX 30 means auto-expire in 30 seconds (prevents deadlocks if the holder crashes). To release:
DEL lock:resource (only if we hold it — use Lua script for atomic check-and-delete). For
stronger guarantees across multiple Redis instances, use the Redlock algorithm (acquire locks
on majority of N independent instances).

Q6: "REDIS VS MEMCACHED — WHEN WOULD YOU CHOOSE EACH?"

Redis: richer data structures (sorted sets, lists, streams), persistence (RDB/AOF), replication,
pub/sub, Lua scripting, cluster mode. Choose Redis for leaderboards, queues, sessions,
complex caching.
Memcached: simpler (strings only), multi-threaded (can use all CPU cores natively), slightly
faster for pure string caching at very high throughput. Choose Memcached for simple
string/object caching where you don't need data structures.
In practice: Redis has largely replaced Memcached because it does everything Memcached
does plus much more.

📋 Redis — Complete Quick Revision

WHAT IS REDIS?
In-memory data structure store. All data in RAM. Sub-millisecond latency. Used as cache,
database, message broker. Single-threaded event loop — 100K-300K ops/sec per instance.

8 DATA STRUCTURES
Strings (cache, counters, locks). Hashes (user sessions, profiles). Lists (queues, feeds). Sets
(unique items, mutual friends via SINTER). Sorted Sets (leaderboards, ranking). Bitmaps (DAU
tracking, 12MB for 100M users). HyperLogLog (unique counts, 12KB for 1B items). Streams
(event log like Kafka).

PERSISTENCE
RDB: periodic snapshots, compact, fast restart, may lose data since last snapshot. AOF: log
every command, minimal loss (1 sec), larger file, slower restart. Production: use BOTH.

CACHING PATTERNS
Cache-Aside (most common): check Redis → miss → query DB → store in Redis + TTL. Write-
Through: write to Redis + DB simultaneously. Write-Behind: write Redis only, async flush to DB.

[Link] 17/18
04/04/2026, 13:40 Redis — Complete Deep Dive from Zero to Expert

EVICTION POLICIES
allkeys-lru (most common) = remove least recently used. allkeys-lfu = least frequently used.
volatile-ttl = closest to expiring. noeviction = return error.

SCALING
Replication: master + read replicas (async). Sentinel: auto-failover watchdog. Cluster: data
sharded across masters using 16,384 hash slots. CRC16(key) % 16384 → slot → node.

BEYOND CACHING
Sessions, rate limiting (INCR+TTL), distributed locks (SETNX+EX), leaderboards (ZADD),
pub/sub, geospatial (GEORADIUS), message queues (BLPOP), analytics (bitmaps/HLL), feature
flags.

TRADE-OFFS
Strengths: fastest possible reads/writes, rich data structures, atomic ops, TTL, pub/sub, Lua
scripting, cluster mode.
Weaknesses: RAM is expensive (data must fit in memory), no complex queries/JOINs,
persistence is best-effort, single-threaded can block on heavy Lua, cross-slot operations fail in
cluster.

WHEN TO USE
Use: caching (always), sessions, leaderboards, rate limiting, locks, counters, real-time features,
geospatial, temp data. Don't use: data > RAM, complex queries, guaranteed durability, relational
data, large blobs.

COMPANIES
Twitter (timeline cache), Instagram (like counters), GitHub (job queues), Snapchat (rate limiting),
Pinterest (feeds), Uber (geospatial), Slack (presence), Stack Overflow (entire site cache).

REDIS VS MEMCACHED
Redis: data structures + persistence + replication + pub/sub. Memcached: simpler, multi-
threaded, slightly faster for pure string cache. Redis has largely replaced Memcached.

[Link] 18/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

N O S Q L D E E P D I V E — T I M E - S E R I E S DATA B A S E

InfluxDB — Complete Deep Dive

From zero to expert: what time-series data is, how InfluxDB stores billions of data points,
the TSM engine, Flux queries, retention policies, downsampling, and real-world monitoring
architectures

1 What Is Time-Series Data? (Starting From Zero)

DEFINITION

Time-series data is a sequence of data points indexed by time. Each data point represents a
measurement or event that occurred at a specific moment. The data arrives in chronological
order, is append-only (you almost never update old readings), and is often queried by time
ranges ("give me the last 5 minutes" or "compare today vs yesterday").

THE THERMOMETER ANALOGY

Imagine a thermometer that records the temperature every second. At 9:00:01 it reads 28°C, at
9:00:02 it reads 28.1°C, at 9:00:03 it reads 28.2°C. After a year, you have 31 million readings
— all timestamped. You never go back and "edit" what the temperature was at 9:00:01. You
only add new readings. And your questions are always time-based: "What was the average
temperature last hour?" or "When did it exceed 40°C?" That's time-series data — and it's
fundamentally different from the data in a user profile or a shopping cart.

EXAMPLES OF TIME-SERIES DATA — IT'S EVERYWHERE

SERVER MONITORING:
timestamp │ host │ cpu_usage │ memory_used │ disk_io
2024-03-15T09:00:01Z │ web-01 │ 45.2% │ 8.1 GB │ 120 MB/s
2024-03-15T09:00:02Z │ web-01 │ 47.8% │ 8.2 GB │ 118 MB/s
2024-03-15T09:00:01Z │ web-02 │ 32.1% │ 6.5 GB │ 95 MB/s
...millions per day...

IOT SENSORS:
timestamp │ sensor_id │ temperature │ humidity │ pressure
2024-03-15T09:00:00Z │ factory-1 │ 28.5°C │ 65% │ 1013 hPa

[Link] 1/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

2024-03-15T09:00:01Z │ factory-1 │ 28.6°C │ 65% │ 1013 hPa

FINANCIAL DATA:
timestamp │ symbol │ price │ volume
2024-03-15T09:30:00Z │ AAPL │ 172.50 │ 1,234,567
2024-03-15T09:30:01Z │ AAPL │ 172.52 │ 987,654

APPLICATION METRICS:
timestamp │ endpoint │ response_time │ status_code
2024-03-15T09:00:00Z │ /api/users │ 45ms │ 200
2024-03-15T09:00:00Z │ /api/products │ 120ms │ 200

Why General-Purpose Databases Fail for Time-Series

WHY SQL/MONGODB/REDIS CAN'T HANDLE THIS WELL

THE VOLUME PROBLEM:

1 server × 10 metrics × 1 reading/sec = 864,000 points/day
100 servers × 50 metrics × 1/sec = 432 MILLION points/day
1000 servers × 100 metrics × 1/sec = 8.64 BILLION points/day

After 1 year: trillions of data points.

After 3 years: must auto-delete old data or disk fills up.

WHY SQL FAILS:

→ INSERT speed: 8.64B inserts/day into a SQL table = too slow
→ SELECT speed: "AVG(cpu) over last hour" scans millions of rows
→ Storage: no time-aware compression — wastes space
→ No auto-cleanup: manual DELETE of old data is slow and locks table

WHY MONGODB FAILS:

→ Documents grow unboundedly (one doc per metric = huge)
→ Sharding by time = hot shard (all writes go to "latest" shard)
→ No built-in downsampling or retention policies

WHY REDIS FAILS:

→ Everything in RAM — trillions of points won't fit
→ No time-range query optimisation
→ No compression

WHAT A TIME-SERIES DB (InfluxDB) DOES DIFFERENTLY:

→ STORAGE: Time-aware compression (timestamps compress 95%+)
→ WRITES: LSM-tree-like engine optimised for sequential time data
→ QUERIES: Time-range queries are O(1) seek + sequential scan
→ CLEANUP: Retention policies auto-delete old data

[Link] 2/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

→ DOWNSAMPLING: Keep per-second data for 1 day, per-minute for 30 days,

per-hour for 1 year — automatic resolution reduction

2 What Is InfluxDB?

DEFINITION

InfluxDB is a purpose-built time-series database designed for high-speed ingestion, time-

range queries, and automatic data lifecycle management. It was built from the ground up for
metrics, events, and analytics — not adapted from a general-purpose database. It's the most
popular open-source time-series database (by DB-Engines ranking).

InfluxDB's Terminology — The Data Model

CRITICAL: UNDERSTAND THESE TERMS

InfluxDB uses unique terminology that confuses beginners. Understanding these terms is
essential before writing any queries.

INFLUXDB TERMINOLOGY MAPPED TO FAMILIAR CONCEPTS

InfluxDB Term │ SQL Equivalent │ What It Is

───────────────────┼───────────────────────┼──────────────────────────────
Bucket │ Database │ Container for all your data
Measurement │ Table │ Name of what you're measuring (e.g., "cpu")
Timestamp │ PRIMARY KEY (auto) │ When the data point was recorded
Field │ Column (value) │ The actual measured value (cpu=45.2%)
Tag │ Column (indexed) │ Metadata for filtering (host="web-01")
Field Set │ Non-indexed columns │ All field key-value pairs in a point
Tag Set │ Indexed columns │ All tag key-value pairs in a point
Point │ Row │ One data record (timestamp + tags + fields)
Series │ Unique combination │ One measurement + unique tag set

EXAMPLE DATA POINT:

cpu,host=web-01,region=us-east usage=45.2,temp=62.5 1710489601000000000

─┬─ ─────────────┬────────── ──────────┬───────── ──────────┬──────────
│ │ │ │

[Link] 3/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

│ │ │ └─ Timestamp
│ │ │ (nanoseconds since epo
│ │ └─ FIELDS (the values)
│ │ NOT indexed. For actual measurements.
│ TAGS (metadata)
└─
│ INDEXED! For filtering and grouping.
│ host, region, sensor_id, etc.
└─ MEASUREMENT NAME
Like a table name.

THE CRITICAL DISTINCTION: TAGS vs FIELDS:

TAGS = indexed, low cardinality (host has ~100 values)
Used in WHERE clauses and GROUP BY. Stored as strings.
Example: host, region, datacenter, sensor_type

FIELDS = NOT indexed, high cardinality (cpu_usage = infinite values)

The actual measured numbers. Can be float, int, string, bool.
Example: cpu_usage, temperature, response_time, price

RULE: If you filter by it → make it a TAG.

If it's the measured value → make it a FIELD.

MISTAKE: Making a high-cardinality value (user_id with millions of values)

a TAG creates "high cardinality" problem → exploding series count
→ degraded performance. Tags should have few distinct values.

INFLUXDB DATA MODEL — ANATOMY OF A DATA POINT

cpu,host=web-01,region=us-east usage=45.2,temp=62.5 1710489601

MEASUREMENT TAGS (indexed!) FIELDS (values) TIMESTAMP

"cpu" host=web-01 usage=45.2 nanosecond precision
region=us-east temp=62.5

WHAT IS A "SERIES"?
A series = unique combination of measurement + tag set
cpu + {host=web-01, region=us-east} = ONE series
100 hosts × 3 regions = 300 series. Each series is stored contiguously on disk.

3 Internal Architecture — The TSM Storage Engine

TSM = TIME-STRUCTURED MERGE TREE

[Link] 4/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

InfluxDB uses the TSM (Time-Structured Merge Tree) engine — a custom storage engine
inspired by LSM Trees (like Cassandra) but optimised specifically for time-series data. It
achieves massive write throughput with excellent compression because timestamps are
sequential and highly compressible.

INFLUXDB TSM ENGINE — WRITE PATH

Incoming Data Points (Line Protocol)

① WAL (Write-Ahead Log) + ② In-Memory Cache

Append-only on disk. Durability guarantee. Sorted by series + time. Serves recent queries.

Write ACK sent here ✅ (sub-millisecond)

cache full

WHY COMPRESSION IS AMAZING: ③ TSM File (on disk)

Timestamps: 9:00:01, 9:00:02, 9:00:03... Immutable, compressed, sorted by series + time.
Delta encoding: +1s, +1s, +1s → 1 byte each!
Float values: 45.2, 45.3, 45.1...
XOR encoding: only store differences.
Result: 90-95% compression ratio!
④ Compaction (background)
1TB raw data → ~50-100GB on disk. Merges multiple TSM files. Better compression. Faster reads.

TSM ENGINE — STEP BY STEP DEEP DIVE

WRITE PATH (very similar to Cassandra's LSM Tree):

Step 1: WAL (Write-Ahead Log)

→ Every write is appended to WAL on disk (sequential write — fast)
→ Provides DURABILITY — if crash, replay WAL to recover
→ Same concept as Cassandra's commit log

Step 2: IN-MEMORY CACHE

→ Simultaneously, data goes into an in-memory sorted cache
→ Organised by SERIES (measurement + tags) then TIME
→ Recent queries served directly from cache (fast!)
→ ACK sent to client immediately

Step 3: TSM FILE FLUSH

→ When cache reaches size threshold → flushed to TSM file
→ TSM file is IMMUTABLE (never modified after writing)
→ Data is COMPRESSED using time-aware algorithms:
- Timestamps: delta-of-delta encoding (sequential → tiny)
- Floats: Facebook Gorilla XOR encoding
- Integers: run-length + zigzag encoding
- Strings: Snappy compression

Step 4: COMPACTION
[Link] 5/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

→ Multiple TSM files merged into fewer, larger files

→ Removes expired data (retention policy)
→ Improves read performance (fewer files to check)

READ PATH:
1. Check in-memory cache (recent data — instant)
2. Check TSM files on disk (use index to jump to right file + offset)
3. TSM files are organised by SERIES — all data for one series
is stored contiguously → sequential reads = fast
4. Time range queries: binary search to start offset, scan forward
→ O(log N) seek + O(K) scan where K = result size

4 Retention Policies & Downsampling — Data Lifecycle

THE DATA LIFECYCLE PROBLEM

At 1 data point per second per sensor, you generate 86,400 points per day per sensor. With
10,000 sensors, that's 864 million points per day. After a year: 315 billion points. You can't
keep all of this at full resolution forever. But you still need historical trends. The solution:
retention policies + downsampling.

RETENTION POLICIES — AUTO-DELETE OLD DATA

RETENTION POLICY:
"Keep data for X duration, then automatically delete it."

CREATE BUCKET "server_metrics"

WITH RETENTION 30d // data older than 30 days is AUTO-DELETED

You can have MULTIPLE retention policies:

- "raw" bucket: 7 days (per-second precision)
- "daily" bucket: 365 days (per-hour averages)
- "archive" bucket: forever (per-day averages)

InfluxDB automatically deletes expired data.

No manual cleanup. No slow DELETE queries. No table locks.
This is WHY time-series DBs exist — lifecycle management is built in.

[Link] 6/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

DOWNSAMPLING — REDUCE RESOLUTION TO SAVE SPACE

DOWNSAMPLING:
"Aggregate high-resolution data into lower-resolution summaries."

Raw data (per-second):

09:00:01 → cpu=45.2
09:00:02 → cpu=47.8
09:00:03 → cpu=43.1
09:00:04 → cpu=46.5
...60 points per minute...

Downsampled to per-minute:
09:00 → cpu_avg=45.6, cpu_max=47.8, cpu_min=43.1

Downsampled to per-hour:
09:00 → cpu_avg=44.2, cpu_max=62.3, cpu_min=28.7

TYPICAL 3-TIER STRATEGY:

┌─────────────────────────────────────────────┐
│ Tier 1: Raw data (per-second) │
│ → Keep for 7 days │
│ → Full precision for recent debugging │
│ │
│ Tier 2: Per-minute aggregates │
│ → Keep for 30 days │
│ → Good enough for weekly trend analysis │
│ │
│ Tier 3: Per-hour aggregates │
│ → Keep for 1 year (or forever) │
│ → Long-term capacity planning │
└─────────────────────────────────────────────┘

Raw: 86,400 points/day → 604,800/week → AUTO-DELETED after 7 days

Minute: 1,440 points/day → 43,200/month → DELETE after 30 days
Hour: 24 points/day → 8,760/year → KEEP forever

Storage savings: 86,400 → 24 = 99.97% reduction!

[Link] 7/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

3-TIER DOWNSAMPLING — DATA LIFECYCLE

TIER 1: Raw TIER 2: Minutes TIER 3: Hours

Per-second precision Per-minute averages Per-hour averages
86,400 points/day aggregate 1,440 points/day aggregate 24 points/day
Retention: 7 days Retention: 30 days Retention: forever
For: real-time dashboards For: weekly trends For: yearly trends
and recent debugging and pattern analysis and capacity planning

5 Querying InfluxDB — Flux & InfluxQL

InfluxDB has two query languages: InfluxQL (SQL-like, older) and Flux (functional, newer, more
powerful). InfluxDB 2.x+ uses Flux as the primary language.

Writing Data — Line Protocol

INSERTING DATA — THE LINE PROTOCOL FORMAT

// Format: measurement,tags fields timestamp

cpu,host=web-01,region=us-east usage=45.2,temperature=62.5 1710489601000000000

cpu,host=web-02,region=eu-west usage=32.1,temperature=58.3 1710489601000000000
memory,host=web-01 used_percent=81.5,available=4096 1710489601000000000

// Batch insert (multiple lines at once — very efficient!)

// InfluxDB can ingest MILLIONS of points per second this way.

// HTTP API:
POST /api/v2/write?bucket=server_metrics
Content-Type: text/plain

cpu,host=web-01 usage=45.2 1710489601000000000

cpu,host=web-01 usage=47.8 1710489602000000000
cpu,host=web-01 usage=43.1 1710489603000000000

Flux Queries (InfluxDB 2.x)

FLUX QUERIES — FROM SIMPLE TO COMPLEX

[Link] 8/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

// BASIC: Get CPU usage for last 1 hour

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "cpu")
|> filter(fn: (r) => r._field == "usage")

// FILTER BY TAG: CPU for specific host

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "cpu")
|> filter(fn: (r) => [Link] == "web-01")
|> filter(fn: (r) => r._field == "usage")

// AGGREGATE: Average CPU per 5-minute window

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage")
|> aggregateWindow(every: 5m, fn: mean)

// GROUP BY: Average CPU per host

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage")
|> aggregateWindow(every: 5m, fn: mean)
|> group(columns: ["host"])

// MAX/MIN: Peak CPU in the last 24 hours

from(bucket: "server_metrics")
|> range(start: -24h)
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage")
|> max()

// ALERT: Find moments where CPU exceeded 90%

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage")
|> filter(fn: (r) => r._value > 90.0)

// MATH: Calculate rate of change (derivative)

from(bucket: "server_metrics")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "network" and r._field == "bytes_sent")
|> derivative(unit: 1s) // bytes per second

// DOWNSAMPLING TASK (runs automatically every 1 hour):

option task = {name: "downsample_cpu", every: 1h}

from(bucket: "raw_metrics")
[Link] 9/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

|> range(start: -1h)

|> filter(fn: (r) => r._measurement == "cpu")
|> aggregateWindow(every: 1m, fn: mean)
|> to(bucket: "downsampled_metrics")

InfluxQL Queries (SQL-like, InfluxDB 1.x)

INFLUXQL — SQL-LIKE SYNTAX (STILL SUPPORTED)

-- Get CPU usage for last hour

SELECT usage FROM cpu WHERE time > now() - 1h

-- Filter by tag
SELECT usage FROM cpu WHERE host = 'web-01' AND time > now() - 1h

-- Aggregate: average per 5 minutes

SELECT MEAN(usage) FROM cpu WHERE time > now() - 1h GROUP BY time(5m)

-- Group by tag + time

SELECT MEAN(usage) FROM cpu WHERE time > now() - 1h GROUP BY host, time(5m)

-- MAX in last 24 hours

SELECT MAX(usage) FROM cpu WHERE time > now() - 24h

-- Continuous query (auto-downsample, InfluxDB 1.x):

CREATE CONTINUOUS QUERY "cq_cpu_1h" ON "mydb"
BEGIN
SELECT MEAN(usage) INTO "downsampled"."cpu_hourly"
FROM "cpu" GROUP BY time(1h), host
END

6 Scaling InfluxDB

INFLUXDB EDITIONS AND SCALING OPTIONS

INFLUXDB OSS (Open Source — Single Node):

→ Free. Runs on one server.
→ Handles millions of points/second on good hardware.

[Link] 10/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

→ No built-in clustering or HA.

→ Good for: small-medium deployments, dev/staging.
→ Limitation: single point of failure, vertical scaling only.

INFLUXDB CLOUD (Managed Service):

→ Fully managed by InfluxData.
→ Auto-scaling, HA, multi-tenancy.
→ Pay per usage (writes, queries, storage).
→ Good for: production without operational overhead.

INFLUXDB ENTERPRISE (Commercial):

→ Clustering with data replication.
→ Meta nodes + Data nodes architecture.
→ HA with automatic failover.
→ Good for: large enterprises, on-premises.

INFLUXDB 3.0 (IOx — New Architecture):

→ Complete rewrite using Apache Arrow + Parquet.
→ Columnar storage for better analytical queries.
→ Designed for unlimited cardinality.
→ Object storage (S3) for cheap infinite storage.
→ Separate compute and storage (cloud-native).

SCALING STRATEGIES:
1. Vertical: more RAM, faster SSDs, more CPU cores
2. Functional sharding: different metrics on different instances
(server metrics on Instance 1, IoT data on Instance 2)
3. Time-based sharding: recent data on fast SSD, old data on HDD
4. InfluxDB Cloud/Enterprise for true horizontal scaling

7 The TICK Stack — Full Monitoring Architecture

WHAT IS THE TICK STACK?

InfluxDB is often used as part of the TICK stack — a complete open-source monitoring
platform. Each letter represents a component: Telegraf (data collection), InfluxDB (storage),
Chronograf (visualization), Kapacitor (alerting). In modern setups, Grafana often replaces
Chronograf for dashboards.

[Link] 11/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

MONITORING ARCHITECTURE — TICK STACK + GRAFANA

DATA SOURCES
Grafana
Servers (CPU/RAM)
Containers (Docker) T I Dashboards
Visualization DevOps /
Databases (PG/MySQL) Telegraf InfluxDB SRE Team
IoT Sensors Data Collector Time-Series Store
Views dashboards
APIs, Apps, Networks 200+ input plugins TSM Engine K Gets alerts
Retention + Downsample
Cloud Services Kapacitor
Alerts (Slack, PD)

8 Read-Heavy vs Write-Heavy

WRITE-HEAVY (INFLUXDB'S SWEET READ-HEAVY (ALSO STRONG)

SPOT)
→ TSM files are pre-sorted by series + time
→ TSM engine = sequential writes, no → Time-range queries = O(log N) seek +
random I/O sequential scan
→ WAL + cache → ACK immediately (sub-ms) → Downsampled tiers serve historical queries
→ Batch ingestion via Line Protocol efficiently
(millions/sec) → Grafana dashboards = continuous time-
→ Timestamps compress 95%+ (delta range reads
encoding) → Examples: dashboards, trend analysis,
→ Examples: server metrics, IoT sensors, alerting
APM → Weak for: ad-hoc queries on non-time
→ Real-world: Tesla ingests sensor data from dimensions
millions of vehicles

9 Challenges & Trade-offs

✅ INFLUXDB STRENGTHS ❌ INFLUXDB WEAKNESSES

→ Purpose-built for time-series — → High cardinality problem — too many
everything is optimised unique tag values = slow

[Link] 12/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

→ Extreme write throughput (millions of → No JOINs — can't correlate across

points/sec) measurements easily
→ 90-95% compression via time-aware → OSS = single node — no free clustering
encoding → Not for general-purpose data — only time-
→ Retention policies — auto-delete old data series
→ Downsampling tasks — built-in resolution → Deletes are expensive — optimised for
reduction append-only
→ Flux — powerful functional query language → Updates are rare — overwrite by same
→ Telegraf — 200+ data collection plugins timestamp

→ Grafana integration — beautiful → Schema changes require careful planning

dashboards (tags vs fields)

→ Nanosecond precision timestamps → Flux learning curve — different from SQL

→ Memory intensive — TSM index needs
RAM

THE HIGH CARDINALITY PROBLEM — #1 PERFORMANCE KILLER

If you make a high-cardinality value a tag (like user_id with millions of distinct values),
InfluxDB creates a separate series for every unique tag combination. 1M users × 10 metrics =
10M series. The in-memory index must track all series → RAM explodes, queries slow down.
Rule: tags should have low cardinality (host, region, datacenter — tens to hundreds of values).
High cardinality values should be fields (not indexed, but stored efficiently).

10 When to Use InfluxDB — And When NOT To

✅ USE INFLUXDB WHEN ❌ DON'T USE INFLUXDB WHEN

→ Server/infrastructure monitoring (CPU, → General-purpose CRUD (use
RAM, disk) PostgreSQL/MongoDB)
→ IoT sensor data (temperature, pressure, → User profiles, shopping carts (not time-
humidity) series)
→ Application performance monitoring → Complex JOINs or relationships (use
(response times, errors) SQL/Neo4j)
→ Financial tick data (stock prices per → Need strong ACID transactions
second) → Full-text search (use Elasticsearch)

[Link] 13/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

→ DevOps metrics (CI/CD pipeline times, → High cardinality dimensions (millions of

deploy frequency) unique tag values)
→ Network monitoring (bandwidth, packet → Data needs frequent updates or deletes
loss) → Caching (use Redis)
→ Data is append-only and timestamped
→ Need auto-expiry and downsampling

11 Real-World Companies Using InfluxDB — And Why

COMPANY WHAT THEY MONITOR WHY INFLUXDB

Tesla Vehicle sensor data from IoT time-series at massive scale. Per-second readings from
millions of cars thousands of sensors per vehicle. Auto-expiry for old data.

IBM Infrastructure monitoring Millions of servers, each emitting dozens of metrics. Need
across global cloud time-range queries for dashboards and alerting.

Cisco Network device metrics Routers, switches emitting bandwidth/latency metrics.

Telegraf collects, InfluxDB stores, Grafana visualizes.

eBay Application performance Response times, error rates across thousands of

monitoring microservices. Downsample old data for long-term trend
analysis.

PayPal Transaction monitoring, Real-time transaction volume tracking. Alerting on

fraud metrics anomalies (sudden spikes or drops).

Hulu Streaming quality metrics Buffer rates, video quality per stream. Millions of
concurrent streams = millions of data points/sec.

Robinhood Financial market data Stock price ticks. Nanosecond timestamp precision. Time-
range queries for charts.

THE PATTERN ACROSS ALL COMPANIES

Every company using InfluxDB has the same data pattern: continuous streams of
timestamped measurements arriving at high speed, queried by time ranges, and needing

[Link] 14/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

automatic lifecycle management. None of them use InfluxDB for user accounts, shopping
carts, or relational data — it's purely for metrics, monitoring, and time-series analytics.

12 InfluxDB vs Other Time-Series Databases

QUERY
DATABASE ARCHITECTURE BEST FOR LIMITATION
LANGUAGE

InfluxDB TSM Engine, Flux / General TSDB, OSS = single node

standalone InfluxQL IoT, APM only

Prometheus Pull-based, local PromQL Kubernetes Not for long-term

storage monitoring storage

TimescaleDB Extension on Full SQL! TSDB + relational Heavier than

PostgreSQL in one purpose-built TSDB

Graphite Whisper files Custom Simple metrics, Older, less efficient

functions legacy

Amazon Serverless (AWS) SQL-like AWS-native Vendor lock-in

Timestream workloads

ClickHouse Columnar SQL Heavy analytics More complex to

(MergeTree) on TS data operate

INTERVIEW TIP: WHEN TO PICK WHICH

InfluxDB: standalone TSDB, IoT, general monitoring, great ecosystem (Telegraf + Grafana).
Prometheus: Kubernetes-native, pull-based, short-term storage (pair with Thanos for long-
term).
TimescaleDB: when you need TSDB + SQL JOINs + relational data in one database.
ClickHouse: when you need heavy SQL analytics on billions of time-series rows.

13 Interview Questions & Model Answers

[Link] 15/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

Q1: "WHY CAN'T YOU JUST USE POSTGRESQL FOR TIME-SERIES DATA?"

You can — but it's 10-100x less efficient. PostgreSQL isn't optimised for the unique
characteristics of time-series: (1) Write speed — InfluxDB's TSM engine handles millions of
inserts/sec; PostgreSQL struggles past hundreds of thousands. (2) Compression — InfluxDB
uses time-aware encoding (delta-of-delta for timestamps, XOR for floats) achieving 90-95%
compression; PostgreSQL stores raw values. (3) Retention — InfluxDB auto-deletes expired
data; PostgreSQL needs manual DELETE (slow, locks table). (4) Downsampling — InfluxDB has
built-in tasks; PostgreSQL needs custom cron jobs. For small-scale time-series, PostgreSQL
works. At scale, a purpose-built TSDB is necessary.

Q2: "WHAT IS THE HIGH CARDINALITY PROBLEM IN INFLUXDB?"

InfluxDB creates a separate series for every unique combination of measurement + tags. If
you use a high-cardinality value as a tag (like user_id with millions of values), the number of
series explodes. The in-memory index must track every series, consuming RAM and slowing
queries. Solution: only use low-cardinality metadata (host, region, sensor_type) as tags. High-
cardinality identifiers should be fields (not indexed). InfluxDB 3.0 (IOx) is redesigned to handle
unlimited cardinality.

Q3: "HOW DOES DOWNSAMPLING WORK AND WHY IS IT IMPORTANT?"

Downsampling aggregates high-resolution data into lower-resolution summaries over time.

Example: keep per-second data for 7 days, then aggregate to per-minute for 30 days, then per-
hour for 1 year. This reduces storage from 86,400 points/day to 24 points/day (99.97%
reduction!) while preserving trends. InfluxDB automates this with tasks (Flux scripts that run on
a schedule) and retention policies that auto-delete old raw data.

Q4: "WHAT'S THE DIFFERENCE BETWEEN TAGS AND FIELDS?"

Tags are indexed metadata used for filtering and grouping — like WHERE host='web-01'. They
should have low cardinality (few distinct values). Fields are the actual measured values
(cpu=45.2%) — NOT indexed. They can have high cardinality (infinite distinct values). Making
a high-cardinality value a tag is the number one performance mistake in InfluxDB.

Q5: "INFLUXDB VS PROMETHEUS — WHEN WOULD YOU CHOOSE EACH?"

Prometheus: Kubernetes-native, pull-based (scrapes targets), designed for cloud-native

monitoring, PromQL is powerful for alerting rules, excellent service discovery. But its local
storage isn't designed for long-term retention — use Thanos or Cortex for that.
InfluxDB: Push-based (agents send data), better for IoT/custom metrics, excellent long-term
storage with retention policies, Telegraf has 200+ input plugins for diverse data sources, better
[Link] 16/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

for non-Kubernetes environments. Choose Prometheus for Kubernetes; InfluxDB for everything
else.

📋 InfluxDB — Complete Quick Revision

WHAT IS INFLUXDB?
Purpose-built time-series database. Designed for timestamped, append-only data. High-
speed ingestion (millions/sec). 90-95% compression. Auto-retention + downsampling. Most
popular open-source TSDB.

DATA MODEL
Bucket (database) → Measurement (table) → Points (rows). Each point: timestamp + tags
(indexed metadata, low cardinality) + fields (values, NOT indexed). Series = measurement +
unique tag set. High cardinality tags = #1 performance killer!

TSM ENGINE
WAL (append-only, durability) + In-memory Cache (serves recent queries, fast ACK) → TSM
Files (immutable, compressed, sorted by series+time) → Compaction (merge, cleanup). Like
Cassandra's LSM but optimised for timestamps.

COMPRESSION
Timestamps: delta-of-delta encoding (sequential → tiny). Floats: XOR encoding (Facebook
Gorilla). Result: 1TB raw → ~50-100GB on disk.

RETENTION & DOWNSAMPLING

Retention: auto-delete data after X days. Downsampling: per-second → per-minute → per-hour
over time. 3-tier strategy: raw (7d) → minutes (30d) → hours (forever). 99.97% storage
reduction.

QUERIES
Flux: from(bucket) |> range(start: -1h) |> filter() |> aggregateWindow(every: 5m, fn:
mean) . InfluxQL: SQL-like syntax. Time-range queries, aggregations, derivatives, downsampling
tasks.

TICK STACK
Telegraf (collect) → InfluxDB (store) → Grafana/Chronograf (visualize) → Kapacitor (alert).
Standard monitoring architecture.

SCALING

[Link] 17/18
04/04/2026, 13:41 InfluxDB — Complete Deep Dive from Zero to Expert

OSS = single node. Cloud = managed, auto-scaling. Enterprise = clustered, HA. 3.0 (IOx) =
Apache Arrow + Parquet, unlimited cardinality, object storage.

WHEN TO USE
Use: server monitoring, IoT sensors, APM, financial ticks, DevOps metrics, network monitoring.
Don't use: user data, CRUD apps, JOINs, full-text search, caching, high-cardinality tags.

COMPANIES
Tesla (vehicle sensors), IBM (infra monitoring), Cisco (network), eBay (APM), PayPal
(transactions), Hulu (streaming quality), Robinhood (market data).

VS ALTERNATIVES
Prometheus: Kubernetes-native, pull-based, short-term. TimescaleDB: TSDB + SQL on
PostgreSQL. ClickHouse: columnar, heavy analytics. InfluxDB = best general-purpose
standalone TSDB.

[Link] 18/18