Module 3

NoSQL
Syllabus:
• Aggregation: The Aggregation Framework Pipeline
Operations $match $project $group $unwind $sort
$limit $skip Using Pipelines MapReduce Example 1:
Finding All Keys in a Collection Example 2:
Categorizing Web Pages MongoDB an MapReduce
Aggregation Commands count distinct group
• Application Design: Normalization versus
Denormalization Examples of Data Representations
Cardinality Friends, Followers, and Other
Inconveniences Optimizations for Data Manipulation
Optimizing for Document Growth Removing Old Data
Planning Out Databases and Collections Managing
Consistency Migrating Schemas.
Aggregation:
The Aggregation Framework in MongoDB is a
powerful tool used for processing and analyzing
data within collections. It works by processing
documents through a pipeline of stages.
Each stage performs an operation on the
documents and passes the results to the next
stage. This allows you to perform complex data
analysis, filtering, and transformation that a
simple find() query cannot.
The aggregation framework's pipeline allows to
break down a complex data analysis task into a
series of manageable, logical steps.
The Pipeline Stages:
• The aggregation pipeline consists of multiple stages, each
starting with a dollar sign ($). The most common stages are:
• $match: Filters documents to pass only the ones that match
the specified criteria to the next stage. This is a highly
efficient stage, especially when used early in the pipeline,
as it can leverage indexes to reduce the number of documents
to process.
• $group: Groups input documents by a specified identifier
expression and then computes an aggregate expression (like a
sum, average, or count) for each group.
• $project: Reshapes each document in the stream, often used
to select fields, add new fields, or exclude fields from the
output.
• $sort: Reorders the documents by a specified field.
• $limit: Restricts the number of documents passed to the next
stage.
Scenario: E-commerce Sales Analysis:
Imagine you are a data analyst for a large online
retail company. Your manager asks you to generate a
report on product sales for the past year.
Specifically, they want to know the top three best-
selling categories based on total revenue for 2023.
They also need this report to be generated quickly
so they can make real-time business decisions.
• Problem: The company's sales data is stored in a
MongoDB collection called sales, with millions of
documents. Each document represents a single
transaction. A simple find() query cannot perform
the required filtering, grouping, and sorting to
$match:
• This is the filtering stage. It's used to
filter documents from a collection based on a
specified query, much like the find() method.
• Detail: You should almost always put $match at
the beginning of your pipeline, as it reduces
the number of documents passed to subsequent
stages, making the entire aggregation more
efficient. It can leverage indexes for a
significant performance boost.
• Example: To process only documents from the
"electronics" category, you would use:
“JavaScript”
$project:
• This is the reshaping stage. It allows you to
select, rename, or create new fields in the
documents that are passed to the next stage. It
can also be used to exclude fields.
• Detail: You can use $project to customize the
output of your pipeline, making it easier to read
and analyze. It's often used to compute new fields
based on existing ones.
• Example: To show only the name and price fields
and create a new field for discountedPrice, you
would use:
JavaScript
{ $project: { name: 1, price: 1, discountedPrice: {
$multiply: ["$price", 0.9] } } }
The $project Stage: Reshaping
Documents:
The $project stage is a powerful pipeline operator
used to reshape and transform documents. It allows
you to:
• Select/Exclude Fields: You can choose which fields
to include or exclude from the documents passed to
the next stage.
• Rename Fields: You can give a field a new name,
which is useful for creating a cleaner output or
preparing for a $group stage.
• Compute New Fields: This is the most powerful
feature. You can perform calculations, string
manipulations, logical operations, and date
conversions to create new fields.
• By default, the _id field is always included in
How $project Works

The $project stage is all about controlling the

fields that are passed from one stage of the
aggregation pipeline to the next. It allows you to:
• Include or Exclude Fields: You can choose which
fields to keep and which to discard.
• Rename Fields: You can give a field a new name
using a syntax like "newFieldName" :
"$oldFieldName".
• Create New Fields with Expressions: This is where
the real power of $project lies. You can use
various expressions (mathematical, date, string,
logical) to compute new values and assign them to
new fields.
The $project stage operates on each document
How $project Works: Types of
Expressions

The document you provided outlines several key types

of expressions available within $project:
• 1. Mathematical Expressions: These are used for
numeric calculations. Operators like $add,
$subtract, $multiply, $divide, and $mod can be
used to perform arithmetic.
• Example: To calculate an employee's total pay by
adding their salary and bonus.
{
$project: {
totalPay: { $add: ["$salary", "$bonus"] }
}
}
How $project Works: Types of
Expressions
2. Date Expressions: These expressions extract components from a
date field, such as the year, month, or day.
Example: To find the year each employee was hired.
{ $project: { hireYear: { $year: "$hireDate" } }}
3. String Expressions:These are used for manipulating string
values, with operators like $substr, $concat, $toLower, and
$toUpper.
Example: To generate a lowercase email address from an employee's
first and last name.
{
$project: {
email: {
$concat: [
{ $toLower: "$firstName" },
".",
{ $toLower: "$lastName" },
"@[Link]"
]
}
}
}
How $project Works: Types of
Expressions
4. Logical Expressions: These operators perform
conditional and boolean logic.
• Comparison Operators: $eq, $ne, $gt, $gte, $lt,
$lte compare two values and return a boolean.
• Boolean Operators: $and, $or, and $not evaluate
multiple boolean expressions.
• Control Flow: $cond and $ifNull allow you to
define conditional logic.
• Example: To determine if an employee is a manager
or not based on a boolean field.
{
$project: {
status: { $cond: ["$isManager", "Manager",
"Employee"] }
}
}
How $project Works: Types of
Expressions
• Projection Behavior and Efficiency:
The $project stage, unlike $group, is a
streaming operator. This means it can process
documents as they arrive and does not need to
collect all documents before producing output.
This makes it highly efficient. However, as the
document notes, if you rename a field in a
$project stage, an index on the original field
name will no longer be usable by a subsequent
$sort or $match stage. This is why it's a best
practice to use $project after you've used a
$match stage that can leverage an existing
index.
$group:
This is the aggregation stage. It groups documents
by a specified identifier (_id) and then performs a
function on the documents within each group (e.g.,
sum, average, count).
Detail: This is the heart of the aggregation
framework for data analysis. You must specify a
grouping key (_id) and at least one accumulator
expression (e.g., $sum, $avg, $count).
Example: To calculate the total number of items sold
and the average price for each category, you would
use:
JavaScript
{ $group: { _id: "$category", totalSold: { $sum:
"$quantity" }, avgPrice: { $avg: "$price" } } }
How Grouping Works:

The $group operator takes two primary parts:

• The Grouping Key (_id): This defines how
documents are grouped. All documents with the
same value for the _id expression will be
considered part of the same group. The grouping
key can be a single field ("$_id": "$country"),
a compound key ("$_id": { "state": "$state",
"city": "$city" }), or a constant value to
group all documents into one.
• The Accumulator Operators: These are the
functions that perform calculations on the
documents within each group. They are used to
How Grouping Works : Types of Accumulator
Operators
The document you provided outlines several key types of
accumulators:
1. Arithmetic Operators: These are used for numeric
calculations.
• $sum: Calculates the sum of all values for a field within a
group. It can sum a simple field ($sum: "$revenue") or a
constant ($sum: 1 to count documents).
• $avg: Calculates the average of all values for a field
within a group.
• Example: To find the total sales revenue and average price
for each product category in an e-commerce store.
[Link]([
{
$group: {
_id: "$category",
totalRevenue: { $sum: "$price" },
averagePrice: { $avg: "$price" },
numProducts: { $sum: 1 }
}
}
])
How Grouping Works : Types of Accumulator
Operators
2. Extreme Operators: These find the maximum or minimum
values.
• $max and $min: These operators are used to find the
highest and lowest values of a field in a group. They
are most efficient when the data is unsorted.
• $first and $last: These operators return the first and
last values encountered in a group. Crucially, their
results are only predictable if the data has been
sorted in a previous pipeline stage. Using a $sort
stage before $group can make them more efficient than
$min and $max for finding extremes.
• Example: To find the highest and lowest-priced product
in each category.
[Link]([
// This approach works on unsorted data
{
$group: {
_id: "$category",
lowestPrice: { $min: "$price" },
highestPrice: { $max: "$price" }
How Grouping Works : Types of Accumulator
Operators
3. Array Operators: These are used to build arrays from
the values in a group.
• $push: This operator adds every value it encounters to
an array. The resulting array will contain all values
for that field from all documents in the group. The
order is not guaranteed unless you pre-sort.
• $addToSet: This is similar to $push, but it only adds a
value to the array if it doesn't already exist. The
resulting array will contain only unique values.
• Example: To find all distinct product brands within
each category and get a list of all prices.
[Link]([
{
$group: {
_id: "$category",
brands: { $addToSet: "$brand" },
prices: { $push: "$price" }
}
}
How Grouping Works : Types of Accumulator
Operators
Grouping Behavior and Efficiency:
The document correctly identifies $group as a
"roadblock" operator. This means it must collect
all documents that match the grouping key before
it can compute the final result and pass it to
the next stage.
• Sharding: In a sharded cluster, the $group
operation is performed in two phases. First,
each shard performs a local grouping on its
subset of the data. Then, a central mongos
router collects the partial results from all
the shards and performs a final, comprehensive
grouping to produce the final, correct result.
This distributed process makes $group powerful
and scalable.
$unwind:
• This is the deconstruction stage. It's used to
deconstruct an array field from the input documents.
For each element in the array, it outputs a separate
document with all the other fields remaining the same.
• Detail: This is especially useful when you need to run
an aggregation on data within an array. For instance,
if each order document has an array of purchased items,
$unwind can create a separate document for each item,
making it easy to group and count products.
• Example: If a document contains { orderId: "A123",
items: ["Laptop", "Mouse"] }, $unwind on the items
array would produce two documents:
JavaScript
{ orderId: "A123", items: "Laptop" }
{ orderId: "A123", items: "Mouse" }
The $unwind stage would be: { $unwind: "$items" }
$sort:
• This is the sorting stage. It reorders the
documents based on a specified field and
direction (ascending or descending).
• Detail: You can sort on one or more fields. A
value of 1 indicates ascending order, and -1
indicates descending order. For performance,
it's best to sort on an indexed field.
• Example: To sort documents by price from
highest to lowest, you would use:
JavaScript
{ $sort: { price: -1 } }
$limit:
• This is the limitation stage. It restricts the
number of documents that are passed to the next
stage.
• Detail: $limit is often used after a $sort to
retrieve the top N documents, as in finding the
top 10 most expensive products.
• Example: To get only the first 10 documents,
you would use:
JavaScript
{ $limit: 10 }
$skip:
• This is the skipping stage. It skips a
specified number of documents and passes the
rest to the next stage.
• Detail: $skip is commonly used with $limit to
implement pagination, allowing you to get a
specific "page" of results.
• Example: To get the second page of 10 documents
(i.e., skipping the first 10), you would use:
JavaScript
{ $skip: 10 }
MapReduce:
MapReduce is a data processing paradigm used to
perform large-scale, parallel computations on
data.
In MongoDB, it's a way to perform aggregation
operations, though it has largely been
superseded by the more efficient Aggregation
Framework. It consists of two main phases:
• Map Phase: The map function processes each
document in a collection. It "maps" the data to
key-value pairs and emits them.
• Reduce Phase: The reduce function collects the
values for each unique key and aggregates them.
MapReduce: Example 1: Finding All
Keys in a Collection
Problem: You have a collection of documents with
varying schemas, and you need to find all the unique
field names (keys) used across the entire
collection. This is a common task for schema
exploration in schema less databases like MongoDB.
• MapReduce Solution:
• Map Phase: The map function iterates through each
key-value pair in a document. For each key it
finds, it emits the key as the output key and 1 as
the value. The value 1 is just a placeholder.
var mapFunction = function() {
for (var key in this) {
emit(key, 1);
}
};
MapReduce: Example 1: Finding
All Keys in a Collection
2. Reduce Phase: The reduce function receives all
the 1s emitted for each unique key. It simply needs
to return a single value to indicate the key's
existence.
var reduceFunction = function(key, values) {
return 1;};
3. Finalize (Optional): After the reduce phase, you
can run a finalize function to format the output.
4. Execution: You run the MapReduce command on your
collection. The result will be a collection
containing all the unique keys found.
[Link]( mapFunction,
reduceFunction, { out: "all_keys" });
This process is a straightforward way to get a
unique list of all keys in a dynamic collection.
MapReduce: Example 2: Categorizing Web Pages
• Problem: You have a collection of website visitor data, with each
document containing the URL of a visited page. You want to group
all pages into categories (e.g., "blog," "product," "contact") and
get a count of how many times each category was visited.
• MapReduce Solution:
• Map Phase: The map function processes each visitor document. It
analyzes the URL string to determine the page's category. For
example, a URL containing "/blog/" would be categorized as "blog."
It then emits the category as the key and 1 as the value.
var mapFunction = function() {
var category;
if ([Link]("/blog/") !== -1) {
category = "blog";
} else if ([Link]("/products/") !== -1) {
category = "product";
} else {
category = "other";
}
emit(category, 1);
MapReduce: Example 2:
Categorizing Web Pages
2. Reduce Phase: The reduce function receives
all the 1s for each category. It sums these
values to get the total count for each category.
var reduceFunction = function(key, values) {
return [Link](values);};
3. Execution: The MapReduce job runs, and the
output is a new collection containing the total
number of visits per category.
[Link]( mapFunction,
reduceFunction, { out:
"page_categories" });
This example shows how MapReduce can be used to
categorize data and perform a simple count, a
task that would be much harder with a simple
MongoDB an MapReduce:
In MongoDB, both MapReduce and the Aggregation Framework are used
to process and analyze data. The Aggregation Framework is generally
the preferred method because it's more efficient, easier to use,
and more scalable.
• Aggregation Commands: The Aggregation Framework uses a pipeline of
commands to process data. These commands are often more intuitive
and perform better than their MapReduce counterparts.
• count: This command is used to quickly get the number of documents
that match a query. It's an efficient way to get a total count
without retrieving the actual documents.
• distinct: This command finds and returns all the unique values for
a specified field in a collection.
Here's an example using these commands in a simple scenario.
• Scenario: An e-commerce site needs to find out how many different
product categories they have and the total number of products in
each category.
• Solution:
distinct: To find the number of unique categories, you can use the
distinct command.
[Link]("category").length
• count: To find the total number of products, you can use the
countDocuments command.
[Link]()
• MapReduce: MapReduce is an older, more manual way to perform
aggregation. It requires you to write custom JavaScript
functions for the map and reduce phases. While powerful,
it's often more complex than using the Aggregation
Framework.
• The Aggregation Framework vs. MapReduce
The Aggregation Framework has a dedicated $group stage that
can perform the same tasks as MapReduce and much more, often
with better performance because it's implemented in native
code rather than JavaScript.
• To perform the "group" operation from the scenario above
(counting products per category), you would use the $group
stage in the Aggregation Framework.
• Solution using the Aggregation Framework's $group:
[Link]([
{
$group: {
_id: "$category",
totalProducts: { $sum: 1 }
}
}
])
 Comparison:
Feature Aggregation Framework MapReduce

Intuitive pipeline stages Requires writing custom

Ease of Use
($group, $match) JavaScript functions

Much faster and more Slower, as it uses

Performance
efficient JavaScript engine

Highly flexible with a wide Flexible, but requires more

Flexibility
range of operators manual coding

Designed for distributed Can be complex to manage on

Scalability
processing on shards sharded clusters

The Aggregation Framework has become the standard for performing complex
data analysis in MongoDB due to its performance, expressiveness, and ease
of use. Typically, only need to use MapReduce for very specific, non-
standard aggregation tasks that can't be handled by the Aggregation
Framework.
Application Design:
Normalization versus Denormalization
• Normalization :
Normalization is the process of structuring a relational
database to reduce data redundancy and improve data
integrity. The goal is to design a database in a way that
stores each piece of information only once. This is achieved
by dividing large tables into smaller, related tables and
defining relationships between them.
• Process: Normalization follows a series of rules called
Normal Forms (NF), from 1NF to 5NF. The most common forms
used in practice are 1NF, 2NF, and 3NF.
• Key Benefits:
• Reduced Data Redundancy: Prevents the same data from being stored
in multiple places. For example, a customer's address is stored
only once in a Customers table, not repeatedly in an Orders table.
• Improved Data Integrity: Ensures that data is consistent. If a
customer's address needs to be updated, it only has to be changed
in one place, which prevents conflicting information.
• Faster Inserts, Updates, and Deletes: Because data is not
duplicated, write operations are quicker and more efficient.
• Drawback: The main drawback is that it requires joining
multiple tables to retrieve data. For a complex query, this
can result in a large number of joins, which can slow down
Normalization versus
Denormalization
• Denormalization :
Denormalization is the process of intentionally adding redundant
data to a normalized database. This is typically done to improve
the read performance of a database, especially for data warehousing
or reporting applications. Denormalization involves combining
tables or adding duplicate data to reduce the number of joins
needed for a query.
• Process: This is not a formal process like normalization; it's a
strategic decision made after a database has been normalized. You
decide what data to duplicate based on the most frequent or
performance-critical queries.
• Key Benefits:
• Faster Reads (Query Performance): By reducing the number of joins, queries
can be executed much faster. For a report that needs customer names, order
dates, and product details, all this information can be stored in a single
denormalized table, eliminating the need for three separate joins.
• Simplified Queries: Queries become simpler to write and easier to manage
because they often involve fewer tables.
• Drawbacks:
• Increased Data Redundancy: Leads to data duplication, which can waste
storage space.
• Slower Inserts, Updates, and Deletes: Because data is duplicated, changes
have to be made in multiple places, which can increase the complexity and
time of write operations.
• Potential for Data Inconsistency: If a piece of duplicated data is not
updated everywhere it appears, it can lead to data integrity issues.
Examples of Data Representations
Data Representations
Data representations refer to the different ways
you can model relationships between documents or
records. The two main approaches are:
• Embedded Data: This model stores related data
within a single document. It's often used for
one-to-one or one-to-many relationships where
the "many" side is not very large and
frequently accessed with the "one" side.
• Referenced Data: This model uses a separate
document for each entity and links them using a
reference, typically an _id. This is the
standard relational approach and is suitable
for many-to-many relationships or one-to-many
relationships where the "many" side is very
large or frequently accessed independently.
• There are three primary ways to represent data relationships in
MongoDB: referenced, embedded, and a hybrid approach. The best
method depends on the relationship type, how often the data is
read versus updated, and how much the related data is expected to
grow.

• 1. Referenced Data Representation: In this approach, you store

related data in separate collections and use a unique identifier
(_id) to link them, similar to foreign keys in relational
databases. This is a highly normalized model.
• Example: Storing students and classes in separate collections.
You'd have a students collection, a classes collection, and a
third studentClasses collection that contains references to the
student and an array of class _ids they are taking.
• studentClasses collection document: {"studentId": "...", "classes":
["class_id_1", "class_id_2"]}.
• Pros:
• Data Consistency: If class information (like a room number) changes, you
only need to update it in one place: the classes collection.
• Efficient Updates: This is ideal for volatile data that changes often.
• Scalability: It's great for relationships where the referenced data can
grow significantly without bound, like user comments on a blog post.
• Cons:
• Slower Reads: Retrieving all the information for a student requires
multiple queries to dereference the _ids from different collections, which
2. Embedded Data Representation
This approach involves storing related data directly
inside a single document. This is a denormalized model
that prioritizes read performance.
• Example: Embedding all of a student's class information
directly within the student's document.
• students collection document: {"name": "John Doe", "classes":
[{"class": "Trigonometry", "credits": 3}, ...]}.
• Pros:
• Fast Reads: All related data is retrieved in a single query.
This reduces the number of server trips and is perfect for
read-heavy applications.
• Atomic Operations: Since all data is in one document, updates
on the entire document are atomic.
• Simpler Queries: Queries are simpler as they don't require _id
dereferencing or joins.
• Cons:
• Data Redundancy: The same class information is duplicated
across multiple student documents. This wastes storage space
and can lead to data inconsistency.
• Complex Updates: If a class detail changes, you have to update
every student document that contains that embedded class, which
3. Hybrid Data Representation
• A hybrid approach combines the best of both worlds by
embedding a small, frequently-accessed subset of data
within the main document, while also including a
reference to the full, separate document for more
detailed information.
• Example: Embedding a class's _id and name within the
student document while storing other details like
credits and room number in a separate classes
collection.
• students collection document: {"name": "John Doe", "classes":
[{"_id": "class_id_1", "class": "Trigonometry"}, ...]}.
• Pros:
• Balanced Performance: This model offers a balance between fast
reads and efficient updates. You get the most common
information in one query but can perform a second query for
less-used details.
• Flexibility: The amount of embedded information can change over
time as application requirements evolve.
• Cons:
• It's a compromise that still requires two queries to get all
the data, which is not as fast as a fully embedded model.
 Pros of Embedding Pros of Referencing

Data Size Small subdocuments Large subdocuments

Data that doesn't change Volatile data that changes

Change Frequency
regularly frequently

When eventual consistency is When immediate consistency is

Consistency
acceptable necessary

Writes vs. Reads Fast reads Fast writes

Documents that grow by a small Documents that grow by a large

Growth
amount amount
Cardinality:
Cardinality describes the nature of the
relationship between two entities. It answers the
question: "How many instances of entity A are
related to how many instances of entity B?"
• One-to-One: Each instance of A relates to exactly
one instance of B.
• One-to-Many: One instance of A relates to many
instances of B.
• Many-to-Many: Many instances of A relate to many
instances of B, and vice-versa.
 Friends, Followers, and Other Inconveniences
This phrase highlights the challenges of
modeling complex relationships in a database,
especially the many-to-many relationships common
in social networks.
• Friends: A "friends" relationship is typically
many-to-many. If Alice is friends with Bob,
then Bob is also friends with Alice. This
symmetric relationship is an inconvenience
because you must model a bidirectional link.
• Followers: A "followers" relationship is
typically one-to-many. If Alice follows Bob,
Bob doesn't necessarily follow Alice back. This
The "Wil Wheaton Effect" and Continuation Documents:
The Wil Wheaton effect refers to the challenge faced when a popular
user (like Wil Wheaton) has so many followers that embedding all
their IDs into a single document would cause that document to
exceed the database's maximum size limit.
To address this, the strategy is to distribute the follower list
across multiple linked documents:
1. The Main User Document:
The main user document holds the initial set of followers and a
special array to point to the additional documents.
• Initial Followers: It contains a field, such as "followers", which
holds as many follower ObjectIds as possible up to the document
size limit.
• Example: {"followers" : [ObjectId("..."), ... ]}
• "To Be Continued" (tbc) Array: This array stores the _ids of the
additional "continuation" documents. This acts as a reference
chain to the rest of the followers.
• Example: "tbc" : [ObjectId("512528ced86041c7dca8191e"),
ObjectId("5126510dd86041c7dca81924")]
2. Continuation Documents:
These are separate, smaller documents created
specifically to store the overflow of followers.
Structure: Each continuation document is identified by
an _id (which is referenced in the main user document's
tbc array) and contains its own "followers" array with
more follower ObjectIds.
Example: A document with _id matching one in the tbc
array, and a "followers" array:
{ "_id" : ObjectId("512528ced86041c7dca8191e"),
"followers" : [ObjectId("..."), ObjectId("..."), ...]}
Purpose: They allow the follower list to scale virtually
infinitely without breaking the document size limit, as
the data is horizontally partitioned across multiple
documents.
3. Application Logic
The key part of this solution is the application logic
needed to manage and retrieve the complete list of
followers.
• Fetching Followers: When an application needs the full
list of a celebrity user's followers, it must:
• Retrieve the main user document.
• Extract the follower IDs from the main document's "followers"
array.
• Iterate through the _ids in the "tbc" array.
• For each _id in tbc, perform a separate database query to fetch
the corresponding continuation document.
• Combine the follower IDs from the main document and all the
continuation documents to get the complete list.
This pattern essentially uses a combination of embedding
(in the initial document) and referencing (via the tbc
array) to overcome the physical size constraints of a
single document while still conceptually linking all the
Optimizations for Data Manipulation :
To optimize your application, you must first know
what its bottleneck is by evaluating its read and
write performance.
Optimizing reads generally involves having the
correct indexes and returning as much of the
information as possible in a single document.
Optimizing writes usually involves minimizing the
number of indexes you have and making updates as
efficient as possible.
There is often a trade-off between schemas that are
optimized for writing quickly and those that are
optimized for reading quickly, so you may have to
decide which is a more important for your
application.
Optimizing for Document Growth:
This optimization deals with minimizing the performance impact when
documents frequently grow in size (e.g., adding items to an
embedded array like a list of followers).
• The Problem: In MongoDB, when a document grows and no longer fits
in its allocated space on disk, the database has to move the
entire document to a new, larger space. This document relocation
is an expensive operation that can cause disk fragmentation and
slow down write operations (updates).
• The Solution (Use PowerOf2Sizes): For collections where documents
are expected to grow rapidly and unpredictably (like the
"followers" collection mentioned in the source text), you can set
the usePowerOf2Sizes storage option (though this option is
specific to the MMAPv1 storage engine, which is now deprecated,
the concept of pre-allocation for growth remains valid in modern
storage engines like WiredTiger). Pre-allocating extra space can
reduce the frequency of document moves.
• Context: For documents that grow a lot (like arrays for
followers), separating them into their own collection
(normalization) allows you to apply growth optimizations only
where needed, keeping the main user documents small and stable.
Removing Old Data:
This refers to managing the ever-increasing volume
of data by implementing strategies to archive or
delete data that is no longer frequently accessed.
• The Problem: Storing vast amounts of historical or
log data that is rarely queried consumes excessive
disk space and slows down backups, recovery, and
general queries (as more data must be scanned).
• The Solution:
• Time-To-Live (TTL) Indexes: Use TTL indexes on a time-
based field (e.g., createdAt) to automatically expire and
delete documents after a specified period. This is
perfect for logs, sessions, or temporary notifications.
• Archiving: Migrate older data to a cheaper, slower
storage tier (e.g., cold storage, a separate cluster, or
a data warehouse) instead of deleting it permanently.
Planning Out Databases and Collections:
This involves making fundamental design decisions about how data is
partitioned across databases and organized into collections to meet
performance and operational requirements.
• The Problem: Poor data organization can lead to inefficient
queries, operational inflexibility, and reduced throughput. For
example, mixing high-read, critical data with high-write,
temporary data in the same database can cause contention.
• The Solution:
• Separation of Concerns: Put high-throughput/volatile collections (like
logs, queues, or a dedicated "followers" collection) in a separate
database or even a separate cluster. This isolates their write load and
allows for independent operational tasks (like compaction or maintenance)
without affecting core application data.
• Sharding Strategy: Plan for sharding (horizontal scaling) from the
beginning if massive scale is anticipated. This involves selecting
appropriate shard keys to ensure even data distribution and efficient
query routing.
• Indexing: Determine which fields will be queried frequently and create the
necessary indexes upfront.
Managing Consistency:
In a distributed system (especially a sharded cluster),
consistency refers to ensuring that all users see the same,
correct version of the data at any given time.
• The Problem: Distributed databases face a trade-off between
Consistency, Availability, and Partition Tolerance (CAP
theorem). In MongoDB, data is replicated, and clients might
read stale data if they read from a secondary replica before
it has finished receiving the latest write from the primary.
• The Solution:
• Write Concerns: Use an appropriate write concern (e.g., wait for
acknowledgment from the majority of replicas) to ensure writes are
durable and consistent across the replica set.
• Read Concerns: Use a read concern (e.g., "majority") to guarantee
that the data being read has been acknowledged by the majority of
the replica set, preventing reads of stale data.
• Transactional Guarantees: For operations involving multiple
documents that must be atomic, use multi-document transactions
(available in MongoDB 4.0+) to ensure all-or-nothing consistency.
Migrating Schemas:
Schema migration is the process of updating the data model (or
structure) of documents in a collection when application
requirements change.
• The Problem: Unlike SQL databases, NoSQL databases are "schema-
less," but an implicit application schema still exists. Changing
the application's expected document structure (e.g., renaming a
field, changing a data type) requires updating existing documents.
• The Solution:
• In-Place Migration (Fast): Use a bulk write operation (like
[Link]()) to quickly apply the schema changes across the
entire collection. This is fast but puts immediate load on the cluster.
• Lazy Migration (Slow/Less Load): Update documents only when they are read
or written to by the application. This spreads the migration load over
time but requires the application code to handle both the old and new
schema formats.
• Two-Phase Writes: For critical changes, write new data in both the old and
new format for a period, read from the old format, and then switch the
application to read and write only the new format. This offers a zero-
downtime transition.