Module 3

NoSQL
Syllabus:
• Aggregation: The Aggregation Framework Pipeline Operations
$match $project $group $unwind $sort $limit $skip Using
Pipelines MapReduce Example 1: Finding All Keys in a Collection
Example 2: Categorizing Web Pages MongoDB an MapReduce
Aggregation Commands count distinct group
• Application Design: Normalization versus Denormalization
Examples of Data Representations Cardinality Friends,
Followers, and Other Inconveniences Optimizations for Data
Manipulation Optimizing for Document Growth Removing Old
Data Planning Out Databases and Collections Managing
Consistency Migrating Schemas.
Textbook2: Chapter7 Chapter8
Aggregation:
The Aggregation Framework in MongoDB is a powerful
tool used for processing and analyzing data within
collections. It works by processing documents through a
pipeline of stages.
Each stage performs an operation on the documents and
passes the results to the next stage. This allows you to
perform complex data analysis, filtering, and
transformation that a simple find() query cannot.
The aggregation framework's pipeline allows to break
down a complex data analysis task into a series of
manageable, logical steps.
The Pipeline Stages:
• The aggregation pipeline consists of multiple stages, each starting with a dollar
sign ($). The most common stages are:
• $match: Filters documents to pass only the ones that match the specified criteria
to the next stage. This is a highly efficient stage, especially when used early in the
pipeline, as it can leverage indexes to reduce the number of documents to process.
• $group: Groups input documents by a specified identifier expression and then
computes an aggregate expression (like a sum, average, or count) for each group.
• $project: Reshapes each document in the stream, often used to select fields, add
new fields, or exclude fields from the output.
• $sort: Reorders the documents by a specified field.
• $limit: Restricts the number of documents passed to the next stage.
• $unwind: Deconstructs an array field from the input documents to output a
document for each element. This is useful for analyzing data within arrays.
Scenario: E-commerce Sales Analysis:
Imagine you are a data analyst for a large online retail company.
Your manager asks you to generate a report on product sales for
the past year. Specifically, they want to know the top three best-
selling categories based on total revenue for 2023. They also
need this report to be generated quickly so they can make real-
time business decisions.
• Problem: The company's sales data is stored in a MongoDB
collection called sales, with millions of documents. Each
document represents a single transaction. A simple find() query
cannot perform the required filtering, grouping, and sorting to
generate this report.
$match:
• This is the filtering stage. It's used to filter documents
from a collection based on a specified query, much like the
find() method.
• Detail: You should almost always put $match at the
beginning of your pipeline, as it reduces the number of
documents passed to subsequent stages, making the
entire aggregation more efficient. It can leverage indexes
for a significant performance boost.
• Example: To process only documents from the
"electronics" category, you would use: “JavaScript”
{ $match: { category: "Electronics" } }
$project:
• This is the reshaping stage. It allows you to select, rename, or
create new fields in the documents that are passed to the next
stage. It can also be used to exclude fields.
• Detail: You can use $project to customize the output of your
pipeline, making it easier to read and analyze. It's often used
to compute new fields based on existing ones.
• Example: To show only the name and price fields and create a
new field for discountedPrice, you would use:
JavaScript
{ $project: { name: 1, price: 1, discountedPrice: { $multiply:
["$price", 0.9] } } }
The $project Stage: Reshaping
Documents:
The $project stage is a powerful pipeline operator used to reshape
and transform documents. It allows you to:
• Select/Exclude Fields: You can choose which fields to include or
exclude from the documents passed to the next stage.
• Rename Fields: You can give a field a new name, which is useful
for creating a cleaner output or preparing for a $group stage.
• Compute New Fields: This is the most powerful feature. You can
perform calculations, string manipulations, logical operations, and
date conversions to create new fields.
• By default, the _id field is always included in the output unless
you explicitly exclude it with "_id": 0.
How $project Works

The $project stage is all about controlling the fields that are passed
from one stage of the aggregation pipeline to the next. It allows you
to:
• Include or Exclude Fields: You can choose which fields to keep and
which to discard.
• Rename Fields: You can give a field a new name using a syntax like
"newFieldName" : "$oldFieldName".
• Create New Fields with Expressions: This is where the real power
of $project lies. You can use various expressions (mathematical, date,
string, logical) to compute new values and assign them to new fields.
The $project stage operates on each document individually. It doesn't
combine documents like $group does.
How $project Works: Types of Expressions

The document you provided outlines several key types of expressions

available within $project:
• 1. Mathematical Expressions: These are used for numeric
calculations. Operators like $add, $subtract, $multiply, $divide, and
$mod can be used to perform arithmetic.
• Example: To calculate an employee's total pay by adding their salary
and bonus.
{
$project: {
totalPay: { $add: ["$salary", "$bonus"] }
}
}
How $project Works: Types of Expressions
2. Date Expressions: These expressions extract components from a date field,
such as the year, month, or day.
Example: To find the year each employee was hired.
{ $project: { hireYear: { $year: "$hireDate" } }}
3. String Expressions:These are used for manipulating string values, with
operators like $substr, $concat, $toLower, and $toUpper.
Example: To generate a lowercase email address from an employee's first and
last name.
{
$project: {
email: {
$concat: [
{ $toLower: "$firstName" },
".",
{ $toLower: "$lastName" },
"@[Link]"
]
}
}
}
How $project Works: Types of Expressions
4. Logical Expressions: These operators perform conditional and
boolean logic.
• Comparison Operators: $eq, $ne, $gt, $gte, $lt, $lte compare two
values and return a boolean.
• Boolean Operators: $and, $or, and $not evaluate multiple boolean
expressions.
• Control Flow: $cond and $ifNull allow you to define conditional logic.
• Example: To determine if an employee is a manager or not based on
a boolean field.
{
$project: {
status: { $cond: ["$isManager", "Manager", "Employee"] }
}
}
How $project Works: Types of Expressions

• Projection Behavior and Efficiency:

The $project stage, unlike $group, is a streaming
operator. This means it can process documents as they
arrive and does not need to collect all documents before
producing output.
This makes it highly efficient. However, as the document
notes, if you rename a field in a $project stage, an index
on the original field name will no longer be usable by a
subsequent $sort or $match stage. This is why it's a best
practice to use $project after you've used a $match stage
that can leverage an existing index.
$group:
This is the aggregation stage. It groups documents by a
specified identifier (_id) and then performs a function on the
documents within each group (e.g., sum, average, count).
Detail: This is the heart of the aggregation framework for data
analysis. You must specify a grouping key (_id) and at least one
accumulator expression (e.g., $sum, $avg, $count).
Example: To calculate the total number of items sold and the
average price for each category, you would use:
JavaScript
{ $group: { _id: "$category", totalSold: { $sum: "$quantity" },
avgPrice: { $avg: "$price" } } }
How Grouping Works:

The $group operator takes two primary parts:

• The Grouping Key (_id): This defines how documents are
grouped. All documents with the same value for the _id
expression will be considered part of the same group. The
grouping key can be a single field ("$_id": "$country"), a
compound key ("$_id": { "state": "$state", "city": "$city" }),
or a constant value to group all documents into one.
• The Accumulator Operators: These are the functions that
perform calculations on the documents within each group.
They are used to create new fields in the output document.
How Grouping Works : Types of Accumulator
Operators
The document you provided outlines several key types of accumulators:
1. Arithmetic Operators: These are used for numeric calculations.
• $sum: Calculates the sum of all values for a field within a group. It can sum
a simple field ($sum: "$revenue") or a constant ($sum: 1 to count
documents).
• $avg: Calculates the average of all values for a field within a group.
• Example: To find the total sales revenue and average price for each
product category in an e-commerce store.
[Link]([
{
$group: {
_id: "$category",
totalRevenue: { $sum: "$price" },
averagePrice: { $avg: "$price" },
numProducts: { $sum: 1 }
}
}
])
How Grouping Works : Types of Accumulator Operators
2. Extreme Operators: These find the maximum or minimum values.
• $max and $min: These operators are used to find the highest and lowest
values of a field in a group. They are most efficient when the data is unsorted.
• $first and $last: These operators return the first and last values encountered
in a group. Crucially, their results are only predictable if the data has
been sorted in a previous pipeline stage. Using a $sort stage before
$group can make them more efficient than $min and $max for finding
extremes.
• Example: To find the highest and lowest-priced product in each category.
[Link]([
// This approach works on unsorted data
{
$group: {
_id: "$category",
lowestPrice: { $min: "$price" },
highestPrice: { $max: "$price" }
}
}
])
How Grouping Works : Types of Accumulator Operators
3. Array Operators: These are used to build arrays from the values in a
group.
• $push: This operator adds every value it encounters to an array. The
resulting array will contain all values for that field from all documents in
the group. The order is not guaranteed unless you pre-sort.
• $addToSet: This is similar to $push, but it only adds a value to the array
if it doesn't already exist. The resulting array will contain only unique
values.
• Example: To find all distinct product brands within each category and
get a list of all prices.
[Link]([
{
$group: {
_id: "$category",
brands: { $addToSet: "$brand" },
prices: { $push: "$price" }
}
}
])
How Grouping Works : Types of Accumulator Operators

Grouping Behavior and Efficiency:

The document correctly identifies $group as a
"roadblock" operator. This means it must collect all
documents that match the grouping key before it can
compute the final result and pass it to the next stage.
• Sharding: In a sharded cluster, the $group operation is
performed in two phases. First, each shard performs a
local grouping on its subset of the data. Then, a central
mongos router collects the partial results from all the
shards and performs a final, comprehensive grouping to
produce the final, correct result. This distributed process
makes $group powerful and scalable.
$unwind:
• This is the deconstruction stage. It's used to deconstruct an array field
from the input documents. For each element in the array, it outputs a
separate document with all the other fields remaining the same.
• Detail: This is especially useful when you need to run an aggregation on
data within an array. For instance, if each order document has an array of
purchased items, $unwind can create a separate document for each item,
making it easy to group and count products.
• Example: If a document contains { orderId: "A123", items: ["Laptop",
"Mouse"] }, $unwind on the items array would produce two documents:
JavaScript
{ orderId: "A123", items: "Laptop" }
{ orderId: "A123", items: "Mouse" }
The $unwind stage would be: { $unwind: "$items" }
$sort:
• This is the sorting stage. It reorders the documents
based on a specified field and direction (ascending or
descending).
• Detail: You can sort on one or more fields. A value of 1
indicates ascending order, and -1 indicates descending
order. For performance, it's best to sort on an indexed
field.
• Example: To sort documents by price from highest to
lowest, you would use:
JavaScript
{ $sort: { price: -1 } }
$limit:
• This is the limitation stage. It restricts the number of
documents that are passed to the next stage.
• Detail: $limit is often used after a $sort to retrieve the
top N documents, as in finding the top 10 most
expensive products.
• Example: To get only the first 10 documents, you would
use:
JavaScript
{ $limit: 10 }
$skip:
• This is the skipping stage. It skips a specified number of
documents and passes the rest to the next stage.
• Detail: $skip is commonly used with $limit to implement
pagination, allowing you to get a specific "page" of
results.
• Example: To get the second page of 10 documents (i.e.,
skipping the first 10), you would use:
JavaScript
{ $skip: 10 }
MapReduce:
MapReduce is a data processing paradigm used to
perform large-scale, parallel computations on data.
In MongoDB, it's a way to perform aggregation
operations, though it has largely been superseded by the
more efficient Aggregation Framework. It consists of two
main phases:
• Map Phase: The map function processes each
document in a collection. It "maps" the data to key-
value pairs and emits them.
• Reduce Phase: The reduce function collects the values
for each unique key and aggregates them. It "reduces"
the data by combining the values into a single result.
MapReduce: Example 1: Finding All Keys in a Collection
Problem: You have a collection of documents with varying
schemas, and you need to find all the unique field names (keys)
used across the entire collection. This is a common task for schema
exploration in schema less databases like MongoDB.
• MapReduce Solution:
• Map Phase: The map function iterates through each key-value
pair in a document. For each key it finds, it emits the key as the
output key and 1 as the value. The value 1 is just a placeholder.
var mapFunction = function() {
for (var key in this) {
emit(key, 1);
}
};
MapReduce: Example 1: Finding All
Keys in a Collection
2. Reduce Phase: The reduce function receives all the 1s emitted for
each unique key. It simply needs to return a single value to indicate
the key's existence.
var reduceFunction = function(key, values) { return 1;};
3. Finalize (Optional): After the reduce phase, you can run a finalize
function to format the output.
4. Execution: You run the MapReduce command on your collection.
The result will be a collection containing all the unique keys found.
[Link]( mapFunction, reduceFunction, {
out: "all_keys" });
This process is a straightforward way to get a unique list of all keys
in a dynamic collection.
MapReduce: Example 2: Categorizing Web Pages
• Problem: You have a collection of website visitor data, with each document containing
the URL of a visited page. You want to group all pages into categories (e.g., "blog,"
"product," "contact") and get a count of how many times each category was visited.
• MapReduce Solution:
• Map Phase: The map function processes each visitor document. It analyzes the URL
string to determine the page's category. For example, a URL containing "/blog/" would
be categorized as "blog." It then emits the category as the key and 1 as the value.
var mapFunction = function() {
var category;
if ([Link]("/blog/") !== -1) {
category = "blog";
} else if ([Link]("/products/") !== -1) {
category = "product";
} else {
category = "other";
}
emit(category, 1);
};
MapReduce: Example 2: Categorizing Web Pages

2. Reduce Phase: The reduce function receives all the 1s for

each category. It sums these values to get the total count for
each category.
var reduceFunction = function(key, values) { return
[Link](values);};
3. Execution: The MapReduce job runs, and the output is a new
collection containing the total number of visits per category.
[Link]( mapFunction, reduceFunction,
{ out: "page_categories" });
This example shows how MapReduce can be used to categorize
data and perform a simple count, a task that would be much
harder with a simple find() query.
MongoDB an MapReduce:
In MongoDB, both MapReduce and the Aggregation Framework are used to process
and analyze data. The Aggregation Framework is generally the preferred method
because it's more efficient, easier to use, and more scalable.
• Aggregation Commands: The Aggregation Framework uses a pipeline of
commands to process data. These commands are often more intuitive and perform
better than their MapReduce counterparts.
• count: This command is used to quickly get the number of documents that match a
query. It's an efficient way to get a total count without retrieving the actual
documents.
• distinct: This command finds and returns all the unique values for a specified field
in a collection.
Here's an example using these commands in a simple scenario.
• Scenario: An e-commerce site needs to find out how many different product
categories they have and the total number of products in each category.
• Solution:
distinct: To find the number of unique categories, you can use the distinct command.
[Link]("category").length
• count: To find the total number of products, you can use the countDocuments
command.
[Link]()
• MapReduce: MapReduce is an older, more manual way to perform
aggregation. It requires you to write custom JavaScript functions for the map
and reduce phases. While powerful, it's often more complex than using the
Aggregation Framework.
• The Aggregation Framework vs. MapReduce
The Aggregation Framework has a dedicated $group stage that can perform the
same tasks as MapReduce and much more, often with better performance
because it's implemented in native code rather than JavaScript.
• To perform the "group" operation from the scenario above (counting
products per category), you would use the $group stage in the Aggregation
Framework.
• Solution using the Aggregation Framework's $group:
[Link]([
{
$group: {
_id: "$category",
totalProducts: { $sum: 1 }
}
}
])
 Comparison:
Feature Aggregation Framework MapReduce

Intuitive pipeline stages ($group, Requires writing custom

Ease of Use
$match) JavaScript functions

Slower, as it uses JavaScript

Performance Much faster and more efficient
engine

Highly flexible with a wide range Flexible, but requires more

Flexibility
of operators manual coding

Designed for distributed Can be complex to manage on

Scalability
processing on shards sharded clusters

The Aggregation Framework has become the standard for performing complex data
analysis in MongoDB due to its performance, expressiveness, and ease of use. Typically,
only need to use MapReduce for very specific, non-standard aggregation tasks that can't
be handled by the Aggregation Framework.
Application Design:
Normalization versus Denormalization
• Normalization :
Normalization is the process of structuring a relational database to reduce data
redundancy and improve data integrity. The goal is to design a database in a way
that stores each piece of information only once. This is achieved by dividing large
tables into smaller, related tables and defining relationships between them.
• Process: Normalization follows a series of rules called Normal Forms (NF), from
1NF to 5NF. The most common forms used in practice are 1NF, 2NF, and 3NF.
• Key Benefits:
• Reduced Data Redundancy: Prevents the same data from being stored in multiple places.
For example, a customer's address is stored only once in a Customers table, not repeatedly in
an Orders table.
• Improved Data Integrity: Ensures that data is consistent. If a customer's address needs to
be updated, it only has to be changed in one place, which prevents conflicting information.
• Faster Inserts, Updates, and Deletes: Because data is not duplicated, write operations
are quicker and more efficient.
• Drawback: The main drawback is that it requires joining multiple tables to retrieve
data. For a complex query, this can result in a large number of joins, which can
slow down read performance.
Normalization versus Denormalization

• Denormalization :
Denormalization is the process of intentionally adding redundant data to a normalized
database. This is typically done to improve the read performance of a database,
especially for data warehousing or reporting applications. Denormalization involves
combining tables or adding duplicate data to reduce the number of joins needed for a
query.
• Process: This is not a formal process like normalization; it's a strategic decision made
after a database has been normalized. You decide what data to duplicate based on the
most frequent or performance-critical queries.
• Key Benefits:
• Faster Reads (Query Performance): By reducing the number of joins, queries can be executed
much faster. For a report that needs customer names, order dates, and product details, all this
information can be stored in a single denormalized table, eliminating the need for three separate
joins.
• Simplified Queries: Queries become simpler to write and easier to manage because they often
involve fewer tables.
• Drawbacks:
• Increased Data Redundancy: Leads to data duplication, which can waste storage space.
• Slower Inserts, Updates, and Deletes: Because data is duplicated, changes have to be made
in multiple places, which can increase the complexity and time of write operations.
• Potential for Data Inconsistency: If a piece of duplicated data is not updated everywhere it
appears, it can lead to data integrity issues.
Examples of Data Representations
Data Representations
Data representations refer to the different ways you can
model relationships between documents or records. The two
main approaches are:
• Embedded Data: This model stores related data within a
single document. It's often used for one-to-one or one-to-
many relationships where the "many" side is not very large
and frequently accessed with the "one" side.
• Referenced Data: This model uses a separate document
for each entity and links them using a reference, typically an
_id. This is the standard relational approach and is suitable
for many-to-many relationships or one-to-many relationships
where the "many" side is very large or frequently accessed
independently.
• There are three primary ways to represent data relationships in MongoDB: referenced,
embedded, and a hybrid approach. The best method depends on the relationship type,
how often the data is read versus updated, and how much the related data is expected to
grow.

• 1. Referenced Data Representation: In this approach, you store related data in

separate collections and use a unique identifier (_id) to link them, similar to foreign keys in
relational databases. This is a highly normalized model.
• Example: Storing students and classes in separate collections. You'd have a students
collection, a classes collection, and a third studentClasses collection that contains
references to the student and an array of class _ids they are taking.
• studentClasses collection document: {"studentId": "...", "classes": ["class_id_1", "class_id_2"]}.
• Pros:
• Data Consistency: If class information (like a room number) changes, you only need to update it in
one place: the classes collection.
• Efficient Updates: This is ideal for volatile data that changes often.
• Scalability: It's great for relationships where the referenced data can grow significantly without
bound, like user comments on a blog post.
• Cons:
• Slower Reads: Retrieving all the information for a student requires multiple queries to dereference
the _ids from different collections, which takes more trips to the server and can be slow.
• Query Complexity: It can make queries more complex, often requiring the use of the $lookup
aggregation stage to "join" the data.
2. Embedded Data Representation
This approach involves storing related data directly inside a single document.
This is a denormalized model that prioritizes read performance.
• Example: Embedding all of a student's class information directly within the
student's document.
• students collection document: {"name": "John Doe", "classes": [{"class":
"Trigonometry", "credits": 3}, ...]}.
• Pros:
• Fast Reads: All related data is retrieved in a single query. This reduces the number of
server trips and is perfect for read-heavy applications.
• Atomic Operations: Since all data is in one document, updates on the entire
document are atomic.
• Simpler Queries: Queries are simpler as they don't require _id dereferencing or joins.
• Cons:
• Data Redundancy: The same class information is duplicated across multiple student
documents. This wastes storage space and can lead to data inconsistency.
• Complex Updates: If a class detail changes, you have to update every student
document that contains that embedded class, which is difficult and prone to errors.
• Document Size Limit: MongoDB has a 16MB document size limit, which can be an
issue if the embedded data grows too large.
3. Hybrid Data Representation
• A hybrid approach combines the best of both worlds by embedding a
small, frequently-accessed subset of data within the main document,
while also including a reference to the full, separate document for more
detailed information.
• Example: Embedding a class's _id and name within the student
document while storing other details like credits and room number in a
separate classes collection.
• students collection document: {"name": "John Doe", "classes": [{"_id":
"class_id_1", "class": "Trigonometry"}, ...]}.
• Pros:
• Balanced Performance: This model offers a balance between fast reads and
efficient updates. You get the most common information in one query but can
perform a second query for less-used details.
• Flexibility: The amount of embedded information can change over time as
application requirements evolve.
• Cons:
• It's a compromise that still requires two queries to get all the data, which is not as
fast as a fully embedded model.
 Pros of Embedding Pros of Referencing

Data Size Small subdocuments Large subdocuments

Data that doesn't change Volatile data that changes

Change Frequency
regularly frequently

When eventual consistency is When immediate consistency is

Consistency
acceptable necessary

Writes vs. Reads Fast reads Fast writes

Documents that grow by a small Documents that grow by a large

Growth
amount amount
Cardinality:
Cardinality describes the nature of the relationship between
two entities. It answers the question: "How many instances
of entity A are related to how many instances of entity B?"
• One-to-One: Each instance of A relates to exactly one
instance of B.
• One-to-Many: One instance of A relates to many
instances of B.
• Many-to-Many: Many instances of A relate to many
instances of B, and vice-versa.
 Friends, Followers, and Other Inconveniences
This phrase highlights the challenges of modeling complex
relationships in a database, especially the many-to-many
relationships common in social networks.
• Friends: A "friends" relationship is typically many-to-many. If
Alice is friends with Bob, then Bob is also friends with Alice. This
symmetric relationship is an inconvenience because you must
model a bidirectional link.
• Followers: A "followers" relationship is typically one-to-many.
If Alice follows Bob, Bob doesn't necessarily follow Alice back.
This asymmetric relationship is easier to model than a
symmetric one.
The "Wil Wheaton Effect" and Continuation Documents:
The Wil Wheaton effect refers to the challenge faced when a popular user (like Wil
Wheaton) has so many followers that embedding all their IDs into a single document would
cause that document to exceed the database's maximum size limit.
To address this, the strategy is to distribute the follower list across multiple linked
documents:
1. The Main User Document:
The main user document holds the initial set of followers and a special array to point to the
additional documents.
• Initial Followers: It contains a field, such as "followers", which holds as many follower
ObjectIds as possible up to the document size limit.
• Example: {"followers" : [ObjectId("..."), ... ]}
• "To Be Continued" (tbc) Array: This array stores the _ids of the additional
"continuation" documents. This acts as a reference chain to the rest of the followers.
• Example: "tbc" : [ObjectId("512528ced86041c7dca8191e"),
ObjectId("5126510dd86041c7dca81924")]
2. Continuation Documents:
These are separate, smaller documents created specifically to store the
overflow of followers.
Structure: Each continuation document is identified by an _id (which is
referenced in the main user document's tbc array) and contains its own
"followers" array with more follower ObjectIds.
Example: A document with _id matching one in the tbc array, and a
"followers" array:
{ "_id" : ObjectId("512528ced86041c7dca8191e"), "followers" :
[ObjectId("..."), ObjectId("..."), ...]}
Purpose: They allow the follower list to scale virtually infinitely without
breaking the document size limit, as the data is horizontally partitioned
across multiple documents.
3. Application Logic
The key part of this solution is the application logic needed to manage and
retrieve the complete list of followers.
• Fetching Followers: When an application needs the full list of a celebrity
user's followers, it must:
• Retrieve the main user document.
• Extract the follower IDs from the main document's "followers" array.
• Iterate through the _ids in the "tbc" array.
• For each _id in tbc, perform a separate database query to fetch the corresponding
continuation document.
• Combine the follower IDs from the main document and all the continuation documents to
get the complete list.
This pattern essentially uses a combination of embedding (in the initial
document) and referencing (via the tbc array) to overcome the physical size
constraints of a single document while still conceptually linking all the data.
Optimizations for Data Manipulation :
To optimize your application, you must first know what its bottleneck is
by evaluating its read and write performance.
Optimizing reads generally involves having the correct indexes and
returning as much of the information as possible in a single document.
Optimizing writes usually involves minimizing the number of indexes
you have and making updates as efficient as possible.
There is often a trade-off between schemas that are optimized for
writing quickly and those that are optimized for reading quickly, so you
may have to decide which is a more important for your application.
Factor in not only the importance of reads versus writes, but also their
proportions: if writes are more important but you’re doing a thousand
reads to every write, you may still want to optimize reads first
Optimizing for Document Growth:
This optimization deals with minimizing the performance impact when documents
frequently grow in size (e.g., adding items to an embedded array like a list of followers).
• The Problem: In MongoDB, when a document grows and no longer fits in its allocated
space on disk, the database has to move the entire document to a new, larger space.
This document relocation is an expensive operation that can cause disk
fragmentation and slow down write operations (updates).
• The Solution (Use PowerOf2Sizes): For collections where documents are expected
to grow rapidly and unpredictably (like the "followers" collection mentioned in the
source text), you can set the usePowerOf2Sizes storage option (though this option is
specific to the MMAPv1 storage engine, which is now deprecated, the concept of pre-
allocation for growth remains valid in modern storage engines like WiredTiger). Pre-
allocating extra space can reduce the frequency of document moves.
• Context: For documents that grow a lot (like arrays for followers), separating them into
their own collection (normalization) allows you to apply growth optimizations only
where needed, keeping the main user documents small and stable.
Removing Old Data:
This refers to managing the ever-increasing volume of data by
implementing strategies to archive or delete data that is no longer
frequently accessed.
• The Problem: Storing vast amounts of historical or log data that is
rarely queried consumes excessive disk space and slows down backups,
recovery, and general queries (as more data must be scanned).
• The Solution:
• Time-To-Live (TTL) Indexes: Use TTL indexes on a time-based field (e.g.,
createdAt) to automatically expire and delete documents after a specified period.
This is perfect for logs, sessions, or temporary notifications.
• Archiving: Migrate older data to a cheaper, slower storage tier (e.g., cold
storage, a separate cluster, or a data warehouse) instead of deleting it
permanently.
Planning Out Databases and Collections:
This involves making fundamental design decisions about how data is partitioned across
databases and organized into collections to meet performance and operational
requirements.
• The Problem: Poor data organization can lead to inefficient queries, operational
inflexibility, and reduced throughput. For example, mixing high-read, critical data with
high-write, temporary data in the same database can cause contention.
• The Solution:
• Separation of Concerns: Put high-throughput/volatile collections (like logs, queues, or a
dedicated "followers" collection) in a separate database or even a separate cluster. This isolates
their write load and allows for independent operational tasks (like compaction or maintenance)
without affecting core application data.
• Sharding Strategy: Plan for sharding (horizontal scaling) from the beginning if massive scale
is anticipated. This involves selecting appropriate shard keys to ensure even data distribution
and efficient query routing.
• Indexing: Determine which fields will be queried frequently and create the necessary indexes
upfront.
Managing Consistency:
In a distributed system (especially a sharded cluster), consistency refers to ensuring
that all users see the same, correct version of the data at any given time.
• The Problem: Distributed databases face a trade-off between Consistency,
Availability, and Partition Tolerance (CAP theorem). In MongoDB, data is
replicated, and clients might read stale data if they read from a secondary replica
before it has finished receiving the latest write from the primary.
• The Solution:
• Write Concerns: Use an appropriate write concern (e.g., wait for acknowledgment from
the majority of replicas) to ensure writes are durable and consistent across the replica set.
• Read Concerns: Use a read concern (e.g., "majority") to guarantee that the data being
read has been acknowledged by the majority of the replica set, preventing reads of stale
data.
• Transactional Guarantees: For operations involving multiple documents that must be
atomic, use multi-document transactions (available in MongoDB 4.0+) to ensure all-or-
nothing consistency.
Migrating Schemas:
Schema migration is the process of updating the data model (or structure) of
documents in a collection when application requirements change.
• The Problem: Unlike SQL databases, NoSQL databases are "schema-less," but an
implicit application schema still exists. Changing the application's expected
document structure (e.g., renaming a field, changing a data type) requires updating
existing documents.
• The Solution:
• In-Place Migration (Fast): Use a bulk write operation (like [Link]()) to
quickly apply the schema changes across the entire collection. This is fast but puts immediate
load on the cluster.
• Lazy Migration (Slow/Less Load): Update documents only when they are read or written to
by the application. This spreads the migration load over time but requires the application code
to handle both the old and new schema formats.
• Two-Phase Writes: For critical changes, write new data in both the old and new format for a
period, read from the old format, and then switch the application to read and write only the
new format. This offers a zero-downtime transition.