Module-1

Chapter-1: Why NoSQL

Why NoSQL?

NoSQL databases are used primarily because they offer flexibility, scalability, and performance that
suit certain types of applications better than traditional relational databases (SQL). Here are key
reasons why NoSQL databases are often chosen:

1. Scalability: NoSQL databases are designed to scale out horizontally, making them ideal for
handling large volumes of data across distributed systems. This contrasts with SQL databases,
which typically scale vertically (adding more resources to a single server).

2. Flexibility in Data Models: NoSQL databases can handle a variety of data formats, such as key-
value pairs, documents, wide-column stores, and graphs. They don't require a predefined schema,
making it easier to adapt to changing data structures.

3. Handling Unstructured Data: NoSQL is better suited for managing unstructured or semi-
structured data, such as JSON or XML, which is common in modern applications like social
media, IoT, and big data platforms.

4. Performance: NoSQL databases can provide faster read/write operations for certain use cases,
especially when data consistency can be relaxed in favor of availability and partition tolerance (as
per the CAP theorem).

5. Big Data and Real-Time Analytics: NoSQL databases are ideal for big data applications and
real-time analytics where the data is vast and constantly changing.

6. High Availability and Fault Tolerance: NoSQL databases are often distributed across many
nodes, providing better fault tolerance and high availability, ensuring that the system continues to
operate even in case of node failures.

7. Schema Evolution: NoSQL databases allow for easy changes to the data structure without
downtime or complex migrations, making them highly flexible for rapidly evolving applications.

For almost as long as we’ve been in the software profession, relational databases have been the
default choice for serious data storage, especially in the world of enterprise applications. If you’re
an architect starting a new project, your only choice is likely to be which relational database to use.
There have been times when a database technology threatened to take a piece of the action, such as
object databases in the 1990’s, but these alternatives never got anywhere.
After such a long period of dominance, the current excitement about NoSQL databases comes as
a surprise.
1.1 The Value of Relational Databases
Relational databases have become such an embedded part of our computing culture that it’s easy to
take them for granted. It’s therefore useful to revisit the benefits they provide.

1.1.1 Getting at Persistent Data

 Probably the most obvious value of a database is keeping large amounts of persistent data.
 Most computer architectures have the notion of two areas of memory: a fast volatile “main
memory” and a larger but slower “backing store”.
 Main memory is both limited in space and loses all data when you lose power or
something bad happens to the operating system.
 Therefore, to keep data around, we write it to a backing store, commonly seen a disk.

 The backing store can be organized in all sorts of ways. For many productivity applications,
it’s a file in the file system of the operating system.

 For most enterprise applications, however, the backing store is a database.

 The database allows more flexibility than a file system in storing large amounts of data in a
way that allows an application program to get at small bits of that information quickly and
easily.

1.1.2 Concurrency
 Enterprise applications tend to have many people looking at the same body of data at once,
possibly modifying that data. Most of the time they are working on different areas of that
data, but occasionally they operate on the same bit of data.

 Concurrency is notoriously difficult to get right, with all sorts of errors that can trap even the
most careful programmers. Since enterprise applications can have lots of users and other
systems all working concurrently, Relational databases help handle this by control- ling all
access to their data through transactions. While this isn’t a cure-all the transactional
mechanism has worked well to contain the complexity of concurrency.

 Transactions also play a role in error handling. With transactions, you can make a change,
and if an error occurs during the processing of the change you can roll back the transaction
to clean things up.
 1.1.3 Integration
 Enterprise applications live in a rich ecosystem that requires multiple applications, written by
different teams, to collaborate in order to get things done. This kind of inter-application
collaboration is awkward because it means pushing the human organizational boundaries.

 Applications often need to use the same data and updates made through one application
have to be visible to others.
 A common way to do this is shared database integration, where multiple applications store
their data in a single database.
 Using a single database allows all the applications to use each others’ data easily, while the
database’s concurrency control handles multiple applications in the same way as it handles
multiple users in a single application.

1.1.4 A Standard Model

 Relational databases have succeeded because they provide the core benefits we outlined
earlier in a standard way.

 As a result, developers and database professionals can learn the basic relational model and
apply it in many projects.

 Although there are differences between different relational databases, the core
mechanisms remain the same: Different vendors’ SQL dialects are similar, transactions
operate in mostly the same way.

1.2 Impedance Mismatch

Impedance mismatch refers to the difficulty developer’s face when trying to store complex, in-
memory data structures (used in applications) in relational databases, which are designed to store
data in simpler, table-like formats. Here’s a simple explanation of the passage:

1. Relational Databases:

 Relational databases organize data into tables made up of rows (also called relations and
tuples in the relational model).
 A row (tuple) contains simple values, like numbers or text, but cannot store complex
structures like lists or nested data.
2. In-Memory Data Structures:

 In-memory data (the data stored and used by your application) can have complex
structures, like objects, lists, or nested records. These structures are rich and flexible,
unlike the simpler format used by relational databases.

3. The Problem (Impedance Mismatch):

 To store these rich in-memory data structures in a relational database, you have to
translate them into a table/row format, which can be frustrating and time-consuming for
developers.
 This mismatch between the two formats is called impedance mismatch.

4. Historical Context:

 In the 1990s, many believed that object-oriented databases (which store data in a way
similar to how in-memory data structures are organized) would replace relational
databases. However, this didn’t happen.
 Object-oriented databases became less popular, while relational databases remained
dominant because of the widespread use of SQL, their integration capabilities, and their
established role in the industry.

5. Object-Relational Mapping (ORM) Tools:

 To help with this mismatch, ORM frameworks (like Hibernate or iBATIS) were
developed. These tools automatically handle the conversion between complex in-memory
data structures and relational tables, saving developer’s time.
 However, ORM tools can create their own problems, especially if developers try to ignore
how the database works, which can lead to poor performance.

6. Relational Databases Today:

 Even though relational databases dominated for a long time, in the 2000s, new
challenges started to emerge, such as the rise of NoSQL databases that are better suited
for certain types of data and applications.
Example of Impedance Mismatch:

Suppose you have a complex in-memory object in a programming language like Java that represents
a Customer. This object has:

 Name (a simple string),

 Address (another object containing street, city, state, etc.),
 A List of Orders (a collection of Order objects, where each Order contains details like
product, price, and quantity).

In a relational database, you can’t directly store this complex structure as it is. You would have to
break it down:

 Store the Name in a Customer table,

 Store the Address in a separate Address table (linked to the Customer table),
 Store each Order in an Orders table.

This process of translating the complex object into multiple tables and rows is the impedance
mismatch. ORM tools like Hibernate help automate this, but they can still struggle with
performance if not used properly.

Figure1.1. An order, which looks like a single aggregate structure in the UI, is split into many rows
from many tables in a relational database.
1.2 Application and Integration Databases
There are two types of database usage: integration databases (used by multiple applications) and
application databases (used by just one application). It also discusses how communication between
applications shifted from using SQL databases to web services, which allowed for more flexible data
structures and easier communication.

1. Integration Databases:

Integration databases are shared between multiple applications, which keeps data consistent
but makes the database more complex and harder to change.

 Multiple applications share a common database. This helps ensure all applications are
working with the same data, which is useful for consistency across different systems.
 However, it makes the database more complex since it has to meet the needs of all the
applications. Changing the database for one application becomes tricky because it could
impact others.
 Different applications may have different needs. For example, one might need a database
index for faster searches, but that could slow down another application that frequently
adds new data.

Example:

Let’s say you have two applications: App A and App B.

 Integration Database: If both App A and App B store their data in a shared database, the
database must be designed to meet the needs of both apps. If App A needs an index for fast
searching, but App B mainly adds new data, the index might slow down App B because
adding data becomes slower with the index. Also, if App A wants to change its data structure,
it must coordinate with App B to avoid breaking anything.

2. Application Databases:

Application databases are used by one application and its team, making them easier to manage
and allowing for flexibility in database design.

 An application database is used by a single application (and team). This makes it easier to
change and maintain the database since only one team needs to know how it’s structured.
 The team can also handle data integrity (making sure the data stays correct) in the
application code itself, reducing the database's complexity.
  Application Database: Instead, if each app has its own application database, App A can
structure and optimize its database without worrying about App B, and vice versa. This gives
both apps more flexibility. For example, App A might use a relational database, while App
B could use a NoSQL database because it needs to handle lots of unstructured data.

3. Shift to Web Services:

 In the 2000s, applications started communicating using web services over HTTP (instead of
directly using a shared database).
 This allowed for more flexible data formats, like XML or JSON, which can represent
complex data structures, including nested records and lists. This is different from SQL,
which needs simpler, table-like data.
 Web services enabled applications to send and receive rich data structures in a single
request or response, reducing the need for multiple database queries.

4. Using Services and Non-Relational Databases:

 When you use services to communicate, the external world no longer cares how you store
your data internally. This gives you the freedom to use different types of databases, even
non-relational ones, because you're not tied to SQL.
 However, despite this flexibility, most teams stuck with relational databases. They were
familiar and worked well for most applications.

1.3 Attack of the Clusters

A cluster is a group of connected computers (often called nodes) that work together as a single
system. Instead of using one powerful machine, a cluster uses multiple smaller machines to share the
workload. If one machine fails, the others can keep the system running smoothly, making clusters
both powerful and reliable.

In the early 2000s, after the dot-com bubble burst, websites started growing massively, both in terms
of data (like logs, social networks, mapping data) and the number of users visiting these sites. This
increase in data and traffic meant that companies needed more computing power. They had two
choices to handle this:
1. Scale Up: Use bigger and more powerful machines (but this gets expensive and has limits).
2. Scale Out: Use lots of smaller, cheaper machines in a cluster. Clusters of small machines can
handle large workloads, and if one machine fails, the others keep things running.

 However, traditional relational databases (like Oracle or SQL Server) are not designed to work
efficiently on clusters. They either need a shared disk (which can become a single point of
failure) or require sharding, where data is split across several servers.

 Sharding causes issues because applications have to manage it, and it can break features like
querying across data shards or keeping data consistent.

 Additionally, the cost of licensing relational databases for clusters is high, which made
companies like Google and Amazon explore alternatives.

 Both companies were dealing with huge amounts of data and needed more scalable solutions, so
they created new types of databases designed for clusters.

 Google developed BigTable and Amazon developed Dynamo, which were more suitable for
large-scale data and clusters.

 Although most companies don’t operate at the massive scale of Google and Amazon, more and
more businesses are handling large datasets and facing similar problems.

 As Google and Amazon shared information about their new databases, others began to follow
their lead, building databases that work better with clusters.

Example:
For example-1: large websites like Facebook or Amazon use clusters to handle the huge amount of
data and users they deal with every day. Each machine in the cluster processes a part of the data, and
together they work faster and more efficiently than one big machine could.

Example-2: Imagine a big e-commerce company, like Amazon, which handles millions of orders
daily. Storing all this data in a single relational database would quickly overwhelm the system.

If they tried to scale up with bigger servers, it would get very expensive. Instead, Amazon uses a
cluster of smaller machines to store and process all this data.

But traditional relational databases don't work well in clusters, so Amazon created Dynamo, which
could spread the data across multiple machines in the cluster while still ensuring the system runs
smoothly, even if some machines fail.
1.4 The Emergence of NoSQL

a. The term NoSQL first appeared in the late 1990s, used by Carlo Strozzi for a relational
database that didn't use SQL (Structured Query Language) but rather Unix commands to
manipulate data. However, this older system has nothing to do with the modern NoSQL we
talk about today.

b. The modern use of the term NoSQL started in 2009 at a meetup in San Francisco, organized
by Johan Oskarsson. Developers were exploring new ways to store data because traditional
relational databases weren’t enough for the big data and cluster computing needs of
companies like Google and Amazon. A developer, Eric Evans, suggested the name “NoSQL”
during this meetup, and it stuck.

c. NoSQL databases started to gain attention because they don't rely on the relational model or
SQL queries. These databases are usually open-source and are designed to handle large-scale
data on clusters (multiple small machines working together). Since relational databases
were originally built to work on single servers, they often struggled with scaling, leading
companies to explore NoSQL options.

d. An important feature of NoSQL databases is that they can store unstructured data without
needing a predefined schema (structure). This is useful for situations where data doesn’t fit
neatly into tables, or the structure of data changes frequently. They also offer flexibility in
terms of consistency, unlike relational databases that focus on ACID transactions for strong
consistency.

e. While NoSQL databases are often used for large-scale data on clusters, they are also
appealing for smaller applications because they can simplify database management and
improve developer productivity by making database interactions more straightforward.
 Chapter 2: Aggregate Data Models

2.1 Aggregates

Aggregate is a term that comes from Domain-Driven Design. In Domain-Driven Design, an

aggregate is a collection of related objects that we wish to treat as a unit. In particular, it is a unit
for data manipulation and management of consistency. Typically, we like to update aggregates with
atomic operations and communicate with our data storage in terms of aggregates.

Aggregate data models are a way of organizing and grouping related data together. Instead of
storing individual pieces of data separately, an aggregate model stores them as a collection, treating
them as a single unit or "aggregate." This makes it easier to work with related data as one package
rather than dealing with separate bits and pieces.

In a relational database, data is stored in simple tables made up of rows (also called tuples), where
each row holds a set of values like a name, address, or phone number. These rows are flat, meaning
you can't nest one row inside another, or put lists or collections of data within a single row.

But with aggregate-oriented databases, things work differently. These databases allow you to store
more complex, nested data structures. Instead of just storing flat rows, you can store a collection of
related data together in a single unit, called an aggregate. For example, instead of storing separate
rows for a customer’s name, address, and order details, you could bundle all of that information into
one aggregate.

2.1.1 Example of Relations and Aggregates

Let's say we're building an e-commerce website where customers can buy items online. We need to
store a lot of information, like details about users, the products we're selling, customer orders,
shipping and billing addresses, and payment information.

To organize this data, we can use two approaches: relational databases (like SQL databases) and
NoSQL databases.

In a relational database, we would create different tables for each type of data. For example, we
might have one table for users, another for products, another for orders, and more for addresses
and payments. These tables would be linked together using relationships, like connecting a
customer's order to their details and their payment.
For a NoSQL database, the approach is different. Instead of splitting everything into separate tables,
we might group related data together in one place (an aggregate). For example, we could store all
the information about an order, including customer details, products, shipping address, and payment,
in a single document. This makes it easier to work with all related data at once, especially when you
want to update or retrieve everything about an order in one go.

Figure 2.1 Data model oriented around a relational database (using UML notation )
 Figure 2.2 Typical data using RDBMS data model

Figure 2.2 presents some sample data for this model.

As we’re good relational soldiers, everything is properly normalized, so that no data is repeated in
multiple tables. We also have referential integrity.

Figure 2.3. An aggregate data model

Now let’s see how this model might look when we think in more aggregate-oriented terms (Figure
2.3)Again, we have some sample data, which we’ll show in JSON format as that’s a common
representation for data in NoSQL land.
// in customers

{ "id":1,

"name":"Martin", "billingAddress":[{"city":"Chicago"}]

// in orders

{ "id":99,

"customerId":1, "orderItems":[

"productId":27, "price": 32.45,

"productName": "NoSQL Distilled"

],

"shippingAddress":[{"city":"Chicago"}] "orderPayment":[

"ccinfo":"1000-1000-1000-1000",

"txnId":"abelif879rft", "billingAddress": {"city": "Chicago"}

],

In this model, we have two main aggregates: customer and order. We’ve used the black-
diamond composition marker in UML to show how data fits into the aggregation structure. The
customer contains a list of billing addresses; the order contains a list of order items, a shipping
address, and payments. The payment itself contains a billing address for that payment.

A single logical address record appears three times in the example data, but instead of using
IDs it’s treated as a value and copied each time. This fits the domain where we would not want the
shipping address, nor the payment’s billing address, to change. In a relational database, we would
ensure that the address rows aren’t updated for this case, making a new row instead. With
aggregates, we can copy the whole address structure into the aggregate as we need to.

The link between the customer and the order isn’t within either aggregate, it’s a relationship
between aggregates. Similarly, the link from an order item would cross into a separate aggregate
structure for products, which we haven’t gone into. We’ve shown the product name as part of the
order item here—this kind of denormalization is similar to the tradeoffs with relational databases,
but is more common with aggregates because we want to minimize the number of aggregates we
access during a data interaction.
 Indeed we could draw our aggregate boundaries differently, putting all the orders for a customer
into the customer aggregate (Figure 2.4).

Figure 2.4 Embed all the objects for customer and the customer’s orders

Using the above data model, an example Customer and Order would look like this:
// in customers

"customer": {

"id": 1,

"name": "Martin",

"billingAddress": [{"city": "Chicago"}],"orders": [

"id":99,
 "customerId":1,"orderItems":[

"productId":27,"price": 32.45,

"productName": "NoSQL Distilled"

],

"shippingAddress":[{"city":"Chicago"}]

"orderPayment":[

"ccinfo":"1000-1000-1000-1000",

"txnId":"abelif879rft", "billingAddress": {"city": "Chicago"}

}],

}]

Like most things in modeling, there’s no universal answer for how to draw your aggregate boundaries.
It depends entirely on how you tend to manipulate your data. If you tend to access a customer
together with all of that customer’s orders at once, then you would prefer a single aggregate.
However, if you tend to focus on accessing a single order at a time, then you should prefer having
separate aggregates for each order

2.1.2 Consequences of Aggregate Orientation

In a relational database (like SQL), data is divided into tables with rows and foreign key
relationships, but it does not understand the idea of an aggregate (a bundle of related data treated as
a single unit). For example, an order includes items, a shipping address, and payment information,
but the relational model just treats them as separate tables linked together without recognizing them
as a single unit or aggregate.

In NoSQL databases, the idea of aggregates is central. Aggregates are important because
they help define how data is used together in an application. This focus is not based on how the
data logically fits together but rather how it’s grouped for interactions within the application.
Relational databases (and some NoSQL ones, like graph databases) are called aggregate-ignorant
because they don’t work with aggregates. This can be helpful when you want to view data from
different perspectives without needing a specific structure.

However, in distributed systems (like those using clusters of servers), aggregates are very
useful. They tell the database what data is typically used together, which helps ensure that related
data is stored on the same server, making it easier and faster to retrieve.

Aggregates also impact how transactions (actions that involve changing multiple pieces of
data) are handled. In relational databases, you can perform ACID transactions, meaning you can
update many rows across many tables in a way that is Atomic, Consistent, Isolated, and Durable.
This ensures that either all changes are made, or none are, and the operation is isolated from others.

In aggregate-oriented NoSQL databases, you can only perform ACID transactions on a single
aggregate at a time. If you need to update multiple aggregates together, you’ll need to manage that
in your code, but this is often not necessary because data is grouped into aggregates with this in
mind. However, aggregate-ignorant databases (like graph databases) usually do support ACID
transactions across multiple pieces of data, similar to relational databases.

2.2 Key-Value and Document Data Models

In both key-value and document databases, data is organized in aggregates, which are chunks of
related data that we access as a whole. These aggregates have a unique key or ID that is used to
retrieve them.

1. Key-Value Databases:

Key-Value Databases store data as blobs that are accessed by a key. The database doesn't
understand the internal structure of the data, giving you the freedom to store anything.

 Opaque Aggregates: The database sees the aggregate as just a blob of data with no structure—
basically, the database doesn’t care what’s inside. This gives us flexibility because we can store
anything we want in that blob.
 Access by Key: To get the data, we can only look it up by its key (like looking up a word in a
dictionary).
 Example: If you store customer information in a key-value database, you can only retrieve all
the data about the customer using their unique ID, not by any specific field (like their name or
age).
 2. Document Databases:

Document Databases store data in a structured format, allowing you to query specific fields inside
the document, not just retrieve by key.

 Structured Aggregates: The database understands the structure of the data inside the
aggregate (for example, it knows there’s a name field, address field, etc.).
 Querying the Data: We can query (search) based on the fields inside the document, not just
by key. Also, we can retrieve only parts of the document if needed.
 Example: If you store a customer record in a document database, you can search for customers
by their names or retrieve just their addresses.

The line between key-value and document databases can be a bit blurry:

 Sometimes, key-value databases allow some structure in the aggregate (like adding metadata
or breaking it into lists).

 Similarly, document databases can be used to look up data by key, just like key-value stores.

In general, the main difference is:

 In key-value databases, we mostly retrieve data using a key.

 In document databases, we mostly search based on the structure of the data (not just the
key).

2.3 Column-Family Stores

Column-Family Stores are a type of NoSQL database that organizes data differently than traditional
relational databases. They are influenced by Google's BigTable and are designed to handle large
amounts of data efficiently, especially for reading specific columns from many rows.

1. Structure:

 Two-Level Map: Instead of thinking of data as tables, it helps to think of it as a two-level

structure. The first level is like a main identifier (like a row key), and the second level
consists of related pieces of data (columns) associated with that identifier.
 Column Families: Each group of related columns is called a column family. Each column
must belong to one column family, and it’s assumed that columns in the same family will
often be accessed together.
2. Row-Oriented vs. Column-Oriented:

 Row-Oriented: Each row represents an aggregate of related data (like a customer's

information), and the column families contain chunks of that data (like profile or order
history).
 Column-Oriented: Each column family defines a record type (e.g., customer profiles), and
you think of a row as a combination of records across different column families.

3. Flexibility:

 In a column-family database, you can add new columns to any row without major changes to
the database structure. This means that different rows can have different columns, making it
flexible for various data types.

Wide and Skinny Rows:

 Skinny Rows have few columns with similar columns across many rows (like a traditional
table).
 Wide Rows have many columns (potentially thousands), where each row can have very
different columns. This is useful for modeling lists, where each item in the list can be its own
column.
4. Sorting and Access:

 Column families can define a sort order for their columns, which helps in accessing data
efficiently. For instance, you could sort orders by a combination of date and ID to retrieve
them in a specific order.
 column key column value

column family

Figure 2.5 Representing customer information in a column-family structure

2.4 Summarizing Aggregate-Oriented Databases

Aggregate-Oriented Databases are types of databases that focus on storing data in groups called
aggregates, which are identified by a key. Here’s how the three main styles differ:

1. Common Features:

Aggregates: All these models use aggregates as the main unit of data. This means that all related
data is stored together, making it easier to manage in a clustered environment.

Atomic Updates: Changes can be made to the whole aggregate at once, providing a basic level of
transaction control.

2. Differences:

Key-Value Model:

 Treats aggregates as a single block of data (opaque).

 You can only access the entire aggregate using its key, with no option to query or retrieve parts
of it.
Document Model:

 Makes the structure of the aggregate visible (transparent).

 Allows for more flexible querying and retrieving specific parts of the aggregate, but lacks a strict
structure. This can limit the database's ability to optimize how it stores and retrieves data.

Column-Family Model:

 Organizes aggregates into groups called column families, which helps structure the data.
 This organization allows the database to optimize access and storage based on the structure of
the data within each column family.

Chapter 3

More Details on Data Models

3.1 Relationships

 Aggregates and Relationships: Aggregates are groups of related data that are often accessed
together. For example, a customer and their orders can be combined into one aggregate. However,
sometimes you may want to handle orders individually, treating them as separate aggregates.

 Linking Aggregates: To connect these separate aggregates, you can store the customer ID within
each order. When you need customer information, you can look it up using this ID. This works fine,
but the database itself doesn’t recognize the relationship between the customer and orders.

 Database Support for Relationships: Some databases, even simpler ones like key-value stores,
allow you to make these relationships visible. For instance, document stores let you create indexes
based on the content of aggregates, while some key-value stores offer metadata for relationships.

 Handling Updates: In aggregate-oriented databases, updates are done within a single aggregate. If
you need to update multiple aggregates at once, you have to manage potential errors yourself. In
contrast, relational databases allow you to update multiple records in one go, ensuring consistency
and reliability (ACID properties).

 Complex Relationships: While aggregate-oriented databases can struggle with complex

relationships, relational databases also have challenges when dealing with many relationships,
especially when it comes to performance with multiple joins in SQL queries.
 NoSQL Databases: There’s a mention of another type of database that fits somewhere between
traditional relational databases and NoSQL databases, which is worth exploring.

3.2 Graph Databases

Graph databases are a type of database designed to store and manage data in a way that emphasizes
relationships between different pieces of information. Instead of using tables like traditional
relational databases, graph databases use structures called nodes and edges.

 Nodes: These are the individual pieces of data, similar to records in a database. For example,
in a social network, a node could represent a person or a post.
 Edges: These are the connections between the nodes. They represent the relationships
between the data. For example, an edge could represent a friendship between two people or a
comment made on a post.

Figure 3.1 An example graph structure

  Graph Databases in NoSQL: Graph databases are different from other NoSQL databases.
While many NoSQL databases focus on handling large records with simple relationships,
graph databases focus on small records with complex relationships.
 Graph Structure: In this context, a "graph" refers to a structure made of nodes (which hold
data) connected by edges (which show relationships). For example, nodes could be names of
people, and edges could represent friendships.
 Complex Queries: Graph databases are great for answering complex questions about
relationships. For instance, you could easily find books in a specific category written by
authors liked by a friend.
 Data Models: The basic model consists of nodes and edges, but different graph databases
offer various features. Some allow you to attach additional information to nodes and edges,
while others keep it simple.
 Efficient Navigation: Graph databases allow for quick navigation through these relationships.
Unlike relational databases, which can slow down when dealing with many connections (due
to complex joins), graph databases make it easier and faster to traverse connections.
 Focus on Relationships: Most queries involve moving through the network of connections.
You usually start by finding a specific node (like a person) and then exploring related nodes
through edges.
 Single Server vs. Clusters: Graph databases often run on a single server rather than
distributed systems. They also need to ensure consistency across multiple nodes and edges,
which requires careful management of transactions.

3.3 Schemaless Databases

 Schemaless databases are a type of NoSQL database that allow you to store data without a
predefined structure. This flexibility makes it easy to adapt to changing needs and handle
diverse data types. However, while this freedom is beneficial, it can lead to confusion in
programs that need to know the exact nature of the data.

 No Fixed Schema: Schemaless databases, a type of NoSQL database, don’t require you to
define a structure (schema) before storing data. In traditional relational databases, you have to
create tables and specify what kind of data each column can hold.

Flexible Data Storage:Key-Value Stores: Store any data under a key.

Document Databases: Allow for various document structures without restrictions.

Column-Family Databases: Let you store different data types under any column.

Graph Databases: Allow easy addition of new connections and properties.

Advantages of Flexibility: This flexibility means you can change what data you store as your
project evolves. You can add new data types or stop storing old ones without losing anything.

 Handling Diverse Records: Schemaless databases can accommodate records with different
fields, avoiding issues like empty columns (sparse tables) or meaningless column names.

 Implicit Schema: Even without a formal schema, applications that use the database often
create their own assumptions about the data structure (an implicit schema). This can lead to
confusion if different parts of the application expect different field names or data types.

 Code Dependency: To understand the data, you often need to look at the application code,
which can be problematic if the code isn’t well-organized. The database itself doesn’t know
about these implicit rules, which can lead to inconsistent data handling.

 Multiple Applications: If multiple applications access the same schemaless database, it can
cause problems because each might have different assumptions about the data. Solutions
include:
 Encapsulating database interactions within one application.
 Clearly separating different data areas for each application.
 Relational Schema Flexibility: While NoSQL advocates criticize relational databases for
being rigid, relational schemas can actually be modified using SQL commands. You can add
new columns as needed, even though this is not commonly done.
 Controlled Changes: Both relational and schemaless databases require careful management
when changing data structures. With schemaless databases, this flexibility only applies within
a specific area of data (aggregate), and changing the boundaries of that area can be complex.

3.4 Materialized Views

Materialized views are a way to store the results of a database query as a physical table. Instead of
running the same complex query every time you need the results, a materialized view saves the data
from that query, making it faster to access later.

 Aggregate-Oriented Data Models: These models keep all related data together, which is
useful for accessing complete records (like orders). However, this can make it difficult to
answer specific questions, such as how many of a certain product sold over time, because you
might need to look through all orders.
 Relational Database Advantage: Relational databases can access data in various ways
because they lack a strict aggregate structure. They use “views” to show data differently
without altering how it’s stored. A view is like a virtual table created by computing data from
base tables.

 Challenges with Views: While views can simplify data access, computing them can be
resource-intensive. This is where materialized views come in.

 Materialized Views: These are precomputed views stored on disk, which means they can be
accessed quickly. They are helpful for data that is frequently read but doesn’t need to be
completely up-to-date all the time.

 Strategies for Building Materialized Views:

 Eager Approach: Update the materialized view immediately when the base data changes
(e.g., when an order is placed, update sales data for products). This keeps the view fresh
but can slow down the system if updates happen frequently.

 Batch Jobs: Update materialized views at set intervals instead of immediately. This
approach reduces the overhead of constant updates, but you’ll need to decide how
outdated the views can be.

 Creating Materialized Views: Materialized views can be built directly in the database or
outside of it, depending on the system's capabilities. You provide the necessary
computation, and the database takes care of it.
 Using Within Aggregates: Materialized views can also be used within the same data
structure (aggregate), such as adding a summary of an order to avoid transferring all
details when only a summary is needed. In column-family databases, you can store
materialized views in separate column families, allowing for atomic updates.
 3.5 Modeling for Data Access

Figure 3.2 Embed all the objects for customer and their orders.

If the requirements are to read the orders or the products sold in each order, the whole object
has to be read and then parsed on the client side to build the results.

When references are needed, we could switch to document stores and then query inside the
documents, or even change the data for the key-value store to split the value object into Customer and
Orderobjects and then maintain these objects’ references to each other.

We can now find the orders independently from the Customer, and with the orderId reference
in the Customer we can find all Orders for the Customer. Using aggregates this way allows for read
optimization, but we have to push the ordered reference into Customer every time with a new Order.

# Customer object
{"
customerId": 1,
"customer": {
"name": "Martin",
"billingAddress": [{"city": "Chicago"}],
"payment": [{"type": "debit","ccinfo": "1000-1000-1000-1000"}],
"orders":[{"orderId":99}]
}
}
# Order object
{"
customerId": 1,
"orderId": 99,
"order":{
"orderDate":"Nov-20-2011",
"orderItems":[{"productId":27, "price": 32.45}],
"orderPayment":[{"ccinfo":"1000-1000-1000-1000",
"txnId":"abelif879rft"}],
"shippingAddress":{"city":"Chicago"}
}
}

Figure 3.3. Customer is stored separately from Order.

  Aggregates can also be used to obtain analytics; for example, an aggregate update may fill in
information on which Orders have a given Product in them.

 This denormalization of the data allows for fast access to the data we are interested in and is
the basis for Real Time BI or Real Time Analytics where enterprises don’t have to rely on
end-of-the-day batch runs to populate data-warehouse tables and generate analytics; now they
can fill in this type of data, for multiple types of requirements, when the order is placed by
the customer.
{"

itemid":27,

"orders":{99,545,897,678}

}{ "

itemid":29,

"orders":{199,545,704,819}

In document stores, since we can query inside documents, removing references to Orders from the

Customer object is possible. This change allows us to not update the Customer object when new

orders are placed by the Customer.

# Customer object

{"

customerId": 1,

"name": "Martin",

"billingAddress": [{"city": "Chicago"}],

"payment": [

{"type": "debit",

"ccinfo": "1000-1000-1000-1000"}

]
}#

Order object

{"

orderId": 99,

"customerId": 1,

"orderDate":"Nov-20-2011",

"orderItems":[{"productId":27, "price": 32.45}],

"orderPayment":[{"ccinfo":"1000-1000-1000-1000",

"txnId":"abelif879rft"}],

"shippingAddress":{"city":"Chicago"}

Since document data stores allow you to query by attributes inside the document, searches
such as “find all orders that include the Refactoring Databases product” are possible, but the decision
to create an aggregate of items and orders they belong to is not based on the database’s query
capability but on the read optimization desired by the application.

When modeling for column-family stores, we have the benefit of the columns being ordered,

allowing us to name columns that are frequently used so that they are fetched first. When using the

column families to model the data, it is important to remember to do it per your query requirements

and not for the purpose of writing; the general rule is to make it easy to query and denormalize the

data during write.

As you can imagine, there are multiple ways to model the data; one way is to store the
Customer and Order in different column-family families (see Figure 3.4). Here, it is important to
note the reference to all the orders placed by the customer are in the Customer column family.
Similar other denormalizations are generally done so that query (read) performance is improved.
 Figure 3.4. Conceptual view into a column data store
When using graph databases to model the same data, we model all objects as nodes and relations
within them as relationships; these relationships have types and directional significance.

Each node has independent relationships with other nodes. These relationships have names
like PURCHASED, PAID_WITH, or BELONGS_TO (see Figure 3.5); these relationship names let
you traverse the graph. Let’s say you want to find all the Customers who PURCHASED a product
with the name Refactoring Database. All we need to do is query for the product node Refactoring
Databases and look for all the Customers with the incoming PURCHASED relationship.
Figure 3.5. Graph model of e-commerce data
This type of relationship traversal is very easy with graph databases. It is especially convenient when
you need to use the data to recommend products to users or to find patterns in actions taken by

users.
 MODULE 2

Chapter4
Distribution Models

 The main reason people are interested in NoSQL databases is because they can run on many
servers at the same time, which makes them better for handling large amounts of data. When data
grows, it becomes harder and more expensive to just upgrade to a bigger server. A better solution
is to add more servers and spread the data across them.

 NoSQL databases are good at scaling this way because they group data into "aggregates" (chunks
of data), which are easy to distribute across different servers.

 By distributing the data across multiple servers, you can handle larger amounts of data, deal with
more requests (both reading and writing data), and make your system more reliable in case some
servers or networks fail. However, managing a system that runs on many servers is more
complex, so it's only worth doing when you really need these benefits.

There are two main ways to spread data across servers: replication and sharding.

1. Replication means making copies of the same data on multiple servers. It can be done
in two ways:

 Master-slave replication: One server (the master) handles all the writes, and the
others (slaves) copy the data from the master.
 Peer-to-peer replication: All servers have the same role and can both read and
write data.

2. Sharding means splitting the data and storing different pieces on different servers.

4.1 SingleServer

 The easiest and simplest option for managing data is to use one server to handle everything—
no distribution of data across multiple servers. This is often the best choice because it avoids
the complexity that comes with using many servers. It’s easier for the people managing the
system and for developers to understand how everything works.

 Even though many NoSQL databases are designed to run on multiple servers, sometimes it’s
better to use just one server if the NoSQL database’s structure fits your application better. For
example, graph databases work best with one server. If your application mainly works with
 groups of related data (called aggregates), using a single-server document or key-value store
can be a good idea because it makes things simpler for developers.

4.2 Sharding

Figure 4.1 Sharding puts different data on separate nodes, each of which does its ownreads and
writes.

 Sharding is a way to split data across different servers to help a busy database handle more
requests. If different users are accessing different parts of the data, you can spread those parts
across multiple servers.

 Ideally, each user talks to just one server, and the load is shared evenly. For example, with ten
servers, each server only handles 10% of the traffic.

 However, it's rare to get perfect balance like that. To come close, you need to group related
data together on the same server and organize it to make accessing it faster. A key question is
how to group data so that one user mostly gets what they need from one server.

 This is where aggregates (related chunks of data) are useful because they combine data that is
often used together, making them a natural choice for sharding.
When placing data on servers, several factors can help performance:

 Location-based grouping: If you know that data will mostly be accessed from a specific

location (like Boston), you can place it in a data center near that location.

 Balancing the load: You want to spread the data evenly across servers, so each one handles

an equal amount of work. This may change over time based on how users access data, so you
might need specific rules for certain situations.

 Sequential access: Sometimes it's helpful to keep data that is likely to be read in order

together on the same server, as Bigtable does by organizing web addresses by reversed
domain names.

 Traditionally, sharding was done manually in the application logic, where data was divided based
on something like customer surnames. This made programming more complicated because the
application had to know how to find data across different servers.

 Also, if you needed to adjust the shards (like adding new ones), you had to change the
application code and move the data. Today, many NoSQL databases offer auto-sharding, where
the database automatically manages data distribution, making it easier for developers.

 Sharding improves performance, especially for applications with a lot of writes. While
replication (copying data to multiple servers) can help with read performance, it doesn't help with
writing data.

 Sharding spreads both reads and writes across servers, allowing horizontal scaling of writes.

 However, sharding doesn't improve system reliability on its own. If a server (node) fails, the data
on that shard becomes unavailable, just like with a single server.

 While the failure only affects the data on that shard, having any part of your data missing is still
a problem. With a single server, you might invest more effort in keeping that one server reliable,
but clusters often use cheaper machines that are more likely to fail.

 Sharding can be a big step, even though it’s made easier when using aggregates. Some databases
are designed to use sharding from the start, so it makes sense to use a cluster right away. Others
start on a single server and only add sharding when needed.

 In those cases, it’s better to start with a single server and switch to sharding only when your
system is nearing its capacity.

 Switching to sharding can be tricky. Some teams wait too long to set it up, and when they finally
try to shard in production, their database struggles to move data to the new shards, causing
downtime.
 4.3 Master-Slave Replication

Figure 4.2 Data is replicated from master to slaves. The master services all writes; reads may
come from either master or slaves.

Master-slave replication is a way to copy data across multiple servers, where one server (the master)
is in charge of the data, and the others (slaves) hold copies of it. The master handles updates (writes),
and the slaves are synced with the master through a replication process.

Master-slave replication is useful for systems with a lot of reads (reading data). You can improve
performance by adding more slaves to handle read requests, while the master focuses on writing and
syncing updates. However, this setup isn't great for systems with a lot of writes, because the master
can still become a bottleneck, although shifting the read load to the slaves helps a bit.

Another benefit is that if the master fails, the slaves can still serve read requests, so users can still
access data. But you won’t be able to write (update) data until the master is restored or a new master
is appointed. The slaves being up to date with the master helps recover from failure quickly, as one
of the slaves can be promoted to master.

This setup can also be helpful if you don’t need to scale out but want better resilience. You can use
the master for both reads and writes, and have a slave as a backup in case the master fails. This gives
you the simplicity of a single-server setup but with the added benefit of a hot backup for better
reliability.
 Masters can be chosen either manually or automatically. With manual selection, you decide in
advance which server will be the master. With automatic selection, the cluster of servers can decide
which one becomes the master, which simplifies things and helps recover more quickly if the master
fails.

To make sure your system can still read data when the master fails, you need to set up separate paths
for reading and writing. This can involve using separate database connections for reads and writes,
which is something not all database tools support. You also need to test thoroughly to ensure that, in
the case of a failure, you can still read data without writing.

While replication offers many advantages, it comes with a downside: inconsistency. Since changes
take time to copy over to the slaves, different clients might see different versions of the data
depending on which slave they are reading from. In the worst case, a client may not be able to read
the latest change it just made. Even if you’re using replication mainly for backup, if the master fails
before syncing with the slaves, any updates that weren’t passed on will be lost.

4.4 Peer-to-Peer Replication

Figure 4.3 Peer-to-peer replication has all nodes applying reads and writes to all the data
 Peer-to-peer replication is a way of managing data across multiple servers where all the servers
(or nodes) are equal. Unlike master-slave replication, there's no single master server. Any server
can handle both reading and writing data, and if one server fails, you can still access your data
through the others. This setup helps with both reads and writes and avoids having a single point
of failure.

 This seems like a great system because you can add more servers to improve performance and
keep running even if some servers fail. But it comes with a major challenge: consistency.
 The biggest issue is the risk of write-write conflicts, which happen when two users try to
update the same piece of data at the same time on different servers. If this happens, the
system could end up with conflicting data. While inconsistencies when reading data can be
temporary (until the system catches up), inconsistencies from writing are more serious
because they create permanent problems.

There are two main ways to handle this:

1. Coordination: When a write happens, the servers can work together to ensure there are no
conflicts. Not all servers need to agree on the write, just a majority, so even if some servers
are down, the write can still succeed. This ensures consistency, but it slows things down
because the servers need to communicate with each other.

2. Accepting inconsistency: In some situations, you might be able to handle conflicting writes
by creating rules to merge them later. This allows for faster performance since you can write
to any server without worrying about conflicts right away, but you have to deal with the
inconsistencies later.

These approaches represent a trade-off between consistency (making sure all servers have the same
data) and availability (making sure the system stays up and running even if some servers fail). You
can choose a solution that fits your needs based on how much you value consistency versus
availability.
 4.5 Combining Sharding and Replication

Figure 4.4. Using master-slave replication together with sharding

Replication and sharding are strategies that can be combined. If we use both master-slave replication
and sharding (see Figure 4.4), this means that we have multiple masters, but each data item only has
a single master. Depending on your configuration, you may choose a node to be a master for some
data and slaves for others, or you may dedicate nodes for master or slave duties.

Figure 4.5: Using Master Slave replication together with sharding