INTRODUCTION TO NO

SQL DATABASE
SOFTWARE ENGINEERING

Fabrice FECKOUNDO TCHINDA MTECH SOFTWARE ENGINEER

1|Page
Course Objectives:
Upon successful completion of this course, you should be able to:
- Students should be able to distinguish the various NoSQL systems and their features.
- Understand the impact of the cluster on database design
- State the CAP theorem and explain it main points.
- Explain where HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem
- Work with the Hadoop Distributed File System (HDFS) as a foundation for NoSQL
technologies
- Warehouse HDFS data using Apache Hive
- Data mine HDFS data with Apache Spark-SQL and Apache Pig
- Describe the design of HBase, MongoDB, Cassandra, Neo4j, and Redis
- Use the data control, definition, and manipulation languages of the NoSQL databases
covered in the course.
- Build projects that use NoSQL databases using the Java Programming Language.
- Compare NoSQL databases with each other and relational systems
List of Figures
Figure 1.1 Relational Databases ..................................................................................................5
Figure 1.2 Distributed Databases .................................................................................................6
Figure 1.3 Scaling a RDBMS to Accommodate more data .........................................................7
Figure 2.4 Column Store NoSQL Model ................................................................................... 17
Figure 2.5 Document NoSQL Model ......................................................................................... 18
Figure 2.6 Graph-Based NoSQL Model .................................................................................... 19
Figure 2.7 Creating a MongoDB Database ................................................................................ 20
Figure 2.8 Inserting New Document into the Collection ............................................................ 21
Figure 2.9 Retrieving Data From MongoDB Database .............................................................. 21
Figure 2.10 The CAP Theorem of NoSQL ................................................................................ 25
Table of Contents
List of Figures ......................................................................................................................................... 1
Table of Contents .................................................................................................................................... 1
MODULE 1 ........................................................................................................................................... 4
GENERAL OVERVIEW OF DATABASES AND DATABASE MANAGEMENT SYSTEMS ........ 4

2|Page
 1.1 Recall on Databases, DBMS, and Relational Databases. ................................................................. 4
1.1.1 Database .................................................................................................................................... 4
1.1.2 Database Management Systems ............................................................................................... 5
1.1.3 Relational Databases ............................................................................................................... 5
[Link] Features of Relational Databases .......................................................................................... 5
1.1.4 Distributed Databases .............................................................................................................. 6
1.1.5 Database Models ..................................................................................................................... 6
1.2 Challenges and Disadvantages of Relational Database Management Systems. ............................. 7
1.3 JSON As A Database ..................................................................................................................... 7
1.3.1 What is JSON ? ....................................................................................................................... 7
1.3.2 JSON Characteristics............................................................................................................... 8
1.3.3 JSON Rules............................................................................................................................. 8
1.3.4 JSON Data - A Name and a Value ........................................................................................... 8
1.3.4 JSON Data – Data Separated By Commas ............................................................................... 8
1.3.4 JSON Data – Curly Braces Holding Objects ............................................................................ 8
1.3.4 JSON Data – Square Brackets Holding Arrays ........................................................................ 9
1.3.2 Creating JSON Objects ........................................................................................................... 9
MODULE 2 - INTRODUCTION TO NoSQL .................................................................................... 10
2.1 General Introduction to NoSQL ................................................................................................... 10
2.1.1 Introduction to NoSQL .......................................................................................................... 10
2.1.2 Introduction to Big Data ........................................................................................................ 11
[Link] Characteristics of Big Data ................................................................................................. 12
[Link] Types of Big Data .............................................................................................................. 12
2.2 Real-Life Examples of NoSQL .................................................................................................... 14
2.3 Differences Between NoSQL and RDBMS .................................................................................. 15
2.4 NoSQL Database Models ............................................................................................................. 16
2.4.1 Key-Value Model .................................................................................................................. 16
2.4.2 Column Store Model ............................................................................................................. 17
2.4.3 Document Store Model ......................................................................................................... 17
2.4.4 Graph-Based Model .............................................................................................................. 18

3|Page
 2.5 Representing NoSQL Databases................................................................................................... 19
2.5.0 Creating Database Clusters .................................................................................................... 19
2.5.1 MongoDB For Document Model ........................................................................................... 19
[Link] Creating a Database and Inserting Records into the MongoDB Database ............................ 20
[Link] Retrieving Records from the MongoDB Database ............................................................... 21
2.5.2 Cassandra For Column Model ............................................................................................... 22
2.5.3 Neo4j For Graph Model ........................................................................................................ 22
2.5.4 – Redis For Key-Value Model ............................................................................................... 23
2.6 Advantages of NoSQL Databases................................................................................................. 23
2.7 Disadvantages of NoSQL Databases ............................................................................................ 23
2.8 The CAP Theorem of NoSQL ..................................................................................................... 24
2.8.1 How HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem ................... 29
[Link] Understanding CP with MongoDB ..................................................................................... 29
[Link] Understanding AP with Cassandra ...................................................................................... 30
2.9 ACID vs CAP .............................................................................................................................. 31
2.9.1 Relationships and Differences between ACID and CAP ............................................................ 32
2.10 Hadoop Distributed File System (HDFS) ................................................................................... 32
2.11 NoSQL Practicals With Java and MongoDB .............................................................................. 35
MODULE 3 - INTRODUCTION TO CLOUDANT – DATABASE AS A SERVICE ...................... 36
3.1 General Introduction to Database As A Service (DBaaS).............................................................. 36
3.1.1 Introduction to NoSQL .......................................................................................................... 36
3.2 Advantages of DBaaS .................................................................................................................. 36
3.3 How To Choose A DBaaS ........................................................................................................... 36
3.4 Introduction To IBM Cloud Database .......................................................................................... 36
3.5 Benefits of IBM Cloudant, a NoSQL DBaaS ................................................................................ 36
3.6 Cloudant Architecture .................................................................................................................. 36
3.7 Cloudant Deployment .................................................................................................................. 36
MODULE 4 – PRACTICALS ON IBM CLOUDANT ....................................................................... 37
4.1 Creating An IBM Cloudant Account ............................................................................................ 37
4.1.1 Introduction to NoSQL .......................................................................................................... 37

4|Page
 4.2 Creating Databases, Collections and Adding Documents .............................................................. 37

TUTORIAL ......................................................................................................................................... 38

MODULE 1
GENERAL OVERVIEW OF DATABASES AND DATABASE MANAGEMENT
SYSTEMS
1.1 Recall on Databases, DBMS, and Relational Databases.
1.1.1 Database
The term database refers to a collection of related data from which the users can efficiently retrieve
the desired information. In addition to the storage and retrieval of data, certain other operations
can also be performed on a database. These operations include adding, updating and deleting data.
A database can be summarily described as a repository for data. It is a collection of nonredundant
data which can be shared by different application systems. Although databases are generally
computerized, instances of non-computerized (paper-based) databases from everyday life can
be cited in abundance. A dictionary, a phone book, a collection of recipes and a TV guide are all
common examples of non-computerized databases. The examples of computerized databases
include customer files, search engines, books catalogue, equipment inventories and sales
transactions.
[Link] Computerized database vs non-computerized database
Computerized database Paper-based database
Can hold vast amount of data Limited by the physical space available
Very fast to find a specific record Can take a while to manually search through all
the record

Can easily search for a specific criterion Difficult to search for a specific criterion; every
record would have to be manually look at
e.g. “All the student who live in Kotto”
Data can be sorted into ascending or Difficult to sort data on more than one criterion
descending order on multiple criteria

The database can be kept secure using The only security will be locking up the record
password

Can easily update or amend a record Changes must be done manually. Record can look
messy if scribbled out

5|Page
 Easy to make a back-up in case of data lost Difficult to make a back-up because every page
would have to be rewritten or photocopied

1.1.2 Database Management Systems

It is an interface between user and database. All these operations on a database are performed
using a database management system (DBMS). Essentially, a DBMS is a computerized record-
keeping system. A DBMS is a collection of programs that manages the database structure and
controls access to the data stored in the database. The DBMS serves as the intermediary between
the end user and the database by translating user requests into the complex computer code. The
end user interacts with the DBMS through an application program.
Some DBMS examples include IBM DB2, MySQL, PostgreSQL, Microsoft Access, SQL Server,
FileMaker, Oracle, RDBMS, dBASE, Clipper, FoxPro and MongoDB (an open source,
nonrelational database management system (DBMS) )
1.1.3 Relational Databases
The relational data model represents the database as a collection of simple two-dimensional tables
called relations. The rows of a relation are referred to as tuples and the columns are referred to as
attributes.

Figure 1.1 Relational Databases

[Link] Features of Relational Databases
1. ACID is a set of properties that guarantees that database transactions are processed reliably.
 Atomicity: Either all or none.
 Consistency: Data is always accurate.
 Isolation: No interfere with other.
 Durability: Commit is persistent.
2. Very rich set of query and functionality.

6|Page
1.1.4 Distributed Databases
A distributed database is a database that consists of two or more files located at different sites
on a computer network. Because the database is distributed, different users can access it without
interfering with one another in a transparent manner. Users access the data in a distributed
database by accessing the WAN, but the user has the impression that the access of data is done on
his local machine. To keep a distributed database up to date, it uses the replication and duplication
processes.

Figure 1.2 Distributed Databases

1.1.5 Database Models
A database model determines logical structure of data. There exist the following types of database
models:
 Hierarchical Model
 Network Model
 Relational Model
 ER Model
 Document Model
 Object Model

1.2 Challenges and Disadvantages of Relational Database Management Systems.

Relational databases seem to solve every problem until the problem to be solved becomes big and
complex.
Below are some of the challenges of a RDBMS:
 RDBMS uses centralized storage not distributed.
 ACID doesn't provide scalability for distributed system.

7|Page
  It is only for structured data.
 The failure of a server or storage device brings down the entire database.
 It needs the schema of your application and its properties (columns, types, etc.) to be
defined up-front before building the application. This does not match well with the agile
development approaches for highly dynamic applications.
 As the data starts to grow larger, you have to scale your database vertically, i.e. adding
more capacity to the existing servers.

Figure 1.3 Scaling a RDBMS to Accommodate more data

It is for some of the above-mentioned problems that it becomes necessary to have a Distributed
Database Management System (DDBMS) and this is where the concept of NoSQL and DBaaS
comes in.
1.3 JSON As A Database
1.3.1 What is JSON ?
JavaScript Object Notation (JSON) is a standard text-based format for representing structured
data based on JavaScript object syntax. It is commonly used for transmitting data in applications
(e.g., sending some data from the server to the client, so it can be displayed on a web page,
processed by a mobile app or software and vice versa). JSON is a text format for storing and
transporting data.
It is easy for humans to read and write and for machines to parse and generate.

8|Page
1.3.2 JSON Characteristics
 It is used while writing JavaScript based applications that includes browser extensions and
websites.
 JSON format is used for serializing and transmitting structured data over network
connection.
 It is primarily used to transmit data between a server and web applications.
 Web services and APIs use JSON format to provide public data.
 It can be used with modern programming languages.
1.3.3 JSON Rules
JSON syntax is derived from JavaScript object notation syntax:
 Data is in name/value pairs
 Data is separated by commas
 Curly braces hold objects
 Square brackets hold arrays
1.3.4 JSON Data - A Name and a Value
JSON data is written as name/value pairs (key/value pairs). A name/value pair consists of a field
name (in double quotes), followed by a colon, followed by a value. JSON names require double
quotes. name:"John"
1.3.4 JSON Data – Data Separated By Commas
Since JSON syntax is derived from JavaScript object notation, you can create an object and assign
data to it.
person = {name:"John", age:25, city:"Douala"};
object1 = { "language" : "Java", "author" : "herbert schildt" };
1.3.4 JSON Data – Curly Braces Holding Objects
It is an unordered set of name/value pairs. Objects are enclosed in curly braces that is, it starts with
'{' and ends with '}'. Each name is followed by ':'(colon) and the key/value pairs are separated by
, comma). The keys must be strings and should be different from each other. Objects should be
used when the key names are arbitrary strings.

{
"id": "2",
"name": "John",

9|Page
 "age": 25,
}
1.3.4 JSON Data – Square Brackets Holding Arrays
Below is an example showing array containing multiple objects –
{
"users": [
{ "name":"John" , "age":"25" },
{ "name":"Peter" , "age":"29" },
{ "name":"Paul" , "age":"31" }
]
}
1.3.2 Creating JSON Objects
JSON is commonly used to exchange data to/from a web server. Generally, when receiving data
from a web server, the data is always a string. Parse the data with [Link](), and the data
becomes a JavaScript object. When using the [Link]() on a JSON derived from an array, the
method will return a JavaScript array, instead of a JavaScript object. This array can now be used
in any programming language and application of your choice.
Generally, JSON is the format that API’s use to send requests and receive responses from servers.
In this course, the notions of JSON will be very important in that it will be necessary to perform
API calls.

10 | P a g e
 MODULE 2 - INTRODUCTION TO NoSQL
2.1 General Introduction to NoSQL
2.1.1 Introduction to NoSQL
Conventional relational databases were the product of a great deal of research and testing to
obtain optimal storage structures. However, their flaw is that they have a rigid structure which
cannot be altered if there are different specifications. To combat this problem, NoSQL databases
were created.
A NoSQL (non-SQL, Not Only SQL or non-relational) is a database that provides a mechanism
for the storage and retrieval of data that is modelled in means other than the tabular relations used
in relational databases.
It refers to a family of databases that vary widely in style and technology, but all share a common
trait in that they are non-relational in nature, meaning they are not a standard row and column
RDBMS. Therefore, a better name to describe these databases would be non-relational. NoSQL
databases provide new ways of storing and querying data that address several issues for modern
applications. Most importantly, most NoSQL databases are geared to handle a different breed of
scale problems that have arisen associated with the “big data” movement. By scale, we are
referring to both the size of the data and the concurrent users acting on that data. NoSQL databases
also are typically more specialized to various use cases and can be much simpler to develop
application functionality than relational databases.
Some examples of NoSQL databases include: MongoDB, CouchDB, CouchBase, Cassandra, HBase,
Redis, Riak, Neo4J.

Figure 2.1 Some Examples of NoSQL Databases

In the late 2000’s, several new databases emerged on the scene, a large number of them from the
open-source communities. Databases like Apache CouchDB, Cassandra, and HBase, as well as

11 | P a g e
Mongo and Riak became more prevalently used in applications, particularly in ones that required
larger scale than a relational database could handle.

Figure 2.2 Rise of NoSQL and DBaaS

Today’s web, mobile, and IoT (Internet of Things) applications use NoSQL to store and process their
data. Below are some of the reasons for doing this:
 Support large numbers of concurrent users (tens of thousands and perhaps millions)
 Deliver highly responsive experiences to a globally distributed base of users
 Server always available – no downtime
 Handle semi- and unstructured data
 Rapidly adapt to changing requirements with frequent updates and new features.
 More customers are going online
 Big data is getting bigger
 The world has gone mobile
 The internet is connecting everything
 Applications are moving to the cloud

2.1.2 Introduction to Big Data

Big Data is a collection of data that is huge in volume yet growing exponentially with time. It is
a data with so large size and complexity that none of traditional data management tools can store
it or process it efficiently. Big data is also a data but with huge size.
Usage Examples include:
 Twitter processing and storing more than 12GB of data daily.
 Facebook generates and processes over 4 petabytes of data daily.
 Google generates over 20 petabytes daily and this includes over 3.5 Billion queries daily.
 The New York Stock Exchange generates about one terabyte of new trade data per day.

12 | P a g e
[Link] Characteristics of Big Data
Big data can be described by the following characteristics:
 Volume
 Variety
 Velocity
 Variability
(i) Volume – The name Big Data itself is related to a size which is enormous. Size of data plays
a very crucial role in determining value out of data. Also, whether a particular data can actually
be considered as a Big Data or not, is dependent upon the volume of data. Hence, ‘Volume’
is one characteristic which needs to be considered while dealing with Big Data solutions.
(ii) Variety – The next aspect of Big Data is its variety.
Variety refers to heterogeneous sources and the nature of data, both structured and unstructured.
During earlier days, spreadsheets and databases were the only sources of data considered by most
of the applications. Nowadays, data in the form of emails, photos, videos, monitoring devices,
PDFs, audio, etc. are also being considered in the analysis applications. This variety of
unstructured data poses certain issues for storage, mining and analyzing data.
(iii) Velocity – The term ‘velocity’ refers to the speed of generation of data. How fast the data is
generated and processed to meet the demands, determines real potential in the data.
Big Data Velocity deals with the speed at which data flows in from sources like business processes,
application logs, networks, and social media sites, sensors, Mobile devices, etc. The flow of data
is massive and continuous.
(iv) Variability – This refers to the inconsistency which can be shown by the data at times, thus
hampering the process of being able to handle and manage the data effectively.
[Link] Types of Big Data
Following are the types of Big Data:
1. Structured
2. Unstructured
3. Semi-structured
a) Structured
Any data that can be stored, accessed and processed in the form of fixed format is termed as a
‘structured’ data. Over the period of time, talent in computer science has achieved greater success
in developing techniques for working with such kind of data (where the format is well known in
advance) and also deriving value out of it. However, nowadays, we are foreseeing issues when a

13 | P a g e
size of such data grows to a huge extent, typical sizes are being in the rage of multiple zettabytes.
One Billion Terabytes = 1 Zettabyte.
Looking at these figures one can easily understand why the name Big Data is given and imagine
the challenges involved in its storage and processing. Examples Of Structured Data
An ‘Employee’ table in a database is an example of Structured Data
Employee_ID Employee_Name Gender Department Salary
2365 Ashu Nyenti Male Finance 650000
3398 Ngono Remi Female Admin 650000
7465 Abdulaye Moussa Male Admin 500000
7500 Hillary Jones Male Finance 500000
7699 Seydou Sane Female Finance 550000

b) Unstructured
Any data with unknown form or the structure is classified as unstructured data. In addition to the
size being huge, un-structured data poses multiple challenges in terms of its processing for
deriving value out of it. A typical example of unstructured data is a heterogeneous data source
containing a combination of simple text files, images, videos etc. Nowadays organizations have
wealth of data available with them but unfortunately, they don’t know how to derive value out of
it since this data is in its raw form or unstructured format. Examples Of Un-structured Data
The output returned by ‘Google Search’

Figure 2.2 Example Of Un-structured Data

14 | P a g e
 c) Semi-structured
Semi-structured data can contain both the forms of data. We can see semi-structured data as
structured in form e.g. a table definition in relational DBMS. Example of semi-structured data is
a data represented in an XML file. Personal data stored in an XML file-
<rec><name>Ashu Nyenti</name><sex>Male</sex><age>35</age></rec>
<rec><name>Agbor Bissong.</name><sex>Female</sex><age>41</age></rec>
<rec><name>Abdulaye Moussa</name><sex>Male</sex><age>29</age></rec>
<rec><name>OumarouJones</name><sex>Male</sex><age>26</age></rec>
<rec><name>Jeremiah Nji</name><sex>Male</sex><age>35</age></rec>

2.2 Real-Life Examples of NoSQL

Global 2000 enterprises are rapidly embracing NoSQL databases to power their mission-critical applications:
 Tesco, Europe’s No. 1 retailer, deploys NoSQL for e-commerce, product catalog, and other applications
 Ryanair, the world’s busiest airline, uses NoSQL to power its mobile app serving over 3 million users.
 Marriott deploys NoSQL for its reservation system that books $38 billion annually
 Gannett, the No. 1 U.S. newspaper publisher, uses NoSQL for its proprietary content management
system, Presto
 GE deploys NoSQL for its Predix platform to help manage the Industrial Internet

2.3 Differences Between NoSQL and RDBMS next class

1. RDBMS is called relational databases while NoSQL is called a distributed database. They
do not have any relations between any of the databases. When RDBMS uses structured
data to identify the primary key, there is a proper method in NoSQL to use unstructured
data.
2. RDBMS is scalable vertically and NoSQL is scalable horizontally. Hence in RDBMS,
servers have to be added and power has to be increased. This makes scalability an RDBMS
expensive. While in NoSQL, we just need to add more machines and this does not make
the database expensive.
3. Maintenance of RDBMS is expensive as manpower is needed to manage the servers added
in the database. NoSQL is mostly automatic and does some repairs on its own. Data
distribution and administration is less in NoSQL.

15 | P a g e
 4. As stated earlier, RDBMS is expensive due to the servers and storage management. NoSQL
databases are open source and cheap when compared with RDBMS. Servers in NoSQL
can be used to manage and transact data with relatively no cost at all.
5. RDBMS has a fixed schema so that data has to be inserted in a proper format. This helps
to get the primary key and foreign key to align the data in the table. Primary keys help to
connect the data from other tables with the common identifier. But in NoSQL, a schema is
not needed and hence data can be inserted in any form. Also, these data can be inserted at
any time and updates can be made without any prior notice. This helps the database to work
flexibly with the users.
6. Data output performance is better in NoSQL due to the support caching in system memory.
Separate infrastructure is not needed and hence this makes the developers tally the data
easily. Whereas in RDBMS, separate infrastructure is needed due to the absence of support
caching. This makes the data tally difficult.
7. NoSQL data structure is never equal due to the absence of schema and the fact that it is
open source. This makes the database be without a proper standard. RDBMS has a proper
schema and data has to be inserted only in the given format. The data structure is equal
from one table to another.
8. RDBMS has stored procedures to understand the data and to know them well. NoSQL,
however, does not have any stored procedure. This makes it difficult for users to identify
the pattern and to learn the data well.
9. Transactions in a RDBMS are written in one location while in a NoSQL, transactions
written in several locations.
10. In a relational database, data arrives from one or few locations whereas in a NoSQL, data
arrives from many locations.

2.4 NoSQL Database Models

Over time, four major types of NoSQL databases emerged: document databases, key-value databases,
wide-column stores, and graph databases.
2.4.1 Key-Value Model
Key–value (KV) stores use the associative array (also called a map or dictionary) as their
fundamental data model. In this model, data is represented as a collection of key–value pairs, such
that each possible key appears at most once in the collection. These databases are organised as
key value pairs, where each key appears exactly once. The keys are usually arranged in a sorted
fashion. Examples of this type of databases are: Aerospike, ArangoDB, Couchbase, Dynamo etc.

16 | P a g e
 Figure 2.3 Key-Value NoSQL Model

2.4.2 Column Store Model

Data is stored in a columnar form. Some examples of this type of database are: Accumulo,
Cassandra, Druid, Vertica etc. It avoids consuming space when storing nulls by simply not storing
a column when a value doesn’t exist for that column. Each unit of data can be thought of as a set
of key/value pairs, where the unit itself is identified with the help of a primary identifier, often
referred to as the primary key. Bigtable and its clones tend to call this primary key the row-key.
This model stores data in tables, and dynamic columns rather than in rows which is more usual
with SQL.

Figure 2.4 Column Store NoSQL Model

17 | P a g e
2.4.3 Document Store Model
With SQL, XML and JSON tend to be tied together, which slows queries down and hampers the
whole process. Since NoSQL doesn’t use the relational model, it doesn’t need to do that, which is
where document-based stores come in. It stores data in documents similar to JSON objects. Each
document contains pairs of fields and values. The values can typically be a variety of types
including things like strings, numbers, booleans, arrays, or objects. Examples of Document based
databases are: MongoDB, HBase, Cassandra, Amazon SimpleDB, Hypertable, etc.
All data is stored in one table, so there’s no need for cross-referencing and instead of storing
information in a table, it’s stored in a document. While this is very similar to a key-value store,
and can sometimes be considered under its umbrella, the difference is that document-based
NoSQL generally has some form of encoding, such as XML.

Figure 2.5 Document NoSQL Model

2.4.4 Graph-Based Model

Graph or network data models essentially treat the relationship between any two pieces of
information as being just as important as the information itself. As such, this type of data model is

18 | P a g e
really made for any information that you’d usually represent on a graph. It uses relationships and
nodes, with the data being the information itself, and the relationship is formed between nodes. It
stores data in nodes and edges. Nodes typically store information about people, places, and things,
while edges store information about the relationships between the nodes. Examples of data include
social relations, public transport links, road maps, network topologies, etc. The following
databases use this model: Neo4j, OrientDB, Facebook Open Graph, FlockDB, etc.

Figure 2.6 Graph-Based NoSQL Model

2.5 Representing NoSQL Databases

2.5.0 Creating Database Clusters
A cluster is a place where you can store your MongoDB databases.

2.5.1 MongoDB For Document Model

MongoDB is a database based on a non-relational document model. The name MongoDB is
derived from the English word “humongous”, which roughly means “gigantic”. MongoDB stores
its data records in structures known as documents. Mongo allows you to group multiple documents
into a structure known as a collection, which can be further grouped into separate databases. A
document is written in BSON, a binary representation of JSON. Like objects in JSON, MongoDB
documents begin and end with curly brackets ({ and }), and contain a number of field-and-value
pairs which typically take the form of field: value. A field’s value can be any one of the data types
used in BSON, or even other structures like documents and arrays.

19 | P a g e
[Link] How To Download And Test MongoDB On Windows
 Go to the page [Link]
 Click on the green “Try Free” button at the top right and create your account.
 On the following page, select “On-premises” and then “MongoDB Community Server“.
 The current database version is now displayed on the right-hand side of the screen and you can click on
the green download button with the preset values
 Create a Shared Cluster
 After setting up, Go to Databases
 Click on the cluster you created and click on CONNECT.
 Select “Connect using MongoDB Compass”
 Copy your connection string and paste in MongoDB Compass
[Link] Creating a Database and Inserting Records into the MongoDB Database
 Open the cluster and click on COLLECTIONS  Click on “Create
Database”

20 | P a g e
 Figure 2.7 Creating a MongoDB Database
 To insert data, select the database, select the collection and click on “Insert
Document”

Figure 2.8 Inserting New Document into the Collection

[Link] Retrieving Records from the MongoDB Database

To query the database, pass, the name-pair values inside {} and click on “Apply”

21 | P a g e
Figure 2.9 Retrieving Data From MongoDB Database
Exercise: Using MongoDB, create a database and collections, insert documents into the collections.
2.5.2 Cassandra For Column Model
Cassandra database is distributed over several machines that operate together. The outermost
container is known as the Cluster. For failure handling, every node contains a replica, and in case
of a failure, the replica takes charge. Cassandra arranges the nodes in a cluster, in a ring format,
and assigns data to them.
Apache Cassandra is an example of NoSQL Database. It is a distributed, decentralized and an
open-source database or a storage system. It is basically used for managing very large amounts of
structured data. There is no single point of failure, providing highly available services. It has the
following features:
a. Decentralized: Every node in the cluster is identical. There are no single points of failure.
b. Fault Tolerance: Since data is replicated to multiple nodes, fault tolerance is pretty high. Also, any
failed nodes are related with no downtime.
c. Performance: The fundamental architecture of Apache Cassandra is very robust. Therefore, it proves
to be better than the alternatives.
d. Scalable: It is linearly scalable. In other words, the throughput is increased as you increase the number
of nodes in the Cassandra cluster.
e. Proven: Apache Cassandra is used at various companies. Some of them are Netflix, GoDaddy, GitHub,
eBay etc.
These features of Apache Cassandra shows that it is actually very powerful and reliable.

2.5.3 Neo4j For Graph Model

Neo4j facilitates personal data storage and management: it allows you to track where private
information is stored and which systems, applications, and users access it. The graph data model
helps visualize personal data and allows for data analysis and pattern detection.

22 | P a g e
Neo4j is a graph database. A graph database, instead of having rows and columns has nodes edges
and properties. It is more suitable for certain big data and analytics applications than row and
column databases or free-form JSON document databases for many use cases. A graph database
is used to represent relationships. You can download Neo4j Windows from
[Link]

2.5.4 – Redis For Key-Value Model

Redis is an in-memory data structure store, used as a distributed, in-memory key–value database,
cache and message broker, with optional durability. Redis supports different kinds of abstract data
structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and
spatial indices. Currently, it is being used by tech-giants like GitHub, Weibo, Pinterest, Snapchat,
Craigslist, Digg, StackOverflow, Flickr. It can be downloaded from [Link]

2.6 Advantages of NoSQL Databases

(i) Flexible Data Model:
NoSQL databases are highly flexible as they can store and combine any type of data, both structured and
unstructured, unlike relational databases that can store data in a structured way only.

(ii) Evolving Data Model :

NoSQL databases allow you to dynamically update the schema to evolve with changing requirements while
ensuring that it would cause no interruption or downtime to your application.

(iii) Elastic Scalability:

NoSQL databases can scale to accommodate any type of data growth while maintaining low cost.

(iv) High Performance:

NoSQL databases are built for great performance, measured in terms of both throughput (it is a measure of
overall performance) and latency (it is the delay between request and actual response).

(v) Open source:

NoSQL databases don’t require expensive licensing fees and can run on inexpensive hardware, rendering
their deployment cost-effective.

2.7 Disadvantages of NoSQL Databases

(i) Lack of Standardization:

23 | P a g e
There is no standard that defines rules and roles of NoSQL databases. The design and query
languages of NoSQL databases vary widely between different NoSQL products – much more
widely than they do among traditional SQL databases.

(ii) Backup of Database:

Backups are a drawback in NoSQL databases. Though some NoSQL databases like MongoDB
provide some tools for backup, these tools are not mature enough to ensure proper complete data
backup solution.

(iii) Consistency:
NoSQL puts a scalability and performance first but when it comes to a consistency of the data
NoSQL doesn’t take much consideration so it makes it little insecure as compared to the relational
database e.g., in NoSQL databases if you enter same set of data again, it will take it without issuing
any error whereas relational databases ensure that no duplicate rows get entry in databases.

2.8 The CAP Theorem of NoSQL

It is very important to understand the limitations of NoSQL database. NoSQL cannot provide
consistency and high availability together.
Why CAP theorem is important?
After the internet boom in 2005, the size of data is growing exponentially day by day. At the early
stages to maintain the ever-changing scale of data and plan the capacity properly the only option
was to increase the capacity vertically which means adding more machines or increasing the
machine capabilities. But is not always feasible and cost-effective. Instead of this, the new concept
is to add the capacity horizontally which means leveraging distributed computing. To standardize
the network, we need to maintain the principles of the CAP theorem.

The CAP theorem (also called Brewer's theorem) states that a distributed database system can only
guarantee two out of these three characteristics: Consistency, Availability, and Partition
Tolerance.

24 | P a g e
 Figure 2.10 The CAP Theorem of NoSQL

a) Consistency
Consistency means that all nodes in the network see the same data at the same time.
Example:
Let’s take an example of MTN Cameroon whereby your ID card has expired, and you want to
update your new ID card number registered with them. You decide to call the customer care
operator and update it with them. When you call, you connect with an operator. This operator
makes the relevant changes in the system. But once you have dropped the phone, you realize you
gave them a wrong ID card number.
So you call the customer care again. This time, you connect with a different customer care operator
and they are able to access your records as well and know that you have recently updated your ID
card number. They make the relevant changes in the ID card number and the rest of your
information is the same as the one you told the last operator.
We call this as Consistency because even though you connect to a different customer care
operator, they were able to retrieve the same information.

25 | P a g e
b) Availability
Availability is a guarantee that every request receives a response about whether it was successful
or failed. However, it does not guarantee that a read request returns the most recent write. The
greater number of users a system can cater to better is the availability.
Example:

Let’s take an example or MTN Cameroon customer service and you are its customer because of
the amazing plans it offers. Besides that, they also provide an amazing customer care service
where you can call anytime and get your queries and concerns answered quickly and efficiently.

26 | P a g e
Whenever a customer calls them, the mobile operator can connect them to one of their customer
care operators.
The customer can evoke any information required by her/him about his accounts like balance,
usage, or other information. We call this Availability because every customer is able to connect
to the operator and get his/her information.

c) Partition Tolerance
Partition Tolerance is a guarantee that the system continues to operate despite arbitrary message
loss or failure of part of the system. In other words, even if there is a network outage in the data
center and some of the computers are unreachable, still the system continues to perform. Out of
these three guarantees, no system can provide more than 2 guarantees. Since in the case of a
distributed systems, the partitioning of the network is must, the tradeoff is always between
consistency and availability.

Example:

Recently MTN keeps sending you messages to update your ID card despite calling them to update
your ID card. So you decide to call the customer care once again. On connecting with the operator
this time, they tell you that they have not been able to update their records due to some issues. So

27 | P a g e
the information lying with the operator might not be up to date, therefore they cannot update the
information with the ID card. We can say here that the service is broken or there is no Partition
tolerance.

As depicted on figure 2.10 above, RDBMS can provide only consistency but not partition
tolerance. While HBASE and Redis can provide Consistency and Partition tolerance. And
MongoDB, CouchDB, Cassandra and Dynamo guarantee only availability but no consistency.
Such databases generally settle down for eventual consistency meaning that after a while the
system is going to be ok.
System designers must take into consideration the CAP theorem while designing or choosing
distributed storages as one needs to be sacrificed from C and A for others.
The CAP theorem categorizes systems into three categories:
CP (Consistent and Partition Tolerant) database: A CP database delivers consistency and
partition tolerance at the expense of availability. When a partition occurs between any two nodes,
the system has to shut down the non-consistent node (i.e., make it unavailable) until the partition
is resolved.
Partition refers to a communication break between nodes within a distributed system. Meaning,
if a node cannot receive any messages from another node in the system, there is a partition
between the two nodes. Partition could have been because of network failure, server crash, or any
other reason.
AP (Available and Partition Tolerant) database: An AP database delivers availability and
partition tolerance at the expense of consistency. When a partition occurs, all nodes remain
available but those at the wrong end of a partition might return an older version of data than others.
When the partition is resolved, the AP databases typically resync the nodes to repair all
inconsistencies in the system.
CA (Consistent and Available) database: A CA delivers consistency and availability in the
absence of any network partition. Often a single node’s DB servers are categorized as CA systems.
Single node DB servers do not need to deal with partition tolerance and are thus considered CA
systems.
In any networked shared-data systems or distributed systems partition tolerance is a must. Network
partitions and dropped messages are a fact of life and must be handled appropriately.
Consequently, system designers must choose between consistency and availability.

28 | P a g e
2.8.1 How HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem
[Link] Understanding CP with MongoDB
Let’s try to understand how a distributed system would work when it decides to give up on
Availability during a partition with the help of MongoDB.

MongoDB is a NoSQL database that stores data in one or more Primary nodes in the form of JSON
files. Each Primary node has multiple replica sets that update themselves asynchronously using
the operation log file of their respective primary node. The replica set nodes in the system send a
heartbeat (ping) to every other node to keep track if other replicas or primary nodes are alive or
dead. If no heartbeat is received within 10 seconds, then that node is marked as inaccessible.
If a Primary node becomes inaccessible, then one of the secondary nodes needs to become the
primary node. Till a new primary is elected from amongst the secondary nodes, the system remains
unavailable to the user to make any new write query. Therefore, the MongoDB system behaves as
a Consistent system and compromises on Availability during a partition.

[Link] Understanding AP with Cassandra

Now let’s also look at how a system compromises on Consistency. For this, we will look at the
Cassandra database which is called a highly available database.
Cassandra is a peer-to-peer system. It consists of multiple nodes in the system. And each node can
accept a read or write request from the user. Cassandra maintains multiple replicas of data in
separate nodes. This gives it a masterless node architecture where there are multiple points of
failure instead of a single point.
The replication factor determines the number of replicas of data. If the replication factor is 3, then
we will replicate the data in three nodes in a clockwise manner.

29 | P a g e
 A situation can occur where a partition
occurs and the replica does not get an
updated copy of the data. In such a
situation the replica nodes will still be
available to the user but the data will be
inconsistent. However, Cassandra also
provides eventual consistency. Meaning,
all updates will reach all the replicas
eventually. But in the meantime, it allows
divergent versions of the same data to exist
temporarily. Until we
update them to the consistent state.
Therefore, by allowing nodes to be available throughout and allowing temporarily inconsistent
data to existing in the system, Cassandra is an AP database that compromises on consistency.

2.9 ACID vs CAP

ACID describes a set of properties which guarantee a database transaction is reliable. It ensures data
integrity. Let’s take a look at the ACID properties.
1. Atomicity : All changes to data are performed as a single operation. That is, all or none,
either all of the operations are performed or one of them is performed. For example, in an
application we are transferring funds from one account to another, the atomicity property ensures
that, if a debit is made successfully from one account, then the associated credit is also done to the
other account.

2. Consistency: For each transaction, the system should move from one consistent state to
another consistent state.

3. Isolation: All transactions should be executed in isolation from other transactions. During
concurrent transaction execution, intermediate transaction results from parallel executed
transactions should be mutually exclusive. Failure of one module should not affect another
transaction

4. Durability: After every successful transaction, the changes made in the database should
persist. Even if the system comprises or failed somehow, still the successfully committed or

30 | P a g e
 aborted operations should persist. Now we can see that, these terms technically refer to different
things. The way in which they are related is that a distributed database system that guarantees the
ACID transactions must choose consistency over availability according to the CAP Theorem (i.e
it is a CP system).
2.9.1 Relationships and Differences between ACID and CAP
The CAP consistency applies to a distributed system, whereas ACID abstracts itself from that level
of the system architecture. CAP consistency may be necessary for the implementation of ACID
atomicity, but it's not the same thing.

ACID Consistency is about database rules , If the data table structure defines that a field value is
unique , Then the consistency system will solve the problem that causes the value of this field to be
non unique in all operations , If a row of records with a foreign key is deleted , Then its foreign key
related records should also be deleted , This is it. ACID Consistency means .

CAP theorem: specifies that a distributed system can provide two services (ex. Availability and
Partition tolerance) but never three. If for example, a service provides Availability and Partitioning
it can never ensure Consistency, not immediately, thus Eventual Consistency is used, which allows
the infrastructure to flux between inconsistency and consistency, however at one point, sooner or
later, the infrastructure will become consistent, resulting in eventual consistency. Cloud services
work in such fashion and Amazon's Simple DB uses eventual consistency. CAP addresses cluster
wide data consistency

ACID features are usually applied to relational DBs. If you want to apply ACID in a distributed
fashion (distributed DB), ACID uses 2PC(two-phase commit) to force consistency across partitions.
However since ACID provides consistency and partitioning, applying the CAP theorem for
(distributed environments) this will mean that availability is compromised. ACID addresses an
individual node's data consistency

2.10 Hadoop Distributed File System (HDFS)

With growing data velocity the data size easily outgrows the storage limit of a machine. A solution
would be to store the data across a network of machines. Such filesystems are called distributed
filesystems. Since data is stored across a network all the complications of a network come
in.

This is where Hadoop comes in. It provides one of the most reliable filesystems.

31 | P a g e
Hadoop is an open-source distributed processing framework that manages data processing and
storage for big data applications.
HDFS (Hadoop Distributed File System) is a unique design that provides storage for extremely
large files with streaming data access pattern and it runs on commodity hardware. Let’s elaborate
the terms:
 Extremely large files: Here we are talking about the data in range of petabytes(1000 TB).
 Streaming Data Access Pattern: HDFS is designed on principle of write-once and read
many-times. Once data is written large portions of dataset can be processed any number
times.
 Commodity hardware: Hardware that is inexpensive and easily available in the market.
This is one of feature which specially distinguishes HDFS from other file system.
Nodes: Master-slave nodes typically forms the HDFS cluster.
1. Master Node:
 Manages all the slave nodes and assign work to them.
 It executes filesystem namespace operations like opening, closing, renaming files and
directories.
 It should be deployed on reliable hardware which has the high config. not on commodity
hardware.
2. Name Node:
 Actual worker nodes, who do the actual work like reading, writing, processing etc.
 They also perform creation, deletion, and replication upon instruction from the master.
 They can be deployed on commodity hardware.
HDFS daemons: Daemons are the processes running in background.
• Name nodes:
 Run on the master node.
 Store metadata (data about data) like file path, the number of blocks, block Ids. etc. 
Require high amount of RAM.
 Store meta-data in RAM for fast retrieval i.e to reduce seek time. Though a persistent
copy of it is kept on disk.
• Data Nodes:
 Run on slave nodes.
 Require high memory as data is actually stored here.
Data storage in HDFS: Now let’s see how the data is stored in a distributed manner.

32 | P a g e
Let’s assume that 100TB file is inserted, then master node (name node) will first divide the file
into blocks of 10TB (default size is 128 MB in Hadoop 2.x and above). Then these blocks are
stored across different data nodes (slave node). Data nodes (slave node) replicate the blocks
among themselves and the information of what blocks they contain is sent to the master. Default
replication factor is 3 means for each block 3 replicas are created (including itself). In [Link]
we can increase or decrease the replication factor i.e we can edit its configuration here.
Note: Master Node has the record of everything, it knows the location and info of each and every
single data nodes and the blocks they contain, i.e. nothing is done without the permission of master
node.
Why divide the file into blocks?
Answer: Let’s assume that we don’t divide, now it’s very difficult to store a 100 TB file on a
single machine. Even if we store, then each read and write operation on that whole file is going to
take very high seek time. But if we have multiple blocks of size 128MB then its become easy to
perform various read and write operations on it compared to doing it on a whole file at once.
So we divide the file to have faster data access i.e. reduce seek time.

Why replicate the blocks in data nodes while storing?

Answer: Let’s assume we don’t replicate and only one yellow block is present on data node D1.
Now if the data node D1 crashes we will lose the block and which will make the overall data
inconsistent and faulty. So we replicate the blocks to achieve fault-tolerance.
Terms related to HDFS:
 Heartbeat : It is the signal that data node continuously sends to name node. If name node
doesn’t receive heartbeat from a data node then it will consider it dead.

33 | P a g e
  Balancing : If a data node is crashed the blocks present on it will be gone too and the
blocks will be under-replicated compared to the remaining blocks. Here master node(name
node) will give a signal to data nodes containing replicas of those lost blocks to replicate
so that overall distribution of blocks is balanced.
 Replication: It is done by data node.
Note: No two replicas of the same block are present on the same data node.
Features:
 Distributed data storage.
 Blocks reduce seek time.
 The data is highly available as the same block is present at multiple data nodes.
 Even if multiple data nodes are down, we can still do our work, thus making it highly
reliable.
 High fault tolerance.
Limitations: Though HDFS provide many features there are some areas where it doesn’t work
well.
 Low latency data access: Applications that require low-latency access to data i.e in the
range of milliseconds will not work well with HDFS, because HDFS is designed keeping
in mind that we need high-throughput of data even at the cost of latency.
 Small file problem: Having lots of small files will result in lots of seeks and lots of
movement from one data node to another data node to retrieve each small file, this whole
process is a very inefficient data access pattern.

2.11 NoSQL Practicals With Java and MongoDB

Using Java as programming language and MongoDB as database, create a CRUD application that registers,
retrieves, update and delete list of students.

MODULE 3 - INTRODUCTION TO CLOUDANT – DATABASE AS A SERVICE

3.1 General Introduction to Database As A Service (DBaaS)
3.1.1 Introduction to NoSQL
3.2 Advantages of DBaaS
3.3 How To Choose A DBaaS
3.4 Introduction To IBM Cloud Database
3.5 Benefits of IBM Cloudant, a NoSQL DBaaS
3.6 Cloudant Architecture

34 | P a g e
 3.7 Cloudant Deployment

MODULE 4 – PRACTICALS ON IBM CLOUDANT

4.1 Creating An IBM Cloudant Account
4.1.1 Introduction to NoSQL
4.2 Creating Databases, Collections and Adding Documents

35 | P a g e