INTRODUCTION TO NO S

SOFTWARE ENGINEERING

Fabrice FECKOUNDO TCHINDA MTECH SOFTWARE ENGINEER

1|Page
Course Objectives:
Upon successful completion of this course, you should be able to:
- Students should be able to distinguish the various NoSQL systems and their features.
- Understand the impact of the cluster on database design
- State the CAP theorem and explain it main points.
- Explain where HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem
- Work with the Hadoop Distributed File System (HDFS) as a foundation for NoSQL
technologies
- Warehouse HDFS data using Apache Hive
- Data mine HDFS data with Apache Spark-SQL and Apache Pig
- Describe the design of HBase, MongoDB, Cassandra, Neo4j, and Redis
- Use the data control, definition, and manipulation languages of the NoSQL databases
covered in the course.
- Build projects that use NoSQL databases using the Java Programming Language.
- Compare NoSQL databases with each other and relational systems
List of Figures
Figure 1.1 Relational Databases ..................................................................................................5
Figure 1.2 Distributed Databases .................................................................................................6
Figure 1.3 Scaling a RDBMS to Accommodate more data .........................................................7
Figure 2.4 Column Store NoSQL Model ................................................................................... 17
Figure 2.5 Document NoSQL Model ......................................................................................... 18
Figure 2.6 Graph-Based NoSQL Model .................................................................................... 19
Figure 2.7 Creating a MongoDB Database ................................................................................ 20
Figure 2.8 Inserting New Document into the Collection ............................................................ 21
Figure 2.9 Retrieving Data From MongoDB Database .............................................................. 21
Figure 2.10 The CAP Theorem of NoSQL ................................................................................ 25
Table of Contents
List of Figures ......................................................................................................................................... 1
Table of Contents .................................................................................................................................... 1
MODULE 1 ........................................................................................................................................... 4
GENERAL OVERVIEW OF DATABASES AND DATABASE MANAGEMENT SYSTEMS ........ 4

2|Page
 1.1 Recall on Databases, DBMS, and Relational Databases. ................................................................. 4
1.1.1 Database .................................................................................................................................... 4
1.1.2 Database Management Systems ............................................................................................... 5
1.1.3 Relational Databases ............................................................................................................... 5
[Link] Features of Relational Databases .......................................................................................... 5
1.1.4 Distributed Databases .............................................................................................................. 6
1.1.5 Database Models ..................................................................................................................... 6
1.2 Challenges and Disadvantages of Relational Database Management Systems. ............................. 7
1.3 JSON As A Database ..................................................................................................................... 7
1.3.1 What is JSON ? ....................................................................................................................... 7
1.3.2 JSON Characteristics............................................................................................................... 8
1.3.3 JSON Rules............................................................................................................................. 8
1.3.4 JSON Data - A Name and a Value ........................................................................................... 8
1.3.4 JSON Data – Data Separated By Commas ............................................................................... 8
1.3.4 JSON Data – Curly Braces Holding Objects ............................................................................ 8
1.3.4 JSON Data – Square Brackets Holding Arrays ........................................................................ 9
1.3.2 Creating JSON Objects ........................................................................................................... 9
MODULE 2 - INTRODUCTION TO NoSQL .................................................................................... 10
2.1 General Introduction to NoSQL ................................................................................................... 10
2.1.1 Introduction to NoSQL .......................................................................................................... 10
2.1.2 Introduction to Big Data ........................................................................................................ 11
[Link] Characteristics of Big Data ................................................................................................. 12
[Link] Types of Big Data .............................................................................................................. 12
2.2 Real-Life Examples of NoSQL .................................................................................................... 14
2.3 Differences Between NoSQL and RDBMS .................................................................................. 15
2.4 NoSQL Database Models ............................................................................................................. 16
2.4.1 Key-Value Model .................................................................................................................. 16
2.4.2 Column Store Model ............................................................................................................. 17
2.4.3 Document Store Model ......................................................................................................... 17
2.4.4 Graph-Based Model .............................................................................................................. 18

3|Page
 2.5 Representing NoSQL Databases................................................................................................... 19
2.5.0 Creating Database Clusters .................................................................................................... 19
2.5.1 MongoDB For Document Model ........................................................................................... 19
[Link] Creating a Database and Inserting Records into the MongoDB Database ............................ 20
[Link] Retrieving Records from the MongoDB Database ............................................................... 21
2.5.2 Cassandra For Column Model ............................................................................................... 22
2.5.3 Neo4j For Graph Model ........................................................................................................ 22
2.5.4 – Redis For Key-Value Model ............................................................................................... 23
2.6 Advantages of NoSQL Databases................................................................................................. 23
2.7 Disadvantages of NoSQL Databases ............................................................................................ 23
2.8 The CAP Theorem of NoSQL ..................................................................................................... 24
2.8.1 How HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem ................... 29
[Link] Understanding CP with MongoDB ..................................................................................... 29
[Link] Understanding AP with Cassandra ...................................................................................... 30
2.9 ACID vs CAP .............................................................................................................................. 31
2.9.1 Relationships and Differences between ACID and CAP ............................................................ 32
2.10 Hadoop Distributed File System (HDFS) ................................................................................... 32
2.11 NoSQL Practicals With Java and MongoDB .............................................................................. 35
MODULE 3 - INTRODUCTION TO CLOUDANT – DATABASE AS A SERVICE ...................... 36
3.1 General Introduction to Database As A Service (DBaaS).............................................................. 36
3.1.1 Introduction to NoSQL .......................................................................................................... 36
3.2 Advantages of DBaaS .................................................................................................................. 36
3.3 How To Choose A DBaaS ........................................................................................................... 36
3.4 Introduction To IBM Cloud Database .......................................................................................... 36
3.5 Benefits of IBM Cloudant, a NoSQL DBaaS ................................................................................ 36
3.6 Cloudant Architecture .................................................................................................................. 36
3.7 Cloudant Deployment .................................................................................................................. 36
MODULE 4 – PRACTICALS ON IBM CLOUDANT ....................................................................... 37
4.1 Creating An IBM Cloudant Account ............................................................................................ 37
4.1.1 Introduction to NoSQL .......................................................................................................... 37

4|Page
 4.2 Creating Databases, Collections and Adding Documents .............................................................. 37

TUTORIAL ......................................................................................................................................... 38

MODULE 1
GENERAL OVERVIEW OF DATABASES AND DATABASE MANAGEMENT
SYSTEMS
1.1 Recall on Databases, DBMS, and Relational Databases.
1.1.1 Database
The term database refers to a collection of related data from which the users can efficiently
retrieve the desired information. In addition to the storage and retrieval of data, certain other
operations can also be performed on a database. These operations include adding, updating and
deleting data.
A database can be summarily described as a repository for data. It is a collection of
nonredundant data which can be shared by different application systems. Although databases are
generally computerized, instances of non-computerized (paper-based) databases from
everyday life can be cited in abundance. A dictionary, a phone book, a collection of recipes and a
TV guide are all common examples of non-computerized databases. The examples of
computerized databases include customer files, search engines, books catalogue, equipment
inventories and sales transactions.
[Link] Computerized database vs non-computerized database
Computerized database Paper-based database
Can hold vast amount of data Limited by the physical space available
Very fast to find a specific record Can take a while to manually search through all
the record

Can easily search for a specific criterion Difficult to search for a specific criterion; every
record would have to be manually look at
e.g. “All the student who live in Kotto”
Data can be sorted into ascending or Difficult to sort data on more than one criterion
descending order on multiple criteria

The database can be kept secure using The only security will be locking up the record
password

Can easily update or amend a record Changes must be done manually. Record can
look messy if scribbled out

5|Page
 Easy to make a back-up in case of data lost Difficult to make a back-up because every page
would have to be rewritten or photocopied

1.1.2 Database Management Systems

It is an interface between user and database. All these operations on a database are performed
using a database management system (DBMS). Essentially, a DBMS is a computerized record-
keeping system. A DBMS is a collection of programs that manages the database structure and
controls access to the data stored in the database. The DBMS serves as the intermediary
between the end user and the database by translating user requests into the complex computer
code. The end user interacts with the DBMS through an application program.
Some DBMS examples include IBM DB2, MySQL, PostgreSQL, Microsoft Access, SQL Server,
FileMaker, Oracle, RDBMS, dBASE, Clipper, FoxPro and MongoDB (an open source,
nonrelational database management system (DBMS) )
1.1.3 Relational Databases
The relational data model represents the database as a collection of simple two-dimensional
tables called relations. The rows of a relation are referred to as tuples and the columns are
referred to as attributes.

Figure 1.1 Relational Databases

[Link] Features of Relational Databases
1. ACID is a set of properties that guarantees that database transactions are processed reliably.
➢ Atomicity: Either all or none.
➢ Consistency: Data is always accurate.
➢ Isolation: No interfere with other.
➢ Durability: Commit is persistent.
2. Very rich set of query and functionality.

6|Page
1.1.4 Distributed Databases
A distributed database is a database that consists of two or more files located at different sites
on a computer network. Because the database is distributed, different users can access it without
interfering with one another in a transparent manner. Users access the data in a distributed
database by accessing the WAN, but the user has the impression that the access of data is done
on his local machine. To keep a distributed database up to date, it uses the replication and
duplication processes.

Figure 1.2 Distributed Databases

1.1.5 Database Models
A database model determines logical structure of data. There exist the following types of
database models:
➢ Hierarchical Model
➢ Network Model
➢ Relational Model
➢ ER Model
➢ Document Model
➢ Object Model

1.2 Challenges and Disadvantages of Relational Database Management Systems.

Relational databases seem to solve every problem until the problem to be solved becomes big
and complex.
Below are some of the challenges of a RDBMS:
➢ RDBMS uses centralized storage not distributed.
➢ ACID doesn't provide scalability for distributed system.

7|Page
 ➢ It is only for structured data.
➢ The failure of a server or storage device brings down the entire database.
➢ It needs the schema of your application and its properties (columns, types, etc.) to be
defined up-front before building the application. This does not match well with the agile
development approaches for highly dynamic applications.
➢ As the data starts to grow larger, you have to scale your database vertically, i.e. adding
more capacity to the existing servers.

Figure 1.3 Scaling a RDBMS to Accommodate more data

It is for some of the above-mentioned problems that it becomes necessary to have a Distributed
Database Management System (DDBMS) and this is where the concept of NoSQL and DBaaS
comes in.
1.3 JSON As A Database
1.3.1 What is JSON ?
JavaScript Object Notation (JSON) is a standard text-based format for representing structured
data based on JavaScript object syntax. It is commonly used for transmitting data in applications
(e.g., sending some data from the server to the client, so it can be displayed on a web page,
processed by a mobile app or software and vice versa). JSON is a text format for storing and
transporting data.
It is easy for humans to read and write and for machines to parse and generate.

8|Page
1.3.2 JSON Characteristics
➢ It is used while writing JavaScript based applications that includes browser extensions
and websites.
➢ JSON format is used for serializing and transmitting structured data over network
connection.
➢ It is primarily used to transmit data between a server and web applications.
➢ Web services and APIs use JSON format to provide public data.
➢ It can be used with modern programming languages.
1.3.3 JSON Rules
JSON syntax is derived from JavaScript object notation syntax:
➢ Data is in name/value pairs
➢ Data is separated by commas
➢ Curly braces hold objects
➢ Square brackets hold arrays
1.3.4 JSON Data - A Name and a Value
JSON data is written as name/value pairs (key/value pairs). A name/value pair consists of a field
name (in double quotes), followed by a colon, followed by a value. JSON names require double
quotes. name:"John"
1.3.4 JSON Data – Data Separated By Commas
Since JSON syntax is derived from JavaScript object notation, you can create an object and
assign data to it.
person = {name:"John", age:25, city:"Douala"};
object1 = { "language" : "Java", "author" : "herbert schildt" };
1.3.4 JSON Data – Curly Braces Holding Objects
It is an unordered set of name/value pairs. Objects are enclosed in curly braces that is, it starts
with '{' and ends with '}'. Each name is followed by ':'(colon) and the key/value pairs are
separated by , comma). The keys must be strings and should be different from each other.
Objects should be used when the key names are arbitrary strings.

{
"id": "2",
"name": "John",

9|Page
 "age": 25,
}
1.3.4 JSON Data – Square Brackets Holding Arrays
Below is an example showing array containing multiple objects –
{
"users": [
{ "name":"John" , "age":"25" },
{ "name":"Peter" , "age":"29" },
{ "name":"Paul" , "age":"31" }
]
}
1.3.2 Creating JSON Objects
JSON is commonly used to exchange data to/from a web server. Generally, when receiving data
from a web server, the data is always a string. Parse the data with [Link](), and the data
becomes a JavaScript object. When using the [Link]() on a JSON derived from an array,
the method will return a JavaScript array, instead of a JavaScript object. This array can now be
used in any programming language and application of your choice.
Generally, JSON is the format that API’s use to send requests and receive responses from
servers. In this course, the notions of JSON will be very important in that it will be necessary to
perform API calls.

10 | P a g e
 MODULE 2 - INTRODUCTION TO NoSQL
2.1 General Introduction to NoSQL
2.1.1 Introduction to NoSQL
Conventional relational databases were the product of a great deal of research and testing to
obtain optimal storage structures. However, their flaw is that they have a rigid structure which
cannot be altered if there are different specifications. To combat this problem, NoSQL databases
were created.
A NoSQL (non-SQL, Not Only SQL or non-relational) is a database that provides a mechanism
for the storage and retrieval of data that is modelled in means other than the tabular relations
used in relational databases.
It refers to a family of databases that vary widely in style and technology, but all share a
common trait in that they are non-relational in nature, meaning they are not a standard row and
column RDBMS. Therefore, a better name to describe these databases would be non-relational.
NoSQL databases provide new ways of storing and querying data that address several issues for
modern applications. Most importantly, most NoSQL databases are geared to handle a different
breed of scale problems that have arisen associated with the “big data” movement. By scale, we
are referring to both the size of the data and the concurrent users acting on that data. NoSQL
databases also are typically more specialized to various use cases and can be much simpler to
develop application functionality than relational databases.
Some examples of NoSQL databases include: MongoDB, CouchDB, CouchBase, Cassandra, HBase,
Redis, Riak, Neo4J.

Figure 2.1 Some Examples of NoSQL Databases

In the late 2000’s, several new databases emerged on the scene, a large number of them from the
open-source communities. Databases like Apache CouchDB, Cassandra, and HBase, as well as

11 | P a g e
Mongo and Riak became more prevalently used in applications, particularly in ones that
required larger scale than a relational database could handle.

Figure 2.2 Rise of NoSQL and DBaaS

Today’s web, mobile, and IoT (Internet of Things) applications use NoSQL to store and process their
data. Below are some of the reasons for doing this:
➢ Support large numbers of concurrent users (tens of thousands and perhaps millions)
➢ Deliver highly responsive experiences to a globally distributed base of users
➢ Server always available – no downtime
➢ Handle semi- and unstructured data
➢ Rapidly adapt to changing requirements with frequent updates and new features.
➢ More customers are going online
➢ Big data is getting bigger
➢ The world has gone mobile
➢ The internet is connecting everything
➢ Applications are moving to the cloud

2.1.2 Introduction to Big Data

Big Data is a collection of data that is huge in volume yet growing exponentially with time. It is
a data with so large size and complexity that none of traditional data management tools can store
it or process it efficiently. Big data is also a data but with huge size.
Usage Examples include:
➢ Twitter processing and storing more than 12GB of data daily.
➢ Facebook generates and processes over 4 petabytes of data daily.
➢ Google generates over 20 petabytes daily and this includes over 3.5 Billion queries daily.
➢ The New York Stock Exchange generates about one terabyte of new trade data per day.

12 | P a g e
[Link] Characteristics of Big Data
Big data can be described by the following characteristics:
➢ Volume
➢ Variety
➢ Velocity
➢ Variability
(i) Volume – The name Big Data itself is related to a size which is enormous. Size of data plays
a very crucial role in determining value out of data. Also, whether a particular data can
actually be considered as a Big Data or not, is dependent upon the volume of data. Hence,
‘Volume’ is one characteristic which needs to be considered while dealing with Big Data
solutions.
(ii) Variety – The next aspect of Big Data is its variety.
Variety refers to heterogeneous sources and the nature of data, both structured and unstructured.
During earlier days, spreadsheets and databases were the only sources of data considered by
most of the applications. Nowadays, data in the form of emails, photos, videos, monitoring
devices, PDFs, audio, etc. are also being considered in the analysis applications. This variety of
unstructured data poses certain issues for storage, mining and analyzing data.
(iii) Velocity – The term ‘velocity’ refers to the speed of generation of data. How fast the data
is generated and processed to meet the demands, determines real potential in the data.
Big Data Velocity deals with the speed at which data flows in from sources like business
processes, application logs, networks, and social media sites, sensors, Mobile devices, etc. The
flow of data is massive and continuous.
(iv)Variability – This refers to the inconsistency which can be shown by the data at times, thus
hampering the process of being able to handle and manage the data effectively.
[Link] Types of Big Data
Following are the types of Big Data:
1. Structured
2. Unstructured
3. Semi-structured
a) Structured
Any data that can be stored, accessed and processed in the form of fixed format is termed as a
‘structured’ data. Over the period of time, talent in computer science has achieved greater
success in developing techniques for working with such kind of data (where the format is well
known in advance) and also deriving value out of it. However, nowadays, we are foreseeing

13 | P a g e
issues when a size of such data grows to a huge extent, typical sizes are being in the rage of
multiple zettabytes. One Billion Terabytes = 1 Zettabyte.
Looking at these figures one can easily understand why the name Big Data is given and imagine
the challenges involved in its storage and processing.
Examples Of Structured Data
An ‘Employee’ table in a database is an example of Structured Data
Employee_ID Employee_Name Gender Department Salary
2365 Ashu Nyenti Male Finance 650000
3398 Ngono Remi Female Admin 650000
7465 Abdulaye Moussa Male Admin 500000
7500 Hillary Jones Male Finance 500000
7699 Seydou Sane Female Finance 550000

b) Unstructured
Any data with unknown form or the structure is classified as unstructured data. In addition to the
size being huge, un-structured data poses multiple challenges in terms of its processing for
deriving value out of it. A typical example of unstructured data is a heterogeneous data source
containing a combination of simple text files, images, videos etc. Nowadays organizations have
wealth of data available with them but unfortunately, they don’t know how to derive value out of
it since this data is in its raw form or unstructured format.
Examples Of Un-structured Data
The output returned by ‘Google Search’

Figure 2.2 Example Of Un-structured Data

14 | P a g e
 c) Semi-structured
Semi-structured data can contain both the forms of data. We can see semi-structured data as
structured in form e.g. a table definition in relational DBMS.
Example of semi-structured data is a data represented in an XML file.
Personal data stored in an XML file-
<rec><name>Ashu Nyenti</name><sex>Male</sex><age>35</age></rec>
<rec><name>Agbor Bissong.</name><sex>Female</sex><age>41</age></rec>
<rec><name>Abdulaye Moussa</name><sex>Male</sex><age>29</age></rec>
<rec><name>OumarouJones</name><sex>Male</sex><age>26</age></rec>
<rec><name>Jeremiah Nji</name><sex>Male</sex><age>35</age></rec>

2.2 Real-Life Examples of NoSQL

Global 2000 enterprises are rapidly embracing NoSQL databases to power their mission-critical applications:
➢ Tesco, Europe’s No. 1 retailer, deploys NoSQL for e-commerce, product catalog, and other
applications
➢ Ryanair, the world’s busiest airline, uses NoSQL to power its mobile app serving over 3 million
users.
➢ Marriott deploys NoSQL for its reservation system that books $38 billion annually
➢ Gannett, the No. 1 U.S. newspaper publisher, uses NoSQL for its proprietary content management
system, Presto
➢ GE deploys NoSQL for its Predix platform to help manage the Industrial Internet

2.3 Differences Between NoSQL and RDBMS

1. RDBMS is called relational databases while NoSQL is called a distributed database.
They do not have any relations between any of the databases. When RDBMS uses
structured data to identify the primary key, there is a proper method in NoSQL to use
unstructured data.
2. RDBMS is scalable vertically and NoSQL is scalable horizontally. Hence in RDBMS,
servers have to be added and power has to be increased. This makes scalability an
RDBMS expensive. While in NoSQL, we just need to add more machines and this does
not make the database expensive.

15 | P a g e
 3. Maintenance of RDBMS is expensive as manpower is needed to manage the servers
added in the database. NoSQL is mostly automatic and does some repairs on its own.
Data distribution and administration is less in NoSQL.
4. As stated earlier, RDBMS is expensive due to the servers and storage management.
NoSQL databases are open source and cheap when compared with RDBMS. Servers in
NoSQL can be used to manage and transact data with relatively no cost at all.
5. RDBMS has a fixed schema so that data has to be inserted in a proper format. This helps
to get the primary key and foreign key to align the data in the table. Primary keys help to
connect the data from other tables with the common identifier. But in NoSQL, a schema
is not needed and hence data can be inserted in any form. Also, these data can be inserted
at any time and updates can be made without any prior notice. This helps the database to
work flexibly with the users.
6. Data output performance is better in NoSQL due to the support caching in system
memory. Separate infrastructure is not needed and hence this makes the developers tally
the data easily. Whereas in RDBMS, separate infrastructure is needed due to the absence
of support caching. This makes the data tally difficult.
7. NoSQL data structure is never equal due to the absence of schema and the fact that it is
open source. This makes the database be without a proper standard. RDBMS has a proper
schema and data has to be inserted only in the given format. The data structure is equal
from one table to another.
8. RDBMS has stored procedures to understand the data and to know them well. NoSQL,
however, does not have any stored procedure. This makes it difficult for users to identify
the pattern and to learn the data well.
9. Transactions in a RDBMS are written in one location while in a NoSQL, transactions
written in several locations.
10. In a relational database, data arrives from one or few locations whereas in a NoSQL, data
arrives from many locations.

2.4 NoSQL Database Models

Over time, four major types of NoSQL databases emerged: document databases, key-value databases,
wide-column stores, and graph databases.
2.4.1 Key-Value Model
Key–value (KV) stores use the associative array (also called a map or dictionary) as their
fundamental data model. In this model, data is represented as a collection of key–value pairs,
such that each possible key appears at most once in the collection. These databases are organised
16 | P a g e
as key value pairs, where each key appears exactly once. The keys are usually arranged in a
sorted fashion. Examples of this type of databases are: Aerospike, ArangoDB, Couchbase,
Dynamo etc.

Figure 2.3 Key-Value NoSQL Model

2.4.2 Column Store Model

Data is stored in a columnar form. Some examples of this type of database are: Accumulo,
Cassandra, Druid, Vertica etc. It avoids consuming space when storing nulls by simply not
storing a column when a value doesn’t exist for that column. Each unit of data can be thought of
as a set of key/value pairs, where the unit itself is identified with the help of a primary identifier,
often referred to as the primary key. Bigtable and its clones tend to call this primary key the
row-key. This model stores data in tables, and dynamic columns rather than in rows which is
more usual
with SQL.

17 | P a g e
 Figure 2.4 Column Store NoSQL Model

2.4.3 Document Store Model

With SQL, XML and JSON tend to be tied together, which slows queries down and hampers the
whole process. Since NoSQL doesn’t use the relational model, it doesn’t need to do that, which
is where document-based stores come in. It stores data in documents similar to JSON objects.
Each document contains pairs of fields and values. The values can typically be a variety of types
including things like strings, numbers, booleans, arrays, or objects. Examples of Document
based databases are: MongoDB, HBase, Cassandra, Amazon SimpleDB, Hypertable, etc.
All data is stored in one table, so there’s no need for cross-referencing and instead of storing
information in a table, it’s stored in a document. While this is very similar to a key-value store,
and can sometimes be considered under its umbrella, the difference is that document-based
NoSQL generally has some form of encoding, such as XML.

18 | P a g e
 Figure 2.5 Document NoSQL Model

2.4.4 Graph-Based Model

Graph or network data models essentially treat the relationship between any two pieces of
information as being just as important as the information itself. As such, this type of data model
is really made for any information that you’d usually represent on a graph. It uses relationships
and nodes, with the data being the information itself, and the relationship is formed between
nodes. It stores data in nodes and edges. Nodes typically store information about people, places,
and things, while edges store information about the relationships between the nodes. Examples
of data include social relations, public transport links, road maps, network topologies, etc. The
following databases use this model: Neo4j, OrientDB, Facebook Open Graph, FlockDB, etc.

19 | P a g e
 Figure 2.6 Graph-Based NoSQL Model

2.5 Representing NoSQL Databases

2.5.0 Creating Database Clusters
A cluster is a place where you can store your MongoDB databases.

2.5.1 MongoDB For Document Model

MongoDB is a database based on a non-relational document model. The name MongoDB is
derived from the English word “humongous”, which roughly means “gigantic”. MongoDB
stores its data records in structures known as documents. Mongo allows you to group multiple
documents into a structure known as a collection, which can be further grouped into separate
databases. A document is written in BSON, a binary representation of JSON. Like objects in
JSON, MongoDB documents begin and end with curly brackets ({ and }), and contain a number
of field-and-value pairs which typically take the form of field: value. A field’s value can be any
one of the data types used in BSON, or even other structures like documents and arrays.

20 | P a g e
[Link] How To Download And Test MongoDB On Windows
➢ Go to the page [Link]
➢ Click on the green “Try Free” button at the top right and create your account.
➢ On the following page, select “On-premises” and then “MongoDB Community Server“.
➢ The current database version is now displayed on the right-hand side of the screen and you can click
on the green download button with the preset values
➢ Create a Shared Cluster
➢ After setting up, Go to Databases
➢ Click on the cluster you created and click on CONNECT.
➢ Select “Connect using MongoDB Compass”
➢ Copy your connection string and paste in MongoDB Compass
[Link] Creating a Database and Inserting Records into the MongoDB Database
➢ Open the cluster and click on COLLECTIONS ➢ Click on “Create
Database”

Figure 2.7 Creating a MongoDB Database

➢ To insert data, select the database, select the collection and click on “Insert
Document”

21 | P a g e
Figure 2.8 Inserting New Document into the Collection

[Link] Retrieving Records from the MongoDB Database

To query the database, pass, the name-pair values inside {} and click on “Apply”

Figure 2.9 Retrieving Data From MongoDB Database

Exercise: Using MongoDB, create a database and collections, insert documents into the collections.
2.5.2 Cassandra For Column Model
Cassandra database is distributed over several machines that operate together. The outermost
container is known as the Cluster. For failure handling, every node contains a replica, and in

22 | P a g e
case of a failure, the replica takes charge. Cassandra arranges the nodes in a cluster, in a ring
format, and assigns data to them.
Apache Cassandra is an example of NoSQL Database. It is a distributed, decentralized and an
open-source database or a storage system. It is basically used for managing very large amounts
of structured data. There is no single point of failure, providing highly available services. It has
the following features:
a. Decentralized: Every node in the cluster is identical. There are no single points of failure.
b. Fault Tolerance: Since data is replicated to multiple nodes, fault tolerance is pretty high. Also, any
failed nodes are related with no downtime.
c. Performance: The fundamental architecture of Apache Cassandra is very robust. Therefore, it proves
to be better than the alternatives.
d. Scalable: It is linearly scalable. In other words, the throughput is increased as you increase the
number of nodes in the Cassandra cluster.
e. Proven: Apache Cassandra is used at various companies. Some of them are Netflix, GoDaddy,
GitHub, eBay etc.
These features of Apache Cassandra shows that it is actually very powerful and reliable.

2.5.3 Neo4j For Graph Model

Neo4j facilitates personal data storage and management: it allows you to track where private
information is stored and which systems, applications, and users access it. The graph data model
helps visualize personal data and allows for data analysis and pattern detection.
Neo4j is a graph database. A graph database, instead of having rows and columns has nodes
edges and properties. It is more suitable for certain big data and analytics applications than row
and column databases or free-form JSON document databases for many use cases. A graph
database is used to represent relationships. You can download Neo4j Windows from
[Link]

2.5.4 – Redis For Key-Value Model

Redis is an in-memory data structure store, used as a distributed, in-memory key–value database,
cache and message broker, with optional durability. Redis supports different kinds of abstract
data structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams,
and spatial indices. Currently, it is being used by tech-giants like GitHub, Weibo, Pinterest,
Snapchat,
Craigslist, Digg, StackOverflow, Flickr. It can be downloaded from [Link]

23 | P a g e
2.6 Advantages of NoSQL Databases
(i) Flexible Data Model:
NoSQL databases are highly flexible as they can store and combine any type of data, both structured and
unstructured, unlike relational databases that can store data in a structured way only.

(ii) Evolving Data Model :

NoSQL databases allow you to dynamically update the schema to evolve with changing requirements while
ensuring that it would cause no interruption or downtime to your application.

(iii) Elastic Scalability:

NoSQL databases can scale to accommodate any type of data growth while maintaining low cost.

(iv) High Performance:

NoSQL databases are built for great performance, measured in terms of both throughput (it is a measure
of overall performance) and latency (it is the delay between request and actual response).

(v) Open source:

NoSQL databases don’t require expensive licensing fees and can run on inexpensive hardware, rendering
their deployment cost-effective.

2.7 Disadvantages of NoSQL Databases

(i) Lack of Standardization:
There is no standard that defines rules and roles of NoSQL databases. The design and query
languages of NoSQL databases vary widely between different NoSQL products – much more
widely than they do among traditional SQL databases.

(ii) Backup of Database:

Backups are a drawback in NoSQL databases. Though some NoSQL databases like MongoDB
provide some tools for backup, these tools are not mature enough to ensure proper complete data
backup solution.

(iii) Consistency:
NoSQL puts a scalability and performance first but when it comes to a consistency of the data
NoSQL doesn’t take much consideration so it makes it little insecure as compared to the
relational database e.g., in NoSQL databases if you enter same set of data again, it will take it
without issuing any error whereas relational databases ensure that no duplicate rows get entry in
databases.

24 | P a g e
2.8 The CAP Theorem of NoSQL
It is very important to understand the limitations of NoSQL database. NoSQL cannot provide
consistency and high availability together.
Why CAP theorem is important?
After the internet boom in 2005, the size of data is growing exponentially day by day. At the
early stages to maintain the ever-changing scale of data and plan the capacity properly the only
option was to increase the capacity vertically which means adding more machines or increasing
the machine capabilities. But is not always feasible and cost-effective. Instead of this, the new
concept is to add the capacity horizontally which means leveraging distributed computing. To
standardize the network, we need to maintain the principles of the CAP theorem.

The CAP theorem (also called Brewer's theorem) states that a distributed database system can
only guarantee two out of these three characteristics: Consistency, Availability, and Partition
Tolerance.

Figure 2.10 The CAP Theorem of NoSQL

a) Consistency
Consistency means that all nodes in the network see the same data at the same time.
Example:
Let’s take an example of MTN Cameroon whereby your ID card has expired, and you want to
update your new ID card number registered with them. You decide to call the customer care
operator and update it with them. When you call, you connect with an operator. This operator

25 | P a g e
makes the relevant changes in the system. But once you have dropped the phone, you realize you
gave them a wrong ID card number.
So you call the customer care again. This time, you connect with a different customer care
operator and they are able to access your records as well and know that you have recently
updated your ID card number. They make the relevant changes in the ID card number and the
rest of your information is the same as the one you told the last operator.
We call this as Consistency because even though you connect to a different customer care
operator, they were able to retrieve the same information.

b) Availability
Availability is a guarantee that every request receives a response about whether it was
successful or failed. However, it does not guarantee that a read request returns the most recent
write. The greater number of users a system can cater to better is the availability.
Example:

26 | P a g e
Let’s take an example or MTN Cameroon customer service and you are its customer because of
the amazing plans it offers. Besides that, they also provide an amazing customer care service
where you can call anytime and get your queries and concerns answered quickly and efficiently.
Whenever a customer calls them, the mobile operator can connect them to one of their customer
care operators.
The customer can evoke any information required by her/him about his accounts like balance,
usage, or other information. We call this Availability because every customer is able to connect
to the operator and get his/her information.

c) Partition Tolerance
Partition Tolerance is a guarantee that the system continues to operate despite arbitrary message
loss or failure of part of the system. In other words, even if there is a network outage in the data
center and some of the computers are unreachable, still the system continues to perform. Out of
these three guarantees, no system can provide more than 2 guarantees. Since in the case of a
distributed systems, the partitioning of the network is must, the tradeoff is always between
consistency and availability.

Example:

27 | P a g e
Recently MTN keeps sending you messages to update your ID card despite calling them to
update your ID card. So you decide to call the customer care once again. On connecting with the
operator this time, they tell you that they have not been able to update their records due to some
issues. So the information lying with the operator might not be up to date, therefore they cannot
update the information with the ID card. We can say here that the service is broken or there is no
Partition tolerance.

As depicted on figure 2.10 above, RDBMS can provide only consistency but not partition
tolerance. While HBASE and Redis can provide Consistency and Partition tolerance. And
MongoDB, CouchDB, Cassandra and Dynamo guarantee only availability but no consistency.
Such databases generally settle down for eventual consistency meaning that after a while the
system is going to be ok.
System designers must take into consideration the CAP theorem while designing or choosing
distributed storages as one needs to be sacrificed from C and A for others.
The CAP theorem categorizes systems into three categories:
CP (Consistent and Partition Tolerant) database: A CP database delivers consistency and
partition tolerance at the expense of availability. When a partition occurs between any two
28 | P a g e
nodes, the system has to shut down the non-consistent node (i.e., make it unavailable) until the
partition is resolved.
Partition refers to a communication break between nodes within a distributed system. Meaning,
if a node cannot receive any messages from another node in the system, there is a partition
between the two nodes. Partition could have been because of network failure, server crash, or
any other reason.
AP (Available and Partition Tolerant) database: An AP database delivers availability and
partition tolerance at the expense of consistency. When a partition occurs, all nodes remain
available but those at the wrong end of a partition might return an older version of data than
others. When the partition is resolved, the AP databases typically resync the nodes to repair all
inconsistencies in the system.
CA (Consistent and Available) database: A CA delivers consistency and availability in the
absence of any network partition. Often a single node’s DB servers are categorized as CA
systems. Single node DB servers do not need to deal with partition tolerance and are thus
considered CA systems.
In any networked shared-data systems or distributed systems partition tolerance is a must.
Network partitions and dropped messages are a fact of life and must be handled appropriately.
Consequently, system designers must choose between consistency and availability.

2.8.1 How HBase, MongoDB, Cassandra, Neo4j, and Redis fit with the CAP theorem
[Link] Understanding CP with MongoDB
Let’s try to understand how a distributed system would work when it decides to give up on
Availability during a partition with the help of MongoDB.

MongoDB is a NoSQL database that stores data in one or more Primary nodes in the form of
JSON files. Each Primary node has multiple replica sets that update themselves asynchronously

29 | P a g e
using the operation log file of their respective primary node. The replica set nodes in the system
send a heartbeat (ping) to every other node to keep track if other replicas or primary nodes are
alive or dead. If no heartbeat is received within 10 seconds, then that node is marked as
inaccessible.
If a Primary node becomes inaccessible, then one of the secondary nodes needs to become the
primary node. Till a new primary is elected from amongst the secondary nodes, the system
remains unavailable to the user to make any new write query. Therefore, the MongoDB system
behaves as a Consistent system and compromises on Availability during a partition.

[Link] Understanding AP with Cassandra

Now let’s also look at how a system compromises on Consistency. For this, we will look at the
Cassandra database which is called a highly available database.
Cassandra is a peer-to-peer system. It consists of multiple nodes in the system. And each node
can accept a read or write request from the user. Cassandra maintains multiple replicas of data in
separate nodes. This gives it a masterless node architecture where there are multiple points of
failure instead of a single point.
The replication factor determines the number of replicas of data. If the replication factor is 3,
then we will replicate the data in three nodes in a clockwise manner.

A situation can occur where a partition occurs and the replica does not get an updated copy of
the data. In such a situation the replica nodes will still be available to the user but the data will
be inconsistent. However, Cassandra also provides eventual consistency. Meaning, all updates

30 | P a g e
 will reach all the replicas eventually. But in the meantime, it allows divergent versions of the
same data to exist temporarily. Until we
update them to the consistent state.
Therefore, by allowing nodes to be available throughout and allowing temporarily inconsistent
data to existing in the system, Cassandra is an AP database that compromises on consistency.

2.9 ACID vs CAP

ACID describes a set of properties which guarantee a database transaction is reliable. It ensures data
integrity. Let’s take a look at the ACID properties.
1. Atomicity : All changes to data are performed as a single operation. That is, all or none,
either all of the operations are performed or one of them is performed. For example, in an
application we are transferring funds from one account to another, the atomicity property ensures
that, if a debit is made successfully from one account, then the associated credit is also done to
the other account.

2. Consistency: For each transaction, the system should move from one consistent state to
another consistent state.

3. Isolation: All transactions should be executed in isolation from other transactions. During
concurrent transaction execution, intermediate transaction results from parallel executed
transactions should be mutually exclusive. Failure of one module should not affect another
transaction

4. Durability: After every successful transaction, the changes made in the database should
persist. Even if the system comprises or failed somehow, still the successfully committed or
aborted operations should persist. Now we can see that, these terms technically refer to different
things. The way in which they are related is that a distributed database system that guarantees
the ACID transactions must choose consistency over availability according to the CAP Theorem
(i.e it is a CP system).
2.9.1 Relationships and Differences between ACID and CAP
The CAP consistency applies to a distributed system, whereas ACID abstracts itself from that level
of the system architecture. CAP consistency may be necessary for the implementation of ACID
atomicity, but it's not the same thing.

31 | P a g e
ACID Consistency is about database rules , If the data table structure defines that a field value is
unique , Then the consistency system will solve the problem that causes the value of this field to
be non unique in all operations , If a row of records with a foreign key is deleted , Then its foreign
key related records should also be deleted , This is it. ACID Consistency means .

CAP theorem: specifies that a distributed system can provide two services (ex. Availability and
Partition tolerance) but never three. If for example, a service provides Availability and Partitioning
it can never ensure Consistency, not immediately, thus Eventual Consistency is used, which allows
the infrastructure to flux between inconsistency and consistency, however at one point, sooner or
later, the infrastructure will become consistent, resulting in eventual consistency. Cloud services
work in such fashion and Amazon's Simple DB uses eventual consistency. CAP addresses cluster
wide data consistency

ACID features are usually applied to relational DBs. If you want to apply ACID in a distributed
fashion (distributed DB), ACID uses 2PC(two-phase commit) to force consistency across
partitions. However since ACID provides consistency and partitioning, applying the CAP theorem
for (distributed environments) this will mean that availability is compromised. ACID addresses an
individual node's data consistency

2.10 Hadoop Distributed File System (HDFS)

With growing data velocity the data size easily outgrows the storage limit of a machine. A
solution would be to store the data across a network of machines. Such filesystems are called
distributed filesystems. Since data is stored across a network all the complications of a network
come in.

This is where Hadoop comes in. It provides one of the most reliable filesystems.
Hadoop is an open-source distributed processing framework that manages data processing
and storage for big data applications.
HDFS (Hadoop Distributed File System) is a unique design that provides storage for extremely
large files with streaming data access pattern and it runs on commodity hardware. Let’s
elaborate the terms:
➢ Extremely large files: Here we are talking about the data in range of petabytes(1000 TB).
➢ Streaming Data Access Pattern: HDFS is designed on principle of write-once and read
many-times. Once data is written large portions of dataset can be processed any number
times.

32 | P a g e
 ➢ Commodity hardware: Hardware that is inexpensive and easily available in the market.
This is one of feature which specially distinguishes HDFS from other file system.
Nodes: Master-slave nodes typically forms the HDFS cluster.
1. Master Node:
➢ Manages all the slave nodes and assign work to them.
➢ It executes filesystem namespace operations like opening, closing, renaming files and
directories.
➢ It should be deployed on reliable hardware which has the high config. not on commodity
hardware.
2. Name Node:
➢ Actual worker nodes, who do the actual work like reading, writing, processing etc.
➢ They also perform creation, deletion, and replication upon instruction from the master.
➢ They can be deployed on commodity hardware.
HDFS daemons: Daemons are the processes running in background.
• Name nodes:
➢ Run on the master node.
➢ Store metadata (data about data) like file path, the number of blocks, block Ids. etc. ➢
Require high amount of RAM.
➢ Store meta-data in RAM for fast retrieval i.e to reduce seek time. Though a persistent
copy of it is kept on disk.
• Data Nodes:
➢ Run on slave nodes.
➢ Require high memory as data is actually stored here.
Data storage in HDFS: Now let’s see how the data is stored in a distributed manner.

33 | P a g e
Let’s assume that 100TB file is inserted, then master node (name node) will first divide the file
into blocks of 10TB (default size is 128 MB in Hadoop 2.x and above). Then these blocks are
stored across different data nodes (slave node). Data nodes (slave node) replicate the blocks
among themselves and the information of what blocks they contain is sent to the master. Default
replication factor is 3 means for each block 3 replicas are created (including itself). In
[Link] we can increase or decrease the replication factor i.e we can edit its configuration
here.
Note: Master Node has the record of everything, it knows the location and info of each and
every single data nodes and the blocks they contain, i.e. nothing is done without the permission
of master node.
Why divide the file into blocks?
Answer: Let’s assume that we don’t divide, now it’s very difficult to store a 100 TB file on a
single machine. Even if we store, then each read and write operation on that whole file is going
to take very high seek time. But if we have multiple blocks of size 128MB then its become easy
to perform various read and write operations on it compared to doing it on a whole file at once.
So we divide the file to have faster data access i.e. reduce seek time.

Why replicate the blocks in data nodes while storing?

Answer: Let’s assume we don’t replicate and only one yellow block is present on data node D1.
Now if the data node D1 crashes we will lose the block and which will make the overall data
inconsistent and faulty. So we replicate the blocks to achieve fault-tolerance.
Terms related to HDFS:

34 | P a g e
 ➢ Heartbeat : It is the signal that data node continuously sends to name node. If name node
doesn’t receive heartbeat from a data node then it will consider it dead.
➢ Balancing : If a data node is crashed the blocks present on it will be gone too and the
blocks will be under-replicated compared to the remaining blocks. Here master
node(name node) will give a signal to data nodes containing replicas of those lost blocks
to replicate so that overall distribution of blocks is balanced.
➢ Replication: It is done by data node.
Note: No two replicas of the same block are present on the same data node.
Features:
➢ Distributed data storage.
➢ Blocks reduce seek time.
➢ The data is highly available as the same block is present at multiple data nodes.
➢ Even if multiple data nodes are down, we can still do our work, thus making it highly
reliable.
➢ High fault tolerance.
Limitations: Though HDFS provide many features there are some areas where it doesn’t work
well.
➢ Low latency data access: Applications that require low-latency access to data i.e in the
range of milliseconds will not work well with HDFS, because HDFS is designed keeping
in mind that we need high-throughput of data even at the cost of latency.
➢ Small file problem: Having lots of small files will result in lots of seeks and lots of
movement from one data node to another data node to retrieve each small file, this whole
process is a very inefficient data access pattern.

2.11 NoSQL Practicals With Java and MongoDB

Using Java as programming language and MongoDB as database, create a CRUD application that
registers, retrieves, update and delete list of students.

MODULE 3 - INTRODUCTION TO CLOUDANT – DATABASE AS A SERVICE

3.1 General Introduction to Database As A Service (DBaaS)
3.1.1 Introduction to NoSQL
3.2 Advantages of DBaaS
3.3 How To Choose A DBaaS
3.4 Introduction To IBM Cloud Database
3.5 Benefits of IBM Cloudant, a NoSQL DBaaS
35 | P a g e
 3.6 Cloudant Architecture

3.7 Cloudant Deployment

MODULE 4 – PRACTICALS ON IBM CLOUDANT

4.1 Creating An IBM Cloudant Account
4.1.1 Introduction to NoSQL
4.2 Creating Databases, Collections and Adding Documents

36 | P a g e