Introduction to Big Data

The quantity of data created by humans is quickly increasing every year as a result of the
introduction of new technology, gadgets, and communication channels such as social
networking sites. Big data is a group of enormous datasets that can't be handled with typical
computer methods. It is no longer a single technique or tool; rather, it has evolved into a
comprehensive subject including a variety of tools, techniques, and frameworks.

What is Big Data?

Big Data is a massive collection of data that continues to increase dramatically over time. It is
a data set that is so huge and complicated that no typical data management technologies can
effectively store or process it. Big data is similar to regular data, except it is much larger. Big
data analytics is the use of advanced analytic techniques to very large, heterogeneous data
sets, which can contain structured, semi-structured, and unstructured data, as well as data
from many sources and sizes ranging from terabytes to zettabytes.

Examples of Big Data

 Every day, 500+ terabytes of fresh data are absorbed into the Facebook systems. This
information is mostly gathered through photo and video uploads, message exchanges, and
the posting of comments, among other things.
 In 30 minutes of flying time, a single Jet engine may create 10+ gigabytes of data. With
thousands of flights every day, the amount of data generated can amount to several
Petabytes.
 Every day, the Stock Exchange creates around a terabyte of new trading data.

Characteristics of Big Data (3 V’s)

Big data can be described by following characteristics as shown in Figure.

 Volume

The term ‘Big Data’ refers to a massive amount of information. The term "volume" refers to a
large amount of data. The magnitude of data plays a critical role in determining its worth.
When the amount of data is extremely vast, it is referred to as 'Big Data.' This means that the
volume of data determines whether or not a set of data may be classified as Big Data. As a
result, while dealing with Big Data, it is vital to consider a certain ‘Volume’.

1
Example: In 2016, worldwide mobile traffic was predicted to be 6.2 Exabytes (6.2 billion GB)
per month. Furthermore, by 2020, we have about 40000 ExaBytes of data.

 Velocity

The term "velocity" refers to the rapid collection of data. Data comes in at a high rate from
machines, networks, social media, mobile phones, and other sources in Big Data velocity. A
large and constant influx of data exists. This influences the data's potential, or how quickly
data is created and processed in order to satisfy needs. Data sampling can assist in dealing
with issues such as 'velocity.' For instance, Google receives more than 3.5 billion queries
every day. In addition, the number of Facebook users is growing at a rate of around 22%
every year.

 Variety

Structured data is just data that has been arranged. It usually refers to data that has been
specified in terms of length and format.

Semi-structured data is a type of data that is semi-organized. It's a type of data that doesn't
follow the traditional data structure. This sort of data is represented by log files.

Unstructured data is just data that has not been arranged. It usually refers to data that doesn't
fit cleanly into a relational database's standard row and column structure. Texts, pictures,
videos etc. are the examples of unstructured data which can’t be stored in the form of rows
and columns.

2
Benefits of Big Data Processing

Ability to process Big Data brings in multiple benefits, such as-

1. Businesses can utilize outside intelligence while taking decisions.

2. Access to social data from search engines and sites like facebook, twitter are enabling
organizations to fine tune their business strategies.
3. Improved customer service (Traditional customer feedback systems are getting replaced
by new systems designed with Big Data technologies.
4. Improved customer service (In these new systems, Big Data and natural language
processing technologies are being used to read and evaluate consumer responses.
5. Early identification of risk to the product/services, if any
6. Better operational efficiency

Big Data technologies can be used for creating a staging area or landing zone for new data
before identifying what data should be moved to the data warehouse. In addition, such
integration of Big Data technologies and data warehouse helps an organization to offload
infrequently accessed data.

Challenges in BIG DATA

 Lack of proper understanding of Big Data

Companies fail in their Big Data initiatives due to insufficient understanding. Employees may
not know what data is, its storage, processing, importance, and sources. Data professionals
may know what is going on, but others may not have a clear picture.

 Data growth issues

One of the most pressing challenges of Big Data is storing all these huge sets of data properly.
The amount of data being stored in data centers and databases of companies is increasing
rapidly. As these data sets grow exponentially with time, it gets extremely difficult to handle.

 Confusion while Big Data tool selection

Companies often get confused while selecting the best tool for Big Data analysis and storage.
Is HBase or Cassandra the best technology for data storage? Is Hadoop MapReduce good
enough or will Spark be a better option for data analytics and storage? These questions bother
companies and sometimes they are unable to find the answers. They end up making poor
3
decisions and selecting an inappropriate technology. As a result, money, time, efforts and
work hours are wasted.

 Lack of data professionals

To run these modern technologies and Big Data tools, companies need skilled data
professionals. These professionals will include data scientists, data analysts and data
engineers who are experienced in working with the tools and making sense out of huge data
sets.

 Securing data

Securing these huge sets of data is one of the daunting challenges of Big Data. Often
companies are so busy in understanding, storing and analyzing their data sets that they push
data security for later stages. But, this is not a smart move as unprotected data repositories
can become breeding grounds for malicious hackers.

 Integrating data from a variety of sources

Data in an organization comes from a variety of sources, such as social media pages, ERP
applications, customer logs, financial reports, e-mails, presentations and reports created by
employees. Combining all this data to prepare reports is a challenging task. This is an area
often neglected by firms. But, data integration is crucial for analysis, reporting and business
intelligence, so it has to be perfect.

4
 Introduction to Hadoop
Hadoop is a framework that allows us to store and process large datasets in parallel and
distributed fashion.

Two major problems in dealing with BIG DATA

• Storage
• Processing

Storage problem resolved by

• HDFS

All big amount of data that we are dumping is gets distributed over different machine. These
machines are interconnected.

Processing problem resolved by

• mapReduce

Hadoop runs applications using the MapReduce algorithm, where the data is processed in
parallel with others. In short, Hadoop is used to develop applications that could perform
complete statistical analysis on huge amounts of data. Hadoop is an Apache open source
framework written in java that allows distributed processing of large datasets across clusters
of computers using simple programming models. The Hadoop framework application works
in an environment that provides distributed storage and computation across clusters of
computers. Hadoop is designed to scale up from single server to thousands of machines, each
offering local computation and storage.

At its core, Hadoop has two major layers namely –

5
 • Processing/Computation layer (MapReduce)
• Storage layer (Hadoop Distributed File System)

The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS) and
provides a distributed file system that is designed to run on commodity hardware. It has many
similarities with existing distributed file systems. However, the differences from other
distributed file systems are significant. It is highly fault-tolerant and is designed to be
deployed on low-cost hardware. It provides high throughput access to application data and is
suitable for applications having large [Link] from the above-mentioned two core
components, Hadoop framework also includes the following two modules − Hadoop
Common − These are Java libraries and utilities required by other Hadoop modules.

Hadoop YARN − this is a framework for job scheduling and cluster resource management.
YARN stands for yet another Resource Negotiator. It manages and schedules the resources,
and decides what should happen in each data node. The central master node that manages all
processing requests is called the Resource Manager. The Resource Manager interacts with
Node Managers; every slave datanode has its own Node Manager to execute tasks.

Benefits of Hadoop for Big Data

 Resilience — Data stored in any node is also replicated in other nodes of the cluster. This
ensures fault tolerance. If one node goes down, there is always a backup of the data
available in the cluster.
 Scalability — Unlike traditional systems that have a limitation on data storage, Hadoop is
scalable because it operates in a distributed environment. As the need arises, the setup can
be easily expanded to include more servers that can store up to multiple petabytes of data.
 Low cost — As Hadoop is an open-source framework, with no license to be procured, the
costs are significantly lower compared to relational database systems. The use of
inexpensive commodity hardware also works in its favor to keep the solution economical.
 Speed — Hadoop’s distributed file system, concurrent processing, and the MapReduce
model enable running complex queries in a matter of seconds.
 Data diversity — HDFS has the capability to store different data formats such as
unstructured (e.g. videos), semi-structured (e.g. XML files), and structured.

6
While storing data, it is not required to validate against a predefined schema. Rather, the data
can be dumped in any format. Later, when retrieved, data is parsed and fitted into any schema
as needed. This gives the flexibility to derive different insights using the same data.

The Hadoop Ecosystem Supplementary Components

Hadoop Ecosystem is a platform or a suite which provides various services to solve the big
data problems. It includes Apache projects and various commercial tools and solutions.

There are four major elements of Hadoop i.e. HDFS, MapReduce, YARN, and Hadoop
Common.

Most of the tools or solutions are used to supplement or support these major elements. All
these tools work collectively to provide services such as absorption, analysis, storage and
maintenance of data etc. The following are a few supplementary components that are
extensively used in the Hadoop ecosystem.

HDFS: Hadoop Distributed File System

YARN: Yet Another Resource Negotiator

MapReduce: Programming based Data Processing

Spark: In-Memory data processing

PIG, HIVE: Query based processing of data services

7
HBase: NOSQL Database

Mahout, Spark MLLib: MachineLearning algorithm libraries

Solar, Lucene: Searching and Indexing

Zookeeper: Managing cluster

Oozie: Job Scheduling

Features of Hadoop

1. Open Source: Hadoop is open-source, which means it is free to use. Since it is an open-
source project the source-code is available online for anyone to understand it or make
some modifications as per their industry requirement.

2. Highly Scalable Cluster: Hadoop is a highly scalable model. A large amount of data is
divided into multiple inexpensive machines in a cluster which is processed parallelly. the
number of these machines or nodes can be increased or decreased as per the enterprise’s
requirements. In traditional RDBMS (Relational Database Management System) the
systems can not be scaled to approach large amounts of data.

3. Fault Tolerance is Available: Hadoop uses commodity hardware(inexpensive systems)

which can be crashed at any moment. In Hadoop data is replicated on various DataNodes
in a Hadoop cluster which ensures the availability of data if somehow any of your
systems got crashed. You can read all of the data from a single machine if this machine
faces a technical issue data can also be read from other nodes in a Hadoop cluster
because the data is copied or replicated by default. By default, Hadoop makes 3 copies of
each file block and stored it into different nodes. This replication factor is configurable
and can be changed by changing the replication property in the [Link] file.

4. High Availability is Provided: Fault tolerance provides High Availability in the Hadoop
cluster. High Availability means the availability of data on the Hadoop cluster. Due to
fault tolerance in case if any of the DataNode goes down the same data can be retrieved
from any other node where the data is replicated. The High available Hadoop cluster also
has 2 or more than two Name Node i.e. Active NameNode and Passive NameNode also
known as stand by NameNode. In case if Active NameNode fails then the Passive node

8
 will take the responsibility of Active Node and provide the same data as that of Active
NameNode which can easily be utilized by the user.

5. Cost-Effective: Hadoop is open-source and uses cost-effective commodity hardware

which provides a cost-efficient model, unlike traditional Relational databases that require
expensive hardware and high-end processors to deal with Big Data. The problem with
traditional Relational databases is that storing the Massive volume of data is not cost-
effective, so the company’s started to remove the Raw data. which may not result in the
correct scenario of their business. Means Hadoop provides us 2 main benefits with the
cost one is it’s open-source means free to use and the other is that it uses commodity
hardware which is also inexpensive.

6. Hadoop Provide Flexibility: Hadoop is designed in such a way that it can deal with any
kind of dataset like structured(MySql Data), Semi-Structured(XML, JSON), Un-
structured (Images and Videos) very efficiently. This means it can easily process any
kind of data independent of its structure which makes it highly flexible. It is very much
useful for enterprises as they can process large datasets easily, so the businesses can use
Hadoop to analyze valuable insights of data from sources like social media, email, etc.
With this flexibility, Hadoop can be used with log processing, Data Warehousing, Fraud
detection, etc.

7. Easy to Use: Hadoop is easy to use since the developers need not worry about any of the
processing work since it is managed by the Hadoop itself. Hadoop ecosystem is also very
large comes up with lots of tools like Hive, Pig, Spark, HBase, Mahout, etc.

8. Hadoop uses Data Locality: The concept of Data Locality is used to make Hadoop
processing fast. In the data locality concept, the computation logic is moved near data
rather than moving the data to the computation logic. The cost of Moving data on HDFS
is costliest and with the help of the data locality concept, the bandwidth utilization in the
system is minimized.

9. Provides Faster Data Processing: Hadoop uses a distributed file system to manage its
storage i.e. HDFS(Hadoop Distributed File System). In DFS(Distributed File System) a
large size file is broken into small size file blocks then distributed among the Nodes
available in a Hadoop cluster, as this massive number of file blocks are processed

9
 parallelly which makes Hadoop faster, because of which it provides a High-level
performance as compared to the traditional DataBase Management Systems.

10. Support for Multiple Data Formats: Hadoop supports multiple data formats like CSV,
JSON, Avro, and more, making it easier to work with different types of data sources.
This makes it more convenient for developers and data analysts to handle large volumes
of data with different formats.

11. High Processing Speed: Hadoop’s distributed processing model allows it to process large
amounts of data at high speeds. This is achieved by distributing data across multiple
nodes and processing it in parallel. As a result, Hadoop can process data much faster
than traditional database systems.

12. Machine Learning Capabilities: Hadoop offers machine learning capabilities through its
ecosystem tools like Mahout, which is a library for creating scalable machine learning
applications. With these tools, data analysts and developers can build machine learning
models to analyze and process large datasets.

13. Integration with Other Tools: Hadoop integrates with other popular tools like Apache
Spark, Apache Flink, and Apache Storm, making it easier to build data processing
pipelines. This integration allows developers and data analysts to use their favorite tools
and frameworks for building data pipelines and processing large datasets.

14. Secure: Hadoop provides built-in security features like authentication, authorization, and
encryption. These features help to protect data and ensure that only authorized users have
access to it. This makes Hadoop a more secure platform for processing sensitive data.

15. Community Support: Hadoop has a large community of users and developers who
contribute to its development and provide support to users. This means that users can
access a wealth of resources and support to help them get the most out of Hadoop.

10
NoSQL Databases

NoSQL is a type of database management system (DBMS) that is designed to handle and
store large volumes of unstructured and semi-structured data. Unlike traditional relational
databases that use tables with pre-defined schemas to store data, NoSQL databases use
flexible data models that can adapt to changes in data structures and are capable of scaling
horizontally to handle growing amounts of data.

The term NoSQL originally referred to “non-SQL” or “non-relational” databases, but the
term has since evolved to mean “not only SQL,” as NoSQL databases have expanded to
include a wide range of different database architectures and data models.

NoSQL databases are generally classified into four main categories:

1. Document databases: These databases store data as semi-structured documents, such as

JSON or XML, and can be queried using document-oriented query languages.

2. Key-value stores: These databases store data as key-value pairs, and are optimized for
simple and fast read/write operations.

3. Column-family stores: These databases store data as column families, which are sets of
columns that are treated as a single entity. They are optimized for fast and efficient
querying of large amounts of data.

4. Graph databases: These databases store data as nodes and edges, and are designed to
handle complex relationships between data.

NoSQL databases are often used in applications where there is a high volume of data that
needs to be processed and analyzed in real-time, such as social media analytics, e-commerce,
and gaming. They can also be used for other applications, such as content management
systems, document management, and customer relationship management.

However, NoSQL databases may not be suitable for all applications, as they may not provide
the same level of data consistency and transactional guarantees as traditional relational
databases. It is important to carefully evaluate the specific needs of an application when
choosing a database management system.

11
Key Features of NoSQL :

1. Dynamic schema: NoSQL databases do not have a fixed schema and can accommodate
changing data structures without the need for migrations or schema alterations.

2. Horizontal scalability: NoSQL databases are designed to scale out by adding more nodes
to a database cluster, making them well-suited for handling large amounts of data and
high levels of traffic.

3. Document-based: Some NoSQL databases, such as MongoDB, use a document-based

data model, where data is stored in semi-structured format, such as JSON or BSON.

4. Key-value-based: Other NoSQL databases, such as Redis, use a key-value data model,
where data is stored as a collection of key-value pairs.

5. Column-based: Some NoSQL databases, such as Cassandra, use a column-based data

model, where data is organized into columns instead of rows.

6. Distributed and high availability: NoSQL databases are often designed to be highly
available and to automatically handle node failures and data replication across multiple
nodes in a database cluster.

7. Flexibility: NoSQL databases allow developers to store and retrieve data in a flexible and
dynamic manner, with support for multiple data types and changing data structures.

8. Performance: NoSQL databases are optimized for high performance and can handle a
high volume of reads and writes, making them suitable for big data and real-time
applications.

Advantages of NoSQL:

There are many advantages of working with NoSQL databases such as MongoDB and
Cassandra. The main advantages are high scalability and high availability.

1. High scalability : NoSQL databases use sharding for horizontal scaling. Partitioning of
data and placing it on multiple machines in such a way that the order of the data is
preserved is sharding. Vertical scaling means adding more resources to the existing
machine whereas horizontal scaling means adding more machines to handle the data.
Vertical scaling is not that easy to implement but horizontal scaling is easy to implement.

12
 Examples of horizontal scaling databases are MongoDB, Cassandra, etc. NoSQL can
handle a huge amount of data because of scalability, as the data grows NoSQL scale
itself to handle that data in an efficient manner.

2. Flexibility: NoSQL databases are designed to handle unstructured or semi-structured data,

which means that they can accommodate dynamic changes to the data model. This makes
NoSQL databases a good fit for applications that need to handle changing data
requirements.

3. High availability : Auto replication feature in NoSQL databases makes it highly available
because in case of any failure data replicates itself to the previous consistent state.

4. Scalability: NoSQL databases are highly scalable, which means that they can handle
large amounts of data and traffic with ease. This makes them a good fit for applications
that need to handle large amounts of data or traffic

5. Performance: NoSQL databases are designed to handle large amounts of data and traffic,
which means that they can offer improved performance compared to traditional relational
databases.

6. Cost-effectiveness: NoSQL databases are often more cost-effective than traditional

relational databases, as they are typically less complex and do not require expensive
hardware or software.

When should NoSQL be used:

1. When a huge amount of data needs to be stored and retrieved.

2. The relationship between the data you store is not that important

3. The data changes over time and is not structured.

4. Support of Constraints and Joins is not required at the database level

5. The data is growing continuously and you need to scale the database regularly to handle
the data.

13
Key-Value Database

A key-value data model or database is also referred to as a key-value store. It is a non-

relational type of database. In this, an associative array is used as a basic database in which
an individual key is linked with just one value in a collection. For the values, keys are special
identifiers. Any kind of entity can be valued. The collection of key-value pairs stored on
separate records is called key-value databases and they do not have an already defined
structure.

When to use a key-value database:

Here are a few situations in which you can use a key-value database:-

 User session attributes in an online app like finance or gaming, which is referred to as
real-time random data access.

 Caching mechanism for repeatedly accessing data or key-based design.

 The application is developed on queries that are based on keys.

Features:

 One of the most un-complex kinds of NoSQL data models.

 For storing, getting, and removing data, key-value databases utilize simple functions.

 Querying language is not present in key-value databases.

 Built-in redundancy makes this database more reliable.

14
Advantages:

 It is very easy to use. Due to the simplicity of the database, data can accept any kind, or
even different kinds when required.

 Its response time is fast due to its simplicity, given that the remaining environment near
it is very much constructed and improved.

 Key-value store databases are scalable vertically as well as horizontally.

 Built-in redundancy makes this database more reliable.

Disadvantages:

 As querying language is not present in key-value databases, transportation of queries

from one database to a different database cannot be done.

 The key-value store database is not refined. You cannot query the database without a key.

Some examples of key-value databases:

Here are some popular key-value databases which are widely used:

 Couchbase: It permits SQL-style querying and searching for text.

 Amazon DynamoDB: The key-value database which is mostly used is Amazon

DynamoDB as it is a trusted database used by a large number of users. It can easily
handle a large number of requests every day and it also provides various security options.

 Riak: It is the database used to develop applications.

 Aerospike: It is an open-source and real-time database working with billions of

exchanges.

 Berkeley DB: It is a high-performance and open-source database providing scalability.

15
Graph Database

A graph database is a type of NoSQL database that is designed to handle data with complex
relationships and interconnections. In a graph database, data is stored as nodes and edges,
where nodes represent entities and edges represent the relationships between those entities.

 Graph databases are particularly well-suited for applications that require deep and
complex queries, such as social networks, recommendation engines, and fraud detection
systems. They can also be used for other types of applications, such as supply chain
management, network and infrastructure management, and bioinformatics.

 One of the main advantages of graph databases is their ability to handle and represent
relationships between entities. This is because the relationships between entities are as
important as the entities themselves, and often cannot be easily represented in a
traditional relational database.

 Another advantage of graph databases is their flexibility. Graph databases can handle
data with changing structures and can be adapted to new use cases without requiring
significant changes to the database schema. This makes them particularly useful for
applications with rapidly changing data structures or complex data requirements.

 However, graph databases may not be suitable for all applications. For example, they
may not be the best choice for applications that require simple queries or that deal
primarily with data that can be easily represented in a traditional relational database.
Additionally, graph databases may require more specialized knowledge and expertise to
use effectively.

Some popular graph databases include Neo4j, OrientDB, and ArangoDB. These databases
provide a range of features, including support for different data models, scalability, and high
availability, and can be used for a wide variety of applications.

The description of components are as follows:

 Nodes: represent the objects or instances. They are equivalent to a row in database. The
node basically acts as a vertex in a graph. The nodes are grouped by applying a label to
each member.

16
 Relationships: They are basically the edges in the graph. They have a specific direction,
type and form patterns of the data. They basically establish relationship between nodes.

 Properties: They are the information associated with the nodes.

Types of Graph Databases:

 Property Graphs: These graphs are used for querying and analyzing data by modelling
the relationships among the data. It comprises of vertices that has information about the
particular subject and edges that denote the relationship. The vertices and edges have
additional attributes called properties.

 RDF Graphs: It stands for Resource Description Framework. It focuses more on data
integration. They are used to represent complex data with well defined semantics. It is
represented by three elements: two vertices, an edge that reflect the subject, predicate and
object of a sentence. Every vertex and edge is represented by URI(Uniform Resource
Identifier).

When to Use Graph Database?

 Graph databases should be used for heavily interconnected data.

 It should be used when amount of data is larger and relationships are present.

 It can be used to represent the cohesive picture of the data.

Document Databases

A Document Data Model is a lot different than other data models because it stores data in
JSON, BSON, or XML documents. in this data model, we can move documents under one
document and apart from this, any particular elements can be indexed to run queries faster.
Often documents are stored and retrieved in such a way that it becomes close to the data
objects which are used in many applications which means very less translations are required
to use data in applications. JSON is a native language that is often used to store and query
data too.

So in the document data model, each document has a key-value pair below is an example for
the same.

17
{

"Name" : "Yashodhra",

"Address" : "Near Patel Nagar",

"Email" : "yahoo123@[Link]",

"Contact" : "12345"

Working of Document Data Model:

This is a data model which works as a semi-structured data model in which the records and
data associated with them are stored in a single document which means this data model is not
completely unstructured. The main thing is that data here is stored in a document.

Features:

 Document Type Model: As we all know data is stored in documents rather than tables or
graphs, so it becomes easy to map things in many programming languages.

 Flexible Schema: Overall schema is very much flexible to support this statement one
must know that not all documents in a collection need to have the same fields.

 Distributed and Resilient: Document data models are very much dispersed which is the
reason behind horizontal scaling and distribution of data.

 Manageable Query Language: These data models are the ones in which query language
allows the developers to perform CRUD (Create Read Update Destroy) operations on the
data model.

Examples of Document Data Models :

 Amazon DocumentDB
 MongoDB
 Cosmos DB
 ArangoDB
 Couchbase Server
 CouchDB

18