HDFS

Distributed File System

When the dataset exceeds the storage capacity of a single machine, then it becomes
mandatory to partition the dataset across several separate machines. The filesystem that
manages the data across the network of machines is called a distributed filesystem.

A distributed filesystem is a filesystem that allows us to store data across multiple machines
or nodes in a cluster and allows multiple users to access data.

Since the DFS is based on the network, all the complications of network programming kick
in, making a distributed file system more complex than the regular filesystem. One of the
biggest challenges in DFS is to tolerate node failure without su ering data loss.

Hadoop comes with a distributed filesystem called Hadoop Distributed Filesystem for
storing vast amounts of data while providing fault tolerance and high availability.

HDFS Tutorial – Introduction

Hadoop Distributed FileSystem (HDFS) is a java based distributed file system used in
Hadoop for storing a large amount of structured or unstructured data, ranging in size from
GigaBytes to PetaBytes, across a cluster of commodity hardware. It is the most reliable
storage known to date on the planet.

In HDFS, data is stored in multiple locations, so if any of the machines fails, then data can
be fetched from other machine containing the copy of data. Thus it is highly fault-tolerant
and ensures no data loss even in the case of hardware failure.

It is the major component of Hadoop, along with MapReduce, YARN, and other common
utilities.

It follows a Write-Once-Read-Many philosophy that simplifies data coherency and enables

high-throughput access.

Why HDFS?

In today’s IT world, almost 75% of the world’s data resides in Hadoop HDFS. It is due to the
following reason:

 HDFS stores data across the commodity hardware due to which there is no need for
high-end machines for storing big data. Thus provides economical storage for storing
big data.
  HDFS follows the most e icient data processing pattern that is Write-Once-Read-
Many-Times pattern. A dataset generated from various sources are copied, and then
the various analysis is performed on that dataset over time. So, it is best for batch
processing.

 HDFS can store data of any size generated from any source in any formats, either
structured or unstructured.

 Its write-one-read-many model relaxes the concurrency control requirements. The

data can be accessed multiple times without any issue regarding data coherency.

 HDFS works on the data locality assumption that is moving of computation to data is
much easier and faster than moving data to the computational unit. HDFS facilitates
locating processing logic near the data rather than moving data to the application
space. Thus this reduces network congestion and overall turnaround time.

So, moving ahead in this HDFS tutorial, let us jump to HDFS Architecture.

HDFS Architecture

Hadoop DFS follows master-slave architecture. The HDFS consists of two types of nodes
that are master node and slave nodes. The master node manages the file system
namespace, that is, it stores the metadata about the blocks of files.

The slave nodes store the user data and are responsible for processing data based on the
instruction from the master node.

HDFS Master

Master in HDFS is the centerpiece of Hadoop HDFS. They are the high-end machines that
store metadata related to all the files stored in HDFS. It manages and maintains the
filesystem namespace and provides instructions to the slave nodes.

The NameNode is the master node in Hadoop HDFS.

HDFS Slave

Slave Nodes are responsible for storing the actual business data. They are the normal
configuration machines (commodity hardware) that stores and processes the datasets upon
instruction from the master node.

The DataNodes are the slave nodes in Hadoop HDFS.

HDFS NameNode

NameNode is the master node. It manages filesystem namespace operations like

opening/closing, renaming files, and directories. NameNode maps data blocks to
DataNodes and records each change made to the filesystem namespace.

HDFS DataNode

DataNodes are the slave nodes that handle read/write requests from HDFS clients.
DataNodes creates, deletes, and replicates data blocks as per the instructions from the
governing name node.

Wondering how data gets stored in HDFS?

Blocks in HDFS

HDFS split the files into block-size chunks called data blocks. These blocks are stored
across multiple DataNodes in the cluster. The default block size is 128 MB. We can configure
the default block size, depending on the cluster configuration.

For the cluster with high-end machines, the block size can be kept large (like 256 Mb or
more). For the cluster with machines having configuration like 8Gb RAM, the block size can
be kept smaller (like 64 Mb).

Also, HDFS creates replicas of blocks based on the replication factor ( a number that defines
the total copies of a block of a file). By default, the replication factor is 3. It means that 3
copies of each block are created and stored across multiple nodes.

If any of the DataNode fails, then the block is fetched from other DataNode containing a
replica of a block. This makes HDFS fault tolerance.

Have you thought how NameNode discovers DataNode failure?

DataNode Failure

All the DataNodes in Hadoop HDFS continuously sends a small heartbeat message (signals)
to NameNode to tell “I am Alive” in every 3 seconds.

If NameNode does not get a heartbeat message from any particular DataNode for more than
10 minutes, then it considers that DataNode as dead and starts creating a replica of blocks
that were available on that DataNode.

NameNode instructs the DataNodes containing a copy of that data to replicate that data on
other DataNodes to balance the replication. In this way, NameNode discovers DataNode
failure.
Want to know how NameNode places replicas on di erent DataNode? Let us explore rack
awareness in HDFS to get an answer to the above question.

Rack Awareness in HDFS

Hadoop HDFS stores data across the cluster of commodity hardware. To provide fault
tolerance, replicas of blocks are created and stored on di erent DataNodes.

NameNode places the replicas of blocks on multiple DataNodes by following the Rack
Awareness algorithm to ensure no data loss even if DataNode or the whole rack goes down.
The NameNode places the first replica on the nearest DataNode.

It stores the second replica on di erent DataNode on the same rack and the third replica on
di erent DataNode on a di erent rack.

If the replication factor is 2, then it places the second replica on a di erent DataNode on a
di erent rack so that if a complete rack goes down, then also the system will be highly
available.

The main purpose of a rack-aware replica placement policy is to improve fault tolerance,
data reliability, availability.

Next in the HDFS tutorial, we discuss some key features of Hadoop HDFS.

Important Features of Hadoop HDFS

1. High Availability

It is a highly available file system. In this file system, data gets replicated among the nodes
in the Hadoop cluster by creating a replica of the blocks on the other slaves present in the
HDFS cluster. So, whenever a user wants to access this data, they can access their data from
the slaves, which contain its blocks.

2. Fault Tolerance

Fault tolerance in Hadoop HDFS is the working strength of a system in unfavorable

conditions. It is highly fault-tolerant. Hadoop framework divides data into blocks.

After that, it creates multiple copies of blocks on di erent machines in the cluster. So, when
any machine in the cluster goes down, then a client can easily access their data from the
other machine, which contains the same copy of data blocks.
3. High Reliability

HDFS provides reliable data storage. It can store data in the range of 100s of petabytes. HDFS
stores data reliably on a cluster. It divides the data into blocks. Then, the Hadoop framework
stores these blocks on nodes present in the cluster.

HDFS also stores data reliably by creating a replica of each and every block present in the
cluster. Hence provides fault tolerance facility.

4. Replication

Data Replication is a unique feature of HDFS. Replication solves the problem of data loss in
an unfavorable condition like hardware failure, crashing of nodes, etc. HDFS maintains the
process of replication at a regular interval of time.

It also keeps creating replicas of user data on di erent machines present in the cluster. So,
when any node goes down, the user can access the data from other machines. Thus, there
is no possibility of losing of user data.

5. Scalability

It stores data on multiple nodes in the cluster. So, whenever requirements increase, you can
scale the cluster. Two scalability mechanisms are available in HDFS: Vertical and Horizontal
Scalability.

6. Distributed Storage

HDFS features are achieved via distributed storage and replication. It stores data in a
distributed manner across the nodes. In Hadoop, data is divided into blocks and stored on
the nodes present in the cluster.

After that, it creates the replica of each and every block and store on other nodes. When the
single machine in the cluster gets crashed, we can easily access our data from the other
nodes which contain its replica.

Next in the HDFS tutorial, we discuss some useful HDFS operations.

HDFS Operation

Hadoop HDFS has many similarities with the Linux file system. We can do almost all the
operation we can do with a local file system like create a directory, copy the file, change
permissions, etc.

It also provides di erent access rights like read, write and execute to users, groups, and
others.
1. Read Operation

When the HDFS client wants to read any file from HDFS, the client first interacts with
NameNode. NameNode is the only place that stores metadata. NameNode specifies the
address of the slaves where data is stored. Then, the client interacts with the specified
DataNodes and read the data from there.

HDFS client interacts with the distributed file system API. Then, it sends a request to
NameNode to send a block location. NameNode first checks if the client has su icient
privileges to access the data or not? After that, NameNode will share the address at which
data is stored in the DataNode.

NameNode provides a token to the client, which it shows to the DataNode for reading the file
for security purposes. When a client goes to DataNode for reading the file, after checking the
token, DataNode allows the client to read that particular block.

After that client opens the input stream and starts reading data from the specified
DataNodes. Thus, in this manner, the client reads data directly from DataNode.

2. Writing Operation

For writing a file, the client first interacts with NameNode. HDFS NameNode provides the
address of the DataNode on which data has to be written by the client.

When the client finishes writing the block, the DataNode starts replicating the block into
another DataNode. Then it copies the block to the third DataNode. Once it creates required
replication, it sends a final acknowledgment to the client. The authentication is the same as
the read operation.

The client just sends 1 copy of data irrespective of our replication factor, while DataNodes
replicate the blocks. Writing of file is not costly because it writes multiple blocks parallelly
multiple blocks on several DataNodes.

Summary

In the HDFS tutorial conclusion, we can say that Hadoop HDFS stores data in a distributed
manner across the cluster of commodity hardware.

Hadoop HDFS is a highly reliable, fault-tolerant, and highly available storage system known
to date. It follows the master-slave architecture where NameNode is the master node, and
the DataNodes are the slave nodes.

Also, the HDFS splits the client’s input file into blocks of size 128 MB, which we can configure
as per our requirement. It also stores replicas of blocks to provide fault tolerance.
NameNode follows rack awareness policy for placing replicas on DataNode to ensure that
no data is lost during machine failure or hardware failure. In addition, the DataNodes sends
a heartbeat message to NameNode to ensure that they are alive.

During file read or write, the client first interacts with the NameNode.

The Hadoop HDFS is scalable, reliable, distributed, fault-tolerant, and highly available
storage system for storing big data.

*****
 What / Why Hadoop, History, Core components of Hadoop

What is Hadoop?

It is an open source software framework for distributed storage & processing of huge
amount of data sets. Open source means it is freely available and even we can change its
source code as per your requirements.

It also makes it possible to run applications on a system with thousands of nodes. It’s
distributed file system has the provision of rapid data transfer rates among nodes. It also
allows the system to continue operating in case of node failure.

Hadoop provides-

 The Storage layer – HDFS

 Batch processing engine – MapReduce

 Resource Management Layer – YARN

Hadoop – History

In 2003, Google launches project Nutch to handle billions of searches. Also for indexing
millions of web pages. In October 2003 Google published GFS (Google File System) paper,
from that paper Hadoop was originated.

In 2004, Google releases paper with MapReduce. And in 2005, Nutch used GFS and
MapReduce to perform operations.

In 2006, Computer scientists Doug Cutting and Mike Cafarella created Hadoop. In
February 2006 Doug Cutting joined Yahoo. This provided resources and the dedicated team
to turn Hadoop into a system that ran at web scale. In 2007, Yahoo started using Hadoop
on a 100 node cluster.

In January 2008, Hadoop made its own top-level project at Apache, confirming its success.
Many other companies used Hadoop besides Yahoo!, such as the New York Times and
Facebook.

In April 2008, Hadoop broke a world record to become the fastest system to sort a terabyte
of data. Running on a 910-node cluster, In sorted one terabyte in 209 seconds.

In December 2011, Apache Hadoop released version 1.0. In August 2013, version 2.0.6 was
available. Later in June 2017, Apache Hadoop 3.0.0-alpha4 is available. ASF (Apache
Software Foundation) manages and maintains Hadoop’s framework and ecosystem of
technologies.
Why Hadoop?

As we have learned the Introduction, Now we are going to learn what is the need of
Hadoop?

It emerged as a solution to the “Big Data” problems-

a. Storage for Big Data – HDFS Solved this problem. It stores Big Data in Distributed
Manner. HDFS also stores each file as blocks. Block is the smallest unit of data in a
filesystem.

Suppose you have 512MB of data. And you have configured HDFS such that it will create
128Mb of data blocks. So HDFS divide data into 4 blocks (512/128=4) and stores it across
di erent DataNodes. It also replicates the data blocks on di erent datanodes.

Hence, storing big data is not a challenge.

b. Scalability – It also solves the Scaling problem. It mainly focuses on horizontal scaling
rather than vertical scaling. You can add extra datanodes to HDFS cluster as and when
required. Instead of scaling up the resources of your datanodes.

Hence enhancing performance dramatically.

c. Storing the variety of data – HDFS solved this problem. HDFS can store all kind of data
(structured, semi-structured or unstructured). It also follows write once and read many
models.

Due to this, you can write any kind of data once and you can read it multiple times for
finding insights.

d. Data Processing Speed – This is the major problem of big data. In order to solve this
problem, move computation to data instead of data to computation. This principle is Data
locality.

Hadoop Core Components

Now we will learn the Apache Hadoop core component in detail. It has 3 core components-

 HDFS

 MapReduce

 YARN(Yet Another Resource Negotiator)

Let’s discuss these core components one by one.

a. HDFS
Hadoop distributed file system (HDFS) is the primary storage system of Hadoop. HDFS
store very large files running on a cluster of commodity hardware. It follows the principle of
storing less number of large files rather than the huge number of small files.

Stores data reliably even in the case of hardware failure. It provides high-throughput access
to the application by accessing in parallel.

Components of HDFS:

 NameNode –It works as Master in the cluster. Namenode stores meta-data. A

number of blocks, replicas and other details. Meta-data is present in memory in the
master. NameNode maintains and also manages the slave nodes, and assigns tasks
to them. It should deploy on reliable hardware as it is the centerpiece of HDFS.

 DataNode – It works as Slave in the cluster. In HDFS, DataNode is responsible for

storing actual data in HDFS. DataNode performs read and write operation as per
request for the clients. DataNodes can also deploy on commodity hardware.

b. MapReduce

MapReduce is the data processing layer of Hadoop. It processes large structured and
unstructured data stored in HDFS. MapReduce also processes a huge amount of data in
parallel.

It does this by dividing the job (submitted job) into a set of independent tasks (sub-job).
MapReduce works by breaking the processing into phases: Map and Reduce.

 Map – It is the first phase of processing, where we specify all the complex logic
code.

 Reduce –It is the second phase of processing. Here we specify light-weight

processing like aggregation/summation.

c. YARN

YARN provides the resource management. It is the operating system of Hadoop. It

responsible for managing and monitoring workloads, also implementing security controls.
Apache YARN is also a central platform to deliver data governance tools across the
clusters.

YARN allows multiple data processing engines such as real-time streaming, batch
processing etc.

Components of YARN:
  Resource Manager – It is a cluster level component and runs on the Master
machine. It manages resources and schedule applications running on the top of
YARN. It has two components: Scheduler & Application Manager.

 Node Manager –It is a node level component. It runs on each slave machine. It
continuously communicate with Resource Manager to remain up-to-date

Advantages of Hadoop

Let’s now discuss various Hadoop advantages to solve the big data problems.

 Scalability –By adding nodes we can easily grow our system to handle more data.

 Flexibility – In this framework, you don’t have to preprocess data before storing it.
You can store as much data as you want and decide how to use later.

 Low-cost – Open source framework is free and runs on low-cost commodity

hardware.

 Fault tolerance – If nodes go down, then jobs are automatically redirected to other
nodes.

 Computing power – It’s distributed computing model processes big data fast. The
more computing nodes you use more processing power you have.

Disadvantages of Hadoop

Some Disadvantage of Apache Hadoop Framework is given below-

 Security concerns – It can be challenging in managing the complex application. If

the user doesn’t know how to enable platform who is managing the platform, then
your data could be a huge risk. Since, storage and network levels Hadoop are
missing encryption, which is a major point of concern.

 Vulnerable by nature – The framework is written almost in java, most widely used
language. Java is heavily exploited by cybercriminals. As a result, implicated in
numerous security breaches.

 Not fit for small data –Since, it is not suited for small data. Hence, it lacks the
ability to e iciently support the random reading of small files.

 Potential stability issues – As it is an open source framework. This means that it is

created by many developers who continue to work on the project. While constantly
improvements are made, It has stability issues. To avoid these issues organizations
should run on the latest stable version.
Conclusion

In conclusion, we can say that it is the most popular and powerful Big data tool. It stores
huge amount of data in the distributed manner.

And then processes the data in parallel on a cluster of nodes. It also provides world’s most
reliable storage layer- HDFS. Batch processing engine MapReduce and Resource
management layer- YARN.

Hence, these daemons ensure Hadoop functionality.

 Big Data Job Roles

1. Big Data Analyst

The Big Data analyst examines huge amounts of information with a complex heterogeneous
or uncertain structure (research results, market trends, customer preferences, etc.).

The analysis of the data can give a di erent level of understanding of the subject of research
and the observed phenomena. The result brings discoveries and new technologies,
substances, and approaches to the phenomena of various spheres of life.

The main competence of a specialist in Big Data is the ability to see logical connections in
the arrays of collected information, and based on this, develop new approaches and
solutions.

With the advent of technologies that make it possible to create multi-gigabyte arrays of
information faster, the profession of Big Data analyst is becoming more and more popular.

Big Data engineers are responsible for storing, transforming, and quickly accessing data. Big
Data analysts are responsible for analyzing data, identifying relationships, and building
models.

2. Database Administrator

The database administrator creates the databases using a DBMS (database management
system), for example, Microsoft SQL Server and Oracle Database.
Having designed the database, the administrator arranges its work, monitors the smooth
operation of the server, and provides users with access to the necessary information,
security, and guidance for their work.

In their work, the database administrators demonstrate knowledge of programming

languages, basic network protocols, and services. Most of the working day passes in front of
a computer monitor.

The first responsibility of the database administrator is to ensure the functioning of the
database.

According to the requirements of the customer, in accordance with the needs of the
company and the features of its activities, the administrator designs and sets up the
database. He or she controls the server, improves the process of storing and processing
information, and provides uninterrupted user access to data.

The database administrator is also responsible for ensuring its security. In an emergency, he
or she must quickly recover the lost data.

The administrator must update the database and back up information from time to time. The
experience of working as a database administrator will help this specialist to apply for a job
as a database management system engineer.

3. Database Developer

A database developer is a specialist who creates databases, debugs them, modernizes

them, and maintains them. To work in this area, higher technical education is required, as
well as knowledge of the SQL query language. Specialists who decide to work in this
profession perform the following tasks:

 Database design (selection of the right tools, analysis of the needs of the company’s
system, etc.)

 Ensuring the e icient use of the created database

 Debugging and maintenance

 Securing data stored on the server

 Analysis of user complaints and suggestions, troubleshooting

 Consultations for system administrators

 Load analysis and updating the database engine

 Collaboration with administrators, programmers, and database architects

These specialists can be ordered to upgrade and subsequently maintain an existing
database in order to increase its performance and security. This employee, if the company
does not have a system administrator, works with the team, where he or she will conduct
consultations and receive complaints.

4. Data Scientist

A data scientist is a specialist in the processing, analyzing, and storage of large amounts of
data, the so-called Big Data.

The data scientist extracts the necessary information from a wide variety of sources using
real-time information flows. He or she finds hidden patterns in data arrays and statistically
analyzes them to make competent business decisions.

The workplace of such a specialist is not one computer or even one server, but a cluster of
servers.

The data scientist uses various methods of working with data:

 Statistical methods

 Database modeling

 Mining methods

 Artificial intelligence applications for working with data

 Database design and development methods

5. Data Architect

A database architect is an experienced technician who manages a huge group of employees.

He or she is engaged in selecting technology for data storage, creating and optimizing
queries, drawing up development plans and technical specifications for subordinates,
designing and optimizing databases, and monitoring database security.

Database architects are valuable specialists who not only develop the structure but also
know how to administer and program the created databases.

The profession is quite complicated because the specialist has to do his or her job in such a
way as to please the management, the technical department, and the employees of the
company.

Everyone puts forward individual requirements, and the database architect should be able
to explain to everyone what can be implemented in the project and what cannot.
In order to get a well-paid job, you need to have an appropriate education. Of course,
education is not easy, and sometimes students need help.

For example, if you need essay help from essay writers, you need to choose an essay writing
service like EssayShark. In this way, getting an education will be much easier.

Summary

If you want to choose a profession related to Big Data, select one of the job roles in big data
to earn money. However, you should know that achieving success in any profession in this
sphere is quite di icult.

*****
 Big Data Analytics Techniques and Types

What is data analysis?

Data analysis, or data analytics, is the process of applying logical and mathematical
techniques to datasets in order to discover patterns and useful information, often to aid in
decision-making.3

It’s often used on an industrial scale, helping organizations make calculated and informed
business decisions. As the Internet of Things (IoT) expands and technology develops, new
forms of data mining and analysis are constantly emerging.
Big data is characterized by the three V’s: Volume, the massive scale of data; velocity, the
speed at which it’s generated and processed; and variety, the wide range of structured and
unstructured data formats.4
Velocity is particularly important, as the need for real-time analysis has led to the
integration of big data with advanced technologies like machine learning and artificial
intelligence.

What are the types of data analysis?

The di erent types of data analysis provide a structured approach to gaining insights from
data, moving from understanding what has happened to predicting what will happen and,
finally, prescribing a course of action. The four main types of data analysis are: 5

Descriptive analytics: Summarizes historical data to explain what happened. This could
include generating reports, KPI dashboards, and summaries. It helps you understand the
past and present state of data.

Diagnostic analytics: Focuses on historical data to determine why something happened.

It involves techniques like regression analysis and A/B testing to find the variables that lead
to a specific outcome.

Predictive analytics: Uses statistical models and machine learning to forecast what is
likely to happen in the future. It can be used for tasks like sales forecasting, risk
assessment, and identifying behavior trends.

Prescriptive analytics: Recommends what should be done to achieve a desired outcome.

It leverages advanced techniques like simulation and natural language processing.

10 examples of big data analytics techniques

Big data analytics techniques function in a two-fold manner: processing data streams as
they emerge and performing batch analysis on data as it accumulates to identify patterns
and trends. As data generation accelerates, these techniques must evolve to handle the
speed, scale, and depth of information.
1. Data mining

Data mining extracts patterns from large data sets by combining methods from statistics
and machine learning, within database management. Today, it is increasingly automated
and integrated with AI, allowing for more complex pattern detection. 6

 Tools: Data-mining tools include programming languages like Python or R, and

proprietary tools like KNIME and RapidMiner, which o er visual workflows and pre-
built algorithms.

 Example: A retail company might use data mining to analyze customer purchasing
histories and identify which segments are most likely to respond to a new product
promotion.

2. Data visualization

Data visualization is the graphical representation of data and information. Through visual
elements like charts, graphs, and dashboards, analysts communicate insights and help
stakeholders make decisions. It can also make complex data understandable and
accessible to non-technical users.7

 Tools: Data visualization softwares like Tableau and Power BI are powerful packages
with built-in charting tools. There are also coding libraries that can power highly-
customized visualizations, like D3.

 Example: A business dashboard can display real-time sales data using a line chart
to show a trend over time or a heat map to show customer density.

3. Cluster analysis

Cluster analysis is a type of data mining that uses unsupervised machine learning in order
to group data points into distinct clusters based on their similarities. The goal is to identify
hidden groupings or structures within the data without any pre-existing labels. 8

 Tools: Common tools and libraries for cluster analysis include Scikit-learn in Python
and R.

 Example: A bank might use cluster analysis to segment customers’ behavior and
detect fraudulent transactions by identifying groupings of unusual activity.

4. A/B testing

This data analysis technique involves comparing a control group with a variety of test
groups to discern what changes will improve or change a given objective variable. 9
  Tools: Software and platforms like Adobe Target, A/B Smartly, and VWO o er
features for creating, monitoring, and analyzing a variety of web-based A/B tests.

 Example: A marketing team could use A/B testing to determine which website
layout or ad copy leads to the highest number of conversions.

5. Regression analysis

Regression analysis is a statistical method for estimating the relationships among

variables. It’s used to understand how the value of a dependent variable changes when one
of the independent variables is varied. 10

 Tools: Statistical software like SAS and SPSS, and programming languages like R
and Python are used for regression analysis.

 Example: A real estate agent could use regression analysis to determine the
relationship between house size and selling price.

6. T-tests

A t-test is a statistical hypothesis test used to determine if there is a significant di erence

between the averages of two groups. It helps to see if they are genuinely di erent or if the
observed di erence is due to chance.11

 Tools: T-tests are available in statistical software like R and Python’s SciPy library.

 Example: A teacher could use a t-test to compare the average test scores of
students who tried a new study method versus those who tried an old method, to
see if there is a meaningful di erence in their performance.

7. Machine learning

Machine learning automates model building for analytics. It enables computers to

automatically learn from data without explicit programming to make inferences,
predictions, and recommendations.12

 Tools: Popular tools and platforms for machine learning include Databricks, KNIME,
and cloud-based services like Google Cloud AI and Amazon SageMaker.

 Example: Machine learning can be used by banks to detect fraud and identify
suspicious credit card transactions automatically.

8. Time-series analysis

Time-series analysis is a statistical technique that analyzes data points collected over a
period of time. The goal is to identify patterns, trends, and seasonal changes in the data. 13
  Tools: Some tools for time-series analysis include Python libraries like Pandas,
Statsmodels, and specialized platforms like Amazon Forecast.

 Example: Weather forecasters might use this kind of analysis to predict future
weather conditions based on past temperature, pressure, and wind patterns.

9. Decision trees

Decision trees are a type of supervised machine learning algorithm that can be used for
both classification and regression. It works like a flow chart, identifying optimal split points
based on feature values to create pure subsets.14

 Tools: Decision-tree algorithms are implemented using various machine learning

libraries, including Scikit-learn in Python and R.

 Example: A business could use a decision tree to predict whether or not a customer
will buy a product, based on factors such as previous purchasing history, age, and
location.

10. Natural language processing

Natural language processing (NLP) is a subset of AI and machine learning that uses
algorithms to analyze, understand, and generate human language. Large language models
(LLMs) and generative AI enable tools to process massive amounts of unstructured text
data, such as emails, social media posts, and customer reviews. 15

 Tools: Popular NLP tools include programming libraries like NLTK and spaCy, and
the IBM Watson platform.

 Example: NLP is used for machine translation services, like Google Translate, to
process input text and return that same information in a number of di erent
languages.
 Characteristics of Big Data (Conitnued…)

Big data empowers organizations to analyze trends, uncover insights, and predict future
outcomes with unprecedented accuracy. From personalized recommendations on e-commerce
platforms to predictive maintenance in manufacturing, big data plays a critical role in shaping the
digital transformation journey.

What Are the Characteristics of Big Data?

Big data is characterized by its unique attributes, commonly referred to as the 5Vs framework:

 Volume: The sheer scale of data generated every second is a defining feature of big data.
For instance, it is estimated that in 2024, humans will generate 94 zettabytes of data
globally.

 Velocity: Big data is generated at unprecedented speeds. Real-time data streams from IoT
devices, social media platforms, and online transactions are prime examples.

 Variety: Big data includes a mix of structured, unstructured, and semi-structured formats,
such as text, images, videos, and sensor-generated data. This variety allows businesses to
derive insights from multiple dimensions.

 Veracity: The accuracy and reliability of big data can vary, making data validation and
cleansing essential steps in the analytics process.

 Value: Ultimately, the goal of big data is to derive meaningful insights that drive
innovation, improve customer experiences, and generate revenue.

By understanding these characteristics, businesses can harness the true potential of big data. For
instance, a retailer analyzing customer buying patterns across multiple channels can use insights
to enhance inventory planning and marketing strategies.
Elements (continued)

As organizations continue to embrace digital transformation, the importance of big data and its
defining features is only expected to grow.

In 2024, industries that effectively utilize big data report 20-30% improvements in
operational efficiency and customer satisfaction, according to McKinsey’s analytics division.
This trend underscores the necessity for businesses to adopt a data-first approach in navigating
the complexities of the modern world.

6. Variability – Inconsistency of Data

Variability refers to the changes in data flow rates, formats, or meanings over time. For instance,
during sales events or trending news, data may spike drastically. Additionally, words or behavior
may have different meanings in different contexts, making interpretation harder.

Example: if you are eating same ice-cream daily and the taste just keep changing.
or
A tweet saying "bad" might be negative, or in slang, it could mean "good." This inconsistency
must be addressed when analyzing large, dynamic datasets.
 Elements – Components - Importance - Connection with Industry
Elements of Big Data
The core elements of big data are known as the Five V's: Volume, Velocity, Variety, Veracity,
and Value. These characteristics define what makes data "big" and how it is effectively managed
and analyzed to extract meaningful insights.

The Five V's of Big Data with Examples

1. Volume

Volume refers to the massive amount of data generated and stored, which often reaches petabytes
or zettabytes and exceeds the capacity of traditional databases.

 Example: Social media platforms like Facebook process billions of interactions, photo
uploads, and comments daily.

 Example: Internet of Things (IoT) devices in a smart factory generating sensor readings
from hundreds of machines every second for predictive maintenance.

2. Velocity

Velocity is the speed at which data is generated, collected, and processed, often in real-time or
near real-time, requiring rapid analysis.

 Example: Financial trading platforms processing millions of transactions per second to

dynamically adjust portfolios and detect fraud instantly.

 Example: Ride-hailing apps like Uber using real-time GPS data to optimize routes and
reduce customer wait times.

3. Variety

Variety refers to the diverse range of data types, formats, and sources. This includes structured
data (like relational databases), semi-structured data (like XML or JSON files), and unstructured
data (like text, images, and audio files).

 Example: A healthcare system integrating structured patient records, semi-structured lab

results (JSON files), and unstructured medical images (X-rays, MRIs) to provide holistic
patient insights.

 Example: Retailers combining structured sales data with unstructured social media
comments to perform sentiment analysis for product development.
4. Veracity

Veracity is the quality, accuracy, and trustworthiness of the data. Big data can be messy and
inconsistent due to its diverse sources, so ensuring data quality is crucial for reliable analysis and
decision-making.

 Example: In market research, ensuring that social media data used for sentiment analysis
is cleansed of slang, typos, and human bias to avoid inaccurate conclusions about a
product's public perception.

 Example: A credit card company validating transaction data to ensure accuracy and
prevent false positives in their fraud detection systems.

5. Value

Value refers to the ability to transform big data into valuable insights that drive business
decisions, improve operations, and enhance customer experiences. Without extracting value, the
data is useless.
 Example: Netflix analyzing user viewing habits to provide personalized content
recommendations, which increases user engagement and retention.
 Example: Manufacturers using sensor data to predict equipment failures before they
happen, enabling predictive maintenance, reducing downtime, and saving costs.
6. Variability – Inconsistency of Data

Variability refers to the changes in data flow rates, formats, or meanings over time. For instance,
during sales events or trending news, data may spike drastically. Additionally, words or behavior
may have different meanings in different contexts, making interpretation harder.

Example: if you are eating same ice-cream daily and the taste just keep changing.
or
A tweet saying "bad" might be negative, or in slang, it could mean "good." This inconsistency
must be addressed when analyzing large, dynamic datasets.

What is Big Data?

Big data refers to the massive volume of structured, semi-structured, and unstructured data that is
too large or complex to be processed by traditional data-processing techniques. With the
proliferation of digital devices, social media, and IoT (Internet of Things), data is being
generated at an unprecedented rate.

According to a report by IDC, the global datasphere is expected to reach 175 zettabytes (ZB)
by 2025, with 90% of the data in the world being unstructured.

This immense volume of data provides valuable insights when analyzed effectively, giving
organizations the opportunity to make more informed decisions, improve operations, and
enhance customer experiences.

Big data involves data sets that cannot be handled by traditional relational databases because of
their sheer size or complexity. This includes data from a variety of sources such as social media
platforms, transactional records, medical records, sensor data from IoT devices, and much more.

It is characterized not just by volume, but also by its velocity (speed of data generation), variety
(types of data), and veracity (accuracy of data), often referred to as the "3 Vs" of big data. With
advancements in data storage and processing technologies, businesses and organizations now
have the ability to extract actionable insights from these vast datasets.
Key Characteristics of Big Data (5Vs Framework)

Big data is defined not only by its size but also by its unique characteristics, which set it apart
from traditional data sets. These characteristics, often referred to as the 5Vs of Big Data, include
Volume, Velocity, Variety, Veracity, and Value.

Understanding these characteristics is crucial for organizations that aim to leverage big data for
analytics and decision-making. Below, we delve deeper into each of these key elements.

1. Volume: The Size of Data

The Volume of data refers to the sheer amount of data generated every second. With the
proliferation of IoT devices, social media, sensors, and digital transactions, the volume of data is
growing exponentially.

This vast volume of data comes from various industries such as healthcare, finance, retail, and
manufacturing, where data is continuously generated.

For example, in healthcare, millions of patient records, diagnostic images, and real-time health
data are generated daily. Similarly, in e-commerce, every user action from product searches to
purchases contributes to a growing dataset that can be analyzed for consumer behavior patterns.

The ability to store and analyze such massive amounts of data is what makes big data analytics
so valuable, as it can provide deeper insights into customer preferences, operational efficiencies,
and market trends.

2. Velocity: The Speed of Data Generation

Big data is not just about the size of data, but how quickly it flows into systems. Data is
produced at an unprecedented speed, driven by factors like real-time data feeds, social media,
and IoT devices.
For example, in stock markets, millions of transactions are processed in fractions of a second.
Real-time data analytics can identify patterns or anomalies in these transactions, which could
influence financial decisions.

Similarly, IoT devices, such as smart sensors in industrial machines, generate real-time data on
equipment performance, which can be used for predictive maintenance. With tools like Apache
Kafka and Apache Flink, companies can manage the high velocity of data and enable real-time
analytics, thus driving quicker decision-making.

3. Variety: The Types of Data

Big data comes in many different forms, including structured data (e.g., data in tables), semi-
structured data (e.g., logs or JSON files), and unstructured data (e.g., videos, audio, social media
posts).

Traditional datasets typically consist of structured data that fits neatly into relational tables, but
big data encompasses a wider range of data types, making it more complex to handle.

For instance, companies in the retail industry collect structured data from customer transactions,
semi-structured data from customer reviews, and unstructured data from social media platforms.
Leveraging data visualization tools such as Tableau or Power BI helps businesses to integrate
and analyze these diverse data types to create actionable insights.

4. Veracity: The Uncertainty of Data

Big data is not always clean and consistent, which can lead to questions regarding its quality and
trustworthiness. This veracity refers to the uncertainty and ambiguity inherent in some data,
especially when sourced from unstructured formats like social media or IoT sensors.

Traditional datasets tend to be cleaner, more reliable, and verified before they are used, whereas
big data often needs advanced cleansing and validation techniques before it can be fully
leveraged.
For example, in customer data, there might be inconsistent or erroneous entries, such as incorrect
addresses or duplicate records, that need to be identified and corrected before analysis.

Ensuring data veracity is critical because any analysis performed on faulty data could lead to
misleading or incorrect insights, ultimately affecting business decisions. As reported by
Gartner, poor data quality costs organizations around $15 million annually.

5. Value: The Insights That Drive Business Decisions

Finally, Value refers to the importance of deriving actionable insights from big data. The value of
big data is not just in its size, but in its ability to help businesses make better decisions, improve
customer experience, enhance operational efficiency, and ultimately increase profitability.

For example, Amazon uses big data analytics to analyze consumer behavior and predict what
products customers are likely to purchase next. By leveraging this data, Amazon can make
personalized recommendations and drive sales.

Similarly, businesses in marketing can use big data to segment customers based on behavior and
tailor campaigns to specific audiences, improving conversion rates.

How Businesses and Industries Rely on Big Data for Decision-Making

Businesses and industries across sectors leverage big data to gain a competitive edge. By
analyzing patterns and correlations in data, companies can make informed decisions that enhance
efficiency, customer satisfaction, and profitability. Some key applications include:

 Retail and E-commerce: Big data is used to analyze customer preferences, optimize
inventory, and deliver personalized shopping experiences. Companies like Amazon rely
heavily on big data for dynamic pricing and product recommendations.

 Healthcare: Big data analytics aids in early disease detection, patient monitoring, and
drug discovery. For instance, AI-driven tools analyze medical records and imaging data to
improve diagnostic accuracy.
  Finance: Banks and financial institutions utilize big data for fraud detection, risk
management, and customer segmentation. Advanced analytics enable them to predict
market trends and personalize services.

 Transportation: In logistics and supply chain management, big data optimizes delivery
routes, reduces costs, and improves operational efficiency. Ride-hailing platforms like
Uber depend on real-time data for driver and rider matching.

 Education: Big data helps educational institutions track student performance, customize
learning experiences, and improve outcomes.

According to a 2023 Gartner report, over 90% of Fortune 500 companies use big data
analytics to drive strategic decision-making. This reliance underlines how critical big data has
become for staying competitive in today’s data-driven world.

Importance of Understanding the Elements of Big Data for Analytics and Applications

Understanding the key elements of big data — volume, variety, velocity, and veracity — is
essential for leveraging it effectively in analytics and applications. Here's why:

1. Improved Decision-Making:

By analyzing large and diverse data sets, organizations can uncover patterns and trends that
would otherwise remain hidden. For example, retailers can use big data to personalize their
marketing strategies by analyzing customer behavior across multiple channels (e.g., online
browsing, in-store visits, and purchase histories). This level of insight would be difficult to
achieve with traditional datasets.

2. Operational Efficiency:

Real-time data analytics allow businesses to optimize their operations by detecting inefficiencies
as they happen. For instance, supply chains can be monitored in real-time to identify bottlenecks
and make quick adjustments, reducing delays and costs.

3. Predictive Analytics:

The ability to analyze historical data and predict future trends is another key benefit of big data
analytics. For example, healthcare providers can use big data to predict patient outcomes based
on medical histories and lifestyle data, enabling more proactive care.
4. Personalization and Customer Insights:

With the volume and variety of data available today, companies can better understand their
customers' preferences and tailor products and services accordingly. Big data makes it possible to
track and analyze customer behavior on a granular level, enhancing customer satisfaction and
driving business growth.

5. Enhanced Innovation:

Big data enables organizations to identify new business opportunities by spotting emerging
trends and unmet needs. In sectors like tech and healthcare, companies can use data to innovate
and stay ahead of the competition.
For example, IoT devices generate vast amounts of data that can be analyzed to develop new
solutions, from smarter cities to connected homes.

6. Industry Applications:

Different industries have adopted big data for various applications. For example:

 Healthcare: Analyzing patient data to improve diagnoses and treatments.

 Retail: Leveraging purchasing patterns to optimize inventory and supply chain

management.

 Finance: Detecting fraudulent transactions in real time by analyzing transaction data.

 Government: Using traffic and infrastructure data to improve urban planning and public
services.

Components of Big Data

Understanding the components of big data is crucial for anyone looking to delve into data
analytics or work with large datasets. Big data is a complex system that involves multiple layers
for data collection, storage, processing, and analysis.
Below is a breakdown of the key components of big data, which play a crucial role in unlocking
insights and supporting decision-making.
1. Data Sources

Big data comes from a multitude of sources, both traditional and modern, making it more diverse
than ever before. These sources generate vast amounts of data continuously, and the challenge
lies in capturing, processing, and making sense of it. Some of the major sources include:

Social Media:

Social platforms such as Facebook, Twitter, Instagram, and LinkedIn produce massive amounts
of data every minute in the form of posts, tweets, likes, and comments. According to recent
statistics, over 500 million tweets are sent each day (Statista).

Sensors:

The rise of Internet of Things (IoT) devices has led to an explosion of data generated by sensors
in vehicles, smart homes, health devices, and industrial equipment. It is estimated that IoT
devices will generate over 79.4 zettabytes of data by 2025 (Statista).
Logs:

Website logs, server logs, application logs, and transactional data from various industries
contribute significantly to big data. For example, e-commerce platforms generate vast amounts
of log data from user activity on their websites.
Financial Data:

Financial institutions, including banks and stock exchanges, generate huge volumes of data
related to transactions, investments, and stock market activities.
Government Data:

Government organizations, ranging from census data to economic indicators, contribute to the
big data landscape. According to a report by the World Economic Forum, open data from
governments is expanding rapidly.

2. Storage Systems
Big data storage is one of the primary concerns for managing massive volumes of information.
Traditional relational databases are ill-equipped to handle the scale and variety of big data. As a
result, new and specialized storage systems have been developed. These systems focus on
scalability, high availability, and flexibility.

Data Lakes:

A data lake is a centralized repository that allows you to store large amounts of raw data in its
native format until it is needed. Data lakes are capable of storing both structured and
unstructured data, unlike traditional storage systems that typically focus on structured data.
According to a 2024 survey by Gartner, 40% of enterprises report adopting data lakes for
big data storage.

NoSQL Databases:

Technologies like MongoDB, Cassandra, and Couchbase are popular NoSQL databases designed
to handle unstructured data. These databases provide flexibility in terms of data models, which is
crucial when dealing with various forms of data.
Cloud Storage:

Cloud services like Amazon S3, Google Cloud Storage, and Microsoft Azure are essential for
storing and managing big data. Cloud storage offers scalability and reduced infrastructure costs,
making it a popular choice for organizations working with big data.

3. Processing Technologies

Once data is collected and stored, it needs to be processed in real-time or batch processing
modes. Big data processing tools help to extract meaningful insights from raw data. The two
most widely used processing frameworks for big data are Hadoop and Apache Spark.
Hadoop:

Hadoop is an open-source platform designed to distribute the processing of massive datasets

across multiple computer clusters, enabling scalable and efficient data analysis. It uses the
MapReduce programming model, enabling the storage and processing of data in a highly
scalable and fault-tolerant manner. Hadoop is widely used in industries like healthcare, finance,
and retail to analyze large datasets.

Apache Spark:

Spark is another open-source processing engine, but it is faster and more advanced than Hadoop
in terms of speed and efficiency, especially for real-time data processing. Spark performs
computations in-memory, reducing time for data retrieval and analysis. According to a report by
Data Science Central, Apache Spark is gaining popularity with 30% year-over-year growth in
adoption across industries.
Stream Processing:

In addition to batch processing, stream processing has become crucial for real-time data
analytics. Tools like Apache Flink, Kafka, and Amazon Kinesis allow businesses to process and
analyze data as it flows in from various sources, enabling faster decision-making.
4. Analytics Platforms

Analytics platforms are the tools and technologies used to derive insights from big data. These
platforms integrate machine learning, AI, and statistical models to analyze data and help
businesses make informed decisions. Among the most commonly utilized platforms are:

Machine Learning and AI:

Machine learning (ML) models are used to predict future trends based on historical data.
Algorithms like regression analysis, decision trees, and neural networks are commonly employed
in big data analytics. In fact, a recent McKinsey report predicts that 70% of businesses will
deploy AI for analytics by 2025.

Business Intelligence (BI) Tools:

BI tools such as Tableau, Power BI, and Qlik Sense provide easy-to-use interfaces for analyzing
and visualizing big data. These platforms enable decision-makers to see insights visually through
dashboards and reports, allowing for quicker decision-making. According to a 2023 survey by
Forbes, over 60% of business leaders use BI tools to gain insights from big data.
Data Science Platforms:

Data science tools like R, Python, and SAS are widely used for deeper analysis and modeling.
With powerful libraries like TensorFlow, Keras, and Pandas, data scientists can uncover trends,
correlations, and predictive models that provide actionable business insights.

Predictive Analytics:

Predictive analytics tools use machine learning algorithms to forecast future outcomes based on
historical data. For instance, retail companies use predictive models to forecast demand, while
healthcare providers use them to predict patient outcomes. According to a recent report by PwC,
the predictive analytics market is expected to grow by 20% annually through 2026.

Overall,

It is the backbone of modern decision-making, influencing everything from business strategies to

scientific advancements. Understanding its key components data sources, storage systems,
processing technologies, and analytics platforms is essential for anyone looking to make sense of
the vast amounts of data generated daily.
The 5Vs (Volume, Velocity, Variety, Veracity, and Value) highlight the challenges and
opportunities in managing Big Data. Technologies like Hadoop and Spark are crucial for
processing large datasets, while machine learning helps derive valuable insights. As the world
generates more data, staying updated with the latest Big Data tools is vital for businesses and
professionals seeking to leverage its power.
 Types of Big Data
Types of Big Data:

Structured Data

 Structured data can be crudely defined as the data that resides in a fixed field within
a record.

 It is type of data most familiar to our everyday lives. for ex: birthday,address

 A certain schema binds it, so all the data has the same set of properties. Structured
data is also called relational data. It is split into multiple tables to enhance the
integrity of the data by creating a single record to depict an entity. Relationships are
enforced by the application of table constraints.

 The business value of structured data lies within how well an organization can utilize
its existing systems and processes for analysis purposes.
Sources of structured data

A Structured Query Language (SQL) is needed to bring the data together. Structured data is
easy to enter, query, and analyze. All of the data follows the same format. However, forcing
a consistent structure also means that any alteration of data is too tough as each record
has to be updated to adhere to the new structure. Examples of structured data include
numbers, dates, strings, etc. The business data of an e-commerce website can be
considered to be structured data.

Name Class Section Roll No Grade

Geek1 11 A 1 A

Geek2 11 A 2 B
Name Class Section Roll No Grade

Geek3 11 A 3 A

Cons of Structured Data

1. Structured data can only be leveraged in cases of predefined functionalities. This

means that structured data has limited flexibility and is suitable for certain specific
use cases only.

2. Structured data is stored in a data warehouse with rigid constraints and a definite
schema. Any change in requirements would mean updating all of that structured
data to meet the new needs. This is a massive drawback in terms of resource and
time management.

Semi-Structured Data

 Semi-structured data is not bound by any rigid schema for data storage and
handling. The data is not in the relational format and is not neatly organized into
rows and columns like that in a spreadsheet. However, there are some features like
key-value pairs that help in discerning the di erent entities from each other.

 Since semi-structured data doesn't need a structured query language, it is

commonly called NoSQL data.

 A data serialization language is used to exchange semi-structured data across

systems that may even have varied underlying infrastructure.

 Semi-structured content is often used to store metadata about a business process

but it can also include files containing machine instructions for computer programs.

 This type of information typically comes from external sources such as social media
platforms or other web-based data feeds.
Semi-Structured Data

Data is created in plain text so that di erent text-editing tools can be used to draw valuable
insights. Due to a simple format, data serialization readers can be implemented on
hardware with limited processing resources and bandwidth.

Data Serialization Languages

Software developers use serialization languages to write memory-based data in files,

transit, store, and parse. The sender and the receiver don’t need to know about the other
system. As long as the same serialization language is used, the data can be understood by
both systems comfortably. There are three predominantly used Serialization languages.

1. XML- XML stands for eXtensible Markup Language. It is a text-based markup language
designed to store and transport data. XML parsers can be found in almost all popular
development platforms. It is human and machine-readable. XML has definite standards for
schema, transformation, and display. It is self-descriptive. Below is an example of a
programmer's details in XML.

<ProgrammerDetails>

<FirstName>Jane</FirstName>

<LastName>Doe</LastName>

<CodingPlatforms>

<CodingPlatform Type="Fav">GeeksforGeeks</CodingPlatform>

<CodingPlatform Type="2ndFav">Code4Eva!</CodingPlatform>

<CodingPlatform Type="3rdFav">CodeisLife</CodingPlatform>

</CodingPlatforms>

</ProgrammerDetails>

<!--The 2ndFav and 3rdFav Coding Platforms are imaginative because Geeksforgeeks is the
best!-->

XML expresses the data using tags (text within angular brackets) to shape the data (for ex:
FirstName) and attributes (For ex: Type) to feature the data. However, being a verbose and
voluminous language, other formats have gained more popularity.

2. JSON- JSON (JavaScript Object Notation) is a lightweight open-standard file format for
data interchange. JSON is easy to use and uses human/machine-readable text to store and
transmit data objects.

"firstName": "Jane",

"lastName": "Doe",

"codingPlatforms": [

{ "type": "Fav", "value": "Geeksforgeeks" },

{ "type": "2ndFav", "value": "Code4Eva!" },

{ "type": "3rdFav", "value": "CodeisLife" }

 ]

This format isn't as formal as XML. It's more like a key/value pair model than a formal data
depiction. Javascript has inbuilt support for JSON. Although JSON is very popular amongst
web developers, non-technical personnel find it tedious to work with JSON due to its heavy
dependence on JavaScript and structural characters (braces, commas, etc.)

3. YAML- YAML is a user-friendly data serialization language. Figuratively, it stands for YAML
Ain’t Markup Language. It is adopted by technical and non-technical handlers all across the
globe owing to its simplicity. The data structure is defined by line separation and
indentation and reduces the dependency on structural characters. YAML is extremely
comprehensive and its popularity is a result of its human-machine readability.

YAML example

A product catalog organized by tags is an example of semi-structured data.

Unstructured Data

 Unstructured data is the kind of data that doesn't adhere to any definite schema or
set of rules. Its arrangement is unplanned and haphazard.

 Photos, videos, text documents, and log files can be generally considered
unstructured data. Even though the metadata accompanying an image or a video
may be semi-structured, the actual data being dealt with is unstructured.

 Additionally, Unstructured data is also known as "dark data" because it cannot be

analyzed without the proper software tools.
Un-structured Data

Summary

Applications data can be classified as structured, semi-structured, and unstructured data.

Structured data is neatly organized and obeys a fixed set of rules. Semi-structured data
doesn't obey any schema, but it has certain discernible features for an organization. Data
serialization languages are used to convert data objects into a byte stream. These include
XML, JSON, and YAML. Unstructured data doesn't have any structure at all. All these three
kinds of data are present in an application. All three of them play equally important roles in
developing resourceful and attractive applications.
 What are the Benefits of using Big Data?

Below are the top advantages of using big data in business –

 Better decision making

 Greater innovations

 Improvement in education sector

 Product price optimization

 Recommendation engines

 Life-Saving application in the healthcare industry

In this competitive business world, the benefits of Big Data shouldn’t be underestimated.
There are endless services o ered by Big Data to the current market. If exploited properly,
Big Data can lead to substantial results.

Almost every company is now moving towards Big Data Analytics due to numerous
reasons. It is helping them in enhancing the overall growth of the organization.

Let’s discuss these advantages of big data in detail and know how they are helping big
businesses to make a profit.

1. Better Decision Making

The main benefit of using Big Data Analytics is that it has boosted the decision-making
process to a great extent. Rather than anonymously making decisions, companies are
considering Big Data Analytics before concluding to any decision.

A variety of customer-centric factors like what the customers want, the solution to their
problems, analyzing their needs according to the market trends, etc. are taken into account
for a better decision making process.

Big Data Case Studies

To understand how big data helps in better decision-making process, DataFlair is providing
you the amazing big data case studies –

a. Big Data Taking a Gamble

Heard about Big Data gaining foothold almost everywhere like the healthcare industry,
financial sectors, government sectors, etc. But now Big Data has entered the Casinos as
well. Surprised? And why not everybody likes gambling though.
Casinos are eventually moving towards Big Data Analytics. The MGM Grand in Las Vegas
has started using Big Data Analytics to provide better gaming experience to its customers.

The relationship between gamblers and the casino is quite delicate. It always has to be a
two-way process. A gambler won’t return to the casino if only the house owners are winning
repeatedly. The house owner must allow the gamblers to win a few games.

But how Big Data can be advantageous for Casinos? It provides them with valuable insights
about their machines enabling them taking better decisions.

These insights include the revenue collected through each machine, sorting the machines
which aren’t being played and then replacing it accordingly, the most popular machines
and at what time, areas in the casino generating great profits and the ones that need to be
rearranged.

b. Big Data Making Customers the King

The success of any organization can be measured in terms of how satisfied and loyal their
customers are. If they provide customers with what they want, there’s nothing that can
hinder their growth.

One of the best marketing policies any organization can follow is letting their customers
decide how the product should be like. Mountain Dew (a soft drink brand) and
Doritos (American brand of flavored tortilla chips) have both used this strategy and have
observed varying levels of success. The key ideology is – let the customers pick what they
want, and supply that.

One such example is that of Tropical Smoothie Cafe, which only had a fruit smoothie
menu. With the help of the insights gained through Big Data Analytics tools, they observed
that there is an increasing demand for veggie smoothies in some other areas and then they
decided to introduce veggie smoothies in their menu as well.

By keeping a track of their sales data, they found that their newly introduced veggie
smoothies have become best sellers within just a few days.

This further helped them in introducing some new veggie smoothies in their menu
according to what customers demanded. The cafe used Big Data to see at what time during
the day did their sales were highest and then launched time-specific o ers such as –
Happy Hours to attract more customers during this time.

2. Big Data in Greater Innovations

Now, the second benefit of Big Data is Innovations.

Innovations are crucial for the success of any organization. And to innovate, you need data,
more and more data. Big Data gives you the freedom to achieve the unthinkable.

Big Data Analytics is used by various firms to create new products and services for their
customers. Companies through Big Data, analyze di erent customers’ opinions about their
products and how their product is perceived.

It gives them information about what they are lacking and what are the significant things to
be kept in mind while developing any new product. This helps them in developing new
products according to customer’s requirement.

Big Data Analytics gives the capabilities of thinking beyond the ordinary.

Big Data Case Study related to Innovation

a. Big Data, Innovation and You

Big Data is the driving force behind every recent IoT (Internet of Things) innovation. Big
Data serves as a backbone to IoT.

One of the most breathtaking IoT applications is the invention of Driverless Cars. With
driverless cars under development, the day it becomes a reality is not far away. As the
name itself says, these cars won’t have drivers and would be sensible enough to drive you
to your destination on its own.

These cars are equipped with tons of devices like sensors, cameras, cloud architecture,
gyroscopes, altimeters, mapping devices, etc.

Through all these devices, driverless cars sense a huge amount of data of tra ic,
pedestrians, conditions of the road such as sharp turns, potholes, speed breakers, etc. and
then immediately process this data and take appropriate driving decisions.

Big Data along with Machine learning and Artificial Intelligence is crucial for the safe and
secure ride of a driverless car.

3. Big Data in Education Sector

Big data benefits the education sector in managing the data related to students of an
educational institute which is unmanageable. It is not used as it should be. Due to its huge
size, it is hard for teachers to exploit it properly.

Big Data Analytics has emerged as a boon to the education sector. It has started bringing
the much-needed transformation in the education system and will surely take it to greater
heights. Analysis of the capabilities of students based on the data can help teachers in
nurturing their future in a better.
Teachers are now aware of the student’s strengths and weaknesses and can guide them
accordingly.

4. Big Data in Product Price Optimization

The advantage of Big Data for companies is that they are using Big Data to optimize the
price they charge their customers. Their goal is to set the prices in such a way that profit is
maximized.

Through Big Data they analyze the prices that have yielded the maximum profits to them
under various historic market conditions. Through Big Data solutions they set their
product’s price according to the customer’s willingness to pay under di erent
circumstances.

Their aim is that the customer should get value for his money. As far as customers think
that way the company will always keep growing. But to make a customer satisfied always,
the company needs to make appropriate advancements in the product according to the
trends in the market and Big Data facilitate them to do so.

5. Big Data in Recommendation Engines

Imagine being able to have recommendations based on your previous as well as current
choices made on various online platforms.

Life is much easier when you have the option of choosing from the things you like. This is
something that has changed the thinking of people towards various online platforms. They
are now more comfortable being on these platforms.

The best example of a Big Data recommendation engine is that of various online shopping
platforms. They analyze every customer’s data and then recommend them accordingly.

These recommendations are majorly based on the activities the customer did when he last
visited the platform and his real-time activities.

Also, suggestions are made to them based on a comparison between the customers who
searched or bought familiar stu . This is how online platforms have broken the physical
barriers between them and their customers. Hasn’t recommendation engines transformed
the online shopping experience? It surely has.

6. Big Data in Life-Saving Application in Healthcare Industry

The advent of Big Data Analytics has o ered numerous benefits to the Healthcare Industry.
It can be regarded as a Revolution in the Making.
According to the Big Data Experts at QUANTZIG (A Global Analytics Solutions
Provider), “Big Data and Advanced Analytics may just be the answer to the hardest of
Healthcare challenges”.

Big Data in healthcare would help practitioners to provide advanced and quality
healthcare to their patients based on the electronic health records of the patient. It
enhances the overall operational e iciency of the healthcare companies and has allowed
them to make the required changes.

Big Data Analytics would allow them to find a better cure for a disease by recognizing
unknown connections and hidden patterns. Even a cure for a disease like cancer can be
made possible by it.

After reading the benefits of Big Data you must know why Big Data is important in today’s
world.

Why Big Data?

Big Data is an integral part of every industry these days.

The true measure of the worth of anything is all about the benefits everyone has gained
from it. And Big Data’s worth is unimaginable.

Big Data is changing the world and is all the hype these days. But what makes it so
important? And more importantly, how can we capitalize on it? Below, we have listed out a
few ways in which you can use it to improve and grow your business or service.

“Information is the Oil of the 21st century, and Analytics is the Combustion Engine.”
– (Peter Sondergaard, Senior Vice President, Gartner)
 Evolution of Big data

Additional Points

Big Data has become ubiquitous, representing the massive volume of structured and
unstructured data generated by various sources.

Origin of Big Data

In ancient times, during the Roman Empire, people used data for making decisions and
planning military strategies. Over the years, technology evolved, like when Herman Hollerith
created the punch card machine in 1884, which helped process data. Dealing with large
amounts of data has always been a challenge. In the late 1990s, the term “big data” was
coined to describe the di iculty of handling huge amounts of data from supercomputers.
Social media platforms such as MySpace, Facebook, and Twitter brought about
unstructured data, leading to the need for new tools like Hadoop and NoSQL databases to
manage and analyse this data e ectively. As technology advanced, more data was
generated from mobile devices and the Internet of Things (IoT) in the 2010s. This created new
challenges in collecting, organising, and analysing di erent types of data. The history of big
data shows how people have dealt with data over time, adapted to new technologies, and
faced the ongoing task of managing large and complex datasets e iciently.

Technological Advancements

In the past few decades, technology has made huge strides in how we store and process
data. We’ve seen the development of advanced storage solutions that can hold massive
amounts of information and processing technologies that can crunch through this data
quickly and e iciently. One significant advancement has been the introduction of
distributed computing frameworks like Hadoop and Spark. These frameworks allow us to
spread out the workload across multiple computers, making it possible to handle enormous
datasets that would be too large for a single machine to manage. Hadoop, for example, uses
a system called HDFS (Hadoop Distributed File System) to store data across a cluster of
computers, while Spark HDFS enables data processing through its in-memory computing
capabilities. These technologies have revolutionized how we work with big data, enabling
businesses to analyze and derive insights from vast amounts of information in a scalable and
e icient manner.

Exploratory Data Analysis and Predictive Modelling

Big data analytics involves the use of advanced techniques to analyze large datasets and
gain valuable insights. Exploratory data analysis (EDA) is a crucial step in this process, where
data is examined to identify patterns, trends, and correlations. EDA helps in understanding
the distribution of data, identifying outliers, and visualising relationships between variables.

Predictive modelling and machine learning algorithms are used to build models that can
forecast future outcomes based on historical data. These algorithms can be trained on large
datasets to identify complex patterns and make accurate predictions. Techniques like
regression, decision trees, and clustering are commonly used in predictive modelling.
Applications Across Industries

Challenges and Opportunities in Big Data

Big data presents both challenges and opportunities for businesses and individuals. One of
the biggest challenges is ensuring data privacy and security. With the increasing amount of
personal data being collected, there is a growing concern about how this data is being used
and protected. Another challenge is scalability and infrastructure requirements. As data
grows, so do the needs for storage and processing power. This requires significant
investments in infrastructure and technology.
However, big data also presents opportunities for innovation and new business models. By
analysing large datasets, businesses can gain valuable insights and make informed
decisions. This can lead to new products and services, improved customer experiences, and
increased competitiveness.

Future Trends

In the coming years, we can expect to see a rise in the use of IoT (Internet of Things) devices
and sensor data. These devices, like smart thermostats and fitness trackers, collect vast
amounts of data that can provide valuable insights for businesses and individuals.

Another trend on the horizon is the adoption of real-time analytics and edge computing.
Real-time analytics allows for immediate data processing and decision-making, while edge
computing brings this processing closer to where the data is generated, reducing latency and
improving e iciency. As big data continues to grow, ethical considerations and responsible
use of this data become increasingly important. Organisations must ensure they are
collecting and using data in a way that respects privacy, security, and fairness. These future
trends in big data will shape how businesses operate and how individuals interact with
technology, paving the way for more e icient and ethical data practices.

Overall, the evolution of big data has been a remarkable journey. From its humble
beginnings in ancient civilisations to the current era of advanced technologies and
applications, big data has transformed the way we live, work, and make decisions. Big data
has enabled businesses to gain valuable insights, improve operations, and innovate
products and services. It has also revolutionised industries such as healthcare, finance, and
retail, allowing for personalised medicine, risk management, and customer analytics. The
transformative impact of big data on various sectors cannot be overstated. It has enabled
real-time decision-making, improved customer experiences, and created new business
models. As big data continues to evolve, it is crucial that we adapt and leverage its potential
to drive growth, innovation, and progress.
 Big Data – Introduction

Day by day the big world of internet is creating 2.5 quintillion bytes of data on regular basis
according to the statistics the percentage of data that has been generated from the last two
years is 90%.

This data comes from many industries like climate information collects by the sensor,
di erent stu from social media sites, digital images and videos, di erent records of the
purchase transaction. This data is big data.

Big Data History

This section of tutorial gives you a clear picture of big data history-

 Research & Development in these Big Data native businesses are very close, and
very close to the research and open source community.

 Each paper on the cost-e icient innovative information processing techniques has
been accompanied by open source adoption within an ever growing ecosystem
called Hadoop.

Two major milestones in the development of Hadoop also added confidence into the Power
of open source and Big Data Technologies.

Only two years after its first release, in 2008, Hadoop won the terabyte sort benchmark in
big data history. This is the first time that either a Java or an open source program has won.
In 2010 Facebook claimed that they had the largest Hadoop cluster in the world with 21 PB
of storage for their social messaging platform.

Facts and Figures

 91% of leaders belongs to marketing believe successful brands use customer data
to drive business decisions.

 The overall percentage of the world’s total data has been created just within the past
two years is 90%.

 87% companies agree capturing and sharing the right data is important to
e ectively measure ROI in their own company.

 500 million calls record daily analyzed by IBM to predict the customer’s churns.

 350 billion annual meter readings converted by IBM through Big Data to better
predict power consumption.
  On Facebook, 30 billion pieces of content are sharing by users in each month.

Big Data Technologies

While the topic of Big Data is broad and encompasses many trends and new technology
developments, the top emerging technologies are given below that are helping users cope
with and handle Big Data in a cost-e ective manner.

1. Apache Hadoop

The backbone of every Big Data solution, It is anticipated that world’s 75% of the data will
be stored in Hadoop by 2017.

2. Apache Spark

Apache Spark is considered as next generation Big Data tool, It is lightening fast cluster
computing engine which is 100 times faster than Hadoop-MapReduce. Learn more about
Apache Spark

3. Apache Flink

Apache Flink is called 4G of Big Data. It is an open source framework that can handle
streaming as well as batch data. Learn more about Apache Flink

Figures Of Big Brands

1. Facebook

Because of more than 950 million users, Facebook is collecting a huge amount of data.
Every time whenever you are clicking a notification, visiting a page, uploading a photo, or
checking out a friend’s link, you’re generating data for the company to track various
records.

Users shared 2.5 billion content items daily (status updates + wall posts + photos + videos
+ comments). 300 million photos are uploaded by users per day. 105 terabytes of data
scanned via Hive, Facebook’s Hadoop query language in every 30 minutes. 70,000 queries
executed on these databases per day. 500+terabytes of new data ingested into the
databases every day.

2. Twitter

Twitter – the second biggest social network generating less social data as compared to
dating app, Tinder. Tinder users swipe 290,278 matches per minute – that is potentially 35
million lovers per hour! on the other hand, twitter users generate 347,222 Tweets each
minute – or 21 million Tweets per hour.
3. Youtube

The video is a big part of our everyday lives on the internet, and although Facebook is also
trying really hard to fit in and it is succeeding, with over 3 billion video views per day but
YouTube is still the king. Every minute users are uploading over 300 hours of new video on
YouTube.

Big Data Use-cases

Financial services

 360-degree complete view of customer

 Risk and fraud monitoring and management

 Real-time transaction tracking and analytics

Healthcare/Life sciences

 Disease diagnosis analysis

 Medical record text analysis

 Genomic analytics

Telecommunications

 Real-time Call detail record CDR processing and analysis

 Customer profile monetization and analysis

 Real-time network element monitoring

 Real-time Network fault analysis

 IVR calls analytics

Digital media

 Real-time ad matching, analysis, and targeting

 Website analytics and conversion tracking

Retail

 Cross-channel marketing

 Customer Clustering and Segmentation

 Click-stream analysis
  Market Basket Analytics

 Real-time Recommendation

 Sentiment Analysis

Law enforcement

 Multimodal surveillance

 Real-time Cybersecurity detection

 Energy and utilities

 Smart meter analytics

 Asset management

To get deep dive into Big data real-life use cases follow this comprehensive guide.
 Hadoop Ecosystem Components.

Hadoop Ecosystem – 15 Must Know Hadoop Components

Hadoop Ecosystem Components

Hadoop Ecosystem is a suite of services that work to solve the Big Data problem. The
di erent components of the Hadoop Ecosystem are as follows:-

1. Hadoop Distributed File System

HDFS is the foundation of Hadoop and hence is a very important component of the
Hadoop ecosystem. It is Java software that provides many features like scalability, high
availability, fault tolerance, cost e ectiveness etc. It also provides robust distributed data
storage for Hadoop. We can deploy many other software frameworks over HDFS.

Components of HDFS:-
There are three major components of Hadoop HDFS are as follows:-

a. DataNode

These are the nodes which store the actual data. HDFS stores the data in a distributed
manner. It divides the input files of varied formats into blocks. The DataNodes stores each
of these blocks. Following are the functions of DataNodes:-

 On startup, DataNode does handshake with NameNode. It verifies the namespace

ID and software version of DataNode.

 Also, it sends a block report to NameNode and verifies the block replicas.

 It sends a heartbeat to NameNode every 3 seconds to tell that it is alive.

b. NameNode

NameNode is nothing but the master node. The NameNode is responsible for managing file
system namespace, controlling the client’s access to files. Also, it executes tasks such as
opening, closing and naming files and directories. NameNode has two major files –
FSImage and Edits log

FSImage – FSImage is a point-in-time snapshot of HDFS’s metadata. It contains

information like file permission, disk quota, modification timestamp, access time etc.
Edits log – It contains modifications on FSImage. It records incremental changes like
renaming the file, appending data to the file etc.

Whenever the NameNode starts it applies Edits log to FSImage. And the new FSImage gets
loaded on the NameNode.

c. Secondary NameNode

If the NameNode has not restarted for months the size of Edits log increases. This, in turn,
increases the downtime of the cluster on the restart of NameNode. In this case, Secondary
NameNode comes into the picture. The Secondary NameNode applies edits log on
FSImage at regular intervals. And it updates the new FSImage on primary NameNode.

2. MapReduce

MapReduce is the data processing component of Hadoop. It applies the computation on

sets of data in parallel thereby improving the performance. MapReduce works in two
phases –

Map Phase – This phase takes input as key-value pairs and produces output as key-value
pairs. It can write custom business logic in this phase. Map phase processes the data and
gives it to the next phase.

Reduce Phase – The MapReduce framework sorts the key-value pair before giving the data
to this phase. This phase applies the summary type of calculations to the key-value pairs.
  Mapper reads the block of data and converts it into key-value pairs.

 Now, these key-value pairs are input to the reducer.

 The reducer receives data tuples from multiple mappers.

 Reducer applies aggregation to these tuples based on the key.

 The final output from reducer gets written to HDFS.

MapReduce framework takes care of the failure. It recovers data from another node in an
event where one node goes down.

3. Yarn

Yarn which is short for Yet Another Resource Manager. It is like the operating system of
Hadoop as it monitors and manages the resources. Yarn came into the picture with the
launch of Hadoop 2.x in order to allow di erent workloads. It handles the workloads like
stream processing, interactive processing, batch processing over a single platform. Yarn
has two main components – Node Manager and Resource Manager.

a. Node Manager

It is Yarn’s per-node agent and takes care of the individual compute nodes in a Hadoop
cluster. It monitors the resource usage like CPU, memory etc. of the local node and
intimates the same to Resource Manager.

b. Resource Manager

It is responsible for tracking the resources in the cluster and scheduling tasks like map-
reduce jobs.

Also, we have the Application Master and Scheduler in Yarn. Let us take a look at them.

Application Master has two functions and they are:-

 Negotiating resources from Resource Manager

 Working with NodeManager to monitor and execute the sub-task.

Following are the functions of Resource Scheduler:-

 It allocates resources to various running applications

  But it does not monitor the status of the application. So in the event of failure of the
task, it does not restart the same.

We have another concept called Container. It is nothing but a fraction of NodeManager

capacity i.e. CPU, memory, disk, network etc.

4. Hive

Hive is a data warehouse project built on the top of Apache Hadoop which provides data
query and analysis. It has got the language of its own call HQL or Hive Query Language.
HQL automatically translates the queries into the corresponding map-reduce job.

Main parts of the Hive are –

 MetaStore – it stores metadata

 Driver – Manages the lifecycle of HQL statement

 Query compiler – Compiles HQL into DAG i.e. Directed Acyclic Graph

 Hive server – Provides interface for JDBC/ODBC server.

Facebook designed Hive for people who are comfortable in SQL. It has two basic
components – Hive Command Line and JDBC, ODBC. Hive Command line is an interface
for execution of HQL commands. And JDBC, ODBC establishes the connection with data
storage. Hive is highly scalable. It can handle both types of workloads i.e. batch processing
and interactive processing. It supports native data type of SQL. Hive provides many pre-
defined functions for analysis. But you can also define your own custom functions called
UDFs or user-defined functions.

5. Pig

Pig is a SQL like language used for querying and analyzing data stored in HDFS. Yahoo was
the original creator of the Pig. It uses pig latin language. It loads the data, applies a filter to
it and dumps the data in the required format. Pig also consists of JVM called Pig Runtime.
Various features of Pig are as follows:-
  Extensibility – For carrying out special purpose processing, users can create their
own custom function.

 Optimization opportunities – Pig automatically optimizes the query allowing users

to focus on semantics rather than e iciency.

 Handles all kinds of data – Pig analyzes both structured as well as unstructured.

a. How does Pig work?

 First, the load command loads the data.

 At the backend, the compiler converts pig latin into the sequence of map-reduce
jobs.

 Over this data, we perform various functions like joining, sorting, grouping, filtering
etc.

 Now, you can dump the output on the screen or store it in an HDFS file.

6. HBase
HBase is a NoSQL database built on the top of HDFS. The various features of HBase are
that it is open-source, non-relational, distributed database. It imitates Google’s
Bigtable and written in Java. It provides real-time read/write access to large datasets. Its
various components are as follows:-

a. HBase Master

HBase performs the following functions:

 Maintain and monitor the Hadoop cluster.

 Performs administration of the database.

 Controls the failover.

 HMaster handles DDL operation.

b. RegionServer

Region server is a process which handles read, writes, update and delete requests from
clients. It runs on every node in a Hadoop cluster that is HDFS DataNode.

HBase is a column-oriented database management system. It runs on top of HDFS. It suits

for sparse data sets which are common in Big Data use cases. HBase support writing
application in Apache Avro, REST and Thrift. Apache HBase has low latency storage.
Enterprises use this for real-time analysis.

The design of HBase is such that to contain many tables. Each of these tables must have a
primary key. Access attempts to HBase tables use this primary key.

As an example lets us consider HBase table storing diagnostic log from the server. In this
case, the typical log row will contain columns such as timestamp when the log gets
written. And server from which the log originated.

7. Mahout
Mahout provides a platform for creating machine learning applications which are scalable.

a. What is Machine Learning?

Machine learning algorithms allow us to create self-evolving machines without being

explicitly programmed. It makes future decisions based on user behavior, past experiences
and data patterns.

b. What Mahout does?

It performs collaborative filtering, clustering, and classification.

 Collaborative filtering – Mahout mines user behavior patterns and based on these
it makes recommendations to users.

 Clustering – It groups together a similar type of data like the article, blogs, research
paper, news etc.

 Classification – It means categorizing data into various sub-departments. For

example, we can classify article into blogs, essays, research papers and so on.

 Frequent Itemset missing – It looks for the items generally bought together and
based on that it gives a suggestion. For instance, usually, we buy a cell phone and its
cover together. So, when you buy a cell phone it will give suggestion to buy cover
also.

8. Zookeeper
Zookeeper coordinates between various services in the Hadoop ecosystem. It saves the
time required for synchronization, configuration maintenance, grouping, and naming.
Following are the features of Zookeeper:-

 Speed – Zookeeper is fast in workloads where reads to data are more than write. A
typical read: write ratio is 10:1.

 Organized – Zookeeper maintains a record of all transactions.

 Simple – It maintains a single hierarchical namespace, similar to directories and

files.

 Reliable – We can replicate Zookeeper over a set of hosts and they are aware of
each other. There is no single point of failure. As long as major servers are available
zookeeper is available.

Why do we need Zookeeper in Hadoop?

Hadoop faces many problems as it runs a distributed application. One of the problems is
deadlock. Deadlock occurs when two or more tasks fight for the same resource. For
instance, task T1 has resource R1 and is waiting for resource R2 held by task T2. And this
task T2 is waiting for resource R1 held by task T1. In such a scenario deadlock occurs. Both
task T1 and T2 would get locked waiting for resources. Zookeeper solves Deadlock
condition via synchronization.

Another problem is of race condition. This occurs when the machine tries to perform two or
more operations at a time. Zookeeper solves this problem by property of serialization.

9. Oozie
It is a workflow scheduler systems for managing Hadoop jobs. It supports Hadoop jobs for
Map-Reduce, Pig, Hive, and Sqoop. Oozie combines multiple jobs into a single unit of work.
It is scalable and can manage thousands of workflow in a Hadoop cluster. Oozie works by
creating DAG i.e. Directed Acyclic Graph of the workflow. It is very much flexible as it can
start, stop, suspend and rerun failed jobs.

Oozie is an open-source web-application written in Java. Oozie is scalable and can execute
thousands of workflow containing dozens of Hadoop jobs.

There are three basic types of Oozie jobs and they are as follows:-

 Workflow – It stores and runs a workflow composed of Hadoop jobs. It stores the
job as Directed Acyclic Graph to determine the sequence of actions that will get
executed.

 Coordinator – It runs workflow jobs based on predefined schedules and availability

of data.

 Bundle – This is nothing but a package of many coordinators and workflow jobs.

How does Oozie work?

Oozie runs a service in the Hadoop cluster. Client submits workflow to run, immediately or
later.

There are two types of nodes in Oozie. They are action node and control flow node.

 Action Node – It represents the task in the workflow like MapReduce job, shell
script, pig or hive jobs etc.

 Control flow Node – It controls the workflow between actions by employing

conditional logic. In this, the previous action decides which branch to follow.

Start, End and Error Nodes fall under this category.

  Start Node signals the start of the workflow job.

 End Node designates the end of job.

 ErrorNode signals the error and gives an error message.

10. Sqoop

Sqoop imports data from external sources into compatible Hadoop Ecosystem
components like HDFS, Hive, HBase etc. It also transfers data from Hadoop to other
external sources. It works with RDBMS like TeraData, Oracle, MySQL and so on. The major
di erence between Sqoop and Flume is that Flume does not work with structured data. But
Sqoop can deal with structured as well as unstructured data.

Let us see how Sqoop works

When we submit Sqoop command, at the back-end, it gets divided into a number of sub-
tasks. These sub-tasks are nothing but map-tasks. Each map-task import a part of data to
Hadoop. Hence all the map-task taken together imports the whole data.
Sqoop export also works in a similar way. Only thing is instead of importing, the map-task
export the part of data from Hadoop to destination database.

11. Flume

It is a service which helps to ingest structured and semi-structured data into

HDFS. Flume works on the principle of distributed processing. It aids in collection,
aggregation, and movement of a huge amount of data sets. Flume has three components
source, sink, and channel.

Source – It accepts the data from the incoming stream and stores the data in the channel

Channel – It is a medium of temporary storage between the source of the data and
persistent storage of HDFS.

Sink – This component collects the data from the channel and writes it permanently to the
HDFS.
12. Ambari

Ambari is another Hadoop ecosystem component. It is responsible

for provisioning, managing, monitoring and securing Hadoop cluster. Following are the
di erent features of Ambari:

 Simplified cluster configuration, management, and installation

 Ambari reduces the complexity of configuring and administration of Hadoop cluster

security.

 It ensures that the cluster is healthy and available for monitoring.

Ambari gives:-
Hadoop cluster provisioning

 It gives step by step procedure for installing Hadoop services on the Hadoop
cluster.

 It also handles configuration of services across the Hadoop cluster.

Hadoop cluster management

 It provides centralized service for starting, stopping and reconfiguring services on

the network of machines.

Hadoop cluster monitoring

 To monitor health and status Ambari provides us dashboard.

 Ambari alert framework alerts the user when the node goes down or has low disk
space etc.

13. Apache Drill

Apache Drill is a schema-free SQL query engine. It works on the top of Hadoop, NoSQL and
cloud storage. Its main purpose is large scale processing of data with low latency. It is a
distributed query processing engine. We can query petabytes of data using Drill. It can
scale to several thousands of nodes. It supports NoSQL databases like Azure BLOB
storage, Google cloud storage, Amazon S3, HBase, MongoDB and so on.

Let us look at some of the features of Drill:-

 Variety of data sources can be the basis of a single query.

 Drill follows ANSI SQL.

 It can support millions of users and serve their queries over large data sets.

 Drill gives faster insights without ETL overheads like loading, schema creation,
maintenance, transformation etc.

 It can analyze multi-structured and nested data without having to do

transformations or filtering.

14. Apache Spark

Apache Spark unifies all kinds of Big Data processing under one umbrella. It has built-in
libraries for streaming, SQL, machine learning and graph processing. Apache Spark is
lightening fast. It gives good performance for both batch and stream processing. It does
this with the help of DAG scheduler, query optimizer, and physical execution engine.

Spark o ers 80 high-level operators which makes it easy to build parallel applications.
Spark has various libraries like MLlib for machine learning, GraphX for graph processing,
SQL and Data frames, and Spark Streaming. One can run Spark in standalone cluster mode
on Hadoop, Mesos, or on Kubernetes. One can write Spark applications using SQL, R,
Python, Scala, and Java. As such Scala in the native language of Spark. It was originally
developed at the University of California, Berkley. Spark does in-memory calculations. This
makes Spark faster than Hadoop map-reduce.

15. Solr And Lucene

Apache Solr and Apache Lucene are two services which search and indexes the Hadoop
ecosystem. Apache Solr is an application built around Apache Lucene. Code of Apache
Lucene is in Java. It uses Java libraries for searching and indexing. Apache Solr is an open
source, blazing fast search platform.

Various features of Solr are as follows –

 Solr is highly scalable, reliable and fault tolerant.

 It provides distributed indexing, automated failover and recovery, load-balanced

query, centralized configuration and much more.

 You can query Solr using HTTP GET and receive the result in JSON, binary, CSV and
XML.

 Solr provides matching capabilities like phrases, wildcards, grouping, joining and
much more.

 It gets shipped with a built-in administrative interface enabling management of solr

instances.

 Solr takes advantage of Lucene’s near real-time indexing. It enables you to see your
content when you want to see it.

So, this was all in the Hadoop Ecosystem. Hope you liked this article.

Summary

The Hadoop ecosystem elements described above are all open system Apache Hadoop
Project. Many commercial applications use these ecosystem elements. Let us summarize
Hadoop ecosystem components. At the core, we have HDFS for data storage, map-reduce
for data processing and Yarn a resource manager. Then we have HIVE a data analysis tool,
Pig – SQL like a scripting language, HBase – NoSQL database, Mahout – machine learning
tool, Zookeeper – a synchronization tool, Oozie – workflow scheduler system, Sqoop –
structured data importing and exporting utility, Flume – data transfer tool for unstructured
and semi-structured data, Ambari – a tool for managing and securing Hadoop clusters, and
lastly Avro – RPC, and data serialization framework.
 Hadoop Cluster
What is a Cluster?

A Cluster is a collection of nodes. Nodes are nothing but a point of connection/intersection

within a network.

A computer cluster is a collection of computers connected with a network, able to

communicate with each other, and works as a single system.

What is Hadoop Cluster?

Hadoop Cluster is just a computer cluster used for handling a vast amount of data in a
distributed manner.

It is a computational cluster designed for storing as well as analyzing huge amounts of

unstructured or structured data in a distributed computing environment.
Hadoop Clusters are also known as Shared-nothing systems because nothing is shared
between the nodes in the cluster except the network bandwidth. This decreases the
processing latency.

Thus, when there is a need to process queries on the huge amount of data, the cluster-wide
latency is minimized.

Let us now study the Architecture of Hadoop Cluster.

Architecture of Hadoop Cluster

The Hadoop Cluster follows a master-slave architecture. It consists of the master node,
slave nodes, and the client node.

1. Master in Hadoop Cluster

Master in the Hadoop Cluster is a high power machine with a high configuration of memory
and CPU. The two daemons that are NameNode and the ResourceManager run on the
master node.

a. Functions of NameNode
NameNode is a master node in the Hadoop HDFS. NameNode manages the filesystem
namespace. It stores filesystem meta-data in the memory for fast retrieval. Hence, it
should be configured on high-end machines.

The functions of NameNode are:

 Manages filesystem namespace

 Stores meta-data about blocks of a file, blocks location, permissions, etc.

 It executes the filesystem namespace operations like opening, closing, renaming

files and directories, etc.

 It maintains and manages the DataNode.

b. Functions of Resource Manager

 ResourceManager is the master daemon of YARN.

 The ResourceManager arbitrates the resources among all the applications in the
system.

 It keeps track of live and dead nodes in the cluster.

2. Slaves in the Hadoop Cluster

Slaves in the Hadoop Cluster are inexpensive commodity hardware. The two daemons that
are DataNodes and the YARN NodeManagers run on the slave nodes.

a. Functions of DataNodes

 DataNodes stores the actual business data. It stores the blocks of a file.

 It performs block creation, deletion, replication based on the instructions from

NameNode.

 DataNode is responsible for serving client read/write operations.

b. Functions of NodeManager

 NodeManager is the slave daemon of YARN.

 It is responsible for containers, monitoring their resource usage (such as CPU, disk,
memory, network) and reporting the same to the ResourceManager.

 The NodeManager also checks the health of the node on which it is running.

3. Client Node in Hadoop Cluster

Client Nodes in Hadoop are neither master node nor slave nodes. They have Hadoop
installed on them with all the cluster settings.

Functions of Client nodes

 Client nodes load data into the Hadoop Cluster.

 It submits MapReduce jobs, describing how that data should be processed.

 Retrieve the results of the job after processing completion.

We can scale out the Hadoop Cluster by adding more nodes. This makes Hadoop linearly
scalable. With every node addition, we get a corresponding boost in throughput. If we have
‘n’ nodes, then adding 1 node gives (1/n) additional computing power.

Single Node Hadoop Cluster VS Multi-Node Hadoop Cluster

1. Single Node Hadoop Cluster

Single Node Hadoop Cluster is deployed on a single machine. All the daemons like
NameNode, DataNode, ResourceManager, NodeManager run on the same machine/host.

In a single-node cluster setup, everything runs on a single JVM instance. The Hadoop user
didn’t have to make any configuration settings except for setting the JAVA_HOME variable.

The default replication factor for a single node Hadoop cluster is always 1.

2. Multi-Node Hadoop Cluster

Multi-Node Hadoop Cluster is deployed on multiple machines. All the daemons in the
multi-node Hadoop cluster are up and run on di erent machines/hosts.

A multi-node Hadoop cluster follows master-slave architecture. The daemons Namenode

and ResourceManager run on the master nodes, which are high-end computer machines.

The daemons DataNodes and NodeManagers run on the slave nodes(worker nodes), which
are inexpensive commodity hardware.
In the multi-node Hadoop cluster, slave machines can be present in any location
irrespective of the location of the physical location of the master server.

Communication Protocols used in Hadoop Cluster

The HDFS communication protocols are layered on the top of the TCP/IP protocol. A client
establishes a connection with the NameNode through the configurable TCP port on the
NameNode machine.

The Hadoop Cluster establishes a connection to the client through the ClientProtocol.
Moreover, the DataNode talks to the NameNode using the DataNode Protocol.

The Remote Procedure Call (RPC) abstraction wraps Client Protocol and DataNode
protocol. By design, NameNode does not initiate any RPCs. It only responds to the RPC
requests issued by clients or DataNodes.

Best Practices for building Hadoop Cluster

The performance of a Hadoop Cluster depends on various factors based on the well-
dimensioned hardware resources that use CPU, memory, network bandwidth, hard drive,
and other well-configured software layers.

Building a Hadoop Cluster is a non-trivial job. It requires consideration of various factors

like choosing the right hardware, sizing the Hadoop Clusters, and configuring the Hadoop
Cluster.

Let us now see each one in detail.

1. Choosing Right Hardware for Hadoop Cluster

Many organizations, when setting up Hadoop infrastructure, are in a predicament as they

are not aware of the kind of machines they need to purchase for setting up an optimized
Hadoop environment, and the ideal configuration they must use.

For choosing the right hardware for the Hadoop Cluster, one must consider the following
points:

1. The volume of Data that cluster will be going to handle.

2. The type of workloads the cluster will be dealing with ( CPU bound, I/O bound).

3. Data storage methodology like data containers, data compression techniques used,
if any.

4. A data retention policy, that is, how long we want to keep the data before flushing it
out.
2. Sizing the Hadoop Cluster

For determining the size of the Hadoop Cluster, the data volume that the Hadoop users will
process on the Hadoop Cluster should be a key consideration.

By knowing the volume of data to be processed, helps in deciding how many nodes will be
required in processing the data e iciently and memory capacity required for each node.
There should be a balance between the performance and the cost of the hardware
approved.

3. Configuring Hadoop Cluster

Finding the ideal configuration for the Hadoop Cluster is not an easy job. Hadoop
framework must be adapted to the cluster it is running and also to the job.

The best way of deciding the ideal configuration for the Hadoop Cluster is to run the
Hadoop jobs with the default configuration available in order to get a baseline. After that,
we can analyze the job history log files to see if there is any resource weakness or the time
taken to run the jobs is higher than expected.

If it is so, then change the configuration. Repeating the same process can tune the Hadoop
Cluster configuration that best fits the business requirements.

The performance of the Hadoop Cluster greatly depends on the resources allocated to the
daemons. For small to medium data context, Hadoop reserves one CPU core on each
DataNode, whereas, for the long datasets, it allocates 2 CPU cores on each DataNode for
HDFS and MapReduce daemons.

Hadoop Cluster Management

On deploying the Hadoop Cluster in production, it is apparent that it should scale along all
dimensions that are volume, variety, and velocity.

Various features that it should be posses to become production-ready are – round the
clock availability, robust, manageability, and performance. Hadoop Cluster management is
the main facet of the big data initiative.

The best tool for Hadoop Cluster management should have the following features:-

 It must ensure 24×7 high availability, resource provisioning, diverse security, work-
load management, health monitoring, performance optimization. Also, it needs to
provide job scheduling, policy management, back up, and recovery across one or
more nodes.
  Implement redundant HDFS NameNode high availability with load balancing, hot
standbys, resynchronization, and auto-failover.

 Enforcing policy-based controls that prevent any application from grabbing a

disproportionate share of resources on an already maxed-out Hadoop Cluster.

 Performing regression testing for managing the deployment of any software layers
over Hadoop clusters. This is to make sure that any jobs or data would not get crash
or encounter any bottlenecks in daily operations.

Benefits of Hadoop Cluster

The various benefits provided by the Hadoop Cluster are:

1. Scalable

Hadoop Clusters are scalable. We can add any number of nodes to the Hadoop Cluster
without any downtime and without any extra e orts. With every node addition, we get a
corresponding boost in throughput.

2. Robustness

The Hadoop Cluster is best known for its reliable storage. It can store data reliably, even in
cases like DataNode failure, NameNode failure, and network partition. The DataNode
periodically sends a heartbeat signal to the NameNode.

In network partition, a set of DataNodes gets detached from the NameNode due to which
NameNode does not receive any heartbeat from these DataNodes. NameNode then
considers these DataNodes as dead and does not forward any I/O request to them.

Also, the replication factor of the blocks stored in these DataNodes falls below their
specified value. As a result, NameNode then initiates the replication of these blocks and
recovers from the failure.

3. Cluster Rebalancing

The Hadoop HDFS architecture automatically performs cluster rebalancing. If the free
space in the DataNode falls below the threshold level, then HDFS architecture
automatically moves some data to other DataNode where enough space is available.

4. Cost-e ective

Setting up the Hadoop Cluster is cost-e ective because it comprises inexpensive

commodity hardware. Any organization can easily set up a powerful Hadoop Cluster
without spending much on expensive server hardware.
Also, Hadoop Clusters with its distributed storage topology overcome the limitations of the
traditional system. The limited storage can be extended just by adding additional
inexpensive storage units to the system.

5. Flexible

Hadoop Clusters are highly flexible as they can process data of any type, either structured,
semi-structured, or unstructured and of any sizes ranging from Gigabytes to Petabytes.

6. Fast Processing

In Hadoop Cluster, data can be processed parallelly in a distributed environment. This

provides fast data processing capabilities to Hadoop. Hadoop Clusters can process
Terabytes or Petabytes of data within a fraction of seconds.

7. Data Integrity

To check for any corruption in data blocks due to buggy software, faults in a storage device,
etc. the Hadoop Cluster implements checksum on each block of the file. If it finds any
block corrupted, it seeks it form another DataNode that contains the replica of the same
block. Thus, the Hadoop Cluster maintains data integrity.

Summary

After reading this article, we can say that the Hadoop Cluster is a special computational
cluster designed for analyzing and storing big data. Hadoop Cluster follows master-slave
architecture.

The master node is the high-end computer machine, and the slave nodes are machines
with normal CPU and memory configuration. We have also seen that the Hadoop Cluster
can be set up on a single machine called single-node Hadoop Cluster or on multiple
machines called multi-node Hadoop Cluster.

In this article, we had also covered the best practices to be followed while building a
Hadoop Cluster. We had also seen many advantages of the Hadoop Cluster, including
scalability, flexibility, cost-e ectiveness, etc.
 Hadoop Architecture

What is Hadoop Architecture?

Hadoop has a master-slave topology. In this topology, we have one master node and
multiple slave nodes. Master node’s function is to assign a task to various slave nodes and
manage resources. The slave nodes do the actual computing. Slave nodes store the real
data whereas on master we have metadata. This means it stores data about data. What
does metadata comprise that we will see in a moment?

Hadoop Application Architecture in Detail

Hadoop Architecture comprises three major layers. They are:-

 HDFS (Hadoop Distributed File System)

 Yarn

 MapReduce

1. HDFS

HDFS stands for Hadoop Distributed File System. It provides for data storage of Hadoop.
HDFS splits the data unit into smaller units called blocks and stores them in a distributed
manner. It has got two daemons running. One for master node – NameNode and other for
slave nodes – DataNode.
a. NameNode and DataNode

HDFS has a Master-slave architecture. The daemon called NameNode runs on the
master server. It is responsible for Namespace management and regulates file access by
the client. DataNode daemon runs on slave nodes. It is responsible for storing actual
business data. Internally, a file gets split into a number of data blocks and stored on a
group of slave machines. Namenode manages modifications to file system namespace.
These are actions like the opening, closing and renaming files or directories. NameNode
also keeps track of mapping of blocks to DataNodes. This DataNodes serves read/write
request from the file system’s client. DataNode also creates, deletes and replicates blocks
on demand from NameNode.

Java is the native language of HDFS. Hence one can deploy DataNode and NameNode on
machines having Java installed. In a typical deployment, there is one dedicated machine
running NameNode. And all the other nodes in the cluster run DataNode. The NameNode
contains metadata like the location of blocks on the DataNodes. And arbitrates resources
among various competing DataNodes.

You must read about Hadoop High Availability Concept

b. Block in HDFS
Block is nothing but the smallest unit of storage on a computer system. It is the smallest
contiguous storage allocated to a file. In Hadoop, we have a default block size of 128MB
or 256 MB.

One should select the block size very carefully. To explain why so let us take an example of
a file which is 700MB in size. If our block size is 128MB then HDFS divides the file into 6
blocks. Five blocks of 128MB and one block of 60MB. What will happen if the block is of
size 4KB? But in HDFS we would be having files of size in the order terabytes to petabytes.
With 4KB of the block size, we would be having numerous blocks. This, in turn, will create
huge metadata which will overload the NameNode. Hence we have to choose our HDFS
block size judiciously.

c. Replication Management

To provide fault tolerance HDFS uses a replication technique. In that, it makes copies of
the blocks and stores in on di erent DataNodes. Replication factor decides how many
copies of the blocks get stored. It is 3 by default but we can configure to any value.
The above figure shows how the replication technique works. Suppose we have a file of
1GB then with a replication factor of 3 it will require 3GBs of total storage.

To maintain the replication factor NameNode collects block report from every DataNode.
Whenever a block is under-replicated or over-replicated the NameNode adds or deletes the
replicas accordingly.

d. What is Rack Awareness?

A rack contains many DataNode machines and there are several such racks in the
production. HDFS follows a rack awareness algorithm to place the replicas of the blocks
in a distributed fashion. This rack awareness algorithm provides for low latency and fault
tolerance. Suppose the replication factor configured is 3. Now rack awareness algorithm
will place the first block on a local rack. It will keep the other two blocks on a di erent rack.
It does not store more than two blocks in the same rack if possible.

2. MapReduce

MapReduce is the data processing layer of Hadoop. It is a software framework that

allows you to write applications for processing a large amount of data. MapReduce runs
these applications in parallel on a cluster of low-end machines. It does so in a reliable and
fault-tolerant manner.

MapReduce job comprises a number of map tasks and reduces tasks. Each task works on
a part of data. This distributes the load across the cluster. The function of Map tasks is to
load, parse, transform and filter data. Each reduce task works on the sub-set of output
from the map tasks. Reduce task applies grouping and aggregation to this intermediate
data from the map tasks.

The input file for the MapReduce job exists on HDFS. The inputformat decides how to split
the input file into input splits. Input split is nothing but a byte-oriented view of the chunk of
the input file. This input split gets loaded by the map task. The map task runs on the node
where the relevant data is present. The data need not move over the network and get
processed locally.
i. Map Task

The Map task run in the following phases:-

a. RecordReader

The recordreader transforms the input split into records. It parses the data into records but
does not parse records itself. It provides the data to the mapper function in key-value pairs.
Usually, the key is the positional information and value is the data that comprises the
record.

b. Map

In this phase, the mapper which is the user-defined function processes the key-value pair
from the recordreader. It produces zero or multiple intermediate key-value pairs.

The decision of what will be the key-value pair lies on the mapper function. The key is
usually the data on which the reducer function does the grouping operation. And value is
the data which gets aggregated to get the final result in the reducer function.

c. Combiner

The combiner is actually a localized reducer which groups the data in the map phase. It
is optional. Combiner takes the intermediate data from the mapper and aggregates them. It
does so within the small scope of one mapper. In many situations, this decreases the
amount of data needed to move over the network. For example, moving (Hello World, 1)
three times consumes more network bandwidth than moving (Hello World, 3). Combiner
provides extreme performance gain with no drawbacks. The combiner is not guaranteed to
execute. Hence it is not of overall algorithm.

d. Partitioner

Partitioner pulls the intermediate key-value pairs from the mapper. It splits them into
shards, one shard per reducer. By default, partitioner fetches the hashcode of the key. The
partitioner performs modulus operation by a number of reducers:
[Link]()%(number of reducers). This distributes the keyspace evenly over the
reducers. It also ensures that key with the same value but from di erent mappers end up
into the same reducer. The partitioned data gets written on the local file system from each
map task. It waits there so that reducer can pull it.

b. Reduce Task

The various phases in reduce task are as follows:

i. Shu le and Sort

The reducer starts with shu le and sort step. This step downloads the data written by
partitioner to the machine where reducer is running. This step sorts the individual data
pieces into a large data list. The purpose of this sort is to collect the equivalent keys
together. The framework does this so that we could iterate over it easily in the reduce task.
This phase is not customizable. The framework handles everything automatically. However,
the developer has control over how the keys get sorted and grouped through a comparator
object.

ii. Reduce

The reducer performs the reduce function once per key grouping. The framework passes
the function key and an iterator object containing all the values pertaining to the key.

We can write reducer to filter, aggregate and combine data in a number of di erent ways.
Once the reduce function gets finished it gives zero or more key-value pairs to the
outputformat. Like map function, reduce function changes from job to job. As it is the core
logic of the solution.

iii. OutputFormat
This is the final step. It takes the key-value pair from the reducer and writes it to the file by
recordwriter. By default, it separates the key and value by a tab and each record by a
newline character. We can customize it to provide richer output format. But none the less
final data gets written to HDFS.

3. YARN

YARN or Yet Another Resource Negotiator is the resource management layer of Hadoop.
The basic principle behind YARN is to separate resource management and job
scheduling/monitoring function into separate daemons. In YARN there is one global
ResourceManager and per-application ApplicationMaster. An Application can be a single
job or a DAG of jobs.

Inside the YARN framework, we have two daemons ResourceManager and NodeManager.
The ResourceManager arbitrates resources among all the competing applications in the
system. The job of NodeManger is to monitor the resource usage by the container and
report the same to ResourceManger. The resources are like CPU, memory, disk, network
and so on.

The ApplcationMaster negotiates resources with ResourceManager and works with

NodeManger to execute and monitor the job.
The ResourceManger has two important components – Scheduler and ApplicationManager

i. Scheduler

Scheduler is responsible for allocating resources to various applications. This is a pure

scheduler as it does not perform tracking of status for the application. It also does not
reschedule the tasks which fail due to software or hardware errors. The scheduler allocates
the resources based on the requirements of the applications.

ii. Application Manager

Following are the functions of ApplicationManager

 Accepts job submission.

 Negotiates the first container for executing ApplicationMaster. A container

incorporates elements such as CPU, memory, disk, and network.

 Restarts the ApplicationMaster container on failure.

Functions of ApplicationMaster:-

 Negotiates resource container from Scheduler.

  Tracks the resource container status.

 Monitors progress of the application.

We can scale the YARN beyond a few thousand nodes through YARN Federation feature.
This feature enables us to tie multiple YARN clusters into a single massive cluster. This
allows for using independent clusters, clubbed together for a very large job.

iii. Features of Yarn

YARN has the following features:-

a. Multi-tenancy

YARN allows a variety of access engines (open-source or propriety) on the same Hadoop
data set. These access engines can be of batch processing, real-time processing, iterative
processing and so on.

b. Cluster Utilization

With the dynamic allocation of resources, YARN allows for good use of the cluster. As
compared to static map-reduce rules in previous versions of Hadoop which provides
lesser utilization of the cluster.

c. Scalability

Any data center processing power keeps on expanding. YARN’s ResourceManager focuses
on scheduling and copes with the ever-expanding cluster, processing petabytes of data.

d. Compatibility

MapReduce program developed for Hadoop 1.x can still on this YARN. And this is without
any disruption to processes that already work.