Chapter 3

Hadoop as Big Data System

3.1. Introduction

Big Data has been a sensation in the IT industry since 2008. Data generation is huge in the
current technological world. Many different sectors like telecom, social media, health care,
insurance, manufacturing, stocks, and many more are generating a huge amount of data.

Before the emergence of Hadoop, processing and storing large volumes of data was
challenging. But with the help of Hadoop and the Big Data ecosystem, industries have been
blessed in understanding large data volumes and driving themselves towards growth.

Here are some instances where you will find Apache Big Data Ecosystem very helpful:

 Costly machines for quality testing are not enough, but capturing data and analyzing it
would reveal patterns for improvement.
 Capturing data and analyzing it will also help you in determining your customer’s
interests.
 Stock markets have tons of data, which will correlate every time and with a Big Data
ecosystem, you can get great insights.
 Forecasting any product for the future is made easy.

Many useful insights on applications have been found with the emergence of the Big Data
ecosystem. Hadoop machine learning is the answer to the dwelling problem of analyzing large
amounts of data. The Hadoop Big Data ecosystem is a combination of different technologies
which has proficiency in solving data-related problems.

What is Hadoop?

Organizations have been using traditional systems like data warehouses and relational
databases for decades to analyze and store their data. But the data generated today are too
huge for those systems to handle.

More semi-structured or unstructured data are usually generated nowadays and the previously
used systems were meant to handle structured data. Those systems were vertically scalable
which required adding more storage, processor, and memory to the system which made it
expensive.

Also, the data from today are stored in different silos, which together when analyzed, will help
you in finding a pattern that might not be possible with those machines. This is where the
Hadoop comes in and turns all your problems into successful solutions.

Apache Hadoop is an open-source framework that deals with large volumes of data in a
distributed environment. It is designed based on the Google File System which was created by
Google. This environment is built of different machines which will work together for providing a
single machine.

Advantages of Hadoop to solve the “Big Data” problems-

1
a. Storage for Big Data – HDFS Solved this problem. It stores Big Data in Distributed Manner.
HDFS also stores each file as blocks. Block is the smallest unit of data in a filesystem.

Suppose you have 512MB of data. And you have configured HDFS such that it will create 128Mb
of data blocks. So HDFS divide data into 4 blocks (512/128=4) and stores it across different
DataNodes. It also replicates the data blocks on different datanodes. Hence, storing big data will
not be a challenge.

b. Scalability – It also solves the Scaling problem. It mainly focuses on horizontal scaling rather
than vertical scaling. You can add extra datanodes to HDFS cluster as and when required.
Instead of scaling up the resources of your datanodes. Hence enhances performance
dramatically.

c. Storing the variety of data – HDFS solved this problem. HDFS can store all kind of data
(structured, semi-structured or unstructured). It also follows write once and read many models.
Due to this, you can write any kind of data once and you can read it multiple times for finding
insights.

d. Data Processing Speed – This is the major problem of big data. In order to solve this problem,
move computation to data instead of data to computation. This principle is Data locality.

Some Disadvantage of Apache Hadoop Framework is given below-

 Security concerns – It can be challenging in managing the complex application. If the

user doesn’t know how to enable platform who is managing the platform, then your
data could be a huge risk. Since, storage and network levels Hadoop are missing
encryption, which is a major point of concern.
 Vulnerable by nature – The framework is written almost in java, most widely used
language. Java is heavily exploited by cybercriminals. As a result, implicated in
numerous security breaches.
 Not fit for small data –Since, it is not suited for small data. Hence, it lacks the ability to
efficiently support the random reading of small files.
 Potential stability issues – As it is an open source framework. This means that it is
created by many developers who continue to work on the project. While constantly
improvements are made, It has stability issues. To avoid these issues organizations
should run on the latest stable version.

Hadoop Ecosystem

The Hadoop Ecosystem is a platform that will give you different services for solving your data
problems. The Hadoop components include Apache projects and other commercial tools for
finding a perfect solution.

All these tools will work together to collectively provide services like analysis, absorption,
storage, and maintenance of your data. If you want to understand the Hadoop Ecosystem, the
following diagram and this article will guide you through it.

These changes have made the organization support the Hadoop Ecosystem massively. Now,
organizations have started hiring more Hadoop developers who are knowledgeable in this
platform for improving their customer experience and solving their big data challenges.

2
3.2. Core Components of the Hadoop Ecosystem
The following Hadoop components in Big Data will help in understanding the ecosystem better
and also help you handle your data efficiently.

HDFS, YARN, and Map Reducer are the core components of the Hadoop Ecosystem.

3.2.1 HDFS

HDFS (Hadoop Distributed File System) is Hadoop’s primary storage system, built for high-
throughput access to large datasets. It runs on inexpensive commodity hardware and stores
data in large blocks to optimize performance. HDFS ensures fault tolerance and high availability
across the cluster.

HDFS Architecture Components:

 NameNode (Master Node): The master node in HDFS that stores metadata (not actual
data), manages file operations and directs clients to nearest DataNode for efficient
access.
 DataNode (Slave Node): Stores actual data blocks, serves read/write requests and
reports to NameNode. Supports replication (default 3) for fault tolerance and scales
storage and performance with more nodes.

High Level Architecture Of Hadoop

3
File Block In HDFS: In HDFS data is always stored in the form of blocks. By default, each block is
128MB in size, although this value can be manually configured depending on the use case
(commonly increased to 256MB or more in modern systems).

Suppose you upload a file of 400MB to HDFS. Hadoop will divide this file into blocks as follows:

128MB + 128MB + 128MB + 16MB = 400MB

This creates four blocks three of 128MB and one of 16MB. Hadoop splits files purely by size, not
content, so a single record can span across two blocks.

A. Comparison with Traditional File Systems

4
  Traditional file systems use small blocks (e.g., 4KB), while HDFS uses large blocks (128MB
or more).
 Larger blocks in HDFS reduce metadata and I/O overhead, improving scalability and
efficiency for big data processing.

B. Replication In HDFS:

HDFS replication ensures data availability and fault tolerance by storing multiple copies of each
block.

 Default Replication Factor: 3 (configurable in [Link])

 If a file is split into 4 blocks, with a replication factor of 3: 4 blocks × 3 replicas = 12 total
blocks

Designed for commodity hardware, where failures are common—replication prevents data loss.
While it increases storage usage, reliability is prioritized over space efficiency.

C. Rack Awareness:

A rack is a group of machines (typically 30–40) in a Hadoop cluster. Large clusters have many
racks. Rack Awareness helps NameNode to:

 Choose the nearest DataNode for faster read/write operations.

 Reduce network traffic by minimizing inter-rack data transfer.

This improves overall performance and efficiency in data access.

3.2.2. YARN

5
YARN stands for Yet Another Resource Negotiator. It is the resource management layer of
Hadoop. It was introduced in Hadoop 2.

YARN is designed with the idea of splitting up the functionalities of job scheduling and resource
management into separate daemons. The basic idea is to have a global ResourceManager and
application Master per application where the application can be a single job or DAG of jobs.

YARN consists of ResourceManager, NodeManager, and per-application ApplicationMaster.

1. ResourceManager

It arbitrates resources amongst all the applications in the cluster.

It has two main components that are Scheduler and the ApplicationManager.

a. Scheduler

 The Scheduler allocates resources to the various applications running in the cluster,
considering the capacities, queues, etc.
 It is a pure Scheduler. It does not monitor or track the status of the application.
 Scheduler does not guarantee the restart of the failed tasks that are failed either due to
application failure or hardware failure.
 It performs scheduling based on the resource requirements of the applications.
6
b. ApplicationManager

 They are responsible for accepting the job submissions.

 ApplicationManager negotiates the first container for executing application-specific
ApplicationMaster.
 They provide service for restarting the ApplicationMaster container on failure.
 The per-application ApplicationMaster is responsible for negotiating containers from the
Scheduler. It tracks and monitors their status and progress.

2. NodeManager:

NodeManager runs on the slave nodes. It is responsible for containers, monitoring the machine
resource usage that is CPU, memory, disk, network usage, and reporting the same to the
ResourceManager or Scheduler.

3. ApplicationMaster:

The per-application ApplicationMaster is a framework-specific library. It is responsible for

negotiating resources from the ResourceManager. It works with the NodeManager(s) for
executing and monitoring the tasks.

3.2.3 MapReduce

It is the data processing layer of Hadoop. It is a software framework for writing applications that
process vast amounts of data (terabytes to petabytes in range) in parallel on the cluster of
commodity hardware.

The MapReduce framework works on the <key, value> pairs.

The MapReduce job is the unit of work the client wants to perform. MapReduce job mainly
consists of the input data, the MapReduce program, and the configuration information. Hadoop
runs the MapReduce jobs by dividing them into two types of tasks that are map tasks and
reduce tasks. The Hadoop YARN scheduled these tasks and are run on the nodes in the cluster.

7
Due to some unfavorable conditions, if the tasks fail, they will automatically get rescheduled on
a different node.

The user defines the map function and the reduce function for performing the MapReduce job.

The input to the map function and output from the reduce function is the key, value pair.

The function of the map tasks is to load, parse, filter, and transform the data. The output of the
map task is the input to the reduce task. Reduce task then performs grouping and aggregation
on the output of the map task.

The MapReduce task is done in two phases-

1. Map phase

a. RecordReader

Hadoop divides the inputs to the MapReduce job into the fixed-size splits called input splits or
splits. The RecordReader transforms these splits into records and parses the data into records
but it does not parse the records itself. RecordReader provides the data to the mapper function
in key-value pairs.

b. Map

In the map phase, Hadoop creates one map task which runs a user-defined function called map
function for each record in the input split. It generates zero or multiple intermediate key-value
pairs as map task output.

The map task writes its output to the local disk. This intermediate output is then processed by
the reduce tasks which run a user-defined reduce function to produce the final output. Once the
job gets completed, the map output is flushed out.

c. Combiner

Input to the single reduce task is the output from all the Mappers that is output from all map
tasks. Hadoop allows the user to define a combiner function that runs on the map output.

Combiner groups the data in the map phase before passing it to Reducer. It combines the
output of the map function which is then passed as an input to the reduce function.

d. Partitioner

When there are multiple reducers then the map tasks partition their output, each creating one
partition for each reduce task. In each partition, there can be many keys and their associated
values but the records for any given key are all in a single partition.

Hadoop allows users to control the partitioning by specifying a user-defined partitioning

function. Generally, there is a default Partitioner that buckets the keys using the hash function.

2. Reduce phase:

8
The various phases in reduce task are as follows:

a. Sort and Shuffle:

The Reducer task starts with a shuffle and sort step. The main purpose of this phase is to collect
the equivalent keys together. Sort and Shuffle phase downloads the data which is written by the
partitioner to the node where Reducer is running.

It sorts each data piece into a large data list. The MapReduce framework performs this sort and
shuffles so that we can iterate over it easily in the reduce task.

The sort and shuffling are performed by the framework automatically. The developer through
the comparator object can have control over how the keys get sorted and grouped.

b. Reduce:

The Reducer which is the user-defined reduce function performs once per key grouping. The
reducer filters, aggregates, and combines data in several different ways. Once the reduce task is
completed, it gives zero or more key-value pairs to the OutputFormat. The reduce task output is
stored in Hadoop HDFS.

c. OutputFormat

It takes the reducer output and writes it to the HDFS file by RecordWriter. By default, it
separates key, value by a tab and each record by a newline character.

9
3.3. Data Access
There are several data access tools in a Hadoop Ecosystem, you can start by using the following
tools:

Apache Pig

The pig was developed to analyze large datasets. Pig helps to overcome any difficulty of writing
maps and reducing functions. And it has two parts:

 Pig Latin - It is a language.

 Runtime - The runtime for executing the code in any environment.

Internally, the code written in Pig converts into MapReduce functions and helps programmers
(familiar with Java).

The compiler in Pig will convert Pig Latin code into MapReduce internally. It will produce a
sequential set of jobs that are an abstraction. It was developed by Yahoo and it will provide you
with a platform to build data flow for ETL, analyzing, and processing large datasets.

Apache Hive

Apache Hive is a distributed data warehouse system developed by Facebook and is used to
analyze structured data. Hive is built on top of Hadoop and operates on the server side of a
cluster. The hive is commonly used by Data Analysts to create reports.

Hive help to perform reading, writing, and managing large data sets in a distributed
environment using a SQL-like interface. Hive supports Data Definition Language (DDL), Data
Manipulation Language (DML), and User-Defined Functions (UDF). Hive is not capable of
handling real-time data and online transaction processing. It has two components:

 Hive Command-Line interface runs HQL commands Java Database Connectivity (JDBC)
 Object Database Connectivity (ODBC) driver establishes connections from data storage.

3.4. Data Storage

HBase and Cassandra are the two popularly used tools in the Hadoop Ecosystem for data
storage. Here is more about them.

HBase

HBase is a column-based NoSQL database and runs on top of HDFS to handle any kind of data.
HBase allows for real-time processing. The HBase is written in Java, and its applications can be
written in REST, Avro, and Thrift APIs.

10
It will support different data types and it will be capable of handling the entire Hadoop
Ecosystem. It is modeled after Google’s BigTable which is a distributed storage system for
coping with larger datasets.

Cassandra

A NoSQL database that is designed for high availability and linear scalability is Cassandra. It is
based on the key-value model and it is developed by Facebook. It provides swift responses to
queries. It has column indexes and supports de-normalization. It will have materialized views
and is influential in caching.

3.5. Interaction - execution & development

Execution and development of the Hadoop Ecosystem in Big Data are done with the following
tools.

Hcatalog

A table management layer that provides integration of hive metadata for other Hadoop
technologies. It allows the user with various processing tools to read and write data easily. It will
provide a tabular view of various formats and provide data availability of different notifications.
The REST APIs for the external system will help you in accessing metadata.

Apache Crunch

11
Apache Crunch is built to pipeline MapReduce programs that are efficient and simple. This
framework is used to write, test, and running MapReduce pipelines. It is developer-focused,
offers a flexible data model, and has minimal abstractions.

Apache Hama

A distributed framework that depends on BSP (Bulk Synchronous Parallel) computing is known
as Apache Hama. It is capable of massive computations like graph, network, and matrix
algorithms. It is a well-suited iterated algorithm that supports YARN. Hama will provide
collaborative filtering unsupervised machine learning applications.

Apache Solr & Lucene

Apache Solr and Apache Lucene are two different services. The relationship between both is
that of a car and its engine. We are incapable of driving an engine, but we can drive a car.
Similarly, Lucene is a programmatic library that may not use as-is, whereas Solr is a complete
application that you can use out-of-box.

3.6 Data Intelligence

The Hadoop tools for understanding data intelligence are explained below:

Apache Drill

Apache Drill is an open-source application for analyzing larger datasets in distributed

environments. It will support various NoSQL databases. And will follow the ANSI SQL combining
different kinds of data stores in one query.

The major reason for this is providing scalability that will help you in processing petabytes and
exabytes of data very effectively. It is a replica of the Google Dremel. Apache Drill has a very
powerful scalability factor that supports many users by serving their query requests over large-
scale data.

Apache Mahout

Apache Mahout is an open-source project, that provides an environment for creating machine
learning applications that are scalable. Apache Mahout performs recommendation,
classification, and clustering machine learning techniques.

Machine learning algorithms allow you to build self-learning machines that will evolve around
themselves, which are being programmed explicitly. According to user behaviors, past
performances, and data patterns, it will make vital decisions. You can also name it AI.

Mahout performs a collaborative filter, classification, and cluster protocols which are explained
below:

12
  Collaborative filter - It will mimic user behaviors, and their characteristics, and the data patterns
accordingly will predict what the users must do.
 Collaborative classification - It will classify and categorize the data into different sub-departments and
categories.
 Collaborative cluster - It will help you with organizing similar data groups.
 Missing frequent item set - The Mahout will check objects that might appear together and suggest if
anything goes missing.

Mahout will provide a command line that will invoke different algorithms. It will have a
predefined library that contains different inbuilt algorithms for various use cases.

Apache Spark

Apache Spark is used for real-time data analytics in a distributed computing environment. It is
written in Scala and supports many different languages. Apache Spark is 100x faster than
MapReduce when processing large-scale data as it will perform in-memory computations and
optimizations.

Also, Apache Spark Framework has its ecosystem with the following components:

 Spark core - It will act as an execution engine for Spark and other APIs that are built on
top of it.
 Streaming API - It will enable Spark to handle real-time data and integrate it with other
data sources.
 MLlib - It is a scalable machine learning library that enables Data Science tasks.
 Spark SQL API - It stores query structured data in DataFrames.
 GraphX - It is a graph computation engine that works with the ecosystem.

3.4. Serialization
Apache Avro and Apache Thrift are some essential Hadoop tools for performing data
serialization.

Apache Avro

A data serialization framework that is language neutral is known as Apache Avro. It is designed
for performing across languages which allow data to potentially outlive the language for reading
and writing it.

Apache Thrift

A language developed for building interfaces that interact with the Hadoop-built technologies is
Apache Thrift. It is used for defining and creating services for many different languages.

3.5. Integration
Some Hadoop Ecosystem tools that will help you with data integration:

Apache Chukwa

13
Apache Chukwa is an open-source data collection system that will help you in monitoring larger
distributed systems. It is built on the HDFS and MapReduce framework. It inherited its
robustness and scalability from the Hadoop Ecosystem.

It includes a powerful and flexible tool for monitoring, analyzing, and displaying results and
making the best use of the collected data. The Kafka routing service plays a major role in moving
data from Kafka to different sinks.

Apache Samza is used for this routing. When Apache Chukwa sends its traffic to Kafka, it
delivers full streams or filtered ones as per our request. Sometimes, we should apply many
filters to the streams which will be written using Chukwa. The router should perform one Kafka
topic for producing a new Kafka.

Apache Sqoop

Apache Sqoop is a structured data ingesting service that can import and export structured data
from RDBMS or Enterprise data warehouses to HDFS or vice versa. It performs with nearly all
relational databases like MySQL, Postgres, SQLite, and more.

When we submit the Sqoop command, the main task converts internally into MapReduce tasks
which are then executed over HDFS. Together, all the Map tasks import the whole data. When
we submit our job, it is mapped and will bring the chuck of data from HDFS. These chunks will
then be exported to a structured destination.

Apache Flume

Intake of the data is a vital part of the Hadoop Ecosystem and Apache Flume plays a major role
in it. It is a service that will help you with the process of intaking unstructured data into HDFS. It
will also provide a solution that is distributed and reliable.

It will also help you with aggregating, moving, and collecting large datasets. It will also help you
with the online streaming of data from different sources into HDFS. A Flume agent will ingest
the streaming data from various sources to HDFS.

There are three main components of the Flume agent as follows:

 Source - It will accept the data from the streamline and will store it in the channel.
 Channel - It will act as local storage. It is temporary storage between the data source and the persistent
data in the HDFS.
 Sink - Sink will collect the data from the channel and will write it permanently to the HDFS.

3.6. Management & Support

Here are some of the Hadoop tools that will help you with support and management of your
Hadoop Ecosystem:

Apache Zookeeper

14
Apache Zookeeper is an open-source that coordinates with various services in a distributed
environment. It was challenging and time-consuming to coordinate between different services
in the Hadoop Ecosystem.

To solve these problems, Zookeeper was introduced. Zookeeper saves up time by following the
process of synchronization, configuration maintenance, grouping, and naming. The Apache
Zookeeper is a replica of the Google Dremel

It will also support different kinds of NoSQL databases and filesystems similar to the Apache
Drill. Rackspace, eBay, and Yahoo are some of the bigshots which are using Apache Zookeeper
and are benefiting from it.

Apache Oozie

Oozie is a workflow scheduler to schedule Hadoop jobs and bind these together as one logical
work. Oozie helps in job scheduling in advance and creating a pipeline of individual jobs to be
executed sequentially or in parallel to achieve a sizable task.

Apache Oozie jobs are of two types:

 Workflow - The Apache Oozie Workflow is a set of actions that need to be performed. It
is performed like a relay race where one task is completed to proceed to the next.
 Coordinator - These jobs will trigger when the data is available in it. It is a response-
stimuli system that will answer when there is a new task available, else will be idle.

Apache Ambari

Apache Ambari is an open-source administration tool deployed on top of the Hadoop cluster
and is responsible for keeping track of running applications and their status. In other words,
consider an open-source web-based management tool that manages, monitors, and provisions
the health of Hadoop clusters.
15
Apache Ambari will provide:

 Provisioning - It will provide a step-by-step installing process for your Hadoop services
across different hosts. It will also handle the configuration of Hadoop services in a
cluster.
 Management - It will give the central management system to start, re-configure, or stop
a Hadoop service across the cluster.
 Monitoring - It will help you with a dashboard for monitoring the health and status of
your clusters. It has an Amber Alert Framework which will notify the user when there is
an emergency.

Conclusion
Organizations can consider using the Hadoop tools for solving most of their Big Data problems.
Hadoop is an open-source platform that can be used if you have the necessary skills. Also,
Hadoop is a highly scalable and easy-to-use software that doesn’t require you to invest a huge
amount in the infrastructure.

The success of the Hadoop Ecosystem is mainly because of the developer communities that
contributed to increasing its productivity. However, inside the Hadoop Ecosystem,
understanding one tool will not help you in coming up with the perfect solution, you need lots
of tools to resolve your Big Data problems, so when you want to achieve high success ensure
you read a set of relatable and needful tools of all types to make use of it.

16