Overview of Hadoop

Hadoop is an open-source distributed computing framework designed to process and store large volumes of data
across clusters of commodity hardware. It was originally developed by Doug Cutting and Mike Cafarella in 2005
and is now maintained by the Apache Software Foundation. Core components of the Hadoop ecosystem include:
• Hadoop Distributed File System (HDFS): A distributed file system designed to store large datasets across
multiple machines.
• MapReduce: A programming model and processing engine for distributed data processing.
• Yet Another Resource Negotiator (YARN): A resource management and job scheduling framework.

Hadoop Distributed File System (HDFS):

HDFS is the primary storage system used by Hadoop for storing large datasets. It is designed to provide high
throughput access to data and is optimized for batch processing workloads. Key features of HDFS include:
• Scalability: HDFS can scale horizontally to accommodate petabytes of data by adding more nodes to the
cluster.
• Fault Tolerance: HDFS replicates data across multiple nodes to ensure data availability in case of node
failures.
• Data Locality: HDFS stores data closer to where it will be processed, reducing network overhead.
Architecture of HDFS

• Master slave architecture

• Comprises of NameNode and a number of DataNodes.
• NameNode is the master that manages the various DataNodes.
• The NameNode manages HDFS cluster metadata
• DataNodes store the data.
• Records and directories are presented by clients to the NameNode.
• These records and directories are managed by NameNode.
• Operations on them, such as their modification or opening and closing them are performed by the
NameNode.
• On the other hand, internally, a file is divided into one or more blocks, which are stored in a group of
DataNodes.
• DataNodes read and write request from the clients.
• DataNodes can also execute operations like the creation, deletion and replication of the blocks depending
on the instructions from the NameNode.
• DataNode and NameNode communicate through continuous signals (“Heartbeat”)
• If signal is not heard by either of the two, the node is considered to have failed and would be no longer
available
• The failed node is replaced and replication scheme is also changed
 • Disk has certain block size
• It is the basic measure of information it can read or compose
• File systems expand by managing information in pieces, which are an indispensable part of the disk block
size.
• File system blocks are commonly a couple of kilobytes in size
• Disk blocks are regularly 512 bytes
• HDFS block are huge in contrast to disk blocks because they have to minimize the expense of seek
operation.
• The time to transfer a huge record made of multiple blocks operate at the disk exchange rate.

MapReduce:
MapReduce is a programming model and processing engine used for parallel processing of large datasets across
a distributed cluster. It consists of two main phases:
• Map: The input data is divided into smaller chunks and processed in parallel across multiple nodes.
• Reduce: The intermediate results produced by the map phase are aggregated and processed to generate the
final output.
Map reduce architecture
 Map reduce algorithm for word count program

Step 1: Setup Hadoop Environment

• Ensure that Hadoop is properly installed and configured on your system.
• Make sure you have Java installed and configured correctly.
• Set up Hadoop environment variables such as HADOOP_HOME, JAVA_HOME, and add Hadoop bin
directory to the system PATH.
Step 2: Create Input Data
• Prepare input data in text format. This can be a single file or multiple files containing text documents,
such as novels, articles, or any text data you want to analyze.
Step 3: Write Mapper Class
• Create a Java class for the Mapper. This class will extend the Mapper class provided by Hadoop.
• Implement the map method, which takes a key-value pair representing a line of text as input.
• Tokenize the input text into words and emit key-value pairs, where the key is the word and the value is 1.
Step 4: Write Reducer Class
• Create a Java class for the Reducer. This class will extend the Reducer class provided by Hadoop.
• Implement the reduce method, which takes a key and an iterable of values as input.
• Sum up the values (counts) for each key (word) to get the total count of occurrences for that word.
• Emit key-value pairs representing the word and its total count.

Step 5: Write Driver Program

• Create a Java class for the driver program.
• Set up the Hadoop job configuration using JobConf.
• Specify input and output paths.
• Set mapper and reducer classes.
• Submit the job to the Hadoop cluster for execution using [Link](conf).
Step 6: Compile and Package
• Compile all Java classes and package them into a JAR file.
• Include any necessary dependencies in the JAR file.
Step 7: Configure Hadoop
• Upload input data to HDFS using hadoop fs -put.
• Create output directory on HDFS using hadoop fs -mkdir.
• Ensure that Hadoop is running and the HDFS is accessible.
Step 8: Execute Job
• Run the Hadoop job using the hadoop jar command, specifying the JAR file, input directory, output
directory, and other parameters as required.
Step 9: View Output
• Once the job completes successfully, view the output generated by the reducer in the specified output
directory on HDFS using hadoop fs -cat or hadoop fs -get.

Yet Another Resource Negotiator (YARN):

YARN is a resource management and job scheduling framework introduced in Hadoop 2.0. It separates the
resource management and job scheduling functions from the MapReduce framework, allowing multiple data
processing engines to run on the same cluster. YARN provides a more flexible and efficient resource allocation
mechanism, enabling better utilization of cluster resources.

Hadoop Ecosystem:
The Hadoop ecosystem consists of a wide range of tools and frameworks that complement the core components
of Hadoop. Some popular components of the Hadoop ecosystem include:
 • Apache Hive: A data warehouse infrastructure built on top of Hadoop for querying and analyzing large
datasets using SQL-like queries.
• Apache HBase: A distributed, scalable NoSQL database for real-time access to data stored in Hadoop.
• Apache Spark: A fast and general-purpose cluster computing framework for processing large datasets.
• Apache Pig: A high-level data flow scripting language and execution framework for parallel data
processing.
• Apache Sqoop: A tool for efficiently transferring data between Hadoop and relational databases.
• Apache Ambari and Apache ZooKeeper: These are used to manage and monitor Hadoop clusters. These
tools provide features such as cluster provisioning, configuration management, and health monitoring to
ensure the stability and performance of the cluster.

The architecture of Hadoop

It is designed to handle large-scale data processing and storage across distributed clusters of commodity hardware.
Here's an overview of the key components and their interactions within the Hadoop ecosystem:

• Application layer - Clients interact with the Hadoop ecosystem to submit jobs and retrieve results.
• Hadoop ecosystem includes various components such as HDFS, MapReduce, and YARN.
• HDFS is responsible for storing large datasets across distributed nodes.
• MapReduce is the programming model and processing engine for parallel processing of data.
• YARN is the resource management and job scheduling framework.
Setting up an Architecture – Real time example
Consider SRM University is looking to optimize its data management system to handle the vast amount of student
and faculty data generated each semester. The university wants a scalable solution that can efficiently store,
process, and analyze this data to improve decision-making and enhance student experiences. what would be the
architecture of Hadoop that could effectively address these requirements?
The architecture of Hadoop for ABC University's data management system would consist of several key
components:
1. Hadoop Distributed File System (HDFS): The foundation of the architecture, HDFS would be used to
store large volumes of student and faculty data across multiple nodes in a distributed manner. This would
ensure scalability and fault tolerance, allowing the university to store petabytes of data reliably.
2. MapReduce: MapReduce would be utilized for processing and analyzing the data stored in HDFS. This
programming model enables parallel processing of data across the distributed cluster, making it suitable
for tasks such as data aggregation, summarization, and analytics.
3. YARN (Yet Another Resource Negotiator): YARN would serve as the resource management layer of
the architecture, responsible for managing and allocating resources (CPU, memory, etc.) to various tasks
running on the Hadoop cluster. It would enable efficient utilization of cluster resources and support
multiple processing frameworks beyond MapReduce, such as Apache Spark and Apache Flink.
4. Apache Hive: Apache Hive would be used as a data warehouse infrastructure built on top of Hadoop,
providing a SQL-like interface for querying and analyzing the data stored in HDFS. This would enable
users at ABC University to perform ad-hoc queries and generate reports using familiar SQL syntax.
5. Apache HBase: Apache HBase would serve as the distributed, scalable NoSQL database for real-time
access to student and faculty data. It would enable random read/write access to individual records stored
in HDFS, making it suitable for applications requiring low-latency access to data, such as student portals
and faculty dashboards.
6. Apache Sqoop: Apache Sqoop would be used for efficiently transferring data between Hadoop and
external data sources such as relational databases, enabling seamless integration of existing data systems
with the Hadoop ecosystem.
7. Apache Flume or Apache Kafka: One of these streaming data ingestion tools would be used to collect
and ingest real-time data streams from various sources within the university, such as student enrollment
systems, faculty databases, and online learning platforms. This would ensure that the Hadoop cluster
remains up-to-date with the latest data for analysis.
Overall, this architecture would provide ABC University with a scalable, fault-tolerant, and cost-effective solution
for managing its vast amounts of student and faculty data, enabling the university to derive valuable insights and
improve decision-making processes.
Apache Pig:
Apache Pig is a high-level scripting language built on top of Hadoop that allows developers to write data
processing jobs using a simple and expressive scripting language called Pig Latin. Pig abstracts the complexities
of MapReduce programming and provides a more user-friendly way to process large datasets.
Real-time Example: Analyzing Web Server Logs
Consider a scenario where a company operates a website and wants to analyze the web server logs to gain insights
into user behavior, such as page views, user interactions, and navigation patterns. The web server logs contain a
vast amount of unstructured data, making it challenging to process and analyze using traditional methods.
To analyze the web server logs using Apache Pig, you can write a Pig Latin script to perform the following tasks:
1. Load the web server logs data from HDFS into Pig.
2. Parse and filter the logs to extract relevant information, such as IP addresses, timestamps, URLs, and
HTTP status codes.
3. Aggregate and summarize the data to calculate metrics such as page views, unique visitors, and popular
pages.
4. Perform additional data transformations and join operations as needed.
5. Store the processed data back into HDFS or another storage system for further analysis or visualization.
By using Apache Pig, developers can write complex data processing pipelines in a concise and readable manner,
enabling efficient analysis of large volumes of web server logs to extract valuable insights and improve the
performance and user experience of the company's website.

Apache Hive:
Apache Hive is a data warehouse infrastructure built on top of Hadoop that provides a SQL-like interface for
querying and analyzing large datasets stored in Hadoop's distributed file system (HDFS). Hive translates SQL
queries into MapReduce jobs, allowing users to interact with Hadoop data using familiar SQL syntax.
Real-time Example: Analyzing Customer Purchase Data
Consider a retail company that wants to analyze its customer purchase data to gain insights into customer
behavior, product sales, and revenue trends. The customer purchase data is stored in HDFS as a collection of
structured files in CSV or JSON format.
To analyze the customer purchase data using Apache Hive, you can write SQL queries to perform the following
tasks:
1. Create an external table in Hive to map to the data files stored in HDFS.
2. Write SQL queries to retrieve information such as customer demographics, product sales, and revenue by
date or category.
 3. Perform data aggregation, filtering, and grouping operations to calculate metrics such as total sales,
average order value, and customer lifetime value.
4. Join the customer purchase data with other datasets, such as product catalogs or customer demographics,
to enrich the analysis.
5. Visualize the results using business intelligence tools or dashboards to gain actionable insights and make
data-driven decisions.
By using Apache Hive, analysts and data scientists can leverage their SQL skills to interact with large datasets
stored in Hadoop, enabling efficient analysis and exploration of customer purchase data to drive business growth
and improve decision-making processes.
Hadoop Commands
List files from root directory
Find the present working directory
Home Directory
 Copy file to and from HDFS
• Find local file system directories
Copy files from local to hdfs
Copy files from hdfs to local
Check and modify replication
To see the configuration files
To print yarn nodes
chmod
find out how much disk space us used,
free, under-replicated, etc.
Disk usage command
Configuration commands
To display the groups
Load balancer
Empty Trash Data
We can remove an entire HDFS directory
using the rm command:
Merge a list of files in one directory on
HDFS into a single file on local file
system
 Remove a file and directory
• Remove a file from HDFS, similar to Unix rm command.
This command does not delete directories. For
recursive delete, use command -rm -r.
• Options:
-r : Recursively remove directories and files
-skipTrash : To bypass trash and immediately delete the
source
-f : Mention if there is no file existing
-rR : Recursively delete directories
• $ hadoop fs -mv /user/hadoop/[Link] /user/text/
• getmerge:
• Merge a list of files in one directory on HDFS
into a single file on local file system. One of
the most important and useful commands
when trying to read the contents of map
reduce job or pig job’s output files.
• $ hadoop fs -getmerge /user/data
Checksum
count
tail
chown
chgrp
change the replication factor
Disk file usage
put Command is used to copy one or multiple files from
local system to the Hadoop File System.
Get Download the file from HDFS to
the local file system.