MODULE-2

5.1 Introducing Hadoop

Introduction

 Today, Big Data has become a buzzword.

 Enterprises across the world are realizing that there is a huge volume of untapped
information.
 This data exists in the form of:
o Structured data
o Semi-structured data
o Unstructured data
 This varied variety of data is spread across networks.

Statistics on Data Generation

1. Data Generated Every Day

 NYSE (New York Stock Exchange) generates 1.5 billion shares and trade data.
 Facebook stores 2.7 billion comments and Likes.
 Google processes about 24 petabytes of data.

2. Data Generated Every Minute

 Facebook users share nearly 2.5 million pieces of content.

 Twitter users tweet nearly 300,000 times.
 Instagram users post nearly 220,000 new photos.
 You Tube users upload 72 hours of new video content.
 Apple users download nearly 50,000 apps.
 Email users send over 200 million messages.
 Amazon generates over $80,000 in online sales.
 Google receives over 4 million search queries.

3. Data Generated Every Second

 Banking applications process more than 10,000 credit card transactions.

 5.1.1 Data: The Treasure Trove

1. Provides business advantages such as generating product recommendations, inventing new

products. analyzing the market, and many, many more, ....

2. Provides few early key indicators that can turn the fortune of business.

3. Provides room for precise analysis. If we have more data for analysis, then we have greater
precision of analysis.

To process, analyze, and make sense of these different kinds of data, we need a system that scales
and addresses the challenges shown in Figure 5.1.

5.2 WHY HADОOР?

 Hadoop is one of the most popular technologies for Big Data processing.
 The key consideration (the rationale behind its huge popularity) is:
Its capability to handle massive amounts of data, different categories of data -fairly quickly.
 The other considerations are (Figure 5.2):
  Low Cost: Hadoop is an open-source framework that uses low-cost commodity hardware to
store large volumes of data.
 High Computing Power: Hadoop uses a distributed computing model to process very large
datasets quickly; more nodes increase processing power.
 Scalability: Hadoop allows easy expansion by adding nodes with minimal administration.
 Storage Flexibility: Hadoop stores data without pre-processing and supports unstructured data
such as images, videos, and free-form text.
 Inherent Data Protection: Hadoop provides fault tolerance by reassigning tasks during node
failure and storing multiple data replicas across the cluster.
 Hadoop makes use of commodity hardware, distributed file system, and distributed computing
as shown in Figure 5.3. In this new design, groups of machine are gathered together; it is known as a
Cluster.

 With this new paradigm, the data can be managed with Hadoop as follows:
 Hadoop distributes data and stores duplicate chunks of each data file across multiple nodes
(for example, 25–30 MB per chunk) as shown in Figure 5.3.
 Each data chunk is processed using locally available computing resources in parallel.
 The Hadoop Framework automatically and intelligently manages failover.

5.3 WHY NOT RDBMS?

 Relational Database Management System (RDBMS) is not suitable for storing

and processing large files, images, and videos.

 RDBMS is not an ideal choice for advanced analytics involving Machine

Learning (ML).
  Figure 5.4 illustrates the cost and storage limitations of RDBMS.
 As the volume of data increases, RDBMS requires a huge investment, making it
expensive and less scalable.

5.4 RDBMS versus HADООР

Table 5.1 describes the difference between RDBMS and Hadoop.

5.5 DISTRIBUTED COMPUTING CHALLENGES

Although there are several challenges with distributed computing, we will focus on two major
challenges.

5.5.1 Нardware Failure

 In a distributed system, several servers are networked together.

 There is a possibility of hardware failure.
 When failure occurs, retrieving stored data becomes a challenge.
 A regular hard disk may fail once in 3 years.
 If 1000 hard disks are used, a few may fail every day.
 Hadoop provides a solution using Replication Factor (RF).
 Replication Factor (RF) means the number of data copies of a given data item/data block
stored across the network.
 Refer Figure 5.5.

5.5.2 How to Process This Gigantic Store of Data?

 In a distributed system, the data is spread across the network on several machines.
 A key challenge is to integrate the data available on several machines prior to processing
it.
 Hadoop solves this problem by using MapReduce Programming.
 MapReduce Programming is a programming model to process the data.
5.6 HISTORY OF HADOOP

 Hadoop was created by Doug Cutting, the creator of Apache Lucene (a commonly used
text search library).
 Hadoop is a part of the Apache Nutch (Yahoo) project (an open-source web search
engine).
 Hadoop is also a part of the Lucene project.
 Refer Figure 5.6 for more details.

5.6.1 The Name "Hadoop"

 The name Hadoop is not an acronym; it is a made-up name.

 Doug Cutting said the name came from a stuffed yellow elephant owned by his child.
 The name was chosen because it is short, easy to spell and pronounce, and has no special
meaning.
 Many Hadoop subprojects and "contrib" modules also have unrelated names.
 These names often follow an animal theme, such as "Pig".

5.7 HADOOP OVERVIEW

 Hadoop is an open-source software framework.

 It is used to store and process massive amounts of data.
 It works in a distributed fashion on large clusters of commodity hardware.
 Hadoop performs two main tasks:
1. Massive data storage.
2. Faster data processing.

5.7.1 Key Aspects of Hadoop

Fgure 5.7 describes the key aspects of Hadoop.

5.7.2 Hadoop Components

Figure 5.8 depicts the Hadoop components.

Hadoop Core Components

1. HDFS:
(a) Storage component.
(b) Distributes data across several nodes.
(c) Natively redundant.
2. MapReduce:
(a) Computational framework.
(b) Splits a task across multiple nodes.
(c) Processes data in parallel.

Hadoop Ecosystem

Hadoop Core Components

1. HDFS:
(a) Storage component.
(b) Distributes data across several nodes.
(c) Natively redundant.

2. MapReduce:
(a) Computational framework.
(b) Splits a task across multiple nodes.
(c) Processes data in parallel.

Hadoop Ecosystem

Hadoop Ecosystem are support projects to enhance the functionality of Hadoop Core
Components.

The Eco Projects are as follows:

1. HIVE
2. PIG
3. SQOOP
4. HBASE
5. FLUME
6. OOZIE
7. MAHOUT

5.7.3 Наdoop Conceptual Layer

 It is conceptually divided into two layers.

 Data Storage Layer stores huge volumes of data.
 Data Processing Layer processes data in parallel.
 It helps to extract richer and meaningful insights from data (Figure 5.9).
5.7.4 High-Level Architecture of Hadoop

 Hadoop follows a distributed Master-Slave Architecture.

 The Master node is called NameNode.
 The slave nodes are called DataNodes.
 Figure 5.10 shows the Master-Slave Architecture of Hadoop Framework.

Key components of the Master Node:

1. Master HDFS:
o Its main responsibility is partitioning the data storage across the slave nodes.
o It keeps track of the locations of data on DataNodes.
2. Master MapReduce:
o It decides and schedules computation tasks on slave nodes.
5.8 USE CASE OF HADOOP

5.8.1 ClickStream Data – Short Notes (Easy Language)

 ClickStream data means mouse clicks made by users on a website.

 It helps to understand customer purchasing behavior.
 Online marketers use ClickStream analysis to improve web pages and advertisements.

Benefits of ClickStream analysis using Hadoop:

1. Hadoop combines ClickStream data with Customer Relationship Management Data

(Customer Demographics Data, Sales Data, Information on Advertising Campaigns).
o This helps to better understand customer behavior.
2. Hadoop can store large amounts of data for many years at low cost.
o It helps in year over year analysis.
3. Business analysts use Apache Pig and Apache Hive for website analysis.
o These tools help to organize, refine, and analyze ClickStream data.

5.10 HDFS (HADOOP DISTRIBUTED FILE SYSTEM)

Key Points of Hadoop Distributed File System (HDFS)

1. Storage component of Hadoop.

2. Distributed File System.
3. Modeled after Google File System.
4. Optimized for high throughput.
o HDFS leverages large block size.
o Moves computation where data is stored.
5. You can replicate a file for a configured number of times.
o Tolerant in terms of both software and hardware.
6. Re-replicates data blocks automatically on nodes that have failed.
7. You can realize the power of HDFS when you perform read or write on large files
(gigabytes and larger).
 8. Sits on top of native file system such as ext3 and ext4, which is described in Figure 5.13.

 Figure 5.14 describes important key points of HDFS.

 Figure 5.15 describes Hadoop Distributed File System Architecture.
 Client Application interacts with NameNode for metadata related activities.
 Client Application communicates with DataNodes to read and write files.
 DataNodes converse with each other for pipeline reads and writes.

Example:

 If the file "[Link]" is of size 192 MB and the default data block size is 64 MB,
 The file will split into three blocks.
 These blocks are replicated across the nodes in the cluster based on the default replication
factor.

5.10.1 HDFS Daemons

510.1.1 NameNode

 DFS breaks a large file into smaller pieces called blocks.

 NameNode uses a rack ID to identify DataNodes in a rack.
 A rack is a collection of DataNodes within the cluster.
 NameNode keeps track of which blocks of a file are placed on which DataNodes.
 NameNode manages file-related operations such as read, write, create, and delete.
 Its main job is managing the File System Namespace.
 File System Namespace is a collection of files in the cluster.
 NameNode stores HDFS namespace.
 File System Namespace includes mapping of blocks to file and file properties.
 This information is stored in a file called FsImage.
 NameNode uses EditLog (transaction log) to record every transaction related to file
system metadata.
 When NameNode starts, it reads FsImage and EditLog from disk.
 It applies all transactions from EditLog to the in-memory representation of FsImage.
 Then it writes the new version of FsImage to disk and truncates the old EditLog.
 There is a single NameNode per cluster.
 [Link] DataNode

 There are multiple DataNodes in a cluster.

 During pipeline read and write, DataNodes communicate with each other.
 DataNode continuously sends a "heartbeat" message to NameNode.
 Heartbeat ensures connectivity between NameNode and DataNode.
 If NameNode does not receive heartbeat from a DataNode,
o It assumes the DataNode has failed.
o NameNode replicates the data of that DataNode within the cluster.
o The system continues to run normally.
 This explains the concept of heartbeat report sent by DataNodes to NameNode.

[Link] Secondary NameNode

 Secondary NameNode takes a snapshot of HDFS metadata at regular intervals (as

specified in Hadoop configuration).
 It has the same memory requirements as NameNode.
 It is better to run NameNode and Secondary NameNode on different machines.
 If NameNode fails, Secondary NameNode can be configured manually to bring up the
cluster.
 Secondary NameNode does not record real-time changes in HDFS metadata.

5.10.2 Anatomy of File Read

Figure 5.18 describes the anatomy of File Read.

Steps Involved in File Read

1. The client opens the file by calling open() on the DistributedFileSystem.

2. DistributedFileSystem communicates with the NameNode to get the location of data
blocks.
o NameNode returns the addresses of the DataNodes where the data blocks are
stored.
o DistributedFileSystem returns an FSDataInputStream to the client.
3. The client calls read() on DFSInputStream.
o It connects to the closest DataNode for the first block of the file.
4. The client repeatedly calls read() to get data from the DataNode.
5. When the end of a block is reached, DFSInputStream closes the connection with that
DataNode.
o It then connects to the best DataNode for the next block.
6. After reading the complete file, the client calls close() on FSDataInputStream to close
the connection.
5.10.3 Anatomy of File Write

Figure 5.19 describes the anatomy of File Write. The steps involved in anatomy of File Write are as
follows:

1. The client calls create() on DistributedFileSystem to create a file.

2. DistributedFileSystem makes an RPC (Remote Procedure Call) to the NameNode to
create a new file.
3. NameNode checks whether the file already exists or not.
o It creates the file without associating any data blocks initially.
o DistributedFileSystem returns an FSDataOutputStream to the client for writing.
4. As the client writes data:
o Data is split into packets by DFSOutputStream.
o Packets are written into a data queue.
o DataStreamer reads from the data queue.
o DataStreamer asks NameNode to allocate new blocks.
o NameNode selects suitable DataNodes based on default replication factor (three).
o These DataNodes form a pipeline.
5. DataStreamer sends packets to the first DataNode.
o The first DataNode stores the packet and forwards it to the second.
o The second forwards it to the third DataNode in the pipeline.
6. DFSOutputStream also maintains an Ack queue.
o A packet is removed from the Ack queue only after acknowledgment from all
DataNodes in the pipeline.
7. When writing is complete, the client calls close().
o This flushes remaining packets to the pipeline.
o It waits for acknowledgments.
o Then it informs NameNode that file creation is complete.
5.10.4 Replica Placement Strategy

[Link] Hadoop Default Replica Placement Strategy

 According to Hadoop Replica Placement Strategy:

1. The first replica is placed on the same node as the client.

2. The second replica is placed on a node in a different rack.
3. The third replica is placed on the same rack as the second replica, but on a different node.

 After deciding replica locations, a pipeline is created.

 This strategy provides good reliability.
 Figure 5.20 describes the typical replica pipeline.

5.10.5 Working with HDFS Commands

 5.10.6 Special Features of HDFS

1. Data Replication:
o Client application does not need to track all blocks.
o HDFS directs the client to the nearest replica.
o This ensures high performance.
2. Data Pipeline:
o Client writes a block to the first DataNode in the pipeline.
o The first DataNode forwards the data to the next DataNode.
o This process continues until all replicas are written to disk.

5.11 PROCESSING DATA WITH HADOOР

MapReduce Programming

MapReduce Programming is a software framework.

 It helps to process massive amounts of data in parallel.

Working of MapReduce Programming:

1. The input dataset is split into independent chunks.

2. Map tasks process these chunks in parallel.
3. Output of map tasks is stored as intermediate data on local disk.
4. The framework automatically shuffles and sorts the mapper output.
5. Sorting is done based on keys.
6. The sorted output becomes input to reduce tasks.
7. Reduce tasks combine the output of various mappers to produce final output.
8. Job inputs and outputs are stored in a file system.
9. The framework handles scheduling, monitoring, and re-executing failed tasks.

 Hadoop Distributed File System and MapReduce Framework run on the same nodes.
 This allows Data Locality (tasks run where data is stored).
 This results in high throughput.
Daemons in MapReduce Programming:

1. One master JobTracker per cluster.

2. One slave TaskTracker per cluster node.

 JobTracker schedules tasks to TaskTrackers.

 It monitors tasks and re-executes them if failure occurs.
 TaskTracker executes the tasks. Refer Figure 5.21.

Job Execution:

 MapReduce applications define map and reduce functions and input/output locations.
 Application and job parameters together form job configuration.
 Hadoop job client submits job (jar/executable) to JobTracker.
 JobTracker schedules tasks to slave nodes.
 It also monitors tasks and provides status to the job client.

5.11.1 MapReduce Daemons

JobTracker and TaskTracker

1. JobTracker:

 Provides connectivity between Hadoop and the application.

 When code is submitted, JobTracker creates an execution plan.
 It decides which task should run on which node.
 It monitors all running tasks.
 If a task fails, it re-schedules the task to another node after predefined retries.
 JobTracker is a master daemon responsible for the overall MapReduce job.
 There is a single JobTracker per Hadoop cluster.

2. TaskTracker:

 Executes individual tasks assigned by JobTracker.

 There is a single TaskTracker per slave node.
 It spawns multiple Java Virtual Machines (JVMs) to run multiple map or reduce tasks in
parallel.
 TaskTracker continuously sends heartbeat message to JobTracker.
 If JobTracker does not receive heartbeat, it assumes TaskTracker has failed.
 The task is then reassigned to another available node.
 After the client submits a job, JobTracker partitions and assigns MapReduce tasks to each
TaskTracker in the cluster.
  Figure 5.22 shows JobTracker and TaskTracker interaction.

5.11.2 Нow Does MapReduce Work?

MapReduce Programming

 MapReduce divides a data analysis task into two parts: map and reduce.
 Figure 5.23 shows how MapReduce Programming works.
 In the example, there are two mappers and one reducer.
 Each mapper works on the partial dataset stored on its node.
 The reducer combines the output from the mappers to produce the final result.

 Figure 5.24 shows the working model of MapReduce Programming.

Steps in MapReduce Working:

1. First,the input dataset is split into several small subsets.

2. Next,the framework creates one master and several worker processes.
o Worker processes are executed remotely.
 3. Multiple map tasks run at the same time.
o Each map task reads its assigned data.
o Map function extracts required data.
o It generates key/value pairs.
4. Map worker uses a partitioner function.
o Partitioner decides which reducer will receive the mapper output.
5. After map tasks finish, master starts reduce workers.
o Reduce workers collect key/value data from map workers.
o Data is shuffled and sorted based on keys.
6. Reduce function is called for each unique key.
o Final output is written to the file.
7. After all reduce tasks are completed, master returns control to the user program.

5.11.3 MapReduce Example

 The famous example of MapReduce Programming is Word Count.

 Example: Count the number of occurrences of similar words across 50 files.
 This can be done using MapReduce Programming.
 Refer Figure 5.25.

Word Count MapReduce Programming using Java requires three things:

1. Driver Class:
o Specifies Job Configuration details.
2. Mapper Class:
o Overrides the Map Function based on the problem statement.
3. Reducer Class:
o Overrides the Reduce Function based on the problem statement.
 5.12 MANAGING RESOURCES AND APPLICATIONS WITH HADOOP YARN (YET ANOTHER RESOURCE
NEGOTIATOR)

Apache Hadoop YARN

 Apache Hadoop YARN is a sub-project of Hadoop 2.x.

 Hadoop 2.x is based on YARN architecture.
 YARN is a general processing platform.
 YARN is not limited to MapReduce only.
 Multiple applications can run on Hadoop 2.x.
 All applications share common resource management.
 Hadoop can be used for different types of processing such as:
o Batch processing
o Interactive processing
o Online processing
 o Streaming processing
o Graph processing
o And others.

5.12.1 Limitations of Hadoop 1.0 Architecture

 In Hadoop 1.0, HDFS and MapReduce are Core Components.

 Other components are built around the core components.

Limitations:

1. Single NameNode manages the entire namespace of the Hadoop Cluster.

2. It has a restricted processing model.
o Suitable mainly for batch-oriented MapReduce jobs.
3. Hadoop MapReduce is not suitable for interactive analysis.
4. Hadoop 1.0 is not suitable for machine learning algorithms, graphs, and other memory
intensive algorithms.
5. MapReduce handles both cluster resource management and data processing.
6. In this architecture, map slots may be full while reduce slots are empty, and vice versa.
o This causes resource utilization issues.
o Proper resource utilization needs improvement.

5.12.2 HDFS Limitation

NameNode Limitation and Solution in Hadoop 2.x

 NameNode stores all file metadata in main memory.

 Main memory has a limit on the number of objects it can store.
 As the system grows, NameNode can become overloaded.
 In Hadoop 2.x, this problem is solved using HDFS Federation.

5.12.3 Нadoop 2: HDFS

HDFS 2 consists of two major components:

1. Namespace Service:
o Manages file-related operations.
o Handles creating files, modifying files, and directories.
2. Block Storage Service:
o Manages DataNode cluster.
 o Handles data block storage and replication.

HDFS 2 Features

1. Horizontal scalability.

2. High availability.

HDFS Federation and High Availability

HDFS Federation:

 Uses multiple independent NameNodes for horizontal scalability.

 NameNodes work independently.
 No coordination is required between NameNodes.
 DataNodes act as common storage for blocks.
 DataNodes are shared by all NameNodes.
 All DataNodes register with each NameNode in the cluster.

High Availability of NameNode:

 Achieved using Passive Standby NameNode.

 In Hadoop 2.x, Active-Passive NameNode handles failover automatically.
 All namespace edits are recorded in shared NFS storage.
 Only one NameNode writes at a time (single writer).
 Passive NameNode reads edits from shared storage and keeps metadata updated.
 If Active NameNode fails, Passive NameNode automatically becomes Active.
 It then starts writing to shared storage.
 Figure 5.26 shows Active-Passive NameNode interaction.
 Figure 5.27 shows Hadoop 1.0 and Hadoop 2.0 architecture.
 5.12.4 Нadoop 2 YARN: Taking Hadoop beyond Batch

 YARN helps to store all data in one place.

 It allows interaction in multiple ways.
 It provides predictable performance and quality of services.
 YARN was originally architected by Yahoo.
 Refer Figure 5.28.

[Link] Fundamental Idea

YARN Architecture

 The main idea of YARN architecture is to divide JobTracker responsibilities into two
parts:
o Resource Management
o Job Scheduling and Monitoring

Daemons in YARN Architecture:

1. Global ResourceManager:

 Distributes resources among different applications.

 It has two components:

(a) Scheduler:
  Allocates resources to running applications.
 It is only a scheduler.
 It does NOT monitor or track application status.

(b) ApplicationManager:

 Accepts job submissions.

 Negotiates resources (container) to run ApplicationMaster.
 Restarts ApplicationMaster if failure occurs.

2. NodeManager:

 Runs on each machine (slave daemon).

 Launches application containers.
 Monitors resource usage (memory, CPU, disk, network).
 Reports resource usage to ResourceManager.

3. ApplicationMaster (Per-application):

 Application-specific component.
 Negotiates required resources from ResourceManager.
 Works with NodeManager to execute and monitor tasks.

[Link] Basic Concepts

Application:

1. Application is a job submitted to the framework.

2. Example – MapReduce Job.

Container:

1. Container is the basic unit of allocation.

2. It provides fine-grained resource allocation across multiple resource types such as
Memory, CPU, disk, network, etc.
o (a) container_0 = 2GB, 1CPU
o (b) container_1 = 1GB, 6CPU
3. Container replaces the fixed map/reduce slots.
YARN Architecture:

Figure 5.29 depicts YARN architecture. The steps involved in YARN architecture are as follows:

1. Client program submits the application.

o It includes specifications to launch ApplicationMaster.
2. ResourceManager launches the ApplicationMaster by assigning a container.
3. ApplicationMaster registers with ResourceManager after starting.
o Client can query ResourceManager for application details.
4. ApplicationMaster negotiates required resource containers using resource-request
protocol.
5. After container allocation, ApplicationMaster asks NodeManager to launch the container.
6. NodeManager runs the application code.
o It sends progress and status to ApplicationMaster.
7. During execution, client communicates directly with ApplicationMaster.
o Client receives status and progress updates.
8. After completion, ApplicationMaster deregisters from ResourceManager.
o It shuts down and releases the container.

CHAPTER 8 : Introduction to MAPREDUCE Programming

8.1 INTRODUCTION

MapReduce Job Execution

 In MapReduce Programming, Jobs (Applications) are divided into map tasks and reduce
tasks.
 These tasks run in a distributed manner on the Hadoop cluster.
 Each task processes a small subset of assigned data.
 MapReduce job takes input files stored in HDFS (Hadoop Distributed File System).

Map Task Responsibilities:

 Loading data
 Parsing data
 Transforming data
 Filtering data

Phases of Map Task:

1. RecordReader
2. Mapper
3. Combiner
4. Partitioner

 Output of map task is called intermediate keys and values.

 These intermediate keys and values are sent to reducer.

Phases of Reduce Task:

1. Shuffle
2. Sort
3. Reducer
4. Output Format

 Hadoop assigns map tasks to the DataNode where data is stored.

 This ensures Data Locality.
 Data Locality means data is not moved over the network.
 Only computational code is moved to process data.
 This saves network bandwidth.

8.2 MAPPER

Mapper and Its Phases – Short Notes (Easy Language)

  A mapper converts input key-value pairs into intermediate key-value pairs.
 Map tasks transform input records into intermediate key-value pairs.

Phases of Mapper:

1. RecordReader:

 Converts byte-oriented input (from InputSplit) into record-oriented view.

 Provides key and value to Mapper.
 Key is usually positional information.
 Value is a chunk of data that forms the record.

2. Map:

 Works on key-value pair from RecordReader.

 Generates zero or more intermediate key-value pairs.
 Key-value pair is decided based on context.

3. Combiner:

 Optional function.
 Improves performance (saves network bandwidth and disk space).
 Applies user-defined aggregate function on mapper output.
 Also called local reducer.

4. Partitioner:

 Takes intermediate key-value pairs from mapper.

 Divides them into shards.
 Sends each shard to a specific reducer based on user-defined logic.
 Keys with same values usually go to the same reducer.
 Partitioned data is stored on local disk and later pulled by reducer.

8.3 REDUCER

Reducer and Its Phases

 The main work of Reducer is to reduce intermediate values that share the same key into a
smaller set of values.
 Reducer has three main phases:
1. Shuffle and Sort
2. Reduce
 3. Output Format

1. Shuffle and Sort:

 Takes output from all partitioners.

 Downloads data to the local machine where reducer runs.
 Data is sorted by keys.
 Similar keys are grouped together.
 This makes it easy for reduce task to process values of the same key.

2. Reduce:

 Works on grouped data from shuffle and sort phase.

 Applies reduce function.
 Processes one key group at a time.
 Iterates all values related to a key.
 Performs operations like aggregation, filtering, and combining data.
 Produces zero or more key-value pairs.
 Output is sent to output format.

3. Output Format:

 Separates key and value using tab (default).

 Writes the final output to a file using record writer.

Figure 8.1 describes the chores of Mapper, Combiner, Partitioner, and Reducer for the word count
problem. The Word Count problem has been discussed under "Combiner" and "Partitioner".
8.4 COMBINER

 Combiner is an optimization technique for MapReduce Job.

 Usually, the reducer class is set as the combiner class.

Difference between Combiner and Reducer:

1. Output of combiner is intermediate data.

o It is sent to the reducer.
2. Output of reducer is final output.
o It is written to output file on disk.

The sections have been designed as follows:

 Objective: What we are trying to achieve.

 Input Data: Data given for processing.
 Act: Statement or command to perform the task.
 Output: Result after executing the task.
8.5 PARTITIONER

Partitioning Phase

 Partitioning phase occurs after map phase and before reduce phase.
 Number of partitions is usually equal to the number of reducers.
 The default partitioner is Hash Partitioner.
8.6 SEARCHING
8.7 SORTING
8.8 COMPRESSION

In MapReduce programming, you can compress the output files. Compression provides two
main benefits:

1. Reduces storage space – compressed files take up less space on HDFS.

2. Speeds up data transfer – smaller files move faster across the network.

You can specify the compression format in the Driver Program as follows:

[Link]("[Link]", true);
[Link]("[Link]",
[Link],
[Link]);

Here:

 codec refers to the implementation of a compression and decompression algorithm.

 GzipCodec is the compression algorithm used for gzip.

This configuration compresses the output files generated by the MapReduce job.