Scribd
Upload
7 views3 pages

Module 2 Combiner, Partitioner, Distributed Cache

This document discusses optimization techniques in MapReduce, specifically Combiners, Partitioners, and Distributed Cache, which enhance performance and reduce network traffic in the Hadoop framework. Combiners aggregate data locally to minimize transfer, Partitioners assign keys to reducers for load balancing, and Distributed Cache provides shared access to small files for efficient processing. Each technique contributes to improved execution speed and overall performance in MapReduce tasks.

Uploaded by

sarveshff2004
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views3 pages

Module 2 Combiner, Partitioner, Distributed Cache

This document discusses optimization techniques in MapReduce, specifically Combiners, Partitioners, and Distributed Cache, which enhance performance and reduce network traffic in the Hadoop framework. Combiners aggregate data locally to minimize transfer, Partitioners assign keys to reducers for load balancing, and Distributed Cache provides shared access to small files for efficient processing. Each technique contributes to improved execution speed and overall performance in MapReduce tasks.

Uploaded by

sarveshff2004
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Module 2

LO2.3: Use of Combiners, Partitioners, and Distributed Cache to Optimize MapReduce Tasks

In MapReduce, optimization techniques are used to improve performance, reduce network traffic,
and speed up processing. Three important techniques are Combiners, Partitioners, and Distributed
Cache. These help in efficient execution of MapReduce programs in the Apache Software Foundation
Hadoop framework.

1 Combiner
Definition

A Combiner is an optional mini-reducer that runs after the Map phase and before the Reduce
phase. Reduce network [Link] performs local aggregation of data on the mapper node.

Purpose

 Reduces the amount of data transferred between Mapper and Reducer.


 Improves performance by minimizing network traffic.

Working

 Mapper produces intermediate key-value pairs.


 Combiner processes these pairs locally.
 It combines values with the same key into a smaller set.
 The reduced data is then sent to the Reducer.

Example: Word Count

Mapper Output:
(word, 1), (word, 1), (data, 1), (word, 1)

Combiner Output:
(word, 3), (data, 1)

Reducer Output:
Final count of each word.

Advantages

 Reduces data transfer.


 Improves execution speed.
 Saves bandwidth.
[Link]
Definition

A Partitioner determines which reducer will process a particular key.

Purpose/Role

 Assign key to reducer


 Distributes data evenly among reducers.
 Ensures load balancing.

Working

 After Combiner, the Partitioner assigns keys to reducers.


 Default partitioner uses hash function:
Partition = hash(key) % number_of_reducers

Example

If reducers = 2
Keys: Apple, Ball, Cat

Partition result:

 Reducer 0 → Apple, Cat


 Reducer 1 → Ball

Advantages

 Ensures efficient parallel processing.


 Prevents overloading of a single reducer.
 Improves overall performance.

[Link] Cache
Definition

Distributed Cache is used to store small files and make them available to all mapper and
reducer nodes.

Purpose

 Provides read-only access to shared data.


 Avoids repeated data transfer.

Working
 Small files (lookup tables, configuration files) are cached.
 These files are copied to each worker node.
 Mappers and reducers can access them locally.

Example

If student ID and name mapping is needed:


Distributed cache stores mapping file.
Mapper reads it locally instead of accessing network repeatedly.

Advantages

 Faster data access.


 Reduces network usage.
 Improves efficiency.

You might also like