Map-Reduce

Map Reduce is an execution model in Hadoop frame work. It divides the process into two
separate phases
1. Mapper
2. Reducer
Mapper
Mapper takes input from raw-input-file (HDFS) and output of the mapper is called
shuffled data (intermediate result). This output will be sent to Reducer which aggregates our
data and writes result into HDFS.
Shuffling and sorting the data
Making multiple rows of same key into single row (it eliminates duplicate keys) and
arranges them in the sorted order of keys.
Example

In high level terminology,

 Mapper input should be in map collection <key, value> pair, its output will also be
in the format of map collection. When input file is submitted, framework will convert each
row of file into <key, value> pairs.
Note: The lines being offset number will become key and entire line will become value.
 Developer logic has to separate the required key and values from the value of input.
 Developer logic in Mapper separate <key, value>.
 Developer logic in Reducer generates aggregate (sum (), avg (), min (), max (), count (),
etc….)
Low Level Flow
Example
 Reducer

NOTE “Map-Reduce API” is available for following languages

1. Java
2. Python
3. C++
4. Ruby
The following map-reduce programs are implemented in JAVA
Structure of Map-Reduce program in Java:

Note The customer Mapper class and Reducer class should be static.
In Mapper class developer logic has to be implemented in map () function.
 In Reducer class developer logic has to be implemented in reduce () function.

Whenever the program is compiled we get 3 separate classes.

1. Main class (or) Driver class
2. Mapper class
3. Reducer class
The following are the common map-reduce classes used in every program.
1. Mapper
2. Reducer 7. Path
3. Job 8. Configuraton
4. Text 9. FileInputFormat
5. IntWritable [Link]
6. LongWritable [Link]

Note:
The above first 10 classes are available in Hadoop–[Link]. If you want to access the
above classes you must configure the Hadoop–[Link]. Path for this above jar file is
(/usr/lib/Hadoop – 0.20) and 11th class is available in commons – cli – [Link], we must
configure this jar file also. Path foe this jar file is (/usr/lib/Hadoop – 0.20/lib).
Mapper : To define mapper class (custom) package is
[Link] . Here Mapper is a class and the remaining is
package. Whenever a java class has extended Mapper class, the java class will get Mapper
functionalities.
Example:
Context
This class is available in Java and also in “map-reduce”. It is inner class, in Mapper and also
in Reducer.
Context of Mapper
It shuffles data that means not allowing duplicates in key place.
Example
context (k1, v1) --------- <k1, v1>
context (k1, v2) --------- <k1, <v1, v2>>
context (k2, v1) --------- <k2, v1>
context (k1, v1) --------- <k1, v1>
context of Reducer
It writes key, value pairs into HDFS. It does not bother weather key is duplicated
or not.
Reducer Class :
To define custom reducer class we need the package. It is
[Link] when java class has extended Reducer class the
class will get reducer functionality.
Example:
Note: Input key type of Reducer is equal to output key type of Mapper. Input value type of
Reducer is equal to output value type of Mapper.

Map-Reduce Data Types

1. Text
2. IntWritable
3. LongWritable
Java data types are compatible with Operating System. Map-Reduce types are
compatible with HDFS. While contexting results into HDFS, we should use Map-Reduce
types.

Input HDFS File

Map-Reduce Types

1. Map-Reduce types are converted into Java type

 2. Apply Java logic (to process)
3. Results (Java Types) are to be converted into Map-Reduce types.
4. Context the results.

Text :
It is equivalent to Java “String” type. Package is [Link]

IntWritable
It is equivalent to Java “int”. Package is [Link]

LongWritable
It is equivalent to Java “long”. Package is [Link]
Converting LongWritable to long Converting long to IntWritable
LongWritable x=7500000000; long a=750000000;
long y=[Link](); LongWritable l=new LongWritable ();
[Link](a);
(or)
long x=7800000000;
LongWritable l=new LongWritable (x);
Example
public static class MyMap extends Mapper < LongWritable, Text, Text, LongWritable >
{
Public void map (LongWritable k, Text v, Context con) throws IOException, Interrupted
Exception
{
String line= [Link]();
String y= [Link](5,9);
int t=[Link] ([Link] (12,14));
[Link] (new Text(y), new IntWritable (t));
}
}
public static class MyReducer extends Reducer < Text, IntWritable, Text, IntWritable >
{
public void reduce (Text y, Iterable <IntWritable> vals, Context con)
throws IOException, InterruptedException
{
int m=0;
for(IntWritable v:vals){
m=[Link](m, [Link]());
[Link](y, new Intwritable(m));
}
}
}
Configuration
To take default parameters of HDFS and Map-Reduce.
Package [Link]
 Example Configuration con=new Configuration ();
GenericOptionsParser
To pass command line arguments of Hadoop jar command.
Example $hadoop jar Desktop/[Link] [Link] file1 dirx;
Desktop/[Link] jar file path
[Link] package & class name
file1 file name (input file name)
dirx Directory name (output directory name)
Example $hadoop jar Desktop/[Link] [Link] /user/Myself/[Link]
/user/urself;
/user/Myself/[Link] arguments 0
/user/urself; arguments 1
Public static void main (String [ ] args)
{
Configuration con=new Configuration ();
String [] files=new GenericOptionParser (con,args).getRemaingArgs();
}
Explaination
GenericOptionParser gop=new GenericOptionParser (con, args);
String [] files=[Link]();
files[0]=/user/Myself/[Link]
files[1]=/user/urself
In the above explanation file[0], file[1] are in operating system file path,
JVM does not know the operating system file path, we must convert this path into
HDFS file path.
Path class
To convert operating system file path compatible with HDFS file path.
Package [Link]
Path p1=new Path (files [0]);
 Path p2=new Path (files [1]);
FileInputFormat class
To specify input file path.
Package [Link]
[Link](j, p1);
Job j=new Job (con, Myjob);
FileOutputFormat class
To specify output file path.
Package [Link]
[Link](j, p2);
Job j= new Job(con, Myjob);
Note FileInputFormat, FileOutputFormat classes contain static methods
like
1. addInputPath()
2. setOuputPath()
Note These methods are static methods. The above methods are calling without
creating instance. Directly call with class name.

Job To define job.

Package [Link]
i. Job j=new Job(con);
[Link](“MyTest”)
ii. Job j=new Job(con, “MyTest”);
[Link]([Link]);
[Link]([Link]); //Main class
[Link]([Link]); //Mapper
[Link]([Link]); //Reducer
 [Link]([Link]); //Reducer
[Link]([Link]); //output key of Mapper
[Link]([Link]); //output value of Mapper

Example:
AIM: Word count program using Map-Reducer Mr/[Link]
Input:
Mr HDFS directory
[Link] HDFS File

Hadoop is a framework

for big data storage

and processing

Hadoop is good for big data Analytics

package [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
public class WordCount
{
public static class MapForWordCount
extends Mapper (LongWritable, Text, Text, IntWritable)
{
Public void map( Long Writable k, Text v, Context con)
throws IOException InterruputException
{
String line=[Link]();
StringTokenizer t =new StringTokenizer(line);
While([Link]())
{
String word=[Link]();
[Link](new Text(word), new IntWritable(l));
} //end of loop
} //end of map()
} //end of Mapper
public static void main( String [] args ) throws Exception
{
Configuration c=new Configuration ();
String [] files=new GenericOptionsParser(c, args).getRemainingArgs();
Path p1=new Path(files[0]);
Path p2=new Path(files[1]);
Job j=new Job (c, “wordcount”);

[Link]([Link]);
[Link]([Link]);
 [Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link]([Link]);
[Link](j,p1);
[Link](j,p2);
[Link]([Link](true)? 0 : 1);
} //end of main function
} //end of main class.
Output: 17 words
Working with delimited files using Map-Reduce
Aim:: Single Grouping with Single Aggregation
Input::

code:
package [Link]
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
import [Link];
public class Emp
{
public static class MapForEmp
extends Mapper < LongWritable, Text, Text, Intwritable >
{

int sal;
String sex,
public void map (LongWritable k, Text v, Context con)
throws IOException,InterruptedException
{
String line=[Link]();
StringTokenizer t=new StringTokenizer(line, “,”);
int i=1;
while([Link]())
{
String word=[Link]();
if(i==3)
sal=[Link](word);
if(i==4)
sex=word;
 i++;
}
if([Link](“f”))
sex= “ female”;
else
sex=”male”;
[Link](new Text(sex), new IntWritable(sal));
}
}
// Reducer
Public static class ReducerForEmp
extends Reducer < Text, IntWritable, Text, IntWritable >
{
Public void reduce ( Text sex, Iterable <IntWritable> salaries, Context con )
throws IOException, InterruptExcaption
{
Int tot=0;
for ( IntWritable sal : salaries)
tot +=[Link]();
[Link] ( sex, new IntWritable (txt));
} //end of reduce function
} //end of reduce class.
Public static void main( Sting [] args) throwsException
{
-----------
----------- //job definition.
}
Output::
 Output of Mapper Output of Reducer

M, < 2000, 4000, 6000, 7000 > <M, 19000>

F, < 3000, 5000, 9000, 7000 > <F, 24000>

How To Execute Map-Reduce Program

Step1 create Java Project
FileNew Java Project
First time
FileNew OthersJava Project
Project NameMyTestscr
Step2
Create Package under
SrcNewPackagePackage name ([Link])
Step3
Create Java class
PackageNewClassClass Name ([Link])
Step4
Supply Java code and Save.
Step5
Create JAR file
Project name -> Export -> Java JAR file
Note Give Jar file name as project name.
Step6
Submit Map-Reduce Jar

Execution Step
 $ Hadoop jar <jar file path> <class path name> <input file path> <output
file path>

Note : How to configure External JAR files

Src (project)  Build Path  Configure Build Path  Libraries  Add External Jars

1. [Link] (for GenericOptionsParser)
2. [Link] (for remaining packages).
Open Eclips  Choose Project  write class.