Scribd
Upload
22 views3 pages

Simple Java MapReduce Example

This document contains code for a Java MapReduce program that counts the occurrences of different log levels (TRACE, DEBUG, etc.) in input log files. The Map class extracts the log level from each line and emits it as a key along with a count of 1. The Reduce class sums the counts for each unique log level. The main method sets up the job configuration and runs the MapReduce job to process the input path and write results to the output path.

Uploaded by

Ravi Chander
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
22 views3 pages

Simple Java MapReduce Example

This document contains code for a Java MapReduce program that counts the occurrences of different log levels (TRACE, DEBUG, etc.) in input log files. The Map class extracts the log level from each line and emits it as a key along with a count of 1. The Reduce class sums the counts for each unique log level. The main method sets up the job configuration and runs the MapReduce job to process the input path and write results to the output path.

Uploaded by

Ravi Chander
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

//Standard Java imports import [Link]; import [Link]; import [Link]; import [Link].

Pattern; //Hadoop imports import import import import import import import import import import import import [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link]; [Link];

import [Link]; import [Link]; import [Link]; /** * Tutorial1 * */ public class Tutorial1 { //The Mapper public static class Map extends MapReduceBase implements Mapper<LongWritable, T ext, Text, IntWritable> { //Log levels to search for private static final Pattern pattern = [Link]("(TRACE)|(DEBUG)|(INFO)| (WARN)|(ERROR)|(FATAL)"); private static final IntWritable accumulator = new IntWritable(1); private Text logLevel = new Text(); public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable > collector, Reporter reporter) throws IOException { // split on space, '[', and ']' final String[] tokens = [Link]().split("[ \\[\\]]"); if(tokens != null) { //now find the log level token for(final String token : tokens) { final Matcher matcher = [Link](token); //log level found if([Link]()) { [Link](token);

//Create the key value pairs [Link](logLevel, accumulator); } } } } } //The Reducer public static class Reduce extends MapReduceBase implements Reducer<Text, IntWr itable, Text, IntWritable> { public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text , IntWritable> collector, Reporter reporter) throws IOException { int count = 0; //code to aggregate the occurrence while([Link]()) { count += [Link]().get(); } [Link](key + "\t" + count); [Link](key, new IntWritable(count)); } } //The java main method to execute the MapReduce job public static void main(String[] args) throws Exception { //Code to create a new Job specifying the MapReduce class final JobConf conf = new JobConf([Link]); [Link]([Link]); [Link]([Link]); [Link]([Link]); // Combiner is commented out to be used in bonus activity //[Link]([Link]); [Link]([Link]); [Link]([Link]); [Link]([Link]); //File Input argument passed as a command line argument [Link](conf, new Path(args[0]));

//File Output argument passed as a command line argument [Link](conf, new Path(args[1])); //statement to execute the job [Link](conf); } }

You might also like