NAME:- MANDAVA SANJANA VENKATA SAI ROLL NO:- 102115186 GROUP:- 4O22
BIG DATA ANALYTICS
UEC 735
LAB ASSIGNMENT -1
Write a word count program using MapReduce programming model.
Theory:-
MapReduce is a programming model used for efficient processing in parallel over large data-sets
in a distributed manner. The data is first split and then combined to produce the final result. The
libraries for MapReduce is written in so many programming languages with various different-
different optimizations. The purpose of MapReduce in Hadoop is to Map each of the jobs and
then it will reduce it to equivalent tasks for providing less overhead over the cluster network and
to reduce the processing power. The MapReduce task is mainly divided into two phases Map
Phase and Reduce Phase.
Components of MapReduce Architecture:
1. Client: The MapReduce client is the one who brings the Job to the MapReduce for processing.
There can be multiple clients available that continuously send jobs for processing to the Hadoop
MapReduce Manager.
2. Job: The MapReduce Job is the actual work that the client wanted to do which is comprised of so
many smaller tasks that the client wants to process or execute.
3. Hadoop MapReduce Master: It divides the particular job into subsequent job-parts.
4. Job-Parts: The task or sub-jobs that are obtained after dividing the main job. The result of all the
job-parts combined to produce the final output.
5. Input Data: The data set that is fed to the MapReduce for processing.
6. Output Data: The final result is obtained after the processing.
CODE:
from [Link] import files
import re
from collections import defaultdict
# Upload the document
uploaded = [Link]()
NAME:- MANDAVA SANJANA VENKATA SAI ROLL NO:- 102115186 GROUP:- 4O22
# Read the uploaded file
for filename in [Link]():
with open(filename, 'r') as file:
text = [Link]()
# Map function
def map_function(text):
words = [Link](r'\w+', [Link]())
return [(word, 1) for word in words]
# Shuffle and Sort function (part of the framework, not the user-defined
part of MapReduce)
def shuffle_sort(mapped_data):
shuffled = defaultdict(list)
for key, value in mapped_data:
shuffled[key].append(value)
return shuffled
# Reduce function
def reduce_function(shuffled_data):
reduced = {}
for key, values in shuffled_data.items():
reduced[key] = sum(values)
return reduced
# Apply MapReduce
mapped_data = map_function(text)
shuffled_data = shuffle_sort(mapped_data)
word_counts = reduce_function(shuffled_data)
# Display the word counts
for word, count in word_counts.items():
print(f'{word}: {count}')
RESULT:-
NAME:- MANDAVA SANJANA VENKATA SAI ROLL NO:- 102115186 GROUP:- 4O22