Write the Mapper file and Reducer file, to find the Average Temperature for each date.
Also, write the command to create necessary input file at required destination.
Assume we have the following date and temperature data:
2024-10-10, 22
2024-10-10, 24
2024-10-11, 18
2024-10-11, 20
2024-10-12, 25
2024-10-12, 30
Solution:
Before running the MapReducde job, please create a input file at HDFS.
Upload the file to HDFS:
You need to place the [Link] file into HDFS using the following command. (This file must be under the same
username, by which you are logged in to the linux system. For e.g., as I am looged in as hduser, so directory will be
/user/hduser.
hdfs dfs -put [Link] /user/hduser/[Link]
To solve the problem of finding the average temperature for each date using Hadoop's MapReduce framework, we need
two key components: A Mapper file and a Reducer file.
Mapper File ([Link])
The Mapper's job is to read each line of input and emit the date and temperature as key-value pairs. For instance, the
input 2024-10-10, 22 will produce the key-value pair (2024-10-10, 22).
Here is the Python code for the Mapper:
#!/usr/bin/env python3
import sys
# Mapper function to emit (date, temperature) pairs
for line in [Link]:
# Remove any leading/trailing whitespace
line = [Link]()
# Split the input line into date and temperature
date, temperature = [Link](',')
# Emit the date as the key and temperature as the value
print(f"{date}\t{temperature}")
Reducer File ([Link])
The Reducer will receive all temperatures for a specific date as input and then calculate the average temperature for
that date.
Here is the Python code for the Reducer:
#!/usr/bin/env python3
import sys
current_date = None
total_temp = 0
count = 0
# Reducer function to compute the average temperature
for line in [Link]:
# Remove any leading/trailing whitespace
line = [Link]()
# Parse the input from mapper
date, temperature = [Link]('\t')
# Convert temperature to integer
temperature = int(temperature)
# If we encounter a new date, output the result for the previous date
if current_date and current_date != date:
# Compute average temperature for the previous date
average_temp = total_temp / count
print(f"{current_date}\t{average_temp:.2f}")
# Reset total_temp and count for the new date
total_temp = 0
count = 0
# Update current_date and accumulate total_temp and count
current_date = date
total_temp += temperature
count += 1
# Output the result for the last date
if current_date:
average_temp = total_temp / count
print(f"{current_date}\t{average_temp:.2f}")
Explanation:
Mapper ([Link]):
Reads each line of input data.
Splits each line into date and temperature.
Emits each date and temperature as key-value pairs.
Reducer ([Link]):
Receives all the temperatures associated with a specific date.
Sums the temperatures and keeps a count.
When a new date is encountered (or at the end of input), the average temperature is calculated and printed.
Check for a New Date (if current_date and current_date != date):
If the current_date has been processed (i.e., it's not None), and a new date is encountered, it means we are
done processing the previous date.
Before moving to the new date, we calculate the average temperature for the current_date.
Input and Output:
Input:
2024-10-10, 22
2024-10-10, 24
2024-10-11, 18
2024-10-11, 20
2024-10-12, 25
2024-10-12, 30
Output:
2024-10-10 23.00
2024-10-11 19.00
2024-10-12 27.50