📘 Instructions: Running Weather Analysis MapReduce Program
1. 📂 Prepare Files
Create a folder on your system (e.g., C:\hadoop\usn\pgm3).
Inside it, keep:
[Link] (mapper program)
[Link] (reducer program)
[Link] (input dataset with date,temperature)
2. 📤 Upload Input to HDFS
1. Start Hadoop:
[Link]
2. Create a directory in HDFS:
hdfs dfs -mkdir /pgm3
3. Upload the dataset:
hdfs dfs -put C:\hadoop\usn\pgm3\[Link] /pgm3/
4. Verify:
hdfs dfs -ls /pgm3
3. 📝 Mapper Program ([Link])
import sys
for line in [Link]:
line = [Link]()
if not line:
continue
parts = [Link](",")
if len(parts) == 2:
date, temp = parts
try:
temp = int(temp)
print(f"{date}\t{temp}")
except ValueError:
continue
4. 📝 Reducer Program ([Link])
import sys
current_date = None
max_temp = -9999
for line in [Link]:
line = [Link]()
if not line:
continue
date, temp = [Link]("\t")
temp = int(temp)
if current_date == date:
max_temp = max(max_temp, temp)
else:
if current_date:
# Decide weather condition
if max_temp >= 35:
condition = "Hot Day"
elif max_temp >= 20:
condition = "Pleasant Day"
else:
condition = "Cold Day"
print(f"{current_date} : {condition}")
current_date = date
max_temp = temp
# Print last record
if current_date:
if max_temp >= 35:
condition = "Hot Day"
elif max_temp >= 20:
condition = "Pleasant Day"
else:
condition = "Cold Day"
print(f"{current_date} : {condition}")
📝 Contents of Input file ([Link])
2025-08-20,35
2025-08-20,37
2025-08-21,22
2025-08-21,25
2025-08-22,15
2025-08-22,18
5. ▶️Run Hadoop Streaming Job
Run from cmd:
hadoop jar C:\hadoop\share\hadoop\tools\lib\[Link] ^
-input /pgm3/[Link] ^
-output /weather/output ^
-mapper " python c:\hadoop\usn\pgm3\[Link]" ^
-reducer " python c:\hadoop\usn\pgm3\[Link]"
⚠️Notes:
Use ^ in Windows for line continuation (or put all in one line).
-output directory must not exist before running (delete old one with hdfs dfs -rm -
r /weather/output).
6. 📥 View Results
hdfs dfs -cat /weather/output/part-00000
Expected output for your sample data:
2025-08-20 : Hot Day
2025-08-21 : Pleasant Day
2025-08-22 : Cold Day
7. 🧪 Local Testing (Optional)
Before Hadoop, test with pipes :
C:\hadoop\usn\pgm3: type [Link] | python [Link] | sort | python [Link]
Output:
2025-08-20 : Hot Day
2025-08-21 : Pleasant Day
2025-08-22 : Cold Day