Scribd
Upload
4 views7 pages

Experiments

The document provides a comprehensive guide on installing Hadoop and performing various file management tasks such as adding, retrieving, and deleting files and directories in HDFS. It also includes the development of multiple MapReduce programs for matrix multiplication, weather data analysis, and movie tag extraction, along with MongoDB functions for counting, sorting, limiting, skipping, and aggregating movie data. Each section contains code snippets executed in Google Colab, demonstrating practical implementations of the discussed concepts.

Uploaded by

officialdeepikamn
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
4 views7 pages

Experiments

The document provides a comprehensive guide on installing Hadoop and performing various file management tasks such as adding, retrieving, and deleting files and directories in HDFS. It also includes the development of multiple MapReduce programs for matrix multiplication, weather data analysis, and movie tag extraction, along with MongoDB functions for counting, sorting, limiting, skipping, and aggregating movie data. Each section contains code snippets executed in Google Colab, demonstrating practical implementations of the discussed concepts.

Uploaded by

officialdeepikamn
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

1.

Install Hadoop and Implement the following file management tasks in Hadoop: Adding
files and directories,Retrieving files, Deleting files and directories. Hint: A typical Hadoop
workflow creates data files (such as log files) elsewhere and copies them into HDFS using
one of the above command line utilities.

Install hadoop on the system as per the installation guide provided in the lab. Then execute the following commands.
These commands/programs are executed in Google Colab using Python.

1. Adding Directories in HDFS


# Create directories in HDFS
!hdfsdfs -mkdir /hadoop_lab
!hdfsdfs -mkdir /hadoop_lab/input
!hdfsdfs -mkdir /hadoop_lab/output
# Display directories
!hdfsdfs -ls /hadoop_lab

OUTPUT
Found 2 items
drwxr-xr-x - root supergroup 0 2026-04-22 05:57 /hadoop_lab/input
drwxr-xr-x - root supergroup 0 2026-04-22 05:57 /hadoop_lab/output

2. AddingFiles to HDFS
# Create a sample local file
!echo "Hadoop file management example in Colab"> [Link]
# Upload file to HDFS directory
!hdfsdfs -put [Link] /hadoop_lab/input
# Remove local file
!rm [Link]
# List files inside HDFS
!hdfsdfs -ls /hadoop_lab/input

OUTPUT
Found 1 items
-rw-r--r-- 1 root supergroup 40 2026-04-22 06:04 /hadoop_lab/input/[Link]

3. Retrieving Files fromHDFS


# Check local file
!ls [Link]

OUTPUT
ls: cannot access '[Link]': No such file or directory

# Display file content


!hdfsdfs -cat /hadoop_lab/input/[Link]

OUTPUT
Hadoop file management example in Colab

# Retrieve file from HDFS


!hdfsdfs -get /hadoop_lab/input/[Link]
# Check local file
!ls [Link]

OUTPUT
[Link]

4. Deleting Files in HDFS


# Remove file from HDFS
!hdfsdfs -rm /hadoop_lab/input/[Link]
#Verify
!hdfsdfs -ls /hadoop_lab/input

OUTPUT
Deleted /hadoop_lab/input/[Link]

5. Deleting Directories in HDFS


# Remove directory and its contents
!hdfsdfs -rm -r /hadoop_lab
# Check root directory
!hdfsdfs -ls /

OUTPUT
Deleted /hadoop_lab

2. Develop a MapReduce program to implement Matrix Multiplication

A = 1 2 B = 5 6
3 4 7 8
Format:
Matrix,row,column,value

%%bash
cat<<EOF > [Link]
A,0,0,1
A,0,1,2
A,1,0,3
A,1,1,4
B,0,0,5
B,0,1,6
B,1,0,7
B,1,1,8
EOF

%%writefile [Link]
#!/usr/bin/env python3
import sys
for line [Link]:
parts = [Link]().split(",")
matrix, row, col, val = parts
if matrix == "A":
print(f"{col}\tA,{row},{val}")
else:
print(f"{row}\tB,{col},{val}")

%%writefile [Link]
#!/usr/bin/env python3
import sys
current_key = None
values = []
for line [Link]:
key, val = [Link]().split("\t")
if key != current_keyandcurrent_keyisnotNone:
total = sum(int([Link](",")[2]) for v in values)
print(f"{current_key}\t{total}")
values = []
current_key = key
[Link](val)
ifcurrent_key:
total = sum(int([Link](",")[2]) for v in values)
print(f"{current_key}\t{total}")

!chmod +x [Link]
!chmod +x [Link]

!hdfsdfs -mkdir /matrix


!hdfsdfs -put [Link] /matrix

!hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming*.jar \


-input /matrix/[Link] \
-output /matrix_output \
-mapper"python3 [Link]" \
-reducer "python3 [Link]" \
-file [Link] \
-file [Link]

!hdfsdfs -cat /matrix_output/part-00000

OUTPUT

0 15
1 21
[Link] a Map Reduce program that mines weather data and displays appropriate
messagesindicating the weather conditions of the day
Format:
Date,Temperature

%%bash
cat<<EOF > [Link]
2026-04-01,15
2026-04-02,28
2026-04-03,35
2026-04-04,10
2026-04-05,22
EOF

%%writefile [Link]
#!/usr/bin/env python3
import sys
for line [Link]:
line = [Link]()
date,temp = [Link](",")
temp = float(temp)
if temp <15:
condition = "Cold Day"
elif temp <= 30:
condition = "Pleasant Day"
else:
condition = "Hot Day"
print(f"{date}\t{condition}")

%%writefile [Link]
#!/usr/bin/env python3
import sys
for line [Link]:
print([Link]())

!chmod +x [Link]
!chmod +x [Link]

!hdfsdfs -mkdir /weather


!hdfsdfs -put [Link] /weather

!hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming*.jar \


-input /weather/[Link] \
-output /weather_output \
-mapper"python3 [Link]" \
-reducer "python3 [Link]" \
-file [Link] \
-file [Link]

!hdfsdfs -cat /weather_output/part-00000

OUTPUT
2026-04-01 Pleasant Day
2026-04-02 Pleasant Day
2026-04-03 Hot Day
2026-04-04 Cold Day
2026-04-05 Pleasant Day

[Link] a MapReduce program to find the tags associated with each movie by analyzing
movie lens data
Format:
UserID,MovieID,Tag

%%bash
cat<<EOF > [Link]
1,1,funny
2,1,action
3,2,romantic
4,3,thriller
5,2,comedy
6,1,classic
EOF

%%writefile [Link]
#!/usr/bin/env python3
import sys
for line in [Link]:
line = [Link]()
fields = [Link](",")
movie = fields[1]
tag = fields[2]
print(f"{movie}\t{tag}")

%%writefile [Link]
#!/usr/bin/env python3
import sys
current_movie = None
tags = []
for line [Link]:
movie, tag = [Link]().split("\t")
ifcurrent_movie == movie:
[Link](tag)
else:
ifcurrent_movie:
print(current_movie, ", ".join(tags))
current_movie = movie
tags = [tag]
ifcurrent_movie:
print(current_movie, ", ".join(tags))
!chmod +x [Link]
!chmod +x [Link]

!/content/hadoop/bin/hdfsdfs -mkdir /movielens


!/content/hadoop/bin/hdfsdfs -put [Link] /movielens

!hadoop jar /content/hadoop/share/hadoop/tools/lib/hadoop-streaming*.jar \


-input /movielens/[Link] \
-output /movie_output \
-mapper"python3 [Link]" \
-reducer "python3 [Link]" \
-file [Link] \
-file [Link]

!/content/hadoop/bin/hdfsdfs -cat /movie_output/part-00000

OUTPUT
1 classic, action, funny
2 romantic, comedy
3 thriller

[Link] Functions: Count – Sort – Limit – Skip – Aggregate using MongoDB

!pip install pymongo mongomock

import mongomock
client = [Link]()
db = client["movieDB"]
collection = db["movies"]

movies = [
{"title": "Inception", "year": 2010, "rating": 8.8},
{"title": "Avatar", "year": 2009, "rating": 7.8},
{"title": "Titanic", "year": 1997, "rating": 7.9},
{"title": "Interstellar", "year": 2014, "rating": 8.6},
{"title": "Joker", "year": 2019, "rating": 8.4}
]
collection.insert_many(movies)

Count Function
print(collection.count_documents({}))

OUTPUT
5

print(collection.count_documents({"rating": {"$gt": 8}}))

OUTPUT
3

Sort Function
for movie [Link]().sort("rating", 1):
print(movie)
OUTPUT
{'title': 'Avatar', 'year': 2009, 'rating': 7.8, '_id': ObjectId('69e89282a3ba10d1763bbdaa')}
{'title': 'Titanic', 'year': 1997, 'rating': 7.9, '_id': ObjectId('69e89282a3ba10d1763bbdab')}
{'title': 'Joker', 'year': 2019, 'rating': 8.4, '_id': ObjectId('69e89282a3ba10d1763bbdad')}
{'title': 'Interstellar', 'year': 2014, 'rating': 8.6, '_id': ObjectId('69e89282a3ba10d1763bbdac')}
{'title': 'Inception', 'year': 2010, 'rating': 8.8, '_id': ObjectId('69e89282a3ba10d1763bbda9')}

for movie [Link]().sort("rating", -1):


print(movie)

OUTPUT
{'title': 'Inception', 'year': 2010, 'rating': 8.8, '_id': ObjectId('69e89282a3ba10d1763bbda9')}
{'title': 'Interstellar', 'year': 2014, 'rating': 8.6, '_id': ObjectId('69e89282a3ba10d1763bbdac')}
{'title': 'Joker', 'year': 2019, 'rating': 8.4, '_id': ObjectId('69e89282a3ba10d1763bbdad')}
{'title': 'Titanic', 'year': 1997, 'rating': 7.9, '_id': ObjectId('69e89282a3ba10d1763bbdab')}
{'title': 'Avatar', 'year': 2009, 'rating': 7.8, '_id': ObjectId('69e89282a3ba10d1763bbdaa')}

Limit Function
for movie [Link]().limit(3):
print(movie)

OUTPUT
{'title': 'Inception', 'year': 2010, 'rating': 8.8, '_id': ObjectId('69e89282a3ba10d1763bbda9')}
{'title': 'Avatar', 'year': 2009, 'rating': 7.8, '_id': ObjectId('69e89282a3ba10d1763bbdaa')}
{'title': 'Titanic', 'year': 1997, 'rating': 7.9, '_id': ObjectId('69e89282a3ba10d1763bbdab')}

Skip Function
for movie [Link]().skip(2):
print(movie)

OUTPUT
{'title': 'Titanic', 'year': 1997, 'rating': 7.9, '_id': ObjectId('69e89282a3ba10d1763bbdab')}
{'title': 'Interstellar', 'year': 2014, 'rating': 8.6, '_id': ObjectId('69e89282a3ba10d1763bbdac')}
{'title': 'Joker', 'year': 2019, 'rating': 8.4, '_id': ObjectId('69e89282a3ba10d1763bbdad')}

Aggregate Function
pipeline = [
{"$group": {"_id": None, "avg_rating": {"$avg": "$rating"}}}
]
for result [Link](pipeline):
print(result)

OUTPUT
{'avg_rating': 8.3, '_id': None}

pipeline = [
{"$group": {"_id": "$year", "total_movies": {"$sum": 1}}}
]
for result [Link](pipeline):
print(result)

OUTPUT
{'total_movies': 1, '_id': 1997}
{'total_movies': 1, '_id': 2009}
{'total_movies': 1, '_id': 2010}
{'total_movies': 1, '_id': 2014}
{'total_movies': 1, '_id': 2019}

You might also like