Task 1: PIG Latin Installation
1. Ensure that the HDFS is running
2. Install required package Pig
COMMAND:- wget [Link]
[Link]
3. Extract and Configure
COMMAND:- tar -xvzf [Link]
4. Move to /usr/local/pig
COMMAND:- sudo mv pig-0.17.0 /usr/local/pig
5. Configure Environment Variables
COMMAND:-
nano ~/.bashrc
6. Check PIG installation
COMMAND:- pig -version
Here, we can see that PIG is successfully installed
Task 2: Basic operation on PIG Latin
● Create a .csv file with the following values
1,2,3,4
5,6,7,8
2,3,4,5
1,2,3,4
• Perform the following basic operations (LOAD, FILTER, LIMIT,
ORDER BY)
COMMAND:- cat lab_4_1_2.csv
● Create a pigdata directory and put the file using HDFS
COMMAND:-
hdfs dfs -put lab_4_1_2.csv /
● Configure the PIG file
COMMAND:-
nano lab_4_1_2.pig
CODE:-
-- Load the CSV file
data = LOAD '/lab_4_1_2.csv' USING PigStorage(',')
AS (c1:int, c2:int, c3:int, c4:int);
-- FILTER: select rows where first column = 1
filtered = FILTER data BY c1 == 1;
-- LIMIT: take only 2 records
limited = LIMIT data 2;
-- ORDER BY: sort by column 2
ordered = ORDER data BY c2 ASC;
-- STORE outputs into HDFS
STORE data INTO '/lab4_output/data' USING PigStorage(',');
STORE filtered INTO '/lab4_output/filtered' USING PigStorage(',');
STORE limited INTO '/lab4_output/limited' USING PigStorage(',');
STORE ordered INTO '/lab4_output/ordered' USING PigStorage(',');
● Start job history server
COMMAND:- mapred -- daemon start historyserver
Run the script
COMMAND:-
pig -x mapreduce lab_4_1_2.pig
● Get the outputs
COMMAND:-
hdfs dfs -ls /lab4_output
Get the name of output file using this command for each of the above
Hdfs dfs -ls /lab4_output/data
hdfs dfs -cat /lab4_output/data/part-m-00000
This is the output for LOAD
This is the output for FILTER
This is the output for LIMIT
This is the output for ORDER
Task 3: Word Count Problem
Write a PIG Latin program to count the frequency of the words in the
document
([Link]) using Hadoop framework.
File Name: [Link]
Nutrition is one of the key determinants of human health and well-being,
influencing
growth, immunity, and disease prevention across all stages of life. Research in
the American Journal of Clinical Nutrition highlights that diets rich in whole
grains, fresh
fruits, vegetables, legumes, and lean proteins provide essential macronutrients
and
micronutrients required for optimal functioning of the body. Micronutrients
such as
iron, zinc, and vitamin A play a crucial role in supporting immunity, while
calcium and
vitamin D are vital for maintaining bone strength. Studies in the Journal of
Nutrition
further emphasize that high consumption of processed foods, refined sugars,
and
saturated fats is associated with obesity, cardiovascular disease, and type 2
diabetes.
Balanced dietary patterns, such as the Mediterranean diet, have been shown to
lower the
risk of chronic illnesses while promoting healthy aging. Personalized nutrition,
guided
by genetic, lifestyle, and metabolic factors, is gaining prominence in recent
nutrition
research.
Steps:
• Save the input file [Link] in Local File System
• Push the file into HDFS using put command
• Open the Grunt Shell in Map Reduce mode
• Create a PIG Latin program (Word Count)
• DUMP the intermediate result for every process.
• STORE the output in HDFS
● Create the [Link] file
COMMAND:- nano [Link]
● Put the file into HDFS
COMMAND:- hdfs dfs -put [Link] /
● Run the grunt
COMMAND:- pig -x mapreduce
● Inserting the pig code into the grunt
CODE:-
-- Load the text file
lines = LOAD '/[Link]' USING TextLoader() AS (line:chararray);
DUMP lines;
-- Split each line into words
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
DUMP words;
-- Convert all words to lowercase for consistency
words_lower = FOREACH words GENERATE LOWER(word) AS word;
DUMP words_lower;
-- Group by word
grouped = GROUP words_lower BY word;
DUMP grouped;
-- Count occurrences
word_count = FOREACH grouped GENERATE group AS word,
COUNT(words_lower) AS
count;
DUMP word_count
-- Store final output into HDFS
STORE word_count INTO '/lab4_output/wordcount' USING PigStorage(',');
INTERMEDIATE RESULT
lines = LOAD '/[Link]' USING TextLoader() AS (line:chararray);
DUMP lines
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
DUMP words;
words_lower = FOREACH words GENERATE LOWER(word) AS word;
DUMP words_lower
grouped = GROUP words_lower BY word;
DUMP grouped
word_count = FOREACH grouped GENERATE group AS word,
COUNT(words_lower) AS
count;
DUMP word_count
Get the output
COMMAND:-
hdfs dfs -ls /lab4_output/wordcount
hdfs dfs -cat /lab4_output/wordcount/part-r-00000