MapReduce Programming
Consider the following text file [Link]
#BillNo, ItemName, UnitCost, NumberOfItems
1,Paste,10,20
1,Brush,15,12
1,Talcum,50,5
2,Bread,30,100
2,Butter,60,50
2,Water,20,120
2,Jam,50,80
3,Bread,35,110
3,Butter,45,55
3,Paste,10,20
3,Brush,15,12
If the following program is executed with the above file as input, what are the
yields of mapper
and reducer in the following program
from [Link] import MRJob
class Bill_Totals(MRJob):
def mapper(self, _, billrow):
if billrow[0] != '#':
tlist = list([Link](','))
yield tlist[0], int(tlist[2]) * int(tlist[3])
def reducer(self, billno, itemtotals):
yield billno, sum(itemtotals)
if __name__ == '__main__':
Bill_Totals.run()
-------------------------------------------------------------------------------------------------------------------------------
#BillNo, ItemName, UnitCost, NumberOfItems
1,Paste,10,20
1,Brush,15,12
1,Talcum,50,5
2,Bread,30,100
2,Butter,60,50
2,Water,20,120
2,Jam,50,80
3,Bread,35,110
3,Butter,45,55
3,Paste,10,20
3,Brush,15,12
Program Breakdown
1. Mapper Function:
○ It reads each line of the input file (billrow).
○ If the first character is not a #, it splits the line by commas to get a list of items.
○ The mapper then yields the bill number (tlist[0]) and the total cost for that
item, which is calculated as UnitCost * NumberOfItems.
2. Reducer Function:
○ The reducer will receive the bill number as the key (billno) and the item
totals (i.e., the costs for each item in the bill) as the values (itemtotals).
○ It then sums up all the item totals for each bill and yields the final total for each
bill.
Analyzing the Mapper Output:
The mapper reads each row from the input file and yields the bill number and the calculated
total cost for each item. Let's go through each row:
1. Row 1 (1,Paste,10,20):
○ UnitCost = 10, NumberOfItems = 20
○ Total = 10 * 20 = 200
○ Mapper Yield: 1, 200
2. Row 2 (1,Brush,15,12):
○ UnitCost = 15, NumberOfItems = 12
○ Total = 15 * 12 = 180
○ Mapper Yield: 1, 180
3. Row 3 (1,Talcum,50,5):
○ UnitCost = 50, NumberOfItems = 5
○ Total = 50 * 5 = 250
○ Mapper Yield: 1, 250
4. Row 4 (2,Bread,30,100):
○ UnitCost = 30, NumberOfItems = 100
○ Total = 30 * 100 = 3000
○ Mapper Yield: 2, 3000
5. Row 5 (2,Butter,60,50):
○ UnitCost = 60, NumberOfItems = 50
○ Total = 60 * 50 = 3000
○ Mapper Yield: 2, 3000
6. Row 6 (2,Water,20,120):
○ UnitCost = 20, NumberOfItems = 120
○ Total = 20 * 120 = 2400
○ Mapper Yield: 2, 2400
7. Row 7 (2,Jam,50,80):
○ UnitCost = 50, NumberOfItems = 80
○ Total = 50 * 80 = 4000
○ Mapper Yield: 2, 4000
8. Row 8 (3,Bread,35,110):
○ UnitCost = 35, NumberOfItems = 110
○ Total = 35 * 110 = 3850
○ Mapper Yield: 3, 3850
9. Row 9 (3,Butter,45,55):
○ UnitCost = 45, NumberOfItems = 55
○ Total = 45 * 55 = 2475
○ Mapper Yield: 3, 2475
10. Row 10 (3,Paste,10,20):
○ UnitCost = 10, NumberOfItems = 20
○ Total = 10 * 20 = 200
○ Mapper Yield: 3, 200
11. Row 11 (3,Brush,15,12):
○ UnitCost = 15, NumberOfItems = 12
○ Total = 15 * 12 = 180
○ Mapper Yield: 3, 180
Mapper Output:
For each bill number, the mapper produces key-value pairs like so:
1, 200
1, 180
1, 250
2, 3000
2, 3000
2, 2400
2, 4000
3, 3850
3, 2475
3, 200
3, 180
Reducer Output:
The reducer will sum up the totals for each bill number. Here's the breakdown:
1. For Bill 1:
○ Totals: 200, 180, 250
○ Sum: 200 + 180 + 250 = 630
○ Reducer Yield: 1, 630
2. For Bill 2:
○ Totals: 3000, 3000, 2400, 4000
○ Sum: 3000 + 3000 + 2400 + 4000 = 12400
○ Reducer Yield: 2, 12400
3. For Bill 3:
○ Totals: 3850, 2475, 200, 180
○ Sum: 3850 + 2475 + 200 + 180 = 7705
○ Reducer Yield: 3, 7705
Final Reducer Output:
1, 630
2, 12400
3, 7705
Conclusion:
● Mapper Output: Key-value pairs where the key is the bill number
and the value is the total cost for each item.
● Reducer Output: Key-value pairs where the key is the bill number
and the value is the total sum of all item totals for that bill.
—----------------------------------------------------------------------------------------------------------------------------
2)
What will the yields of mapper and reducer of the following program if the
input file contains
1000 four-digit integers
from [Link] import MRJob
class SortIntegers(MRJob):
def mapper(self, key, line):
for integer in [Link]():
yield None, int(integer)
def reducer(self, key, values):
integers = list(values)
[Link]()
# [Link](reverse=True)
for integer in integers:
yield None, integer
if __name__ == '__main__':
[Link]()
—----------------------------------------------------------------------------------------------------------------------------
Understanding Term Frequency-Inverse Document Frequency (TF-IDF)
TF-IDF is a statistical measure used to evaluate how important a word is to a document in a
collection or corpus. It combines two components: Term Frequency (TF) and Inverse
Document Frequency (IDF).
1. Term Frequency (TF)
Term Frequency is a measure of how frequently a term appears in a document. It is calculated
as:
Where:
● t is the term.
● d is the document.
● The numerator counts the occurrences of the term in the document.
● The denominator is the total number of terms in the document.
2. Inverse Document Frequency (IDF)
IDF measures how important a term is in the context of the entire corpus. If a term appears in
many documents, it is considered less informative. IDF is calculated as:
Where:
t is the term.
D is the total number of documents in the corpus.
The denominator is the number of documents containing the term t, and the numerator is
the total number of documents in the corpus.
3. TF-IDF Calculation
The TF-IDF score for a term t in document d is the product of the term frequency (TF) and the
inverse document frequency (IDF):
Q3: Compute the TF-IDF values for the words in Dataset 02. Ignore stop words and use
only the stem words:
Dataset 02
The sun rises in the east and sets in the west. The bright sun provides warmth and light to the
world
The moon appears at night and reflects the light of the sun. The night sky is filled with stars and
the cool breeze
Plants need sunlight for photosynthesis. The sun's rays are essential for growth, providing
energy and warmth
Solution:-