Applications
TF-IDF
• Understanding the significance of a word in a text is crucial for
analyzing and interpreting large volumes of data. This is where the
term frequency-inverse document frequency (TF-IDF) technique
comes into play.
Terminology: Key Terms Used in TF-IDF
• t: term (word)
• d: document (set of words)
• N: count of corpus
• corpus: the total document set
What is Term Frequency (TF)?
The frequency with which a term occurs in a document is measured by term frequency (TF).
A term’s weight in a document is directly correlated with its frequency of occurrence. The TF
formula is:
What is Document Frequency (DF)?
• The significance of a document within a corpus is gauged by its
Document Frequency (DF). DF counts the number of papers that
contain the phrase at least once, as opposed to TF, which counts the
instances of a term in a document. The DF formula is:
• DF(t)=occurrence of t in documents
What is Inverse Document Frequency (IDF)?
• The informativeness of a word is measured by its inverse document
frequency, or IDF. All terms are given identical weight while
calculating TF, although IDF helps scale up uncommon terms and
weigh down common ones (like stop words). Where N is the total
number of documents and DF(t) is the number of documents
containing the term t. The IDF formula is:
What is TF-IDF?
• TF-IDF stands for Term Frequency-Inverse Document Frequency, a
statistical measure used to evaluate how important a word is to a
document in a collection or corpus. It combines the importance of a
term in a document (TF) with the term’s rarity across the corpus (IDF).
The formula is:
Numerical Calculation of TF-IDF
• Documents:
• “The sky is blue.”
• “The sun is bright today.”
• “The sun in the sky is bright.”
• “We can see the shining sun, the bright sun.”
Step 1: Calculate Term Frequency (TF)
Step 2: Calculate Inverse Document Frequency
(IDF)
Documents:
“The sky is blue.”
“The sun is bright today.”
“The sun in the sky is bright.”
“We can see the shining sun, the
bright sun.”
N: # of documents=4
DF(t)=occurrence of t in documents
Step 3: Calculate TF-IDF
Why is TF-IDF used in machine learning?
• The most significant problem faced by text processing is that machine
learning models tend to only deal with numerical values. Therefore,
we must vectorize the text to convert it into numbers. This is a crucial
step in machine learning.
• The tf-idf score converts words into numbers that can be fed to
algorithms like Naive Bayes and Support Vector Machines,
significantly improving the results of more straightforward techniques
like word counts.
Example
• Tf-Idf is critical in search and ranking applications. Tf-idf provides
results that are most pertinent to your search. Consider your search
engine as someone searching for “the red car.” The outcomes will be
presented in relevant order. In other words, the most pertinent
articles about red cars will be ranked higher because the words “red”
and “car” receive a higher score from tf-idf. Due to it’s importance,
every search engine you have used probably incorporates tf-idf scores
into its algorithm.
Map Reduce
• Map Reduce is a framework in which we can write applications to run
huge amount of data in parallel.
Phases of MapReduce
• MapReduce model has three major and one optional phase.
• Mapping
• Shuffling and Sorting
• Reducing
• Combining
Example
Let us take a real-world example to comprehend the power of
MapReduce. Twitter receives around 500 million tweets per day,
which is nearly 3000 tweets per second. The following
illustration shows how Tweeter manages its tweets with the help
of MapReduce.
Page Rank Algorithm
• Modern search engines employ methods of ranking the results to provide the "best" results first that
are more elaborate than just plain text ranking. One of the most known and influential algorithms
for computing the relevance of web pages is the Page Rank algorithm used by the Google search
engine.
• It was invented by Larry Page and Sergey Brin while they were graduate students at Stanford, and
it became a Google trademark in 1998. The idea that Page Rank brought up was that, the
importance of any web page can be judged by looking at the pages that link to it.
• If we create a web page i and include a hyperlink to the web page j, this means that we
consider j important and relevant for our topic. If there are a lot of pages that link to j, this means
that the common belief is that page j is important. If on the other hand, j has only one backlink, but
that comes from an authoritative site k, (like [Link], [Link], [Link])
we say that k transfers its authority to j.
example
• Suppose for instance, that we have a small Internet consisting of just 4 web sites [Link],
[Link], [Link], [Link], referencing each other in the manner suggested by the
picture:
Formula
The PageRank value for a page p is dependent on the PageRank values for each page q contained in the set Bu (the set
containing all pages linking to page p), divided by the number L(q) of links from page q. The algorithm involves a
damping factor for the calculation of the PageRank.
In PageRank, the damping factor (commonly 0.85) represents the
Here there are 4 pages, so N=4.
probability that a "random surfer" will continue to click links on a web
Damping factor D=0.85 page, with the remaining probability (1 - 0.85 = 0.15) being the chance
Intitial Page Rank=1/N=1/4=0.25 they will jump to a completely random page.
Example2 Calculate rank score of each node page. Given d=0.85
In this example, p5 is dead end, so we have to handle this
situation by redistributing to its page rank to all pages in each
interation.
Dead link adjustment=
Results of interation 1