Index
Sr.
Practical Name Date Pg. No. Sign
No.
Perform text preprocessing – tokenization,
1 stop-word removal, stemming, and 1
lemmatization.
Compute term frequency (TF), inverse
2 document frequency (IDF), and TF–IDF 3
weighting for a given dataset.
Implement Boolean and Vector Space
3 retrieval models and compute cosine 5
similarity between documents and queries.
Perform text classification and clustering
4 using algorithms such as Naïve Bayes, k- 7
Means, or Decision Tree.
Evaluate retrieval performance using
5 Precision, Recall, and F-measure, and 9
analyze ranking results for different models.
EXPERIMENT 1
Aim: Perform text preprocessing – tokenization, stop-word removal, stemming, and
lemmatization.
Objective: To process raw text data by breaking it down into fundamental units, removing
noise, and reducing words to their base root forms for efficient indexing.
Theory:
Text preprocessing is the foundational step in Information Retrieval to normalize data before
indexing. It involves:
Tokenization: Breaking down a stream of text into smaller units called tokens (words,
phrases, or symbols).
Stop-word Removal: Filtering out highly common, uninformative words (e.g., "is",
"the", "and") to save index space and speed up processing.
Stemming: A heuristic process that chops off the ends of words to reduce them to their
base form (e.g., "fascinating" to "fascin").
Lemmatization: A more advanced morphological analysis that uses a vocabulary to
return the dictionary base form (lemma) of a word (e.g., "better" to "good").
Implementation:
1
Output:
Result: Successfully applied tokenization, stop-word removal, stemming, and lemmatization
using the NLTK library to preprocess the raw text corpus for indexing.
2
EXPERIMENT - 2
Aim: Compute term frequency (TF), inverse document frequency (IDF), and TF–IDF
weighting for a given dataset.
Objective: To quantify the importance of terms within a document relative to a larger corpus
to improve document ranking.
Theory:
In the Vector Space Model, simple term counts are insufficient because common words
dominate the weights. TF-IDF solves this:
Term Frequency (TF): Measures how frequently a term appears in a specific
document.
Inverse Document Frequency (IDF): Measures how rare or informative a term is
across the entire corpus. It heavily penalizes common words.
TF-IDF Weighting: The product of TF and IDF. A high weight is achieved by a high
term frequency in the given document and a low document frequency of the term in the
whole collection.
Implementation:
3
Output:
Result: Successfully computed term frequencies and inverse document frequencies to
generate a TF-IDF matrix, accurately converting textual documents into weighted numerical
vectors.
4
EXPERIMENT - 3
Aim: Implement Boolean and Vector Space retrieval models and compute cosine similarity
between documents and queries.
Objective: To measure the semantic similarity between a user search query and a collection
of documents to rank them by relevance.
Theory:
The Vector Space Model represents documents and queries as vectors in a multidimensional
space, where each dimension corresponds to a distinct term.
Cosine Similarity: Instead of measuring the physical distance between vectors
(which is biased by document length), Cosine Similarity measures the cosine of the
angle between the query vector and the document vector.
A cosine value of 1 means the vectors point in the exact same direction (high
similarity), while 0 means they are orthogonal (no shared terms).
Implementation:
5
Output:
Result: Successfully transformed text documents and a search query into vector
space, and calculated cosine similarity scores to accurately determine document
relevance.
6
EXPERIMENT 4
Aim: Perform text classification and clustering using algorithms such as Naïve Bayes, k-
Means, or Decision Tree.
Objective: To automatically classify textual data into predefined categories using a
probabilistic machine learning model.
Theory:
Multinomial Naïve Bayes is a popular supervised learning algorithm for text classification
(like spam filtering or sentiment analysis).
It is based on Bayes’ Theorem, which calculates the probability of a document
belonging to a specific class given the terms it contains.
It is "Naïve" because it assumes that the presence of a particular feature (word) in a
class is completely independent of the presence of any other feature.
Implementation:
7
Output:
Result: Successfully trained a Multinomial Naïve Bayes classifier using vectorized text data
and correctly predicted the spam/ham classification of an unseen test string.
8
EXPERIMENT 5
Aim: Evaluate retrieval performance using Precision, Recall, and F-measure, and analyze
ranking results for different models.
Objective: To quantitatively evaluate the accuracy and effectiveness of an information
retrieval system's search results.
Theory:
Evaluating an IR system requires comparing its retrieved results against a known set of
"Ground Truth" relevant documents.
Precision: The fraction of the retrieved documents that are actually relevant.
(Measures accuracy/quality).
Recall: The fraction of all relevant documents in the corpus that were successfully
retrieved. (Measures completeness).
F-Measure (F1-Score): The harmonic mean of Precision and Recall, providing a
single metric that balances both concerns, especially useful when there is an uneven
class distribution.
Implementation:
9
Output:
Result: Successfully evaluated the IR system’s performance by applying a
confidence threshold and calculating precision, recall, and F-measure scores for
the retrieved documents.
10