Lab-2 Date:

a. Write a python program to perform Parts of Speech tagging using nltk.

Aim: To write a python program to perform Parts of Speech tagging using nltk.
Parts of Speech (PoS) tagging is a core task in NLP, It gives each word a grammatical category such as nouns,
verbs, adjectives and adverbs. Through better understanding of phrase structure and semantics, this technique
makes it possible for machines to study human language more accurately.

Theory:
PoS tagging is essential in many NLP applications like machine translation, sentiment analysis and information
retrieval. It serves as a link between language and machine understanding, enabling the creation of complex
language processing systems.
Parts of Speech tagging is a linguistic activity in Natural Language Processing (NLP) wherein each word in a
document is given a particular part of speech (adverb, adjective, verb etc.) or grammatical category. Through the
addition of a layer of syntactic and semantic information to the words, this procedure makes it easier to
understand the sentence's structure and meaning.
In NLP applications, POS tagging is useful for machine translation, named entity recognition and information
extraction, among other things. It also works well for clearing out ambiguity in terms with numerous meanings
and revealing a sentence's grammatical structure.
Example of POS Tagging
Consider the sentence: "The quick brown fox jumps over the lazy dog."
After performing POS Tagging:
 "The" is tagged as determiner (DT)
 "quick" is tagged as adjective (JJ)
 "brown" is tagged as adjective (JJ)
 "fox" is tagged as noun (NN)
 "jumps" is tagged as verb (VBZ)
 "over" is tagged as preposition (IN)
 "the" is tagged as determiner (DT)
 "lazy" is tagged as adjective (JJ)
 "dog" is tagged as noun (NN)
By offering insights into the grammatical structure, this tagging helps machines in understanding not just
individual words but also the connections between them inside a phrase. For many NLP applications like text
summarization, sentiment analysis, this kind of data is essential.
Workflow of POS Tagging in NLP
 Tokenization: The input text is divided into individual tokens, representing words or sub-words.
Tokenization is the foundational step in most NLP tasks which enables further analysis at the word level.
 Loading a Language Model: Tools like NLTK or SpaCy requires a pre-trained language model to
perform POS tagging. These models are trained on large datasets and provide insights into the
grammatical rules and structure of the language.
 Text Pre-processing: The text is then cleaned to improve accuracy. Common pre-processing steps
include converting text to lowercase, removing xspecial characters and eliminating irrelevant content.
 Linguistic Analysis: This stage involves parsing the sentence to understand the grammatical role of each
token. It lays the groundwork for assigning the appropriate part of speech by interpreting the sentence’s
syntactic structure.
 POS Tagging: Each token is then assigned a specific part-of-speech label. This is based on its role in the
sentence and contextual clues provided by surrounding words.
  Result Evaluation: Finally, the POS-tagged output is reviewed to ensure accuracy. Any
misclassifications or anomalies are identified and corrected as needed.

Implementation of Parts-of-Speech tagging using NLTK

1. Installing packages

import nltk
from [Link] import word_tokenize
from nltk import pos_tag
[Link]('punkt')
[Link]('averaged_perceptron_tagger')

2. Implementation
 The sentence is stored in the variable text.
 The text is tokenized into words using word_tokenize(text) before applying POS tagging.
 pos_tag(words) assigns grammatical tags (e.g., noun, verb) to each word.
 The original sentence is printed for reference.
 A loop prints each word alongside its predicted part-of-speech tag.

import nltk
from [Link] import word_tokenize
from nltk import pos_tag
[Link]('punkt')
[Link]('averaged_perceptron_tagger')
[Link]('punkt_tab')
[Link]('averaged_perceptron_tagger_eng')

text = "The quick brown fox jumps over the lazy dog."

words = word_tokenize(text)
pos_tags = pos_tag(words)

print("Words with POS tags:")

print(pos_tags)

Output:

[nltk_data] Downloading package punkt to /root/nltk_data...

[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] /root/nltk_data...
[nltk_data] Package averaged_perceptron_tagger is already up-to-
[nltk_data] date!
[nltk_data] Downloading package punkt_tab to /root/nltk_data...
[nltk_data] Package punkt_tab is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger_eng to
[nltk_data] /root/nltk_data...
[nltk_data] Unzipping taggers/averaged_perceptron_tagger_eng.zip.
Words with POS tags:
[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'),
('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN'), ('.', '.')]
b. Write a python program to perform lemmatization using nltk.

Aim: To Write a python program to perform lemmatization using nltk.

Theory:
 Lemmatization is an important text pre-processing technique in Natural Language
Processing (NLP) that reduces words to their base form known as a "lemma." For
example, the lemma of "running" is "run" and "better" becomes "good."
Unlike stemming which simply removes prefixes or suffixes, it considers the word's
meaning and part of speech (POS) and ensures that the base form is a valid word. This
makes lemmatization more accurate as it avoids generating non-dictionary words.
Lemmatization is important for various reasons in NLP:
 Improves accuracy: It ensures words with similar meanings like "running" and "ran"
are treated as the same.
 Reduced Data Redundancy: By reducing words to their base forms, it reduces
redundancy in the dataset. This leads to smaller datasets which makes it easier to
handle and process large amounts of text for analysis or training machine learning
models.
 Better NLP Model Performance: By treating all similar word as same, it improves the
performance of NLP models by making text more consistent. For example, treating
"running," "ran" and "runs" as the same word improves the model's understanding of
context and meaning.
Lemmatization Techniques
 There are different techniques to perform lemmatization each with its own
advantages and use cases:
1. Rule Based Lemmatization
 In rule-based lemmatization, predefined rules are applied to a word to remove suffixes
and get the root form. This approach works well for regular words but may not handle
irregularities well.
 For example:
 Rule: For regular verbs ending in "-ed," remove the "-ed" suffix.
 Example: "walked" -> "walk"
 While this method is simple and interpretable, it doesn't account for irregular word
forms like "better" which should be lemmatized to "good".
 2. Dictionary-Based Lemmatization
 It uses a predefined dictionary or lexicon such as WordNet to look up the base form of
a word. This method is more accurate than rule-based lemmatization because it
accounts for exceptions and irregular words.
 For example:
 'running' -> 'run'
 'better' -> 'good'
 'went' -> 'go
 "I was running to become a better athlete and then I went home," -> "I was run to
become a good athlete and then I go home."
 By using dictionaries like WordNet this method can handle a range of words
effectively, especially in languages with well-established dictionaries.
 3. Machine Learning-Based Lemmatization
  It uses algorithms trained on large datasets to automatically identify the base form of
words. This approach is highly flexible and can handle irregular words and linguistic
nuances better than the rule-based and dictionary-based methods.
For example:
 A trained model may deduce that “went” corresponds to “go” even though the suffix
removal rule doesn’t apply. Similarly, for 'happier' the model deduces 'happy' as the
lemma.
 Machine learning-based lemmatizers are more adaptive and can generalize across
different word forms which makes them ideal for complex tasks involving diverse
vocabularies.
 For more details regarding these techniques refer to: Python - Lemmatization
Approaches with Examples

Implementation of Lemmatization in Python

Lets see step by step how Lemmatization works in Python:
 Step 1: Installing NLTK and Downloading Necessary Resources
 In Python, the NLTK library provides an easy and efficient way to implement
lemmatization. First, we need to install the NLTK library and download the necessary
datasets like WordNet and the punkt tokenizer.

!pip install nltk

import nltk

[Link]('punkt_tab')

[Link]('wordnet')

[Link]('omw-1.4')

[Link]('averaged_perceptron_tagger_eng')

Step 2: Lemmatizing Text with NLTK

Now we can tokenize the text and apply lemmatization using

NLTK's WordNetLemmatizer.
from [Link] import word_tokenize
from [Link] import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

text = "The cats were running faster than the dogs."

tokens = word_tokenize(text)

lemmatized_words = [[Link](word) for word in tokens]

print(f"Original Text: {text}")
print(f"Lemmatized Words: {lemmatized_words}")

Output:
Original Text: The cats were running faster than the dogs.

Lemmatized Words: ['The', 'cat', 'were', 'running', 'faster', 'than',

'the', 'dog', '.']