DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING (AIML)

LABORATORY MANUAL

SUBJECT CODE: AL3501

SUBJECT NAME: NATURAL LANGUAGE PROCESSING LABORATORY REGUALATION:
R2021

P a g e 1|1
 SYLLABUS

Course
COURSE NAME L T P C
Code
AL3501 NATURAL LANGUAGE PROCESSING 0 0 2 3

COURSE OBJECTIVES:
• To learn the fundamentals of natural language processing.
• To learn the word level analysis methods.
• To explore syntactic analysis concepts.
• To understand semantics and pragmatics.
• To learn to analyze discourses and Lexical Resources.

EXPERIMENTS
1 Word Analysis

2 Word Generation.

3 Morphological Analysis

4 N-Grams and N-Grams Smoothing.

5 POS Tagging using Hidden Markov Model

6 POS Tagging using Viterbi Decoding

7 Building a POS Tagger

8 Chunking and Building Chunker

9 Minimum Edit Distance Computation

TOTAL: 30 PERIODS
COURSE OUTCOMES:

At the end of this course, the students will be able to:

CO1: Tag a given text with basic Language features.

CO2: Implement a rule based system to tackle morphology/syntax of a language.

CO3: Design a tag set to be used for statistical processing for real-time applications.

P a g e 2|2
CO4: Compare and contrast the use of different statistical approaches for different types of NLP
applications.

CO5: Use tools to process natural language and design innovative NLP applications.

LIST OF EXPERIMENTS

[Link]. NAME OF THE EXPERIMENT PAGE NO

1. Word Analysis 5

2. Word Generation 10

3. Morphological analysis 13

4. N-Grams and N-Grams Smoothing 18

5. POS Tagging using Hidden Markov Model –Virtual Lab 22

6. POS Tagging using Viterbi Decoding – Virtual Lab 27

7. Building a POS Tagger 31

8. Chunking and Building Chunker 35

9. Minimum Edit Distance

Computation 40

P a g e 3|3
EX NO.: 1
WORD ANALYSIS

Aim
To study and implement various text preprocessing techniques such as
tokenization, filtration, script validation, stop word removal, and stemming using
Python.

Software Requirements
 Python 3.x
 NLTK Library
 Jupyter Notebook / VS Code / Spyder

Theory
Text preprocessing is an important step in Natural Language Processing (NLP).
Raw text data often contains punctuation marks, special symbols, stop words, and
different word forms that may affect analysis. Text preprocessing converts
unstructured text into a clean and structured format suitable for NLP applications.
Tokenization
Tokenization is the process of splitting text into smaller units called tokens. These
tokens may be words, sentences, or characters. It is the first step in most NLP
tasks.
Example:
Input: Natural Language Processing is interesting.
Output: [Natural, Language, Processing, is, interesting]
Filtration
Filtration removes unwanted elements such as punctuation marks, special
characters, and extra spaces from the text. This helps improve the quality of the
data for further processing.
Script Validation
Script validation ensures that the input text follows the expected language and
character format. It helps identify invalid characters and ensures correct text
processing.
Stop Word Removal

P a g e 4|4
Stop words are frequently occurring words such as a, an, the, is, and are that carry
little semantic meaning. Removing stop words reduces the size of the text and
improves processing efficiency.
Example:
Original: The machine learning model is trained using the data.
Processed: machine learning model trained using data
Stemming
Stemming reduces words to their root form by removing prefixes or suffixes.

Word Stem

playing play

running run

connected connect

Stemming helps reduce vocabulary size and improves the performance of NLP
systems.

Algorithm
Algorithm for Text Preprocessing
1. Read the input text.
2. Perform tokenization.
3. Remove punctuation marks and special symbols.
4. Validate the text script.
5. Remove stop words.
6. Apply stemming.
7. Display the processed output.

Program 1: String Handling

Code

Output

Program 2: Tokenization and Filtering

Code

P a g e 5|5
Input File

Output

Program 3: Stop Word Removal

Code

Output

Program 4: Stemming Using Porter Stemmer

Code

Output

Augmented Questions
1: Perform lemmatization on a given text and compare the results with stemming.
2: Remove URLs, hashtags, punctuation, and stop words from social media text
and display the cleaned output.

Viva Voce Questions

1. What is tokenization?
2. What are stop words?
3. Define stemming.
4. Differentiate stemming and lemmatization.
5. What is script validation?
6. Why is text preprocessing necessary?
7. Mention any two applications of NLP.

Result

P a g e 6|6
Thus, various text preprocessing techniques such as tokenization, filtration, script
validation, stop word removal, and stemming were studied and successfully
implemented using Python.

EX. NO.: 2
WORD GENERATION

Aim
To study and implement word generation by producing appropriate word forms
from a given root word and grammatical features.

Software Requirements
 Python 3.x
 Jupyter Notebook / VS Code
 Morphological Processing Tools (if available)

Theory
Word Generation is the process of producing a valid word form from a given root
word and a set of grammatical features such as number, gender, tense, person,
and case. It is the reverse process of morphological analysis.
In Natural Language Processing, word generation is widely used in machine
translation, text generation, grammar checking systems, and conversational
agents.
For example, given the root word play and grammatical information indicating third
-person singular present tense, the generated word is plays.
Similarly, given the root word boy and plural information, the generated word is
boys.
Word generation helps NLP systems produce grammatically correct words based
on linguistic rules.
Examples

Root Word Features Generated Word

play Verb, Present, 3rd Person Singular plays

walk Verb, Past Tense walked

P a g e 7|7
Root Word Features Generated Word

boy Noun, Plural boys

girl Noun, Plural girls

Word generation is extensively used in machine translation systems and language

generation applications.

Algorithm
Algorithm for Word Generation
1. Read the root word.
2. Read the grammatical features.
3. Identify the word category (noun/verb).
4. Apply the appropriate morphological rules.
5. Generate the required word form.
6. Display the generated word.

Program
Code

Output

Augmented Questions
1: Generate singular and plural forms of regular and irregular nouns.
2: Generate different verb forms (present, past, and continuous) from a given root
word.

Viva Voce Questions

1. What is word generation?
2. How is word generation different from morphological analysis?
3. What is a root word?
4. Mention any two grammatical features used in word generation.
5. Give an example of noun generation.
6. Give an example of verb generation.
7. Mention any two applications of word generation.

P a g e 8|8
Result
Thus, the process of word generation was studied and implemented successfully
using Python.

EX NO.: 3
MORPHOLOGICAL ANALYSIS

Aim
To study and implement morphological analysis techniques such as regular
expressions, stop word removal, synonym extraction, and stemming using Python.

Software Requirements
 Python 3.x
 NLTK Library
 Regular Expression (re) Module
 Jupyter Notebook / VS Code / Spyder

Theory
Morphological Analysis is the process of analyzing the internal structure of words
and identifying their root words, prefixes, suffixes, and grammatical properties. It
plays an important role in Natural Language Processing by helping computers
understand the meaning and structure of words.
Morphological analysis is used to determine the relationship between different
forms of a word and classify words based on their grammatical categories.
Regular Expressions
Regular Expressions (Regex) are patterns used to search, match, and manipulate
text. They are widely used for extracting information such as email addresses,
phone numbers, hashtags, and specific word patterns from text.
Stop Word Removal
Stop words are commonly used words such as the, a, an, is, and are that
contribute little meaning to text analysis. Removing them helps reduce
unnecessary processing and improves efficiency.
Synonym Extraction
Synonyms are words that have similar meanings. Identifying synonyms helps
improve information retrieval, text mining, and semantic analysis tasks.
P a g e 9|9
Example:
 Happy → Joyful
 Begin → Start
 Large → Big

Stemming
Stemming reduces words to their root form by removing suffixes and prefixes.
Example:

Word Stem

eating eat

dancing danc

connected connect

Morphological analysis helps NLP systems process language more effectively by

reducing word variations and identifying meaningful word structures.

Algorithm
Algorithm for Morphological Analysis
1. Read the input text.
2. Apply regular expressions to process text patterns.
3. Remove stop words from the text.
4. Extract synonyms using WordNet.
5. Apply stemming to reduce words to their root forms.
6. Display the processed results.

Program 1: Regular Expression

Code

Output

Program 2: Stop Word Removal

Code

P a g e 10 | 10
Output

Program 3: Synonym Extraction

Code

Output

Program 4: Stemming
Code

Output

Augmented Questions
1: Extract prefixes, suffixes, and root words from a list of words.
2: Identify synonyms and antonyms of a given word using WordNet.

Viva Voce Questions

1. What is morphological analysis?
2. What is a regular expression?
3. What are stop words?
4. What is stemming?
5. What is a synonym?
6. What is WordNet?
7. Mention any two applications of morphological analysis.

Result
Thus, morphological analysis techniques such as regular expressions, stop word
removal, synonym extraction, and stemming were studied and successfully
implemented using Python.

EX. NO 4
N-GRAMS AND N-GRAM SMOOTHING

P a g e 11 | 11
Aim
To study and implement N-Gram models and N-Gram smoothing techniques using
Python.

Software Requirements
 Python 3.x
 NLTK Library
 Jupyter Notebook / VS Code / Spyder

Theory
An N-Gram is a sequence of N consecutive words occurring in a text. N-Gram
models are statistical language models used to predict the next word in a
sequence based on the previous words.
N-Grams play an important role in Natural Language Processing applications such
as speech recognition, machine translation, predictive text input, spelling
correction, and text generation.
Based on the value of N, N-Grams are classified as:

N Type Example

1 Unigram Machine

2 Bigram Machine Learning

3 Trigram Machine Learning Model

4 Four-Gram Machine Learning Model Works

Example
Sentence:
Machine learning is an important part of Artificial Intelligence.
Bigrams:
 Machine learning
 learning is
 is an
 an important
 important part
 part of
 of Artificial

P a g e 12 | 12
  Artificial Intelligence

N-Gram Smoothing
In language modeling, some word combinations may not appear in the training
data and therefore receive a probability of zero. This problem is known as the zero
-frequency problem.
Smoothing techniques are used to assign a small probability to unseen word
sequences.
One commonly used method is Laplace (Add-One) Smoothing.

Smoothing improves the robustness of language models by handling unseen

words and word sequences.

Algorithm
Algorithm for N-Gram Generation
1. Read the input sentence.
2. Tokenize the sentence into words.
3. Select the value of N.
4. Generate N-Gram sequences.
5. Display the generated N-Grams.
Algorithm for N-Gram Smoothing
1. Calculate N-Gram frequencies.
2. Identify unseen word combinations.
P a g e 13 | 13
 3. Apply Laplace smoothing.
4. Compute adjusted probabilities.
5. Display the smoothed probabilities.

Program 1: Bigram Generation

Code

Output

Program 2: Trigram Generation

Code

Output

Program 3: Laplace Smoothing

Code

Output

Augmented Questions
1: Generate unigram, bigram, and trigram models for a given paragraph.
2: Implement Laplace smoothing and calculate probabilities for unseen bigrams.

Viva Voce Questions

1. What is an N-Gram?
2. Differentiate unigram, bigram, and trigram.
3. What is the zero-frequency problem?
4. Why is smoothing required in language models?
5. What is Laplace smoothing?
6. Mention two applications of N-Gram models.
7. What is the role of N-Grams in predictive text systems?

Result
Thus, N-Gram models and N-Gram smoothing techniques were studied and
successfully implemented using Python.

P a g e 14 | 14
EX. NO.: 5
PART-OF-SPEECH TAGGING USING HIDDEN MARKOV MODEL (HMM)

Aim
To study and implement Part-of-Speech (POS) tagging using the Hidden Markov
Model (HMM).

Software Requirements
This experiment shall be conducted using the IIIT Hyderabad Virtual Lab – POS
Tagging using Hidden Markov Model. Students will perform the experiment
through the interactive simulation environment provided in the virtual lab platform.
Instructions
1. Open the Virtual Lab link in a web browser with a stable internet connection.
2. Read the objective, theory, and procedure provided in the virtual lab before
execution.
3. Observe the training corpus, transition probabilities, and emission probabilities
displayed in the simulation.
4. Enter the given test sentence in the input section of the simulator.
5. Execute the simulation and observe the POS tagging process step by step.
6. Analyze how the Hidden Markov Model assigns tags based on contextual
probabilities.
7. Record the generated POS tags and corresponding observations in the laboratory
record.
8. Take screenshots of important outputs obtained from the virtual lab.
9. Repeat the experiment using different input sentences and compare the results.
10. Write the inference and result based on the obtained POS tagging output.

Theory
Part-of-Speech (POS) Tagging is the process of assigning grammatical categories
such as noun, verb, adjective, adverb, and pronoun to each word in a sentence.

P a g e 15 | 15
Example:

Word POS Tag

Ram NNP

is VBZ

reading VBG

a DT

book NN

In a Hidden Markov Model (HMM), the actual POS tags are considered hidden
states, while the words in the sentence are observable outputs. HMM uses
probability values to determine the most likely sequence of tags for a given
sentence.
The model uses two probabilities:
Transition Probability
The probability of moving from one tag to another.
Example:
P(Verb | Noun)
Emission Probability
The probability of observing a word given a particular tag.
Example:
P(reading | Verb)
HMM-based POS tagging is widely used because it considers both the current
word and the context provided by neighboring words.

HMM Model
The probability of a tag sequence is computed using:

P a g e 16 | 16
Algorithm
Algorithm for POS Tagging using HMM
1. Read the input sentence.
2. Tokenize the sentence into words.
3. Load the POS tagging model.
4. Calculate transition probabilities.
5. Calculate emission probabilities.
6. Assign the most probable POS tag to each word.
7. Display the tagged output.

Program
Code

Output
Program 2: POS Tagging Multiple Sentences

P a g e 17 | 17
Code

Output
.:.

Common POS Tags

Tag Meaning

NN Noun

NNS Plural Noun

NNP Proper Noun

VB Verb

VBD Past Tense Verb

VBG Verb Gerund

JJ Adjective

RB Adverb

DT Determiner

IN Preposition

Augmented Questions
1: Perform POS tagging on a paragraph and count the frequency of each POS tag.
2: Extract all nouns, verbs, adjectives, and adverbs from a given text using POS
tagging.

Viva Voce Questions

1. What is POS tagging?
2. What is a Hidden Markov Model?
3. What are hidden states in HMM POS tagging?
4. What is transition probability?
5. What is emission probability?
6. Give any two POS tags and their meanings.
7. Mention any two applications of POS tagging.

Result
Thus, Part-of-Speech tagging using the Hidden Markov Model (HMM) was studied
and successfully implemented using Python.

P a g e 18 | 18
EX. NO.: 6
POS TAGGING USING VITERBI DECODING

Aim
To study and implement the Viterbi Decoding algorithm for Part-of-Speech (POS)
tagging.

Software Requirements
This experiment shall be conducted using the IIIT Hyderabad Virtual Lab – POS
Tagging using Viterbi Decoding. The virtual lab provides an interactive
environment to understand sequence decoding using the Viterbi algorithm.
Instructions
1. Open the Virtual Lab website using the prescribed URL.
2. Carefully study the aim, theory, and algorithm given in the virtual lab.
3. Enter the sample sentence in the simulation input field.
4. Execute the Viterbi decoding process and observe the step-by-step
computation.
5. Analyze transition probabilities and emission probabilities used during
decoding.
6. Observe how the Viterbi algorithm determines the most probable sequence
of POS tags.
7. Record all intermediate observations and final outputs in the observation
notebook.

P a g e 19 | 19
 8. Capture screenshots of the decoding process and final tagged sentence.
9. Test the experiment with different sentences to understand contextual
tagging behavior.
10. Write the result and inference based on the generated POS tag sequence.

Theory
The Viterbi Algorithm is a dynamic programming algorithm used to determine the
most probable sequence of hidden states in a Hidden Markov Model (HMM).
In POS tagging, the hidden states are the POS tags, and the observed states are
the words in a sentence. Since a word may belong to multiple grammatical
categories, the Viterbi algorithm helps identify the most likely tag sequence based
on transition and emission probabilities.
The algorithm efficiently computes the best path by storing intermediate
probabilities and avoiding repeated calculations.
Example
Sentence:
Time flies like an arrow.
Possible tags may exist for some words. The Viterbi algorithm selects the tag
sequence with the highest probability among all possible combinations.
Advantages
 Efficient computation using dynamic programming.
 Finds the most probable tag sequence.
 Widely used in NLP and speech recognition applications.

Viterbi Formula
The Viterbi algorithm computes the highest probability path using:

P a g e 20 | 20
Algorithm
Algorithm for Viterbi Decoding
1. Read the input sentence.
2. Tokenize the sentence into words.
3. Initialize transition and emission probabilities.
4. Compute probabilities for each state.
5. Store the highest probability path.
6. Trace back the best sequence of tags.
7. Display the tagged sentence.

Program
Code

Output

Program 2: POS Tagging Example

Code

Output

Augmented Questions
1: Implement the Viterbi algorithm to determine the most probable POS tag
sequence for a sentence.
2: Compare POS tagging results obtained with and without Viterbi decoding.

Viva Voce Questions

1. What is the Viterbi algorithm?
2. Why is Viterbi Decoding used in HMM?
3. What are hidden states in POS tagging?
4. What is transition probability?

P a g e 21 | 21
 5. What is emission probability?
6. What is dynamic programming?
7. Mention any two applications of the Viterbi algorithm.

Result
Thus, the Viterbi Decoding algorithm for POS tagging was studied and
implemented successfully using Python.

EX. NO 7:
BUILDING A POS TAGGER

Aim
To study and implement a Part-of-Speech (POS) Tagger using Python and the
NLTK library.

Software Requirements
 Python 3.x
 NLTK Library
 Jupyter Notebook / VS Code / Spyder

Theory
Part-of-Speech (POS) Tagging is the process of assigning grammatical categories
such as noun, verb, adjective, adverb, pronoun, and preposition to words in a
sentence.
A POS Tagger is a software tool that automatically assigns these tags based on
lexical information and contextual analysis.
POS tagging plays an important role in Natural Language Processing because
many NLP tasks such as machine translation, information extraction, question
answering, and text summarization require knowledge of the grammatical
structure of sentences.
Example
Sentence:
The boy plays cricket.
POS Tagged Output:

Word POS Tag

P a g e 22 | 22
 Word POS Tag

The DT

boy NN

plays VBZ

cricket NN

A POS tagger analyzes each word and assigns the most appropriate grammatical
category.

Common POS Tags

Tag Description

NN Noun

NNS Plural Noun

NNP Proper Noun

VB Verb

VBD Past Tense Verb

VBG Gerund Verb

JJ Adjective

RB Adverb

DT Determiner

IN Preposition

Algorithm
Algorithm for Building a POS Tagger
1. Import the required NLTK libraries.
2. Read the input sentence.
3. Tokenize the sentence into words.
4. Apply POS tagging.
5. Assign grammatical tags to each word.
6. Display the tagged output.
7. Analyze the generated tags.

P a g e 23 | 23
Program 1: POS Tagging
Code

Output

Program 2: POS Tagging for Multiple Sentences

Code

Output

Augmented Questions
1: Train and test a POS tagger using a custom text corpus.
2: Evaluate the accuracy of a POS tagger using a test dataset.

Viva Voce Questions

1. What is a POS Tagger?
2. What is POS tagging?
3. What does the tag NN represent?
4. What does the tag VBZ represent?
5. Why is POS tagging important in NLP?
6. Mention any two applications of POS tagging.
7. Which Python library is commonly used for POS tagging?

Result
Thus, a Part-of-Speech (POS) Tagger was built and implemented successfully
using Python and the NLTK library.

P a g e 24 | 24
EX. NO.: 8
CHUNKING AND BUILDING A CHUNKER

Aim
To study and implement chunking techniques for extracting meaningful phrases
from text using Python.

Software Requirements
 Python 3.x
 NLTK Library
 Jupyter Notebook / VS Code / Spyder

Theory
Chunking, also known as shallow parsing, is the process of grouping words into
meaningful phrases based on their Part-of-Speech (POS) tags. Instead of
analyzing the complete grammatical structure of a sentence, chunking identifies
important phrase structures such as noun phrases and verb phrases.
A chunk is a collection of related words that together form a meaningful unit in a
sentence.
Example
Sentence:
The little yellow dog barked at the cat.
POS Tags:

Word Tag

The DT

little JJ

yellow JJ

dog NN

barked VBD

at IN

the DT

cat NN

P a g e 25 | 25
Noun Phrases Identified:
 The little yellow dog
 the cat
Chunking helps in extracting important information from text and is widely used in
information extraction and text mining applications.
Types of Chunks
1. Noun Phrase (NP)
2. Verb Phrase (VP)
3. Prepositional Phrase (PP)
Chunk Grammar
Chunk patterns are defined using POS tags.
Example:
NP: {<DT>?<JJ>*<NN>}
Where:
 DT → Determiner
 JJ → Adjective
 NN → Noun
This pattern identifies a noun phrase consisting of an optional determiner, zero or
more adjectives, and a noun.

Algorithm
Algorithm for Chunking
1. Read the input sentence.
2. Perform POS tagging.
3. Define chunk grammar rules.
4. Apply the chunk parser.
5. Identify meaningful phrases.
6. Display the chunk structure.
7. Analyze the extracted chunks.

Program 1: Noun Phrase Chunking

Code
P a g e 26 | 26
Output

Program 2: Chunk Tree Visualization

Code

Output

Augmented Questions
1: Extract noun phrases from a paragraph using chunk grammar rules.
2: Build a chunker to identify noun phrases and verb phrases in a given text.

Viva Voce Questions

1. What is chunking in NLP?
2. What is meant by shallow parsing?
3. What is a noun phrase?
4. What is chunk grammar?
5. What is the purpose of a chunker?
6. Mention any two applications of chunking.
7. Differentiate POS tagging and chunking.

Result
Thus, chunking techniques were studied and a chunker was successfully built
using Python to extract meaningful phrases from text.

EX. NO.: 9 CONTENT BEYOND SYLLABUS

MINIMUM EDIT DISTANCE COMPUTATION

Aim
To study and implement the Minimum Edit Distance (MED) algorithm for
measuring the similarity between two words.

Software Requirements
 Python 3.x
P a g e 27 | 27
  Jupyter Notebook / VS Code / Spyder

Theory
Minimum Edit Distance (MED) is a technique used in Natural Language
Processing to determine the minimum number of editing operations required to
transform one word into another.
The three basic edit operations are:
1. Insertion – Adding a character.
2. Deletion – Removing a character.
3. Substitution – Replacing one character with another.
Each operation is assigned a cost, usually 1. The minimum total cost required to
convert one word into another is called the Minimum Edit Distance.
Example
Convert:
CAT → CUT
Operation:
 Replace A with U
Minimum Edit Distance = 1
Applications
 Spell Checking
 Auto Correction
 Information Retrieval
 DNA Sequence Analysis
 Machine Translation
 Text Similarity Measurement

Minimum Edit Distance Formula

The recurrence relation used in MED computation is:

P a g e 28 | 28
Algorithm
Algorithm for Minimum Edit Distance
1. Read two input strings.
2. Create a distance matrix.
3. Initialize the first row and first column.
4. Compare characters of both strings.
5. Compute insertion, deletion, and substitution costs.
6. Select the minimum cost for each cell.
7. Display the final minimum edit distance.

Program
Code

Output

Explanation

Augmented Questions
P a g e 29 | 29
1: Develop a spell checker that suggests the closest matching word using
Minimum Edit Distance.
2: Compute and compare the Minimum Edit Distance for multiple pairs of words
and rank them based on similarity.

Viva Voce Questions

1. What is Minimum Edit Distance?
2. What are the basic edit operations used in MED?
3. What is the purpose of MED in NLP?
4. What is insertion operation?
5. What is substitution operation?
6. Mention any two applications of MED.
7. What is the Minimum Edit Distance between "CAT" and "CUT"?

Result
Thus, the Minimum Edit Distance algorithm was studied and implemented
successfully to compute the similarity between two strings.

P a g e 30 | 30