UNIT-II

N- gram model in nlp

In Natural Language Processing (NLP), an N-gram model is a type of statistical language

model that predicts the next item (typically a word) in a sequence based on the preceding N-1 items. It
works by analyzing the frequency of sequences of 'N' words (or characters) in a given text or speech
corpus (a large collection of text or speech data).

N-grams can be of various types based on the value of 'n':

• Unigrams (1-grams) are single words

• Bigrams (2-grams) are pairs of consecutive words
• Trigrams (3-grams) are triplets of consecutive words

N-gram

• An N-gram is a contiguous sequence of 'N' items (words, letters, or syllables) extracted from a given
sample of text or speech.

• The value of 'N' determines the order of the N-gram, influencing the length of the word sequence under
consideration.

• Common examples include:

o Unigram (N=1): A single word, like "cat" or "house".

o Bigram (N=2): A sequence of two words, like "hot dog" or "machine learning".

o Trigram (N=3): A sequence of three words, like "Natural Language Processing".

• N-grams can be extended to 4-grams, 5-grams, and beyond, with the choice of N depending on the
specific NLP task and available data.
 N-gram language model

• An N-gram model estimates the probability of a word, given the preceding N-1 words, based on the
frequency of these sequences in the training data. This is based on the Markov assumption, which
simplifies the problem by assuming the probability of a word depends only on a limited window of
preceding words,

• The probability of a sequence of words is calculated by multiplying the conditional probabilities of

each word in the sequence, given its preceding words within the N-gram window.

• For example, in a bigram model, the probability of the phrase "I want coffee" would be approximated
as P("I") * P("want"|"I") * P("coffee"|"want").

• N-gram models are trained by counting the occurrences of different N-grams and (N-1)-grams in a text
corpus and using these counts to calculate the conditional probabilities,

Applications in NLP

N-gram models have a wide range of applications in NLP

• Language Modelling: Predicting the next word in a sequence, used in predictive text input and
autocomplete functions.

• Speech Recognition: Enhancing accuracy by modeling spoken language patterns and correcting errors
based on probability.

• Machine Translation: Generating more natural and accurate translations by considering word sequences
and context.

• Text Classification and Sentiment Analysis: Identifying patterns that help categorize text or determine
its emotional tone.

• Information Retrieval and Search Engine Algorithms: Recognizing relevant word patterns to improve
search results and ranking.

• Spell Checking and Grammar Correction: Rectifying errors by suggesting more probable word
sequences.

• Plagiarism Detection: Identifying similar phrases and structures between documents.

Advantages

• Simplicity: Easy to understand and implement.

• Efficiency: Can be computationally lightweight for smaller N-gram values, suitable for baseline models
and rapid prototyping.

• Local Context Capture: Effectively captures short-range dependencies between words, preserving their
immediate sequence.

• Strong Baseline Performance: Can provide competitive performance for various NLP tasks as a starting
point.

Challenges and limitations

• Data Sparsity: As N increases, the number of possible N-grams grows exponentially. This makes it
difficult to encounter all possible sequences in the training data, leading to low or zero probabilities for
unseen N-grams. Smoothing techniques, like Laplace smoothing, are used to address this by assigning
small, non-zero probabilities to unseen N-grams.

• Lack of Semantic Understanding: N-gram models focus on patterns rather than deep meaning,
struggling with context and semantic relationships.

• Limited Long-Range Context: N-gram models primarily consider a fixed window of preceding words,
making them less effective at capturing long-range dependencies crucial for understanding complex
sentences or paragraphs.

• Scalability Challenges: Large N-gram models can demand significant memory and storage due to the
exponential growth of possible N-grams with increasing N.

N-gram models and neural networks

While N-gram models are effective for certain tasks, neural language models have emerged as
more powerful tools in many NLP applications. Neural networks, like Recurrent Neural Networks
(RNNs) and Transformer-based models (e.g., BERT and GPT), can better capture long-range
dependencies and contextual understanding through techniques like word embedding. However, N-
gram models still serve as valuable baselines and find use in specific applications, particularly when
simplicity, speed, or integration with existing systems are important considerations

Smoothing Techniques:
POS(Parts-Of-Speech) Tagging

Parts of Speech (PoS) tagging is a core task in NLP, It gives each word a grammatical category such
as nouns, verbs, adjectives and adverbs. Through better understanding of phrase structure and
semantics, this technique makes it possible for machines to study human language more accurately.
 PoS tagging is essential in many NLP applications like machine translation, sentiment analysis and
information retrieval. It serves as a link between language and machine understanding, enabling the
creation of complex language processing systems.

POS(Parts-Of-Speech) Tagging
Parts of Speech tagging is a linguistic activity in Natural Language Processing (NLP) wherein each
word in a document is given a particular part of speech (adverb, adjective, verb etc.) or grammatical
category. Through the addition of a layer of syntactic and semantic information to the words, this
procedure makes it easier to understand the sentence's structure and meaning.
In NLP applications, POS tagging is useful for machine translation, named entity recognition and
information extraction, among other things. It also works well for clearing out ambiguity in terms
with numerous meanings and revealing a sentence's grammatical structure.
Example of POS Tagging
Consider the sentence: "The quick brown fox jumps over the lazy dog."
After performing POS Tagging:
• "The" is tagged as determiner (DT)
• "quick" is tagged as adjective (JJ)
• "brown" is tagged as adjective (JJ)
• "fox" is tagged as noun (NN)
• "jumps" is tagged as verb (VBZ)
• "over" is tagged as preposition (IN)
• "the" is tagged as determiner (DT)
• "lazy" is tagged as adjective (JJ)
• "dog" is tagged as noun (NN)
By offering insights into the grammatical structure, this tagging helps machines in understanding not
just individual words but also the connections between them inside a phrase. For many NLP
applications like text summarization, sentiment analysis, this kind of data is essential.
Workflow of POS Tagging in NLP
• Tokenization: The input text is divided into individual tokens, representing words or subwords.
Tokenization is the foundational step in most NLP tasks which enables further analysis at the word
level.
• Loading a Language Model: Tools like NLTK or SpaCy requires a pre-trained language model
to perform POS tagging. These models are trained on large datasets and provide insights into the
grammatical rules and structure of the language.
• Text Preprocessing: The text is then cleaned to improve accuracy. Common preprocessing steps
include converting text to lowercase, removing xspecial characters and eliminating irrelevant
content.
• Linguistic Analysis: This stage involves parsing the sentence to understand the grammatical role
of each token. It lays the groundwork for assigning the appropriate part of speech by interpreting
the sentence’s syntactic structure.
• POS Tagging: Each token is then assigned a specific part-of-speech label. This is based on its
role in the sentence and contextual clues provided by surrounding words.
• Result Evaluation: Finally, the POS-tagged output is reviewed to ensure accuracy. Any
misclassifications or anomalies are identified and corrected as needed.

Types of POS Tagging in NLP

Assigning grammatical categories to words in a text is known as Part-of-Speech (PoS)
tagging and it is an essential aspect of Natural Language Processing (NLP). Different PoS tagging
approaches exist, each with a unique methodology. Here are a few typical kinds:

Rule-Based Tagging
Rule-based POS tagging assigns grammatical tags to words using a predefined set of rules, as opposed
to machine learning-based methods that require training on annotated corpora. These rules are crafted
based on morphological features (like word endings) and syntactic context, making the approach
highly interpretable and transparent.
Example
a rule might specify that words ending in “-tion” or “-ment” should be tagged as nouns, based on
common suffix patterns found in English.
• Rule: Assign the POS tag "Noun" to words ending in -tion or -ment.
• Text: "The presentation highlighted the key achievements of the project's development."
Tagged Output:
• "The" : Determiner (DET)
• "presentation" : Noun (N)
• "highlighted" : Verb (V)
• "the" : Determiner (DET)
• "key" : Adjective (ADJ)
• "achievements" : Noun (N)
• "of" : Preposition (PREP)
• "the" : Determiner (DET)
• "project's" : Noun (N)
• "development" : Noun (N)
In this case, the rule-based tagger correctly identifies "presentation," "achievements," and
"development" as nouns by applying suffix-based rule. While simple, this example illustrates how
rule-based systems can handle a wide range of linguistic patterns using structured, interpretable logic.

Transformation Based tagging

Transformation-Based Tagging (TBT) is a method for refining POS tags through a series of
context-based transformations. Unlike statistical taggers that rely on probabilities or rule-based
taggers that apply static rules, TBT starts with initial tags and improves them iteratively by applying
transformation rules.
Example
a rule might state: “Change a word’s tag from Verb to Noun if it follows a determiner like ‘the’.”
• Text: "The cat chased the mouse."
• Initial Tags: "The" – DET, "cat" – N, "chased" – V, "the" – DET, "mouse" – N
• Transformation Rule Applied: Change “chased” from Verb to Noun because it follows “the”.
• Updated Tags: "chased" becomes Noun.
Statistical POS Tagging
Statistical POS tagging is a computational linguistics approach that uses probabilistic models
to assign grammatical categories (e.g., noun, verb, adjective) to words in a text. Unlike rule-based
methods, which rely on handcrafted rules, statistical tagging learns patterns from large annotated
corpora using machine learning techniques.
These models estimate the probability of a tag given a word and its context, enabling them to resolve
linguistic ambiguities and adapt to complex grammatical structures.
Advantages of POS tagging
Advantages Description

Text Simplification Helps deconstruct complex sentences for easier understanding.

Improved Information Enables more accurate indexing and searching based on

Retrieval grammatical categories.

Named Entity Recognition Serves as a precursor for identifying names, places and
(NER) organizations.

Syntactic Parsing Assists in analyzing sentence structure and word relationships.

Disadvantages of POS Tagging

Disadvantages Description

Ambiguity Words may have multiple meanings depending on context.

Idiomatic Expressions Informal or non-standard phrases are hard to tag correctly.

Out-of-Vocabulary Words Unseen words can lead to incorrect tagging.

Domain Dependence Models may not generalize well outside their training domain.
Hidden Markov Model

HMM (Hidden Markov Model) is a Stochastic technique for POS tagging. Hidden Markov models are
known for their applications to reinforcement learning and temporal pattern recognition such as speech,
handwriting, gesture recognition, musical score following, partial discharges, and bioinformatics.

Let us consider an example proposed by [Link] Serrano and find out how HMM selects an appropriate
tag sequence for a sentence.

In this example, we consider only 3 POS tags that are noun, model and verb. Let the sentence “ Ted
will spot Will ” be tagged as noun, model, verb and a noun and to calculate the probability associated
with this particular sequence of tags we require their Transition probability and Emission probability.

The transition probability is the likelihood of a particular sequence for example, how likely is that a
noun is followed by a model and a model by a verb and a verb by a noun. This probability is known as
Transition probability. It should be high for a particular sequence to be correct.

Now, what is the probability that the word Ted is a noun, will is a model, spot is a verb and Will is a
noun. These sets of probabilities are Emission probabilities and should be high for our tagging to be
likely.

Let us calculate the above two probabilities for the set of sentences below

• Mary Jane can see Will

• Spot will see Mary
• Will Jane spot Mary?
• Mary will pat Spot
Note that Mary Jane, Spot, and Will are all names.
In the above sentences, the word Mary appears four times as a noun. To calculate the emission
probabilities, let us create a counting table in a similar manner.

Words Noun Model Verb

Mary 4 0 0

Jane 2 0 0

Will 1 3 0

Spot 2 0 1

Can 0 1 0
 See 0 0 2

pat 0 0 1

Now let us divide each column by the total number of their appearances for example, ‘noun’ appears
nine times in the above sentences so divide each term by 9 in the noun column. We get the following
table after this operation.

From the above table, we infer that

Words Noun Model Verb

Mary 4/9 0 0

Jane 2/9 0 0

Will 1/9 3/4 0

Spot 2/9 0 1/4

Can 0 1/4 0

See 0 0 2/4

pat 0 0 1

The probability that Mary is Noun = 4/9

The probability that Mary is Model = 0

The probability that Will is Noun = 1/9

The probability that Will is Model = 3/4

In a similar manner, you can figure out the rest of the probabilities. These are the emission probabilities.
 Next, we have to calculate the transition probabilities, so define two more tags <S> and <E>. <S> is
placed at the beginning of each sentence and <E> at the end as shown in the figure below.

Let us again create a table and fill it with the co-occurrence counts of the tags.

N M V <E>

<S> 3 1 0 0

N 1 3 1 4

M 1 0 3 0

V 4 0 0 0

In the above figure, we can see that the <S> tag is followed by the N tag three times, thus the first entry
is [Link] model tag follows the <S> just once, thus the second entry is 1. In a similar manner, the rest of
the table is filled.

Next, we divide each term in a row of the table by the total number of co-occurrences of the tag in
consideration, for example, The Model tag is followed by any other tag four times as shown below, thus
we divide each element in the third row by four.
 N M V <E>

<S> 3/4 1/4 0 0

N 1/9 3/9 1/9 4/9

M 1/4 0 3/4 0

V 4/4 0 0 0

These are the respective transition probabilities for the above four sentences. Now how does the HMM
determine the appropriate sequence of tags for a particular sentence from the above tables? Let us find
it out.

Take a new sentence and tag them with wrong tags. Let the sentence, ‘ Will can spot Mary’ be tagged
as-

• Will as a model
• Can as a verb
• Spot as a noun
 • Mary as a noun
Now calculate the probability of this sequence being correct in the following manner.

The probability of the tag Model (M) comes after the tag <S> is ¼ as seen in the table. Also, the
probability that the word Will is a Model is 3/4. In the same manner, we calculate each and every
probability in the graph. Now the product of these probabilities is the likelihood that this sequence is
right. Since the tags are not correct, the product is zero.

1/4*3/4*3/4*0*1*2/9*1/9*4/9*4/9=0

When these words are correctly tagged, we get a probability greater than zero as shown below

Calculating the product of these terms we get,

3/4*1/9*3/9*1/4*3/4*1/4*1*4/9*4/9=0.00025720164

For our example, keeping into consideration just three POS tags we have mentioned, 81 different
combinations of tags can be formed. In this case, calculating the probabilities of all 81 combinations
seems achievable. But when the task is to tag a larger sentence and all the POS tags in the Penn Treebank
project are taken into consideration, the number of possible combinations grows exponentially and this
task seems impossible to achieve. Now let us visualize these 81 combinations as paths and using the
transition and emission probability mark each vertex and edge as shown below.
The next step is to delete all the vertices and edges with probability zero, also the vertices which do not
lead to the endpoint are removed. Also, we will mention-

Now there are only two paths that lead to the end, let us calculate the probability associated with each
path.

<S>→N→M→N→N→<E> =3/4*1/9*3/9*1/4*1/4*2/9*1/9*4/9*4/9=0.00000846754

<S>→N→M→N→V→<E>=3/4*1/9*3/9*1/4*3/4*1/4*1*4/9*4/9=0.00025720164

Clearly, the probability of the second sequence is much higher and hence the HMM is going to tag each
word in the sentence according to this sequence.
 Neural Models
Neural models have revolutionized the field of NLP by leveraging deep learning techniques
to create more sophisticated and accurate language models. These models include Recurrent Neural
Networks (RNNs), Transformer-based models, and large language models.
1. Recurrent Neural Networks
Recurrent Neural Networks (RNNs) are a type of neural network designed for sequential data, making
them well-suited for language modelling. RNNs maintain a hidden state that captures information
about previous inputs, allowing them to consider the context of words in a sequence.
LSTMs and GRUs are advanced RNN variants that address the vanishing gradient problem, enabling
the capture of long-range dependencies in text. LSTMs use a gating mechanism to control the flow
of information, while GRUs simplify the gating mechanism, making them faster to train.
2. Transformer-based Models
The Transformer model,has revolutionized NLP. Unlike RNNs, which process data sequentially, the
Transformer model processes the entire input simultaneously, making it more efficient for parallel
computation.

• Self-Attention Mechanism: This mechanism allows the model to weigh the importance of
different words in a sequence, capturing dependencies regardless of their distance in the text. Each
word's representation is updated based on its relationship with all other words in the sequence.
• Encoder-Decoder Structure: The Transformer consists of an encoder and a decoder. The encoder
processes the input sequence and generates a set of hidden representations. The decoder takes these
representations and generates the output sequence.
• Positional Encoding: Since Transformers do not process the input sequentially, they use
positional encoding to retain information about the order of words in a sequence. This encoding
adds positional information to the input embedding, allowing the model to consider the order of
words.