Scribd
Upload
4 views15 pages

Understanding N-gram Models and Smoothing

The document discusses simple N-gram models, which are sequences of words used to estimate the probability of a word based on its preceding words. It covers concepts like perplexity for evaluating language models and Laplace smoothing to handle unseen contexts in word sequences. The document also highlights the importance of adjusting probabilities to avoid underflow issues during calculations.

Uploaded by

sowmithreddy.t2022
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
4 views15 pages

Understanding N-gram Models and Smoothing

The document discusses simple N-gram models, which are sequences of words used to estimate the probability of a word based on its preceding words. It covers concepts like perplexity for evaluating language models and Laplace smoothing to handle unseen contexts in word sequences. The document also highlights the importance of adjusting probabilities to avoid underflow issues during calculations.

Uploaded by

sowmithreddy.t2022
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd

Simple N-gram models

An n-gram is a sequence of n words:


• 2-gram(bigram): is a two-word sequence of words like “please turn”, “turn
your”, or ”your homework”
• 3-gram (trigram) is a three-word sequence of words like “please turn your”, or
“turn your homework”.
• probabilistic model that can estimate the probability of a word given the n-1
previous words, and thereby also to assign probabilities to entire sequences.
Other terms in numerator and denominator
cancel out bcuz they are equal

Other terms in numerator and denominator


cancel out bcuz they are equal
Whenever we multiply,
underflow problem
occurs. We can overcome
this problem by adding
log of all probabilities.

=log(3/7)+log(3/6)+log(3/5)+log(3/3) = -2.0513

=log(3/7)+log(2/4)+log(3/6)+log(2/5)+log(3/5) = -3.6607
ESTIMATING PARAMETERS AND EVALUATING LANGUDAGE MODELS

• Perplexity: the inverse probability of the test set, normalized by the number of words.
• Lower perplexity -> better model, higher perplexity -> more confused for prediction
LAPLACE SMOOTHING
• What do we do with words that are in our vocabulary (they are not unknown words) but
appear in a test set in an unseen context (for example they appear after a word they never
appeared after in training)?
• Smoothing is the process of flattening a probability distribution implied by a language model
so that all reasonable word sequences can occur with some probability. This often involves
broadening the distribution by redistributing weight from high probability regions to zero
probability regions.
• The simplest way to do smoothing is to add one to all the bigram counts, before we
normalize them into probabilities. All the counts that used to be zero will now have a count of
1, the counts of 1 will be 2, and so on.
• V = (no. of unique words in corpus-1) and N = total no. of times w is occurring in corpus
With Laplace smoothing
How much a smoothing algorithm has changed the
original counts?

You might also like