Language Modelling
ITS69204 Computer Vision and NLP
ITS69204
Topic 3
Language Modelling
Outline
3.1 What is Language Modelling (LM)
3.2 How does Language Model Works
3.3 Types of Language Models
3.3.1 Statistical Language Models
3.3.2 Neural Language Models
3.4 Language Model Evaluation
3.5 Challenges with Language Model
3.6 Common Examples of Language Model
2
ITS69204
Topic 3
Language Modelling
Dictionary
Corpus - Body of text, singular. Corpora is the plural of this. Example: A collection
of medical journals.
Token- Each "entity" that is a part of whatever was split up based on rules. For
examples, each word is a token when a sentence is "tokenized" into words. Each
sentence can also be a token, if you tokenized the sentences out of a paragraph.
Lexicon - Words and their meanings. Example: English dictionary.
However, various fields will have different lexicons.
Example: To a financial investor, the first meaning for the word "Bull" is someone who is confident about the
market, as compared to the common English lexicon, where the first meaning for the word "Bull" is an
animal. As such, there is a special lexicon for financial investors, doctors, children, mechanics, and so on.
3
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM)
• Language Model learns to predict the probability of a sequence of words.
Why learn
probability of words?
4
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM)
• Example of popular NLP application, Machine Translation.
• Machine translation takes a bunch of words from a language and convert these
words into another language.
• There can be many potential translations that a system might return.
• Thus, need to compute the probability of each of these translations to
understand which one is the most accurate.
5
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM)
• Word ordering:
p(the elephant is big) > p(big the is elephant)
• Probability of the first sentence is bigger than the second sentence.
• The ability to model the rules of a language as a probability gives great power
for NLP related tasks:
• Machine Translation
• Spell Correction
• Speech Recognition
• Sentiment analysis (etc.)
• Each of those tasks require use of language model.
6
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM) (Implementation)
1) Machine Translation
• Example: translating a sentence related to height. It would probably state
P(tall man) > P (large man)
• Why? 👉 the ʻlargeʼ might also refer to weight or general appearance thus, not as
probable as ʻtallʼ.
7
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM) (Implementation)
2) Spelling Correction
• Spell corrected sentence
• Example: Put your name into form
P(name into form) > P (name into from)
8
ITS69204
Topic 3
Language Modelling
3.1 What is Language Modelling (LM)
3) Speech Recognition
• Example: Call my nurse
P(call my nurse) > P (call miners)
• Example: I have no idea
P(no idea) > P (no eye deer)
9
ITS69204
Topic 3
Language Modelling
3.2 How Does Language Model Works?
Steps:
1. Determine probability of the next word by analyzing the text in data (interpret the
data by feeding it through algorithms. The algorithms are responsible for creating rules for the
context in natural language).
2. The models then are prepared for prediction of words by learning the features
and characteristics of a language (the model prepares itself for understanding phrases and
predict the next words in sentences)
3. For training a language model, a number of probabilistic approaches are used
(approaches vary depends on its purpose. E.g.: Search Query prediction vs. Google Docs prediction. The
approach followed to train the model would be unique in both cases.)
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model
Generally there are 2 types of Language Model:
1. Statistical Model - use traditional statistical techniques like N-grams, Hidden
Markov Models (HMM) and certain linguistic rules to learn the probability
distribution of words.
2. Neural Language Model - These are new players in the NLP town and have
surpassed the statistical language models in their effectiveness. They use
different kinds of Neural Networks to model language
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
How do N-Gram Language Models work?
within any
Predicts the of a given sequence
probability N-gram of the
words
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
How do N-Gram Language Models work?
P(w|h) What’s the probability of seeing the word w,
given a history of previous words h,
where the history contain n-1 words
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
How do N-Gram Language Models work?
P(w|h) What’s the probability of seeing the word w,
given a history of previous words h,
where the history contain n-1 words
Two ways to estimate the probability:
1. Apply chain-rule of probability
2. then apply strong simplification assumption to compute P(w1…ws)
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
Computing Probability of Bi Gram (example)
Training corpus:
what’s the probability of:
<s> I am a human </s> I
<s> I am not a flower </s> I am
<s> I I study in Taylors </s> I s t u dy ?
👉
👉
👉
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
Computing Probability of 4-Grams (example)
Training Toy corpus:
This is the house that Jack built. what’s the probability of:
This is the malt
That lay in the house that Jack built.
this is the house ?
This is the rat,
That ate the malt
That lay in the house that Jack built.
This is the cat,
That killed the rat,
That ate the malt
That lay in the house that Jack built.
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Statistical Model (N-Gram)
What’s chain-rule?
It tells us how to compute joint probability of a sequence using conditional
probability of previous words.
P(w1…ws) = P(w1).P(w2|w1). P(w3|w1 w2). P(w3|w1 w2 w3)… P(wn|w1… wn-1).
But, we don’t know the conditional probabilities for complex conditions (up to n-1 words).
So we use, simplification assumption:
P(wk|w1… wk-1) = P(wk|wk-1)
where we approximate history of the word (wk), by looking only at last word of the content
(a.k.a Markov assumption)
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Neural Language Model (NLM)
• Recently, the use of neural networks in the development of language models has become
very popular, to the point that it may now be the preferred approach.
• The use of neural networks in language modeling is often called Neural Language
Modeling, or NLM for short.
• Neural network approaches are achieving better results than classical methods both on
standalone language models and when models are incorporated into larger models on
challenging tasks like speech recognition and machine translation.
A key reason for the leaps in improved performance may be the method’s ability
to generalize.
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Neural Language Model (NLM)
Nonlinear neural network models solve some of the shortcomings
of traditional language models:
allow conditioning on increasingly large context sizes with
only a linear increase in the number of parameters,
alleviate the need for manually designing backoff orders,
support generalization across different contexts.
Neural Network Methods in Natural Language Processing, 2017 (pp.109).
Backoff: go back to a n-1 gram level to calculate the probabilities when encounter a word with prob=0.
1
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Neural Language Model (NLM)
• Specifically, a word embedding is adopted that uses a real-valued vector to represent each
word in a project vector space.
• This learned representation of words based on their usage allows words with a similar
meaning to have a similar representation.
Neural Language Models (NLM) address the n-gram data sparsity issue through
parameterization of words as vectors (word embeddings) and using them as
inputs to a neural network.
The parameters are learned as part of the training process. Word embeddings
obtained through NLMs exhibit the property whereby semantically close words
are likewise close in the induced vector space.
Character-Aware Neural Language Model, 2015
2
ITS69204
Topic 3
Language Modelling
3.3 Types of Language Model – Neural Language Model (NLM)
• Specifically, a word embedding is adopted that uses a real-valued vector to represent each
word in a project vector space.
• This learned representation of words based on their usage allows words with a similar
meaning to have a similar representation.
2
ITS69204
Topic 3
Language Modelling
3.4 Language Model Evaluation
• However, the extrinsic evaluation approach requires multiple tests on models
which are expensive.
• An alternative is the intrinsic evaluation, which is about testing the LM itself not
on some particular task or application.
• The popular intrinsic evaluation is perplexity.
• As perplexity is a bad approximation to an extreme extrinsic evaluation ( in cases
where the test dataset does NOT look just like the training set) .
• Thus, it is useful only at the early stages of experiment. So later in experiment
extrinsic evaluation should also be used.
Best model is the one that best predicts an unseen test set, or assigns on
average the probability to all sentences that is sees.
2
ITS69204
Topic 3
Language Modelling
3.5 Challenges with Language Model
• Formal languages (like a programming language) are precisely defined (all the
words and their usage is predefined in the system)
• Anyone who knows a specific programming language can understand what’s
written without any formal specification.
• However, Natural language isn’t designed; it evolves according to the
convenience and learning of an individual.
• There are several terms in natural language that can be used in a number of
ways, hence cause ambiguity but can still be understood by humans.
2
ITS69204
Topic 3
Language Modelling
3.5 Challenges with Language Model
• Machines only understand the language of numbers.
• For creating language models, it is necessary to convert all the words into a
sequence of numbers (a.k.a encodings)
• Encodings can be simple or complex.
• Generally, a number is assigned to every word and this is called label-encoding.
• Example: In a sentence
“I love to play cricket on weekends”
every word is assigned a number [1, 2, 3, 4, 5, 6,7]. This is a.k.a one-hot-encoding.
2
ITS69204
Topic 3
Language Modelling
3.6 Common Examples of Language Model
• Language models are the cornerstone of Natural Language Processing (NLP)
technology. We have been making the best of language models in our routine,
without even realizing it.
• Some of the examples of language models:
1. Speech Recognization
Voice assistants such as Siri and Alexa are examples of how language
models help machines in processing speech audio.
2. Machine Translation
Google Translator and Microsoft Translate are examples of how NLP
models can help in translating one language to another.
2
ITS69204
Topic 3
Language Modelling
3.6 Common Examples of Language Model
• Some of the examples of language models:
3. Sentiment Analysis
Example: to allow businesses to understand a customer’s intent behind
opinions or attitudes expressed in the text. Hubspot’s Service Hub is an
example of how language models can help in sentiment analysis.
4. Text Suggestions
Google services such as Gmail or Google Docs use language models to help
users get text suggestions while they compose an email or create long text
documents, respectively.
2
ITS69204
Topic 3
Language Modelling
3.6 Common Examples of Language Model
• Some of the examples of language models:
5. Parsing Tools
Parsing involves analyzing sentences or words that comply with syntax or
grammar rules. Spell checking tools are perfect examples of language
modelling and parsing.
2
ITS69204
Topic 3
Language Modelling
Thank You