04 - Language
Modelling + RNNs
Benjamin Akera
13-03-2024
Modern Neural Nets are very Big
Large Neural Nets are the cornerstone of modern NLP
systems
Building these large models is difficult
It took a long time and much
work to make deep neural
networks practical!
We have models with very many parameters: Some
approaches
● Regularization
● Dropout
● Vectorization
● Optimizers
● Parameter initialization
What is Language Modelling
Language Modelling: The task of predicting what word comes next
A system that does this is called a Language Model
What is Language Modelling
Also can be thought of as a system that assigns a probability to a
piece of text
For example, if we have some text: (x1….xn)
We use Language Models everyday
Any other
examples?
We use
Language
Models
everyday
N Gram Language Models
Kampala is the only place where ______
Question: How do we learn a language model?
Answer: (Pre Deep Learning): Learn an n-gram model Language model
Definition: An n-gram is a chunk of n-consecutive words
● Unigrams: “Kampala”, “is”, “the”, ”only”
● Bi-grams: “Kampala is”, “the only”, “place where”
● Tri-grams: “Kampala is the”, “is the only place”
● Four-grams: “Kampala is the only place”
N Gram models in practice
● You can build a simple N Gram model in practice over a 1.7million
word corpus (Reuters) in a few seconds on your laptop
Today the ___
This is assignment 1
Sparsity Problem: with N-Gram language models
Recall this example?
Sparsity Problems: with N-Gram language models
Sparsity Problem 1
What if “students opened their “w” Partial solution: Add small
never occurred in the data? Then w delta to every w in the vocab.
has a probability of 0 This is called smoothing
Sparsity Problem 2
Problem: What if “students Partial solution: Just
opened their” never occurred in the condition an “opened their”
data, then we can’t calculate any instead. This is called backoff
probability of w!
Note: Increasing n makes sparsity problems worse. Typically we cant have
an n bigger than 5
Storage Problems: with N-Gram language models
Sparsity Problem 1
Storage: Need to store count for all
n-grams you saw in the corpus
Note: Increasing n or increasing the corpus increases the model size
N Gram models in practice
You can build a simple trigram Language Model over a 1.7 million word
corpus (Reuters) in a few seconds on your laptop.
Homework 1: Read the medium article below and try to reproduce
Today the ____
Sparsity Problem: Not much
granularity in the probability
Company 0.153 distribution
bank 0.153
price 0.077
Italian 0.039
Emirate 0.039
….
Reproduce this
Generating text with N Gram language Models
While you can use an n-gram language model to generate text, it is
today the price of gold per ton , while production of shoe lasts and shoe industry , the
bank intervened just after it considered and rejected an imf demand to rebuild
depleted european stocks , sept 30 end primary 76 cts a share .
Surprisingly grammatical
…but incoherent. We need to consider more than three words at a time if we want to model language
well. But increasing n worsens sparsity problem, and increases model size
Can we do Better?
Recurrent Neural Networks ( RNNs)
A type of Neural Network designed to work on
sequences.
Problem with vanilla feedforward networks: fixed
size input and output, fixed number of
computational steps.
Solution: Recurrent networks that combine the
input vector with their state vector and update the
state vector in addition to producing output.
Recurrent neural networks instead of operating on
vectors operate on sequences of vectors.
Recurrent Neural Networks - Variants
Vanilla Neural Net
Recurrent Neural Networks - Variants
E.g Image Captioning,
mapping from an image to a
sequence of words
Recurrent Neural Networks - Variants
E.g Sentiment classification, mapping
from a sequence of words to a sentiment
Recurrent Neural Networks - Variants
E.g Machine Translation, mapping from a
sequence of words to a sequence of words
Recurrent Neural Networks - Variants
E.g Video Classification: Frame
Level
Recurrent Neural Networks
We can process a sequence
of vectors x
By applying a recurrence
formula at every time step:
Recurrent Neural Networks
The state consists of a single “hidden” vector h:
Character Level Language Model Example
Vocabulary:
[h, e, l, l, o]
Example training sequence: “hello”
Training: Backpropagation Through Time
Unfolding through time
To train the network we need to unfold it through time and apply backpropagation algorithm.
Training: Backpropagation Through Time
Unfolding through time
To train the network we need to unfold it through time and apply backpropagation algorithm.
Problem: very deep network architecture - vanishing or exploding gradients.
We have to limit the number of steps we unroll the network for.
Homework 2: Read about the vanishing gradients problem in RNNs and write a paragraph about it.
Basic Code: Keras
RNNs and Long Term Dependencies
Vanilla RNNs work well, but struggle with long term dependencies
E.g Predicting the last word in a sentence, generating plausible text
LSTMs ( Long-Short-Term Memory) Networks
Vanilla RNN
LSTM Network
LSTM Architecture
LSTMs include a "memory cell" that holds the
unit's state.
LSTM units have the ability to remove or add
information to the cell state, regulated by
structures called gate
LSTM Architecture: Sigmoid Activation Function
.Gates are a way to optionally let
information through. They are composed
out of a sigmoid neural net layer and a
pointwise multiplication operation.
The sigmoid layer outputs numbers
between zero and one, describing how
much of each component should be let
through.
LSTM Architecture: Forget Gates
The forget gate layer looks at previous
output and input vector and outputs a
vector of values in range (0,1) to decide
which information to delete from cell state.
LSTM Architecture: Hidden States
We multiply the old state by ft, forgetting
the things we decided to forget earlier.
Then we add ~Ct. These are the new
candidate values, scaled by how much we
decided to update each state value.
LSTM Architecture: Output
A sigmoid layer decides which part of the
state vector will we output. We put the cell
state through tanh to squash the values
into (-1,1) range.
LSTM Variants
Many variants of the architecture, but
performance is similar.
A recent and increasingly popular variation on
LSTM architecture is GRU - Gated Recurrent Unit.
GRU combines the forget and input gates into a
single “update gate”, merges the cell state and
hidden state, and makes some other changes
Resources:
1. [Link]
2. [Link]
Assignments
1. Read this medium article below and try to reproduce a word vectors LLM
2. Read about the vanishing gradients problem in RNNs and write a paragraph
about it.
3. Do some research and code up a basic RNN and LSTM in your DL framework of
choice - submit colab notebooks
Recap Next Time
● Limitations of Word Vectors ● Code walk through
● RNNs ● Limitations of RNNs/LSTMs
● LSTMs ● Sequence-Sequence Models
● GRUs ● Encoder-Decoder Architecture
● Attention & Memory