Language Models
From RNN to BERT
Neural LM
Unlimited History: Recurrent Neural Networks
- The RNNs are neural nets that can deal with sequences of
variable length. They are able to this by defining a recurrence
relation over timesteps:
20 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
Neural LM
Unlimited History: Recurrent Neural Networks
- The activation value of the hidden layer depends on the
current input as well as the activation value of the hidden
layer from the previous time step.
- In this way, Recurrent neural language models (RNNLMs)
avoid the limited context constraint inherent in traditional N-
gram models, since the hidden state embodies information
about all of the preceding words all the way back to the
beginning of the sequence.
21 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
Neural LM
Unlimited History: Recurrent Neural Networks
- The activation value of the hidden layer depends on the
current input as well as the activation value of the hidden
layer from the previous time step:
22 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
Neural LM
Unlimited History: Recurrent Neural Networks
23 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
Neural LM
Unlimited History: Recurrent Neural Networks
24 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
RNN variants
Gated RNNs
- Credit assignment issue
- Vanishing gradient issue
25 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
RNN Variants
Gated RNNs
Bi-directional RNNs
26 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
RNN Features
Gated RNNs
Bi-directional RNNs
27 Credit: Speech and Language Processing. Daniel Jurafsky & James H. Martin.
Magic and Reality
Magic: Flexible
- complex networks can be treated as modules that can
be combined in creative ways.
- Naturally suited for variable-length sequence processing
- Theoretically handling long-term dependencies
Reality: hard to train
- Long-range dependencies still tricky, despite gating
- Limited contextual information are captured
- Loss of hierarchy
- Sequentiality prohibits parallelization within instances
- Training is slow
28
Transformers
Attention is All You Need
- Transformer is a pure attention-based model consisting of:
- The encoding component is a stack of encoders.
- The decoding component is a stack of decoders of the
same number.
Transformer Block
- The encoders are identical Transformer Block.
- The encoder architecture is also used in BERT.
39
Transformers
Attention is All You Need
- Transformer is a pure attention-based model consisting of:
- The encoding component is a stack of encoders.
- The decoding component is a stack of decoders of the
same number.
Transformer Block
- The encoders are identical Transformer Block.
- The encoder architecture is also used in BERT.
40
Transformer Block
41 Credit: The Illustrated Transformer. Jay Alammar.
Transformer Block
42 Credit: The Illustrated Transformer. Jay Alammar.
Transformer Block
43 Credit: The Illustrated Transformer. Jay Alammar.
Transformer Block
Attention is All You Need
- Attention: indexing/referring contents using similarity
- e.g. Encoder-Decoder Attention
44
Transformer Block
Attention is All You Need
- Attention: indexing/referring contents using similarity
- Self-Attention
45
Transformer Block
Attention is All You Need
- Attention: indexing/referring contents using similarity
- Convolution v.s. Self-Attention
46
Transformer Block
Attention is All You Need
- Attention: indexing/referring contents using similarity
- Self-Attention (similar with convolution)
- Constant path length between any two positions
- Variable-sized perceptive field
- Gating/multiplication enables crisp error propagation
- Trivial to parallelize (per layer)
- Long-distance context has “equal opportunity”
- Self-attention is the method the Transformer uses to bake
the “understanding” of other relevant words into the one
we’re currently processing.
47
BERT
Bidirectional Encoder Representations from Transformers:
BERT
- BERT ≈ Encoder of Transformer
Combining the Two Worlds:
- BERT ≈ ELMo + Transformer
- ELMo’s (Embeddings from Language Models) language
model was bi-directional, but based on (slow) RNNs
- Transformer purely relies on (parallelized) attention,
but only trains a forward language model.
- Why: Language models only use left context or right
context, but language understanding is bidirectional.
65
BERT
Handling the bi-directionality:
- Transformer’s encoder is only a forward language model
- If they were to perform bidirectional self-attention, then the
model would learn that the next word in the sentence is the
target and would predict it always, with 100% accuracy.
66
BERT
Handling the bi-directionality:
- Transformer’s encoder is only a forward language model
- If they were to perform bidirectional self-attention, then
the model would learn that the next word in the sentence is
the target and would predict it always, with 100%
accuracy.
- BERT solves this problem using two “new” paradigms:
- Masked Language Model
- Next Sentence Prediction (Two-Sentence Tasks)
67
Masked Language Model
Predict random words from within the sequence
- Transformer’s encoder is only a
forward language model
- If they were to perform
bidirectional self-attention,
then the model would learn
that the next word in the
sentence is the target and
would predict it always, with
100% accuracy.
- BERT solves this problem using
two “new” paradigms:
- Masked Language Model
- Two-sentence Tasks
68 Credit: The Illustrated BERT. Jay Alammar.
Masked Language Model
Predict random words from within the sequence
- In this specific case, 15% of the words that were fed in
as input were masked.
- Too little masking: Too expensive to train
- Too much masking: Not enough context
69 Credit: The Illustrated BERT. Jay Alammar.
Masked Language Model
Predict random words from within the sequence
- In this specific case, 15% of the words that were fed in
as input were masked.
- But not all tokens were masked in the same way:
- 80% of the time, replace with [MASK]
- went to the store → went to the [MASK]
- 10% of the time, replace random word
- went to the store → went to the running
- 10% of the time, keep same
- went to the store → went to the store
70 Credit: The Illustrated BERT. Jay Alammar.
Masked Language Model
Predict random words from within the sequence
- In this specific case, 15% of the words that were fed in
as input were masked.
- But not all tokens were masked in the same way since:
- If the model had been trained on only predicting
‘<MASK>’ tokens and then never saw this token during
fine-tuning, it would have thought that there was no need
to predict anything.
- The model would have only learned a contextual
representation of the ‘<MASK>’ token and this would
have made it learn slowly (since only 15% of the input
tokens are masked).
71 Credit: The Illustrated BERT. Jay Alammar.
Next Sentence Prediction
To learn relationships between sentences
- predict whether Sentence B is actual sentence that
proceeds Sentence A, or a random one
- BERT ≈ Encoder of Transformer with Input Specifics
- [CLS]
- [SEP]
7 Credit: The Illustrated BERT. Jay Alammar.
Next Sentence Prediction
Predict random words from within the sequence
- Transformer’s encoder is only a forward language model
- If they were to perform bidirectional self-attention, then the
model would learn that the next word in the sentence is the
target and would predict it always, with 100% accuracy.
- BERT solves this problem using two “new” paradigms:
- Masked Language Model
- Two-sentence Tasks
73 Credit: The Illustrated BERT. Jay Alammar.
7
Two-Stage Fine-Tuning
Fine-tuning for downstream tasks
- The BERT paper shows a number of ways to use BERT for
different tasks.
74