Diving into Fine-Tuning

through BERT
Siksha ‘O’ Anushandhan, ITER
Center for Data Science
Table of Contents

1 BERT Architecture

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 2 / 15
 Scaling Up Transformers: Architecture Comparison
BERTlarge
LARGEST MODEL

BERTbase
24
Transformer 12
encoder layers
encoder layers

6
encoder layers
dmodel = 768 dmodel = 1024
dmodel = 512 A = 12 heads A = 16 heads
A = 8 heads 110M params 340M params
Baseline model
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 3 / 15
BERT Architecture

Architecture
The BERT model only has encoder layers and no decoder stack
a multi-head self-attention mechanism that allows each token to learn to
understand all the surrounding tokens (masked or not) [Bidirectional method].
This enables BERT to understand the context on both sides of each token.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 4 / 15
BERT Architecture

Architecture
The BERT model only has encoder layers and no decoder stack
a multi-head self-attention mechanism that allows each token to learn to
understand all the surrounding tokens (masked or not) [Bidirectional method].
This enables BERT to understand the context on both sides of each token.

Training method

The model was trained with two parts.

Masked language modeling (MLM)
Next-Sentence Prediction (NSP)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 4 / 15
Masked Language Modeling?

Masked Language Modeling (MLM) is

a self-supervised learning technique
Goal: Predict missing (masked) words
in a sentence
Helps model understand context from
both directions

Basic idea of MLM

Some words in a sentence are replaced
with [MASK]
Model tries to predict the original word
Uses both left and right context

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 5 / 15
Masked Language Modeling?

Masked Language Modeling (MLM) is Example

a self-supervised learning technique input sentence: “the cat sat on the mat”
Goal: Predict missing (masked) words masked sentence: “the cat [MASK] on the
in a sentence mat”
prediction: “sat”
Helps model understand context from
both directions
How MLM Works
Basic idea of MLM Select 15% of tokens randomly
Some words in a sentence are replaced Apply masking strategy:
with [MASK] 80% → Replace with [MASK]
10% → Replace with random word
Model tries to predict the original word 10% → Keep unchanged
Uses both left and right context Model predicts original tokens

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 5 / 15
How MLM Works
Example
sentence: Natural language processing is a field of artificial intelligence that focuses on the
interaction between computers and humans through language. It involves tasks such as text
classification, machine translation, question answering, and sentiment analysis. Modern
models like BERT have significantly improved the performance of many NLP applications by
learning deep contextual representations.
Total words : 60, 15% of 9 ≈ 9 words

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 6 / 15
How MLM Works
Example
sentence: Natural language processing is a field of artificial intelligence that focuses on the
interaction between computers and humans through language. It involves tasks such as text
classification, machine translation, question answering, and sentiment analysis. Modern
models like BERT have significantly improved the performance of many NLP applications by
learning deep contextual representations.
Total words : 60, 15% of 9 ≈ 9 words

Suppose selected words: processing, artificial, interaction, computers, classification,

translation, BERT, performance, contextual

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 6 / 15
How MLM Works
Example
sentence: Natural language processing is a field of artificial intelligence that focuses on the
interaction between computers and humans through language. It involves tasks such as text
classification, machine translation, question answering, and sentiment analysis. Modern
models like BERT have significantly improved the performance of many NLP applications by
learning deep contextual representations.
Total words : 60, 15% of 9 ≈ 9 words

Suppose selected words: processing, artificial, interaction, computers, classification,

translation, BERT, performance, contextual
Modified input: Natural language [MASK] is a field of [MASK] intelligence that focuses on
the [MASK] between [MASK] and humans through language. It involves tasks such as text
[MASK], machine [MASK], question answering, and sentiment analysis. Modern models like
BERT have significantly improved the city of many NLP applications by learning deep
[MASK] representations.
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 6 / 15
The Tokenization Problem

? How should we split "unaffable" into pieces a model can understand?

Option A Word-level Option B Char-level Option C Subword ✓

"unaffable" → [u] [n] [a] [f] [f] [un] [##aff] [##able]

[unaffable] [a] [b] [l] [e]
✓3 tokens, meaningful
✓Simple ✓No OOV ✓No OOV
✗ OOV! Model never saw it ✗ 9 tokens loses meaning ✓Best of both!
✗ No morphology ✗ Very slow to train

BERT uses Option C–WordPiece tokenization with a vocabulary of ∼30,000 subword units.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 7 / 15
WordPiece: How It Learns to Merge

Step 1 Start with characters Step 3 Merge the best pair

Vocabulary = every character in corpus If "un" has the highest score ⇒ add "un"
e.g. a, b, c, ..., z, á, ü, ... to vocab.
Remove "u"+"n" entries from all training
words.
Step 2 Score every adjacent pair

P(AB) Step 4 Repeat until vocab size V

score(A, B) =
P(A) · P(B)
High score ⇒ pair appears together more BERT stops at V = 30,000 tokens.
than by chance Result: frequent words stay whole,
rare words get split into subwords.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 8 / 15
Worked Example: Tokenizing "unhappiness"
Walk-through

Watch how BERT’s WordPiece tokenizer splits a word it has never seen whole:

Step 1 – Look up whole word unhappiness ← Not in vocab → split

Step 2 – Try known prefix un happiness ← "un" IS in vocab!

Step 3 – Split remainder un happy ness ← "happy" in vocab, split rest

Step 4 – Final (BERT) un ##happy ##ness ← ## = continuation of

word

The ## prefix signals the token is NOT the start of a word. "happy" ̸= "##happy"!

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 9 / 15
Class Exercise: Match the Tokenization!

Exercise:
Guess BERT token for the word: tokenization, COVID-19, GPT, running, transformer,
[MASK]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 10 / 15
Class Exercise: Match the Tokenization!

Exercise:
Guess BERT token for the word: tokenization, COVID-19, GPT, running, transformer,
[MASK]

Word BERT Tokens Why?

tokenization token ##ization "token" common; "##ization" is a known

suffix
COVID-19 CO ##VID - 19 Rare compound → broken to pieces
GPT G ##P ##T Acronyms often char-split if unseen
running run ##ing Common word → kept whole
transformer transform ##er Root transform in vocab + suffix
[MASK] [MASK] Special tokens added explicitly to vocab

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 10 / 15
WordPiece Example (Step-by-Step)

Sentence: playing played

Step 1: Character Tokens

playing → [p] [l] [a] [y] [i] [n]
[g]
played → [p] [l] [a] [y] [e] [d]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 11 / 15
WordPiece Example (Step-by-Step)

Sentence: playing played

Step 1: Character Tokens

playing → [p] [l] [a] [y] [i] [n]
[g]
played → [p] [l] [a] [y] [e] [d]
Step 2: Frequencies
Characters: p = 2, l = 2, a = 2, y =
2, i = 1, n = 1, g = 1, e = 1, d = 1
Pairs: pl = 2, la = 2, ay = 2, in =
1, ng = 1, ed = 1

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 11 / 15
WordPiece Example (Step-by-Step)

Sentence: playing played Step 4: Key Scores

1
(i, n) = 1×1 = 1.0
Step 1: Character Tokens
1
playing → [p] [l] [a] [y] [i] [n] (n, g ) = 1×1 = 1.0
2
[g] (p, l) = 2×2 = 0.5
played → [p] [l] [a] [y] [e] [d]
Step 5: Merging
Step 2: Frequencies i + n → in
Characters: p = 2, l = 2, a = 2, y = in + g → ing
2, i = 1, n = 1, g = 1, e = 1, d = 1
p + l + a + y → play
Pairs: pl = 2, la = 2, ay = 2, in =
e + d → ed
1, ng = 1, ed = 1
Final Tokens
Step 3: Scoring Formula
h i playing → [play, ##ing]
freq(xy ) P(xy )
Score(x, y ) = freq(x)×freq(y ) or P(x)×P(y )
played → [play, ##ed]
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 11 / 15
BERT’s Choice: Learned Positional Embedding
BERT vs Transformer

Original Transformer Sinusoidal BERT Learned

Fixed formula no parameters Trainable matrix Epos ∈ R512×768

Works for any sequence length Max sequence length = 512 tokens
All models share the same PE PE is learned from pre-training data
Less flexible cannot adapt to data Slightly better NLP performance

BERT chooses learned PE because the 512-token limit is acceptable for NLP tasks.
Flexibility > generalization for downstream NLP.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 12 / 15
What is Next-Sentence Prediction?
BERT Pre-training Task 2

NSP: Given two sentences A and B, predict whether B is the ACTUAL next sentence after A.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 13 / 15
What is Next-Sentence Prediction?
BERT Pre-training Task 2

NSP: Given two sentences A and B, predict whether B is the ACTUAL next sentence after A.

Text processing
50% of the training pair is actual.
50% of the training pair, second
sentence is random.
[CLS] is a binary classification token
added to the beginning of the first
sequence.
[SEP] is a separation token that
signals the end of a sequence.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 13 / 15
What is Next-Sentence Prediction?
BERT Pre-training Task 2

NSP: Given two sentences A and B, predict whether B is the ACTUAL next sentence after A.

Text processing IsNext ✓

[CLS] The man went to the store. [SEP]
50% of the training pair is actual.
He bought a gallon of milk. [SEP]
50% of the training pair, second → B follows A in the original text.
sentence is random. Label: IsNext = TRUE
[CLS] is a binary classification token
added to the beginning of the first NotNext ✗
sequence. [CLS] The man went to the store. [SEP]
Penguins are flightless birds. [SEP]
[SEP] is a separation token that
→ B is a random sentence – no connection.
signals the end of a sequence.
Label: IsNext = FALSE
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 13 / 15
NSP Worked Example: Full Input Construction
Step-by-step

Input sentences:
A: "I love machine learning." B: "It is a fascinating field."

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 14 / 15
NSP Worked Example: Full Input Construction
Step-by-step

Input sentences:
A: "I love machine learning." B: "It is a fascinating field."
Constructed BERT input:
[CLS] I love machine learn ##ing . [SEP] It is a fascin ##ating
field . [SEP]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 14 / 15
NSP Worked Example: Full Input Construction
Step-by-step

Input sentences:
A: "I love machine learning." B: "It is a fascinating field."
Constructed BERT input:
[CLS] I love machine learn ##ing . [SEP] It is a fascin ##ating
field . [SEP]

tok: E[CLS] EI Elove Emachine Elearn E##ing E. E[SEP] EIt Eis Ea Efascin E##ating Efield E. E[SEP]
Seg: EA EA EA EA EA EA EA EA EB EB EB EB EB EB EB EB
Pos: E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 14 / 15
NSP Worked Example: Full Input Construction
Step-by-step

Input sentences:
A: "I love machine learning." B: "It is a fascinating field."
Constructed BERT input:
[CLS] I love machine learn ##ing . [SEP] It is a fascin ##ating
field . [SEP]

tok: E[CLS] EI Elove Emachine Elearn E##ing E. E[SEP] EIt Eis Ea Efascin E##ating Efield E. E[SEP]
Seg: EA EA EA EA EA EA EA EA EB EB EB EB EB EB EB EB
Pos: E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15

vi = Etoki + Esegi + Eposi → BERT layers → IsNext / NotNext

Notice: "learning" → [learn, ##ing] WordPiece in action inside the NSP example!
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 14 / 15
Next-sentence prediction (NSP)

Continue to code...

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 15 / 15