Getting Started with the

Architecture
of the Transformer Model
Siksha ‘O’ Anushandhan, ITER
Center for Data Science
Table of Contents

5 Positional Encoding
1 Introduction: The Transformer Revolution
6 Multi-Head Attention
2 Transformer Architecture Overview 7 Post-layer normalization
3 The Encoder Stack 8 Feedforward network
4 Input Embedding 9 The Decoder stack

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 2 / 33
The Language Revolution

The Digital Age of Language

Language is the essence of human communication. Our daily lives now rely on digitalized NLP
functions:

Web search engines Web page generation

Email systems Speech-to-text transcripts
Social networks & posts Text-to-speech services
Smartphone texting Chatbots & assistants
Real-time translations And many more...

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 3 / 33
The Birth of Transformers

December 2017

Google Brain & Google Research published

”Attention Is All You Need” by Vaswani et al.

The Transformer Outperformed

was born existing NLP models

Key Achievement
The Transformer trained faster and obtained higher evaluation results than previous
architectures
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 4 / 33
From RNNs to Transformers

The Old Way: RNNs The New Way: Transformers

Sequential processing Parallel processing
Long-distance dependency problems Direct word-to-word attention
Slow training Fast training
Limited parallelization Fully parallelizable

Attention replaced recurrence!

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 5 / 33
Modern Transformer Models

GPT-4
OpenAI

PaLM
Google Large Language
Original
Models (LLMs)
Transformer
The beginning
(2017)
LaMBDA of a new era!
Google

ChatGPT
OpenAI

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 6 / 33
Transformer Architecture

The Structure
6-layer encoder stack Output
6-layer decoder stack
Each layer contains

ENCODER

DECODER
sublayers
Output of layer ℓ = Input
of layer ℓ + 1

No RNN, LSTM, or CNN!

Recurrence has been Input
completely abandoned

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 7 / 33
Attention mechanism

Q (Query) K, V (Keys, Values)

Word-to-Word Operations the the 0.05

The attention mechanism determines

how each word relates to all other words cat cat 0.68

in a sequence
sat sat 0.15

Example: “The cat sat on the mat.”

on on 0.06

The cat sat on the mat.

the the 0.04
Attention calculates relationships
between “cat” and all words mat mat 0.02

Attention weights show how strongly ”cat” focuses on each word

Higher values = stronger connections
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 8 / 33
Encoder layer structure

Structural Consistency
All N=6 (can be 12, 24, etc) layers identical in
structure
Different learned weights per layer
Each layer explores different word associations

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 9 / 33
Encoder layer structure

Input Embedding
The input embedding sublayer converts the input tokens to vectors of
dimension dmodel = 512 using learned embeddings in the original
Transformer model.

A tokenizer will transform a sentence into tokens.

Each tokenizer has its methods, such as Byte Pair Encoding
(BPE), word piece, and sentence piece methods.
The Transformer initially used BPE, but other models use other
methods.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 10 / 33
Example of embedding

Text : the Transformer is an innovative NLP model!

Tokenized text : [’the’, ’transform’, ’er’, ’is’, ’an’, ’innovative’, ’n’, ’l’, ’p’, ’model’, ’ !’]

Text : The cat slept on the [Link] was too tired to get up.
Tokenized text : [1996, 4937, 7771, 2006, 1996, 6411, 1012, 2009, 2001, 2205, 5458,
2000, 2131, 2039, 1012]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 11 / 33
Example of embedding

Text : the Transformer is an innovative NLP model!

Tokenized text : [’the’, ’transform’, ’er’, ’is’, ’an’, ’innovative’, ’n’, ’l’, ’p’, ’model’, ’ !’]

Text : The cat slept on the [Link] was too tired to get up.
Tokenized text : [1996, 4937, 7771, 2006, 1996, 6411, 1012, 2009, 2001, 2205, 5458,
2000, 2131, 2039, 1012]

There is not enough information in the tokenized text at this point to go further.
Need to apply some embedding method.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 11 / 33
Word embedding

skip-gram architecture of the word2vec

A skip-gram will focus on a center word in a window of words and predict context words. For
example, if word(i) is the center word in a two-step window, a skip-gram model will analyze
word(i-2), word(i-1), word(i+1), and word(i+2). Then, the window will slide and repeat the
process.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 12 / 33
Word embedding

skip-gram architecture of the word2vec

A skip-gram will focus on a center word in a window of words and predict context words. For
example, if word(i) is the center word in a two-step window, a skip-gram model will analyze
word(i-2), word(i-1), word(i+1), and word(i+2). Then, the window will slide and repeat the
process.

Suppose we need to perform embedding for the following sentence:

The black cat sat on the couch and the brown dog slept on the rug.
The word embedding vector of black and brown should be similar.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 12 / 33
Word embedding

skip-gram architecture of the word2vec

A skip-gram will focus on a center word in a window of words and predict context words. For
example, if word(i) is the center word in a two-step window, a skip-gram model will analyze
word(i-2), word(i-1), word(i+1), and word(i+2). Then, the window will slide and repeat the
process.

Suppose we need to perform embedding for the following sentence:

The black cat sat on the couch and the brown dog slept on the rug.
The word embedding vector of black and brown should be similar.

black = [[−0.01206071 0.11632373 ... − 0.03184025 − 0.04273562]]

brown = [[1.35794589e − 02 − 2.18823571e − 02 ... 9.73121971e − 02
− 4.90022525e − 02]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 12 / 33
Word embedding

skip-gram architecture of the word2vec

A skip-gram will focus on a center word in a window of words and predict context words. For
example, if word(i) is the center word in a two-step window, a skip-gram model will analyze
word(i-2), word(i-1), word(i+1), and word(i+2). Then, the window will slide and repeat the
process.

Suppose we need to perform embedding for the following sentence:

The black cat sat on the couch and the brown dog slept on the rug.
The word embedding vector of black and brown should be similar.

black = [[−0.01206071 0.11632373 ... − 0.03184025 − 0.04273562]]

brown = [[1.35794589e − 02 − 2.18823571e − 02 ... 9.73121971e − 02
− 4.90022525e − 02]]

cosine similarity(black, brown) = [[0.01088095]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 12 / 33
Word embedding

1 ! pip install gensim # Version Gensim 4.0.0 and above

2 import torch
3 import nltk
4 nltk . download ( ’ punkt_tab ’)

1 import nltk
2 from nltk . tokenize import sent_tokenize , word_tokenize
3 s = " The black cat sat on the couch and the brown dog slept on the
rug . "
4 data = [[ word . lower () for word in word_tokenize ( sentence ) ] for
sentence in sent_tokenize ( s ) ]
5 print ( data )

[[ ’ the ’ , ’ black ’ , ’cat ’ , ’sat ’ , ’on ’ , ’the ’ , ’ couch ’ , ’and ’ , ’the ’ ,

’ brown ’ , ’dog ’ , ’ slept ’ , ’on ’ , ’the ’ , ’rug ’ , ’. ’]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 13 / 33
Skip Gram model
1 # Creating Skip Gram model
2 model = gensim . models . Word 2 Vec ( data , min_count = 1 , vector_size = 5
1 2 , window = 5 , sg = 1 )

Parameter Description
data This is the corpus (training data) for the Word2Vec model. It should be an iterable of
lists of words. Each inner list represents a sentence or a document, and each element in
the inner list is a word (string). In our case, data is [[’the’, ’black’, ’cat’, ...,
’rug’, ’.’]].
min count Ignores all words with total frequency lower than this.
vector size Dimensionality of the word vectors.
window Maximum distance between the current and predicted word within a sentence.
sg {0, 1}, Training algorithm: 1 for skip-gram; otherwis Continuous Bag of
Words(CBOW).

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 14 / 33
Skip Gram model

Word vector embedding

1 word 1 = ’ black ’
2 word 2 = ’ brown ’
3 a = model . wv [ word 1 ]
4 b = model . wv [ word 2 ]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 15 / 33
Skip Gram model

Word vector embedding

1 word 1 = ’ black ’
2 word 2 = ’ brown ’
3 a = model . wv [ word 1 ]
4 b = model . wv [ word 2 ]

Word cosine similarity

1 # compute cosine similarity
2 cos_lib = cosine_similarity ( a . reshape ( 1 ,-1 ) , b . reshape ( 1 ,-1 ) )
3 print ( f " Cosine similarity : { cos_lib } " )

Cosine similarity : [[0.01088095]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 15 / 33
Positional Encoding

The word black is in position 2, and the word brown is in position 10

Our problem is to find a way to add a value to the word embedding

of each word so that it has that information.

Positional Encoding (PE)

 
pos
PE (pos, 2i) = sin 2i

 10000
dmodel

pos
PE (pos, 2i + 1) = cos 2i
10000 dmodel

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 16 / 33
Final Input

Input x1 = black x2 = brown

Embedding vector y1 = black y2 = brown

+ +
Positional encoding pe(2) pe(10)
= =
Final Input pc(black) pc(brown)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 17 / 33
positional encoding
Define function to get positional encoding vector

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 18 / 33
positional encoding
Define function to get positional encoding vector
1 import numpy as np
2 def positional_encoding ( pos , d_model ) :
3 pe = np . zeros (( 1 , d_model ) )
4 for i in range ( 0 , d_model , 2 ) :
5 pe [ 0 ][ i ] = np . sin ( pos / ( 1 0 0 0 0 ** (( 2 * i ) / d_model ) ) )
6 if i + 1 < d_model :
7 pe [ 0 ][ i + 1 ] = np . cos ( pos / ( 1 0 0 0 0 ** (( 2 * i ) /
d_model ) ) )
8 return pe

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 18 / 33
positional encoding
Define function to get positional encoding vector
1 import numpy as np
2 def positional_encoding ( pos , d_model ) :
3 pe = np . zeros (( 1 , d_model ) )
4 for i in range ( 0 , d_model , 2 ) :
5 pe [ 0 ][ i ] = np . sin ( pos / ( 1 0 0 0 0 ** (( 2 * i ) / d_model ) ) )
6 if i + 1 < d_model :
7 pe [ 0 ][ i + 1 ] = np . cos ( pos / ( 1 0 0 0 0 ** (( 2 * i ) /
d_model ) ) )
8 return pe

Find positional encoding vector for black and brown.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 18 / 33
positional encoding
Define function to get positional encoding vector
1 import numpy as np
2 def positional_encoding ( pos , d_model ) :
3 pe = np . zeros (( 1 , d_model ) )
4 for i in range ( 0 , d_model , 2 ) :
5 pe [ 0 ][ i ] = np . sin ( pos / ( 1 0 0 0 0 ** (( 2 * i ) / d_model ) ) )
6 if i + 1 < d_model :
7 pe [ 0 ][ i + 1 ] = np . cos ( pos / ( 1 0 0 0 0 ** (( 2 * i ) /
d_model ) ) )
8 return pe

Find positional encoding vector for black and brown.

1 d_model = model . vector_size
2 pe_a = positional_encoding ( pos = 2 , d_model = d_model )
3 pe_b = positional_encoding ( pos = 1 0 , d_model = d_model )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 18 / 33
Final Input
Find Cosine-similarity

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 19 / 33
Final Input
Find Cosine-similarity
1 cos_pos_enc_only = cosine_similarity ( pe_a , pe_b )
2 print ( f " Cosine similarity of positional encodings :
{ cos_pos_enc_only } " )

Positional similarity :[[0.8600013]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 19 / 33
Final Input
Find Cosine-similarity
1 cos_pos_enc_only = cosine_similarity ( pe_a , pe_b )
2 print ( f " Cosine similarity of positional encodings :
{ cos_pos_enc_only } " )

Positional similarity :[[0.8600013]]

Find final input for the model and the cosine similarity

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 19 / 33
Final Input
Find Cosine-similarity
1 cos_pos_enc_only = cosine_similarity ( pe_a , pe_b )
2 print ( f " Cosine similarity of positional encodings :
{ cos_pos_enc_only } " )

Positional similarity :[[0.8600013]]

Find final input for the model and the cosine similarity
1 pos_embd_a = a . reshape ( 1 , -1 ) + pe_a
2 pos_embd_b = b . reshape ( 1 , -1 ) + pe_b
3
4 cos_sim_combined = cosine_similarity ( pos_embd_a , pos_embd_b )
5 print ( f " Cosine similarity of combined embeddings :
{ cos_sim_combined } " )

Cosine similarity of combined embeddings : [[0.85996798]]

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 19 / 33
Multi-Head Attention

Why Multiple Heads?

Runs 8 attention mechanisms in parallel
Each head learns different perspectives, like syntactic
relation, semantic relation, positional patterns.
Provides broader, in-depth analysis
Speeds up calculations H3
Precludes recurrence H4 H2

Example: “The cat sat on the mat” H5 MHA H1

Head 1: subject-verb
Head 2: positional
H6 H8
Head 3: object relations H7
Head 4: long-range deps

Each head specializes in 8 Parallel Heads

different linguistic phenomena
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 20 / 33
8-Head Configuration
Head 1
dk = 64, X 1 ∈ Rn×64
n : number of words Head 2
dk = 64, X 2 ∈ Rn×64
Head 3
dk = 64, X 3 ∈ Rn×64
Head 4
Input: dk = 64, X 4 ∈ Rn×64 Concatenate → Output
X ∈ Rn×512 Rn×512
Head 5
dk = 64, X 5 ∈ Rn×64
Head 6
dk = 64, X 6 ∈ Rn×64
Head 7
dk = 64, X 7 ∈ Rn×64
Head 8
dk = 64, X 8 ∈ Rn×64

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 21 / 33
Core Idea of attention

Attention mechanism

Input: X ∈ Rn×dmodel

Compute:
Q = XW Q W Q , W K , W V are
K = XW K learnable matrices,
and W Q , W K , W V ∈
V = XW V Rdmodel ×dmodel
Q, K , V ∈ Rn×dmodel

Compute attention
T

Attention(Q, K , V ) = softmax QK
√ V
dk

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 22 / 33
Core Idea of attention
Multi-head Attention
Attention mechanism
Input: X ∈ Rn×dmodel

Input: X ∈ Rn×dmodel
Head i: (for i = 1, 2, . . . , m)
headi = Attention(Qi , Ki , Vi )
Q KT

Compute: = softmax √i di Vi
k
Q = XW Q W Q , W K , W V are Qi , Ki , Vi ∈ Rn×dk
K = XW K learnable matrices,
and W Q , W K , W V ∈
V = XW V Rdmodel ×dmodel
Q, K , V ∈ Rn×dmodel Compute attention
Heads = Concat(head1 , . . . , headm )
Compute attention
T

Attention(Q, K , V ) = softmax QK
√ V Final output
dk
output = Heads × W O
W O ∈ Rdmodel ×dmodel
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 22 / 33
Multi-head attention implementation
Define all parameters
1 import numpy as np
2 np . random . seed ( 4 2 )
3 n , d_model , num_heads = 5 , 8 , 2
4 d_k = d_model // num_heads
5 print ( " d_k per head : " , d_k ) # 4

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 23 / 33
Multi-head attention implementation
Define all parameters
1 import numpy as np
2 np . random . seed ( 4 2 )
3 n , d_model , num_heads = 5 , 8 , 2
4 d_k = d_model // num_heads
5 print ( " d_k per head : " , d_k ) # 4

Create X (∈ Rn×dmodel ) by using any random function.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 23 / 33
Multi-head attention implementation
Define all parameters
1 import numpy as np
2 np . random . seed ( 4 2 )
3 n , d_model , num_heads = 5 , 8 , 2
4 d_k = d_model // num_heads
5 print ( " d_k per head : " , d_k ) # 4

Create X (∈ Rn×dmodel ) by using any random function.

1 X = np . random . randn (n , d_model )
2 print ( " Input X shape : " , X . shape ) # (5 ,8)

Define Weight matrices randomly (W Q , W K , W V )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 23 / 33
Multi-head attention implementation
Define all parameters
1 import numpy as np
2 np . random . seed ( 4 2 )
3 n , d_model , num_heads = 5 , 8 , 2
4 d_k = d_model // num_heads
5 print ( " d_k per head : " , d_k ) # 4

Create X (∈ Rn×dmodel ) by using any random function.

1 X = np . random . randn (n , d_model )
2 print ( " Input X shape : " , X . shape ) # (5 ,8)

Define Weight matrices randomly (W Q , W K , W V )

1 W_Q = np . random . randn ( d_model , d_model )
2 W_K = np . random . randn ( d_model , d_model )
3 W_V = np . random . randn ( d_model , d_model )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 23 / 33
Multi-head attention implementation
Compute Q, K , V

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 24 / 33
Multi-head attention implementation
Compute Q, K , V
1 Q = X @ W_Q
2 K = X @ W_K
3 V = X @ W_V
4 print ( " Q shape : " , Q . shape ) # (5 , 8)

Split into heads

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 24 / 33
Multi-head attention implementation
Compute Q, K , V
1 Q = X @ W_Q
2 K = X @ W_K
3 V = X @ W_V
4 print ( " Q shape : " , Q . shape ) # (5 , 8)

Split into heads

1 Q = Q . reshape (n , num_heads , d_k )
2 K = K . reshape (n , num_heads , d_k )
3 V = V . reshape (n , num_heads , d_k )
4 print ( " Q split shape : " ,
Q . shape ) # (5 , 2 , 4)

1 Q = Q . transpose ( 1 , 0 , 2 )
2 K = K . transpose ( 1 , 0 , 2 )
3 V = V . transpose ( 1 , 0 , 2 )
4 print ( " Q transposed shape : " ,
Q . shape ) # (2 , 5 , 4)
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 24 / 33
Multi-head attention implementation
Compute Q, K , V
1 Q = X @ W_Q
2 K = X @ W_K
3 V = X @ W_V
4 print ( " Q shape : " , Q . shape ) # (5 , 8)

Split into heads

1 Q = Q . reshape (n , num_heads , d_k )
2 K = K . reshape (n , num_heads , d_k ) Alternative method
3 V = V . reshape (n , num_heads , d_k ) 1 Q = np . array ([ Q [: , i : i + d_k ] for i in
4 print ( " Q split shape : " , range ( 0 , d_model , d_k ) ])
Q . shape ) # (5 , 2 , 4) 2 K = np . array ([ K [: , i : i + d_k ] for i in
1 Q = Q . transpose ( 1 , 0 , 2 ) range ( 0 , d_model , d_k ) ])
2 K = K . transpose ( 1 , 0 , 2 ) 3 V = np . array ([ V [: , i : i + d_k ] for i in
3 V = V . transpose ( 1 , 0 , 2 ) range ( 0 , d_model , d_k ) ])
4 print ( " Q transposed shape : " , 4 print ( Q . shape ) # (2 ,5 ,4)
Q . shape ) # (2 , 5 , 4)
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 24 / 33
Multi-head attention implementation

Compute Multi-head outputs

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 25 / 33
Multi-head attention implementation

Compute Multi-head outputs

1 from scipy . special import softmax
2 head_outputs = []
3
4 for h in range ( num_heads ) :
5 Q_h = Q [ h ]
6 K_h = K [ h ]
7 V_h = V [ h ]
8 # Attention scores
9 score = softmax (( Q_h @ K_h . T ) / np . sqrt ( d_k ) ) @ V_h # (5 , 5)
10 head_outputs . append ( score )
11 head_outputs = np . array ( head_outputs )
12 print ( head_outputs . shape )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 25 / 33
Multi-head attention implementation

Join all multi-head outputs (By using concatenate function)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 26 / 33
Multi-head attention implementation

Join all multi-head outputs (By using concatenate function)

1 multi_head_output = np . concatenate ( head_outputs , axis = - 1 )
2 print ( " Concatenated shape : " , multi_head_output . shape ) # (5 ,
8)

Compute final output

1 W_O = np . random . randn ( d_model , d_model )
2 final_output = multi_head_output @ W_O
3 print ( " Final output shape : " , final_output . shape ) # (5 ,8)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 26 / 33
Post-layer normalization (Add and Norm)

Post-layer normalization
output = LayerNormalization(X + MHA(X ))

output = LayerNormalization(X + final output)

LayerNormalization

For any vector x,

x −µ
LayerNormalization(x) = γ. √ +β
σ2 + ϵ

where, µ, σ 2 : mean and variance of x, respectively.

γ, β: learnable parameter.
ϵ: small constant for stability.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 27 / 33
Post-layer normalization implementation
Compute X + final output

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

1 mean = np . mean ( residual , axis = 1 , keepdims = True )
2 print ( mean . shape ) # (5 ,1)
3 variance = np . var ( residual , axis = 1 , keepdims = True )
4 print ( variance . shape ) # (5 ,1)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

1 mean = np . mean ( residual , axis = 1 , keepdims = True )
2 print ( mean . shape ) # (5 ,1)
3 variance = np . var ( residual , axis = 1 , keepdims = True )
4 print ( variance . shape ) # (5 ,1)

Normalize it for a very small ϵ

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

1 mean = np . mean ( residual , axis = 1 , keepdims = True )
2 print ( mean . shape ) # (5 ,1)
3 variance = np . var ( residual , axis = 1 , keepdims = True )
4 print ( variance . shape ) # (5 ,1)

Normalize it for a very small ϵ

1 epsilon = 1e - 6
2 normalized = ( residual - mean ) / np . sqrt ( variance + epsilon )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

1 mean = np . mean ( residual , axis = 1 , keepdims = True )
2 print ( mean . shape ) # (5 ,1)
3 variance = np . var ( residual , axis = 1 , keepdims = True )
4 print ( variance . shape ) # (5 ,1)

Normalize it for a very small ϵ

1 epsilon = 1e - 6
2 normalized = ( residual - mean ) / np . sqrt ( variance + epsilon )

Compute the final normalized output for γ = 1 and β = 0

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Post-layer normalization implementation
Compute X + final output
1 residual = X + final_output

Compute mean and variance per token

1 mean = np . mean ( residual , axis = 1 , keepdims = True )
2 print ( mean . shape ) # (5 ,1)
3 variance = np . var ( residual , axis = 1 , keepdims = True )
4 print ( variance . shape ) # (5 ,1)

Normalize it for a very small ϵ

1 epsilon = 1e - 6
2 normalized = ( residual - mean ) / np . sqrt ( variance + epsilon )

Compute the final normalized output for γ = 1 and β = 0

1 gamma = np . ones ( d_model )
2 beta = np . zeros ( d_model )
3 output = gamma * normalized + beta
4 print ( " Final Output Shape : " , output . shape ) # (5 ,8)
(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 28 / 33
Exercise

Exercise:
1 Define ReLU function.
2 Define Softmax function.
3 Define LayerNormalization function.

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 29 / 33
Feedforward network

The FFN is a position-wise network (Each token vector is

processed independently.)
The FFN contains two layers and applies a ReLU activation
function.
FFN(x) = W2 .ReLU(W1 x + b1 ) + b2

W1 ∈ Rdmodel ×dff , W2 ∈ Rdff ×dmodel

Usually, dff = 4 × dmodel
b1 ∈ Rdff , b2 ∈ Rdmodel
The output of the FFN goes to post-LN. Then the output is
sent to the next layer of the encoder stack.
Post-LN
FNN(x) Post-LN Next Layer
output

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 30 / 33
Feedforward network implementation

Define W1 , W2 randomly and b1 , b2 as zeros, and dff = 4 × dmodel .

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 31 / 33
Feedforward network implementation

Define W1 , W2 randomly and b1 , b2 as zeros, and dff = 4 × dmodel .

1 d_ff = 4 * d_model
2 W1 = np . random . randn ( d_model , d_ff )
3 b1 = np . zeros ( d_ff )
4 W2 = np . random . randn ( d_ff , d_model )
5 b2 = np . zeros ( d_model )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 31 / 33
Feedforward network implementation

Define W1 , W2 randomly and b1 , b2 as zeros, and dff = 4 × dmodel .

1 d_ff = 4 * d_model
2 W1 = np . random . randn ( d_model , d_ff )
3 b1 = np . zeros ( d_ff )
4 W2 = np . random . randn ( d_ff , d_model )
5 b2 = np . zeros ( d_model )

Compute FNN(x)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 31 / 33
Feedforward network implementation

Define W1 , W2 randomly and b1 , b2 as zeros, and dff = 4 × dmodel .

1 d_ff = 4 * d_model
2 W1 = np . random . randn ( d_model , d_ff )
3 b1 = np . zeros ( d_ff )
4 W2 = np . random . randn ( d_ff , d_model )
5 b2 = np . zeros ( d_model )

Compute FNN(x)
1 fnn_hidden = np . maximum ( 0 , output @ W 1 + b 1 )
2 print ( fnn_hidden . shape )
3 fnn_output = fnn_hidden @ W 2 + b 2
4 print ( fnn_output . shape )

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 31 / 33
Feedforward network implementation

Compute Layer normalization of FNN(x)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 32 / 33
Feedforward network implementation

Compute Layer normalization of FNN(x)

1 gamma 1 = np . ones ( d_model )
2 beta 1 = np . zeros ( d_model )
3 epsilon 1 = 1e - 6
4 output 1 = LayerNormalization ( output , gamma 1 , beta 1 , epsilon 1 ,
fnn_output )
5 print ( output 1 . shape ) # (5 ,8)

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 32 / 33
The decoder stack
Masked attention
When predicting token at position t, the model must NOT
see tokens at positions > t.
QK T + M
 
Attention(Q, K , V ) = softmax √ V
dk
where (
0 if j ≤ i
Mij =
−∞ if j > i

Masked multi-head attention

Qi KiT + M
 
headi = Attention(Qi , Ki , Vi ) = softmax √ Vi
dk

(ITER, Center for Data Science) Natural Language Processing with Transformer Dr. Debashis Bhowmik 33 / 33