Scribd
Upload
11 views40 pages

PyTorch Transformer Encoder Guide

The document provides an overview of transformer models using PyTorch, detailing the architecture of encoder-only, decoder-only, and encoder-decoder transformers. It includes explanations of components such as multi-head self-attention, feed-forward layers, and the cross-attention mechanism, along with code snippets for implementing these models. Additionally, it highlights practical applications and next steps for training and working with pre-trained transformers.

Uploaded by

teeravac vac
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views40 pages

PyTorch Transformer Encoder Guide

The document provides an overview of transformer models using PyTorch, detailing the architecture of encoder-only, decoder-only, and encoder-decoder transformers. It includes explanations of components such as multi-head self-attention, feed-forward layers, and the cross-attention mechanism, along with code snippets for implementing these models. Additionally, it highlights practical applications and next steps for training and working with pre-trained transformers.

Uploaded by

teeravac vac
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Encoder

transformers
TRANSFORMER MODELS WITH PYTORCH

James Chapman
Curriculum Manager, DataCamp
The original transformer

TRANSFORMER MODELS WITH PYTORCH


The original transformer

TRANSFORMER MODELS WITH PYTORCH


Encoder-only transformers
Transformer body: encoder stack with N
encoder layers

Encoder layer

Multi-head self-attention

Feed-forward (sub)layers

Layer normalizations, dropouts

Transformer head

TRANSFORMER MODELS WITH PYTORCH


Encoder-only transformers
Transformer body: encoder stack with N
encoder layers

Encoder layer

Multi-head self-attention

Feed-forward (sub)layers

Layer normalizations, dropouts

Transformer head: process encoded inputs to


produce output prediction

Supervised task: classification, regression

TRANSFORMER MODELS WITH PYTORCH


Feed-forward sublayer in encoder layers
2 x fully connected + ReLU activation

d_ff : dimension between linear layers

forward() : processes attention outputs to

class FeedForwardSubLayer([Link]):
capture complex, non-linear patterns
def __init__(self, d_model, d_ff):
super().__init__()
self.fc1 = [Link](d_model, d_ff)
self.fc2 = [Link](d_ff, d_model)
[Link] = [Link]()

def forward(self, x):


return self.fc2([Link](self.fc1(x)))

TRANSFORMER MODELS WITH PYTORCH


Encoder layer
class EncoderLayer([Link]):
def __init__(self, d_model, num_heads, d_ff, dropout):
super().__init__()
self.self_attn = MultiHeadAttention(d_model, num_heads)
self.ff_sublayer = FeedForwardSubLayer(d_model, d_ff)
self.norm1 = [Link](d_model)
self.norm2 = [Link](d_model)
[Link] = [Link](dropout)

def forward(self, x, src_mask):


Multi-headed self-attention
attn_output = self.self_attn(x, x, x, src_mask)
x = self.norm1(x + [Link](attn_output)) Feed-forward sublayer
ff_output = self.ff_sublayer(x)
x = self.norm2(x + [Link](ff_output)) Layer normalizations and dropouts
return x
forward() :

mask prevents processing padding tokens

TRANSFORMER MODELS WITH PYTORCH


Masking the attention process

TRANSFORMER MODELS WITH PYTORCH


Encoder transformer body
class TransformerEncoder([Link]):
def __init__(self, vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length):
super().__init__()
[Link] = InputEmbeddings(vocab_size, d_model)
self.positional_encoding = PositionalEncoding(d_model, max_seq_length)
[Link] = [Link](
[EncoderLayer(d_model, num_heads, d_ff, dropout) for _ in range(num_layers)]
)

def forward(self, x, src_mask):


x = [Link](x)
x = self.positional_encoding(x)
for layer in [Link]:
x = layer(x, src_mask)
return x

TRANSFORMER MODELS WITH PYTORCH


Encoder transformer head
class ClassifierHead([Link]): Classification head
def __init__(self, d_model, num_classes):
super().__init__() Tasks: text classification, sentiment
[Link] = [Link](d_model, num_classes)
analysis, NER, extractive QA, etc.
def forward(self, x): fc : fully connected linear layer
logits = [Link](x)
Transforms encoder hidden states into
return F.log_softmax(logits, dim=-1)
num_classes class probabilities
class RegressionHead([Link]): Regression head
def __init__(self, d_model, output_dim):
super().__init__()
Tasks: estimate text readability, language
[Link] = [Link](d_model, output_dim)
complexity, etc.
def forward(self, x): output_dim is 1 when predicting a single
return [Link](x)
numerical value

TRANSFORMER MODELS WITH PYTORCH


Let's practice!
TRANSFORMER MODELS WITH PYTORCH
Decoder
transformers
TRANSFORMER MODELS WITH PYTORCH

James Chapman
Curriculum Manager, DataCamp
From original to decoder-only transformer

TRANSFORMER MODELS WITH PYTORCH


From original to decoder-only transformer
Autoregressive sequence generation: text
generation and completion

TRANSFORMER MODELS WITH PYTORCH


From original to decoder-only transformer
Autoregressive sequence generation: text
generation and completion

Masked multi-head self-attention

Hide later tokens in sequence

TRANSFORMER MODELS WITH PYTORCH


From original to decoder-only transformer
Autoregressive sequence generation: text
generation and completion

Masked multi-head self-attention

Hide later tokens in sequence

Decoder-only transformer head

Linear + Softmax over vocabulary

Predict most likely next tokens

TRANSFORMER MODELS WITH PYTORCH


Masked self-attention/causal attention
Key to autoregressive or causal behavior
Triangular (causal) attention mask

TRANSFORMER MODELS WITH PYTORCH


Masked self-attention/causal attention
Key to autoregressive or causal behavior
Triangular (causal) attention mask

Token only pays attention to prior tokens in


the sequence

TRANSFORMER MODELS WITH PYTORCH


Masked self-attention/causal attention
Key to autoregressive or causal behavior
Triangular (causal) attention mask

Token only pays attention to prior tokens in


the sequence
"favorite": "orange", "is", "my", "favorite"
Enforced causal attention: predict next
word to generate, e.g., "fruit"

tgt_mask = (1 - [Link](
[Link](1, seq_len, seq_len), diagonal=1)
).bool()

TRANSFORMER MODELS WITH PYTORCH


Decoder layer
class DecoderLayer([Link]):
def __init__(self, d_model, num_heads, d_ff, dropout):
super().__init__()
self.self_attn = MultiHeadAttention(d_model, num_heads)
self.ff_sublayer = FeedForwardSubLayer(d_model, d_ff)
self.norm1 = [Link](d_model)
self.norm2 = [Link](d_model)
[Link] = [Link](dropout)

def forward(self, x, tgt_mask):


attn_output = self.self_attn(x, x, x, tgt_mask)
x = self.norm1(x + [Link](attn_output))
ff_output = self.ff_sublayer(x)
x = self.norm2(x + [Link](ff_output))
return x

TRANSFORMER MODELS WITH PYTORCH


Decoder transformer body and head
class TransformerDecoder([Link]):
def __init__(self, vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length):
super(TransformerDecoder, self).__init__()
[Link] = InputEmbeddings(vocab_size, d_model)
self.positional_encoding = PositionalEncoding(d_model, max_seq_length)
[Link] = [Link]([DecoderLayer(d_model, num_heads, d_ff, dropout) for _ in range(num_layers)])
[Link] = [Link](d_model, vocab_size)

def forward(self, x, tgt_mask):


x = [Link](x)
x = self.positional_encoding(x)
for layer in [Link]:
x = layer(x, tgt_mask)
x = [Link](x)
return F.log_softmax(x, dim=-1)

[Link] : output linear layer with vocab_size neurons

Add [Link] and softmax activation in forward pass

TRANSFORMER MODELS WITH PYTORCH


Instantiating the decoder-only transformer
decoder = TransformerDecoder(vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length=seq_length)
output = decoder(input_sequence, tgt_mask)

tensor([[[ -9.4692, -9.8429, -9.3077, ..., -9.9523, -10.2669, -9.7084],


[ -9.1556, -9.6133, -10.0923, ..., -9.3810, -9.0420, -9.1780],
...,
[ -9.5327, -10.3534, -9.8443, ..., -9.8170, -8.8491, -8.8322],
[ -9.6086, -9.6336, -10.1595, ..., -9.8550, -9.9955, -8.7121]],

[[ -9.5865, -8.0360, -8.5056, ..., -9.9855, -9.5677, -9.0352],


[ -9.7213, -8.6451, -8.3779, ..., -9.2994, -9.2601, -9.8509],
...,
[ -9.0471, -9.7410, -10.0160, ..., -10.0195, -9.4651, -8.9605],
[ -9.5767, -10.2692, -8.8394, ..., -8.3458, -9.1479, -10.0650]]],
grad_fn=<LogSoftmaxBackward0>)

TRANSFORMER MODELS WITH PYTORCH


Let's practice!
TRANSFORMER MODELS WITH PYTORCH
Encoder-decoder
transformers
TRANSFORMER MODELS WITH PYTORCH

James Chapman
Curriculum Manager, DataCamp
Encoder meets decoder

TRANSFORMER MODELS WITH PYTORCH


Encoder meets decoder

TRANSFORMER MODELS WITH PYTORCH


Cross-attention mechanism

1. Information processed throughout decoder

2. Final hidden states from encoder block

TRANSFORMER MODELS WITH PYTORCH


Modifying the DecoderLayer
class DecoderLayer([Link]):
def __init__(self, d_model, num_heads, d_ff, dropout):
super().__init__()
1. Information processed throughout decoder self.self_attn = MultiHeadAttention(
d_model, num_heads)
2. Final hidden states from encoder block self.cross_attn = MultiHeadAttention(
d_model, num_heads)
...

def forward(self, x, y, tgt_mask, cross_mask):


x : decoder information flow, becomes self_attn_output = self.self_attn(x, x, x,
tgt_mask)
cross-attention query x = self.norm1(x + [Link](self_attn_output))

y : encoder output, becomes cross- cross_attn_output = self.cross_attn(x, y, y,

attention key and values cross_mask)


x = self.norm2(x + [Link](cross_attn_output))
...

TRANSFORMER MODELS WITH PYTORCH


Modifying DecoderTransformer

Decoder-only Encoder-decoder
class TransformerDecoder([Link]): class TransformerDecoder([Link]):
... ...
def forward(self, x, tgt_mask): def forward(self, x, y, tgt_mask, cross_mask):
x = [Link](x) x = [Link](x)
x = self.positional_encoding(x) x = self.positional_encoding(x)
for layer in [Link]: for layer in [Link]:
x = layer(x, tgt_mask) x = layer(x, y, tgt_mask, cross_mask)
x = [Link](x) x = [Link](x)
return F.log_softmax(x, dim=-1) return F.log_softmax(x, dim=-1)

TRANSFORMER MODELS WITH PYTORCH


Encoder meets decoder

TRANSFORMER MODELS WITH PYTORCH


Transformer head

jugar (to play): 0.03

viajar (to travel): 0.96

dormir (to sleep): 0.01

For other tasks, different activations may


be required

TRANSFORMER MODELS WITH PYTORCH


Everything brought together!

TRANSFORMER MODELS WITH PYTORCH


Everything brought together!
class InputEmbeddings([Link]): class Transformer([Link]):
... def __init__(self, vocab_size, d_model, num_heads,
class PositionalEncoding([Link]): num_layers, d_ff, max_seq_len, dropout):
... super().__init__()
class MultiHeadAttention([Link]):
... [Link] = TransformerEncoder(vocab_size,
class FeedForwardSubLayer([Link]): d_model, num_heads, num_layers,
... d_ff, dropout, max_seq_len)
class EncoderLayer([Link]): [Link] = TransformerDecoder(vocab_size,
... d_model, num_heads, num_layers,
class DecoderLayer([Link]): d_ff, dropout, max_seq_len)
...
def forward(self, x, src_mask, tgt_mask, cross_mask):
encoder_output = [Link](x, src_mask)
class TransformerEncoder([Link]):
decoder_output = [Link](x, encoder_output,
...
tgt_mask, cross_mask)
class TransformerDecoder([Link]):
return decoder_output
...
class ClassificationHead([Link]):
...

TRANSFORMER MODELS WITH PYTORCH


Let's practice!
TRANSFORMER MODELS WITH PYTORCH
Congratulations!
TRANSFORMER MODELS WITH PYTORCH

James Chapman
Curriculum Manager, DataCamp
Chapter 1
model = [Link](
d_model=1536,
nhead=8,
num_encoder_layers=6,
num_decoder_layers=6
)

class InputEmbeddings([Link]): ...


class PositionalEncoding([Link]): ...
class MultiHeadAttention([Link]): ...

TRANSFORMER MODELS WITH PYTORCH


Encoder-only transformer Decoder-only transformer

TRANSFORMER MODELS WITH PYTORCH


Chapter 2 - Encoder-decoder transformer

TRANSFORMER MODELS WITH PYTORCH


What next?

Training Pre-trained transformers

Distributed AI Model Training in Python Working with Hugging Face

Introduction to LLMs in Python

Fine-Tuning with Llama 3


Additional Resources

Attention Is All You Need

TRANSFORMER MODELS WITH PYTORCH


Let's practice!
TRANSFORMER MODELS WITH PYTORCH

You might also like