Transformers
Contextual Embedding
Contexts (adjectives) are attending to word dish
Transformers
Introduced in the paper "Attention Is All You Need" (Vaswani et al., 2017), the
Transformer model eliminates recurrence (RNNs, LSTMs) and relies entirely on
self-attention and positional encoding to process sequences in parallel.
Key Components
Self-Attention Mechanism: Helps the model focus on different parts of a
sentence while processing each word.
Multi-Head Attention: Allows the model to capture multiple relationships between
words.
Positional Encoding: Since transformers do not have a sequential structure like
RNNs, positional encoding helps retain the order of words.
Feedforward Layers: Fully connected layers for transforming representations.
Layer Normalization & Residual Connections: Helps in stabilizing training.
Transformer-Based Models in NLP
(a) Encoder-Only Models (Used for classification, Named Entity
Recognition, etc.)
BERT (Bidirectional Encoder Representations from Transformers)
Pretrained using the Masked Language Model (MLM) and Next Sentence
Prediction (NSP)
Used in tasks like sentiment analysis and question answering.
Variants: RoBERTa, ALBERT, DistilBERT
b) Decoder-Only Models (Used for text generation)
GPT (Generative Pretrained Transformer)
Pretrained using causal language modeling (predicts next word based on previous
words).
Used in text completion, summarization, and chatbots.
Variants: GPT-2, GPT-3, GPT-4
(c) Encoder-Decoder Models (Used for machine
translation, text summarization, etc.)
T5 (Text-to-Text Transfer Transformer)
Converts all NLP tasks into a text-to-text format.
Example: Given "summarize: input text", the model generates a summary.
BART (Bidirectional and Auto-Regressive Transformers)
Similar to T5 but focuses on sequence-to-sequence pretraining.
Positional Encoding
Since Transformers don’t process words sequentially like RNNs, positional
encoding is added to retain word order.
The encoding is computed using sine and cosine functions:
Overall Structure
A Transformer consists of an encoder-decoder architecture, mainly used for
sequence-to-sequence tasks like machine translation. However, models like BERT (only
encoder) and GPT (only decoder) use variations of this architecture.
Encoder (N Layers)
● Processes input sequences
● Extracts contextual features for each token
Decoder (N Layers)
● Generates output sequences
● Uses attention to focus on important parts of the input
Each encoder and decoder layer consists of self-attention, feedforward layers, and residual
connections.
Transformer Encoder
Each encoder block consists of:
Self-Attention Layer (Multi-Head Attention)
Feedforward Neural Network
Layer Normalization & Residual Connections
(a) Self-Attention (Scaled Dot-Product Attention)
Transformer Decoder
The decoder is similar to the encoder but has an additional encoder-decoder attention
layer.
Decoder Components:
Masked Self-Attention: Prevents the decoder from "cheating" by only allowing it to
attend to earlier tokens.
Encoder-Decoder Attention: Allows the decoder to focus on relevant encoder outputs.
Feedforward Network: Similar to the encoder.
Shifting target sequence
When training a language model (e.g., for machine translation), the decoder generates
output one token at a time. However, if we pass the entire target sequence at once, the
model could see future words, which is not allowed in autoregressive generation.
To prevent this, we shift the target sequence one position to the right and mask future
tokens.
Why Masked head self attention
In standard self-attention, every word attends to every other word. This works fine
for encoders but not for decoders.
Problem:
During training, if the decoder sees the entire target sentence, it could cheat by
using future words to predict the next word.
Solution:
We use masked self-attention, which blocks future words, ensuring that each word
can only see previous words
Masked attention
In masked attention, the softmax is applied only to past words, while future tokens
are masked (set to -∞) to prevent the model from attending to them.
Multi-Head Cross-Attention
Unlike self-attention, which processes the same sequence (e.g., input or output)
independently, cross-attention allows the decoder to focus on relevant parts of the
encoder’s output.
This is crucial for sequence-to-sequence tasks like machine translation, text
summarization, and speech recognition.
Final Output (Softmax Layer)
For tasks like text generation, the decoder output is passed through a linear layer
+ softmax function to predict the next word probability.
Why is the Transformer Better?
Parallelization: Unlike RNNs, transformers process all words simultaneously.
Long-range Dependencies: Captures context better than LSTMs.
Scalability: Used in large models like GPT-4.
Libraries for Implementing Transformers
Hugging Face Transformers (pip install transformers)
Provides pretrained transformer models like BERT, GPT, T5, etc.
from transformers import pipeline
summarizer = pipeline("summarization")
text = "Transformers have changed NLP by replacing RNNs with self-attention."
print(summarizer(text))
Custom transformer models can be implemented using TensorFlow/Keras or
PyTorch.
[Link]