MTech DSML (PES)

Arun Sharma K
 How ML What are the What are
Basics of Text Utilizing LLMs
processing
models SOTA DL LLMs and
and RAGs Foundation
process test models RAGs
Day Plan

Need and Workflow NLP fundamentals

Applications

Hands on Typical Q&A

questions

Participate
Sessions

1 2
• Unstructured Vs structured • Feature engineering
• Need for NLP • Frequency based
• Typical applications • Sematic based
• Aspects of NLP • Hands-on
• Workflow
• Clean-up
• EDA
• Hands-on
NLP:Session-1
Structured Vs Unstructured
Need for NLP

Growing volume of text, streaming audio, video,

clickstream, sensor and log data ~90% of worldwide
data will be unstructured by 2025 [IBM]
Need for NLP

Huge emphasize from Organizations to focus on develop

robust strategies for effectively

• Managing

• Storing

• Analyzing

Harness the data for better, data-driven decisions

Value
•25% possible annual savings

•31% reduction in 30-day readmission for certain patients

•USD 500,000 savings per year

•30% fewer fraud incidents

•Nearly USD 4 million reduction in expenditure

•90% quicker time-to-value for big data analytics

[Link]
What is NLP

text  Structured data

NLP Understanding (NLU)

Generation (NLG)
NLU Vs NLG
• NLG is the inverse of NLU
• NLG maps from meaning to text, while
NLU maps from text to meaning
• NLG is easier than NLU because a NLU
system cannot control the complexity
of the language structure it receives as
input while NLG links the complexity of
the structure of its output.

[Link]
Typical Applications of NLP
• Spell Checking
• Text Classification
• Sentiment Analysis/opinion mining
• Question Answering
• Automatic Summarization
• Text suggestion
• Machine Translation(statistical machine translation)
• Speech Recognition
 Workflow
• In house documents
• Encoding/
Application Embedding method • Mode
• Web-based Expertise sensitive • POS tagging (??)
• Augmentation • MLOPS strategies
• Digitized

Injest the Feature Model

EDA Clean-up Deployment
text corpus Engineering development

• Noise Vs Signal Denoise the data • Choosing different

• Special characters models
Organization • Clean-up strategy • Emoticons Application • Fine-tune/ retrain
• Numbers sensitive model
Annotation • Evaluate issues • Spelling corrections • Model evaluation
• normalization
• Replace non-native words
Steps in preprocessing
• Removing punctuations
• Tokenization
• Remove stop words
• Stemming
• Lemmatizing
• POS tagging
• Vectorizing Data
• Bag of words/CountVectorizer
• N-Grams
• TF-IDF
Tokenization
• Sentence tokenization split collection of sentences to individual
sentences
• Word tokenization split sentence into words
• Stop words words that do not hold much importance in processing
compared to keywords
• Nltk stop words
• Adding new stop words
Stemming
• Stemming: heuristically removing the affixes of a word, to get its stem
(root) word
• Stem (root) is the part of the word to which you add inflectional
(changing/deriving) affixes such as (-ed,-ize, -s,-de,mis)
• Stemming :reducing inflection in words to their root forms such as
mapping a group of words to the same stem even if the stem itself is
not a valid word in the Language

[Link]
Lemmatization
• Lemmatization: unlike Stemming, reduces the inflected words
properly ensuring that the root word belongs to the language. In
Lemmatization root word is called Lemma.

• A lemma is the dictionary form, or citation form of a set of words.

• runs, running, ran are all forms of the word run, therefore run is the
lemma of all these words
Hands-on link

NLP S1 and S2 content and notebooks

REGEX
Regex Sequences of characters that form search patterns

Regex Description Example Matches

. Any single character except newline a.b → "acb", "a_b"
\d Any digit (0-9) \d → "5", "123"
\D Any non-digit character \D → "a", "@"
\w Any word character (a-z, A-Z, 0-9, _) \w → "a", "1", "_"
\W Any non-word character \W → "!", " "
\s Any whitespace (space, tab, newline) \s → " ", "\t"
\S Any non-whitespace character \S → "a", "9"
^ Start of a string ^Hello → "Hello World"
$ End of a string end$ → "The end"
* 0 or more repetitions a* → "", "aaaa"
+ 1 or more repetitions a+ → "a", "aaa"
{n} Exactly n repetitions a{3} → "aaa"
() Grouping (abc)+ → "abcabc"
[] Character set [a-z] → "a", "z"
 Examples
Regex to extract email Regex to extract phone#

^[a-zA-Z0-9._]+@[a-zA-Z0-9]+.[a-zA-Z]{2,}$ ^\d{10}$
• ^ asserts the start of the string. • ^ asserts the start.

• [a-zA-Z0-9._]+ matches the username (1+ alphanumeric, • \d{10} matches exactly 10 digits.
dots, underscores).
• $ asserts the end.
• @ matches the literal "@" character.

• [a-zA-Z0-9]+ matches the domain name.

• .[a-zA-Z]{2,} matches the top-level domain (e.g., ".com").

• $ asserts the end of the string.

Feature engineering
Encoding and Embedding types
Application Complexity, Performance, and available data size
Unstructured to
structured
Frequency Semantic
based based

• word2Vec
• Count vectorization
• Glove
• TF-IDF vectorization
• ELMO
• N-gram
• BERT, SBERT

• GPT

• Custom embedding
 Bag of words
documents
• I like to eat apples
corpus • I hate bananas
• I like apples and bananas

Vocabulary ={and, apples, bananas, eat, hate, I, like, to}

And Apples bananas eat hate I like to TARGET

1 0 1 0 1 0 1 1 1 1
2 0 0 1 0 1 1 0 0 0
3 1 1 1 0 0 1 1 0 1

I like to eat apples  [0,1,0,1,0,1,1,1]

TF-IDF Statistical measure used to evaluate
• TF(t,d) = count of ‘t’ in d / number of words in d how important a word is to a
document in a collection or corpus.
• IDF(t) = log(N/(df(t) + 1))
• df(t) = occurrence of t in documents
• N= total number of documents in the corpus
• TF-IDF(t, d) = tf(t, d) * log(N/(df + 1))

Term Frequency Inverse Document Frequency

This measures the frequency of a word This measures the importance of the
in a document. This highly depends on document in the whole corpus. DF is
the length of the document and the the count of occurrences of term t in
generality of word the document set N
 Bag of words
documents
• I like to eat apples
corpus • I hate bananas
• I like apples and bananas
Vocabulary ={and, apples, bananas, eat, hate, I, like, to}
And Apples bananas eat hate I like to TARGET
1 0 1 0 1 0 1 1 1 1
2 0 0 1 0 1 1 0 0 0
3 1 1 1 0 0 1 1 0 1

TF (I)= 1/5 =0.2 TF (Apples)= 1/5 =0.2 TF (to)= 1/5 =0.2

IDF (I)= ln(3/(3+1))=-0.28 IDF (Apples)= ln(3/(2+1))=0 IDF (to)= ln(3/(1+1))=0.4
TF-IDF(I)=0.2*-0.28=-0.05754 TF-IDF(Apple)=0.2*0=0 TF-IDF(to)=0.2*0.4=0.08

I like to eat apples  [0, 0, 0, 0.08, 0, -0.05754, 0, 0.08]

 Bag of words-N grams
documents
• I like to eat apples
corpus • I hate bananas
• I like apples and bananas
Vocabulary ={and, apples, bananas, eat, hate, I, like, to, I like, like to, to eat, eat apples, I hate, hate
bananas, like apples, apples and, and bananas }

# And Apples bananas eat hate I like to I like like to to eat eat I hate like apples and TARG
apples hate bananas apples and bananas ET
1 0 1 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1

2 0 0 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0

3 1 1 1 0 0 1 1 0 1 0 0 0 0 0 1 1 1 1

I like to eat apples  [0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]

NLP: Text embeddings
Session-2
Word embedding
• Basic CountVectors limitations
• The product is good [1,1,1,1]
• Is the product good [1,1,1,1]
• Vocabulary [good, is, product, this]
• Count-Based Vector Space Model
• Arrangement of same words in a different order (syntactic) could
change the meaning. Which is not captured by CountVectors
• Sequencing issue
Word embedding
• Have a great time [0,1,1,1]
• Have a good time [1,0,1,1]
• Vocabulary[good, great, have, time]
• Semantically similar sentences.
• Different representations by CountVectors/OHE
• Each word is treated as an orthogonal dimension and similarity
between words is not captured
• This technique leads to a high dimensional sparse matrix
 Word embedding: the idea
16 colors

Orange [1,0,0……]:  input to NN

Hidden layer
Orange [1,0,0……]: the OHE way

Color embeddings
[1,0.5,0]
Orange [R,G,B]:[1,0.5,0] the feature vector way

• LD representation
• Captures similarity
• compare orange and Blue  [1,0.5,0]-[0,0,1] large difference
• compare orange and red  [1,0.5,0]-[1,0,0] less difference
• Essentially a FA type DR method
Context matters
• We watched a monkey swing effortlessly between the trees.
• My son tends to monkey around with the settings on the phone,
causing unintended changes.

• he is a rich man
• he is rich manuel
Adjacent words add to/ modify the meaning of the word
 Word embedding
• Word embeddings are word representation as vectors in a multi-dimensional
space, that allows words with similar meaning to have a similar representation
• leads to a low dimensional dense matrix
• Each word is represented by a real-valued vector, often tens or hundreds of dimensions.
• This is contrasted to the thousands or millions of dimensions required for sparse word
representations
• Convert input OHE matrix (HDS) to real-valued vector(LDD)
• semantic-Based Vector Space Model

Distributional semantics
Distributional hypothesis: linguistic items with similar distributions have similar meanings.
 Word embedding: the idea
16 colors

Orange [1,0,0……]:  input to NN

Hidden layer
Orange [1,0,0……]: the OHE way

Color embeddings
[1,0.5,0]
Orange [R,G,B]:[1,0.5,0] the feature vector way

• LD representation
• Captures similarity Word embedding A DR technique applied to text data
• compare orange and Blue  [1,0.5,0]-[0,0,1] large difference
• compare orange and red  [1,0.5,0]-[1,0,0] less difference
• Essentially a FA type DR method
 Word embedding: the idea

1. Each word is represented with a lower-dimensional vector (3 instead of 9).

2. Similar words have similar vectors here. There’s a smaller distance between the embeddings for “girl” and
“princess”, than from “girl” to “prince”. In this case, distance is defined by Euclidian distance.
3. The embedding matrix is much less sparse (less empty space),
4. We could add more words to the vocabulary without increasing the dimensionality. For instance, the word “child”
might be represented with [0.5, 1, 0].
5. Relationships between words are captured and maintained, e.g. the movement from king to queen, is the same as
the movement from boy to girl, and could be represented by [+1, 0, 0].
[Link]
Outcomes of WE
• Recognizes words that are similar,
• Naturally captures the relationships between words as we use them

• The relationships between words in the embedding space lend themselves to unusual
word algebra, allowing words to be added and subtracted, and the results actually
making sense. For instance, in a well-defined word embedding model, calculations such
as (where [[x]] denotes the vector for the word ‘x’)
Word embedding
• Word embeddings are dense vector representations of words that
capture meaning based on word usage patterns in large corpora
• The central idea of word embedding is that similar words are typically
surrounded by the same “context” words
• So similar words have similar embeddings
• Word embeddings don’t “understand” definitions of words (like in a
dictionary). Instead, they understand patterns of usage (statistical).
Word embedding

V1
Word embedding

V2
V1 V3
V4
V5

Relative positions matter

Word embedding

• What is petrichor (spelt as pet-re-ko)

Word embedding

• What is petrichor (spelt as pet-re-ko)

After the soft drizzle, the petrichor rising from the parched soil was intoxicating

For the farmers, petrichor was more than a scent—it was a signal that life was beginning again.

When she opened the window, the petrichor wafted in, evoking memories of childhood monsoons
Word embedding

• What is petrichor (spelt as pet-re-ko)

After the soft drizzle, the petrichor rising from the parched soil was intoxicating

For the farmers, petrichor was more than a scent—it was a signal that life was beginning again.

When she opened the window, the petrichor wafted in, evoking memories of childhood monsoons

Less data noisy association

Large data association with relevant words such as smell, monsoon, rain, drizzle
 Different Embedding approaches
Embedding
[I am learning NLP] ML/MLP
extraction

0.1 0 0.2 0
1. Word2Vec 0.3 0.4 0.3 0.1 n × m Tensor for ‘m’ words/tokens
0.8 0.6 0.4 0
2. Glove 0.9 0.2 0.7 0.7
, , ,
| | | | How to use this Tensor as input?
3. FastText | | | |
| | | | 1) Aggregation Not a smart idea
4. ELMO (Sequential model-based) 0 0.5 0.1 0.2
2) Sequential models
5. BERT ( Transformer-based)
‘n’ dimensional
3) Transformers
matrix for each word
Google News is a news aggregator app developed by Google. It presents a continuous, customizable flow of
articles organized from thousands of publishers and magazines. Google News is available as an app on Android,
iOS, and the Web. Google released a beta version in September 2002 and the official app in January 2006.’
 Sequential models Hidden states
Context vector

RNN, LSTM, GRU

[START]

Embedding TR-1 TR-2 TR-3 TR-4

[I am learning NLP]
extraction
SM SM SM SM

0.1 0 0.2 0 0.1 0 0.2 0

0.3 0.4 0.3 0.1 0.3 0.4 0.3 0.1
0.8 0.6 0.4 0 0.8 0.6 0.4 0
0.9 0.2 0.7 0.7 0.9 0.2 0.7 0.7
, , , |
| | | | | | |
| Classification
| | | | | | |
Head
| | | | | | | |
0 0.5 0.1 0.2 0 0.5 0.1 0.2
 Classification Head
Transformers Hidden states

BERT, GPT, T5

Embedding
[I am learning NLP]
extraction
Transformer (ENCODER)

0.1 0 0.2 0 0.1 0 0.2 0

0.3 0.4 0.3 0.1 0.3 0.4 0.3 0.1
0.8 0.6 0.4 0 0.8 0.6 0.4 0
0.9 0.2 0.7 0.7 0.9 0.2 0.7 0.7
, , , |
| | | | | | |
| | | | | | | |
| | | | | | | |
0 0.5 0.1 0.2 0 0.5 0.1 0.2
 word2vec
• Developed by Google
• Word2vec NNet models to get word
embedding
• Not a Deep NNet.
• Has only one hidden layer
• Two approaches
• CBOW The central idea of word embedding training is that
• Skip-gram similar words are typically surrounded by the same
“context” words in normal use.
• terms  ‘target’ and ‘context’

[Link]
CBOW
• Uses a neural network to predict a target word,
given a context of words
• Create embeddings for the target word
• Has a projection/ averaging layer
• No activation function for the hidden layer
• SoftMax AF for the output layer.
• Weights are learnt for vocabulary classification
(given context words, predict the probability of
target words)
• Once trained on a corpus, the OL can be
discarded since we are interested only in the
EL/HL output
Skip-gram
• Uses a neural network to predict context
words, given a target word
• Create embeddings for the target word
• No activation function for the hidden layer
• SoftMax AF for the output layer.
• Once trained on a corpus, the OL can be
discarded since we are interested only in
the EL/HL output
CBOW Vs. Skip-gram
• Skip-gram: works well with small amount of the training data,
represents well even rare words or phrases.
• CBOW: several times faster to train than the skip-gram, slightly better
accuracy for the frequent words.
Applications of W2V
• Analyzing Survey Responses
• Recommendation systems
GloVe
• Global Vectors for Word Representation
• Developed by Stanford University
• Glove is an unsupervised learning algorithm for obtaining vector
representations for words. Training is performed on aggregated global
word-word co-occurrence statistics from a corpus, and the resulting
representations showcase interesting linear substructures of the word
vector space ([Link]
• The word vector representation is in terms of a ratio of probability
GloVe
• I love programing.
• I love Math.
• I tolerate Biology

Create co-occurrence matrix

• Calculate co-occurrence probabilities Plove-I=2/3
Ptolerate-I=1/3
• Calculate word vectors such that

f(x) = (x / xmax)^α if x < xmax

1 if x ≥ xmax

A new weighted least squares regression model xmax = 100, α = 0.75

 Glove
Step 1: Build a Word Co-occurrence Matrix
• Count how often each word appears with every other word in a context window.
• Step 2: Learn Embeddings from the Co-occurrence Stats
• GloVe tries to find word vectors such that the dot product of 2 ‘word’ vectors equals the log of how
often those two words appear together.

This turns out a matrix factorization problem.

Mathematically: wi⋅wj+bi+bj≈log(Xij)
Where:
X_ij = number of times word j appears in the context of word i
w_i, w_j = word vectors
b_i, b_j = bias terms
This is optimized over the entire vocabulary
GloVe Vs word2vec

Feature GloVe Word2Vec

Type Count-based (matrix factorization) Predictive (shallow neural network)
Yes — uses full corpus co-
Uses context? Yes — via sliding window
occurrence
Context prediction (CBOW/Skip-
Trains on Word-word co-occurrence matrix
gram)
Classification (predict
Loss type Regression on log-counts
context/word)
GloVe Vs word2vec
• No clear winner
• GloVe does better in word analogy task
• We have better options
• contextualized word-embeddings
The problem: polysemy
• will you read this article
• Can you read this article
• I heard that you read this article (same word different tenses)

• to sanction is to permit
• put sanction on means not permit (same word different meaning)

• W2V and Glove lead to same vector for the words read/sanction in these
the sentences

• Issue with polysemous words

Embeddings from Language Models
(ELMo)
• Unlike word2vec and GLoVe, the ELMo assigns a vector to a
token/word that is a function of the entire sentence containing that
word.
• ELMo word representations take the entire input sentence into
equation for calculating the word embeddings
• Hence, the same word can have different vectors under different
contexts
More advanced Embedding techniques
• BERT
• GPT-*
Sequential modeling
Sequential modeling
● Conventional prediction
● Y= f(xi :{i=1..k})
● Sequential models
● Yn= f(xn-m :{m=1..j})
Sequential models
● The thief, after stealing, was looking(ed) for a place to hide
● The thief, after stealing, were looking(ed) for a place to hide
● To use the correct state verb, we need to ‘remember’ if the subject is
singular or plural
● To use correct action verb tense, we need to ‘remember’ the state verb
● Sequential models use internal memory and memory manipulation for
this
Non-Sequential Vs. Sequential
● Relational
○ In non-sequential, all the inputs are independent of each other

○ In sequence, all the inputs are related

● Input/output size
○ Fixed in NS

○ Varies in S
Activation Functions in RNNs
Sigmoid
Sigmoid and tanh
Sigmoid, tanh, ReLu
Sequential model ANNs

RNN LSTM GRU

How vanilla RNN works
● Words get transformed into machine-
readable vectors(embeddings)
● RNN processes the sequence of vectors one
by one ht= tanh(Wht-1+Uxt+bh)
● While processing, it passes the previous
hidden state to the next step of the
sequence
● The hidden state acts as the neural
networks memory
How vanilla RNN works Context vector

the thief after stealing was looking for place to hide

0.5
−0.7
0.8

(batch_size,time_steps,10) (batch_size, 10)

Vanilla RNN
● Current state ht=f(ht-1,xt)
● With activation function: ht= tanh(Wht-1+Uxt+bh)
● Output state yt= g(V ht+bo)
○ W is the weight at previous hidden state,

○ U is the weight at current input state,

○ V is the weight at the output state

[Link]
neural-networks-tutorial-part-1-introduction-
to-rnns/
Vanilla RNN
● Limitation:
○ can capture only short-term relations

○ Looses sight of long-term dependencies due to multiple

operations (vanishing gradient)
● How to solve this problem
○ A way to decide which part of the sequence to remember

○ A way to decide what aspect of the current input to remember

● LSTM solves this problem through “gate(-ing)”

approach [Link]
RNN forward propagation
0.488311
0.508526 0.152139
0.461771 0.165761
0
0.195104
0 ho W
0
+
0.456738
Ux1 +Who+b T W 0.501887

+
0.543609
0.53384
Ux2 +Wh1+b
T
0.56074 W

U
0.49956 0.77321
0.579995
+ Ux3 +Wh2+b
T
U
0.833438 0.493182
U
x1 x3 0.551826
x2
0.92 0.609265
0.56 0.19
0.38 0.03 0.23
0.19 0.45 0.24
I LOVE NLP

0.239 0.314
U= 0.423 w= 0.329 b=0.4
0.524 0.428
LSTM
● The intuition:

● Amazing, the movie is outstanding. I have

watched it twice already and will watch it again.

● Amazing, the movie is outstanding. I have

watched it twice already and will watch it again
LSTM
● The Forget gate decides what is Ct-1 Ct

relevant to keep from prior steps.

ht-1 ht

● The input gate decides what

information is relevant to add from xt

the current step.

● The output gate determines what the

next hidden state should be.
Intuition for each gate and updates
Forget Gate Input Gate
Decides which parts of the previous cell state (𝑐𝑡−1 ) Determine how much new information from the current input
are no longer relevant. (𝑥𝑡) is added to the cell state.

Intuition: Forgetting unnecessary information helps Intuition: Capture relevant new information without
the model avoid "clutter." overwriting important long-term knowledge.

Output Gate Cell State Update

Filters the updated cell state to produce the Combines the effects of forgetting old information and adding
hidden state (ℎ𝑡), which serves as the short-term new information to update long-term memory.
memory and output.

Intuition: Controls what parts of the cell state are Intuition: The cell state acts as a "memory bank," retaining key
immediately relevant for the task. information across time steps.
 Forget Gate
Decides which parts of the previous cell state (𝑐𝑡−1 ) are no
longer relevant.
Intuition: Forgetting unnecessary information helps the
model avoid "clutter."

Cell State Update

Combines the effects of forgetting old
information and adding new information to
update long-term memory.

Intuition: The cell state acts as a "memory

bank," retaining key information across time
steps.
Input Gate
Determine how much new information from the
current input (𝑥𝑡) is added to the cell state.
Intuition: Capture relevant new information
without overwriting important long-term
knowledge.
 Cell State Update
Combines the effects of forgetting old
information and adding new information to
update long-term memory.
Intuition: The cell state acts as a "memory
bank," retaining key information across time
steps.
 LSTM

Forget gate Input Gate Output Gate

• This gate decides what information • To update the cell state, we have the input gate. The output gate decides what the next hidden
should be forgotten or remembered. • First, pass the previous hidden state and current input into state should be.
• Information from the previous hidden a sigmoid function. Remember that the hidden state contains
state and information from the current • That decides which values will be updated by transforming information on previous inputs.
input is passed through the sigmoid the values to be between 0 and 1. The hidden state is also used for predictions.
function. • Also pass the hidden state and current input into the tanh First, pass the previous hidden state and the
• Values come out between 0 and 1. function current input into a sigmoid function. Then we
• The closer to 0 means to forget, and • Multiply the tanh output with the sigmoid output. pass the newly modified cell state to the tanh
the closer to 1 means to remember • The sigmoid output will decide which information is function.
important to keep from the tanh output Multiply the tanh output with the sigmoid
output to decide what information the hidden
• First, the cell state gets pointwise multiplied by the forget vector. state should carry. The output is the hidden
• This has a possibility of dropping values in the cell state if it gets multiplied by values near 0. state
• Then we take the output from the input gate and do a pointwise addition which updates the cell state to new
values that the neural network finds relevant. That gives us our new cell state
GRU The reset gate is used to decide how much past information to
forget

● Gated Recurrent Unit

● Uses two gates to decide
● How much past information to ht-1
z
r 1-z
remember
ℎ
● what new information to add

xt

The update gate acts similar to the forget and input gate of an
h
LSTM.
It decides what new information to add
Bidirectional LSTM Vs Transformer

As opposed to directional models, which read the text input sequentially (left-to-right or right-to-
left), the Transformer encoder reads the entire sequence of words at once. Therefore it is considered
bidirectional, though it would be more accurate to say that it’s non-directional. This characteristic
allows the model to learn the context of a word based on all of its surroundings (left and right of the
word).
Greedy Vs Beam
Greedy
First word  “The”

“the” Model “cat” (0.4)

“rat” (0.3)
“dog” (0.2)

Next word  “cat”

“the” +” cat” Model “is” (0.26)

“was” (0.12)
“could” (0.05)

Next word  “is” The sequence

the cat is ……
Greedy Vs Beam Beam

First word  “The” Next word  “cat”

“the” +” cat” Model “is” (0.26)

“cat” (0.4) “was” (0.12)
“the” Model
“rat” (0.3) “could” (0.05)
“dog” (0.2)
“the” +”rat” Model “ran” (0. 6)
is was colud “hid” (0.22)
0.26 0.12 0.05 “tremble” (0.01)
cat (0.4) 0.104 0.048 0.02
ran hid tremble
rat (0.3) 0.6 0.22 0.01 “the” +”dog” Model “barked” (0. 4)
0.18 0.066 0.003 “chased” (0.3)
barked chased ate “ate” (0.09)
0.4 0.3 0.09 The sequence
dog(0.2) 0.08 0.06 0.018
the cat is ……
The rat ran …….
The dog barked……
References
● [Link]

● [Link]

● [Link]
step-by-step-explanation-44e9eb85bf21

● [Link]
lessons-learned-c62fb1d3485b
Transformers
Elements of Transformers
Encoder
• Word embedding
• Position encoding
• Multi-headed self-Attention
• Skip Connections and Layer Normalization
Transformers • Feed forward

Processes the input sequence and encodes its representation

Decoder
• Position encoding
• Multi-headed self-Attention
• Masked Multi-headed attention
• Encoder-Decoder attention
• Skip Connections and Layer Normalization
• Linear and SoftMax
Generates the output sequence (such as the translated sentence)
by attending to both the encoded input and the previously
generated output
Encode-Decoder

[ H1, H2, H3,---------Hn] Decoder

Encoder
[ T1, T2, T3,---------Tm]

[ output sequence]

राहुल एक अ छा लड़का है
[ T1, T2, T3,---------Tn]

[ input sequence]
Rahul is a good boy
Transformer types
Encoder-Decoder machine translation, summarization,
and text generation
• BART, M2M, mBART
Bidirectional and Auto-Regressive Transformers

Transformers Encoder only Semantic analysis, NER, SA, Semantic

• BERT Search, classification
Bidirectional Encoder Representations from Transformers

Decoder only Text generation, Chatbot, Code

• GPT*
generation
 Attention types
Self-attention
• Understand the relevance of one word to another to
A capture the meaning Encoder
t
Focus on input sequence only
t
[[],[],[],…..[]]
e
n Masked Self-attention Decoder
t • Hide the future words from the decoder during training
• Mask padding (add high negative value)
i
[[],[],[],…..[]]
o Focus on output sequence only
n
Encoder-Decoder [[],[],[],…..[]] Decoder
• Helps the decoder to focus on specific words will predicting
the next word Encoder
[[],[],[],…..[]]
decoder focuses on relevant parts of the input
sequence, produced by the encoder, when
[[],[],[],…..[]]
generating each token
Contextualized Embeddings using self-
attention

0.32

0.21
Lamborghini  [0.65,0.74]
Grapes  [0.32,0.21]
Contextualized Embeddings using self-
attention

0.32

0.21
• Where should the word ‘apple’ fit?

• On its own, it can fit into any of the 3 categories

• How do you decide where to fit apple?

 Contextualized Embeddings using self-
attention

0.32

0.21

Apples are generally grown in regions with temperate climates where winters are cold
enough to allow the trees to go dormant and summers are warm enough to promote
fruit growth.
Words that establish what apple refers to fruit, trees, grown, climate region
 Contextualized Embeddings using self-
attention

0.32

0.21

Apple launched its new iPhone*** in September. iPhone*** is Apple’s premium model, popular for its build quality and
software integration. A model class apart from its competitors.
Words that establish what apple refers to:

premium, popular, iPhone, quality, class, Integration, apart, competitors

 Contextualized Embeddings using self-
attention

0.32

0.21

Apple Inc. is a leading technology company known for its innovative products. It has revolutionized consumer electronics
with focus on sleek design, user-friendly interfaces, and seamless ecosystem integration.

Words that establish what Apple refers to: innovative, technology, consumer,
electronics, sleek, user-friendly, seamless, ecosystem
Multi-headed self-attention
The meaning of the word depends on words that may or may not be in the immediate neighbourhood

Self-Attention is a way to determine this dependency

One of the core elements of Transformers is Multi-headed Self-attention

Amit love supercars. He is driving a Bugatti

Each word will look at what word is relevant to it

As part of this exercise, multiple relationships might stem out

Each Self-attention captures one of such relations

Multi-headed attention captures multiple relations

 Multi headed self-attention
The meaning of the word depends on words that may or may not be in the immediate neighbourhood

Self-Attention is a way to determine this dependency

One of the core elements of Transformers is Multi-headed Self-attention

amit loves supercars he is driving a bugatti
AM-1 Amit ‘s attitude
amit
About what
AM-2
loves Entity matching
supercars AM-3
Pronoun reference
he AM-4 Multi-headed
is Self-Attention

driving AM-5 Action

a
bugatti AM-6 Motivation

Amit is driving a Bugatti since he loves supercars

Transformers

The encoder’s inputs first The decoder has SA and FF and

flow through a self- also , a layer between them: an
attention layer. It helps the attention layer that helps the
encoder look at other words
All encoders have the same architecture. in the input sentence as it
decoder focus on relevant parts
of the input sentence
encodes a specific word
All decoders have the same architecture.
 Self Attention
One key property of the Transformer, which is that the word in
each position flows through its own path in the encoder
{positional encoding}, dependencies captured only in SA.

Steps in SA:

• Create three vectors from each of the encoder’s input

vectors. For each word, we create a Query vector, a Key
vector, and a Value vector
• These vectors are created by multiplying the embedding
by three matrices that we trained during the training
Each word is embedded into a vector of size 512 process
Self Attention
Weight matrices obtained during
training
512

64

Steps in SA:
STEP-1
• Create three vectors from each of the
encoder’s input vectors. For each word, we
create a Query vector, a Key vector, and a
Value vector
• These vectors are created by multiplying
the embedding by three matrices that we
trained during the training process
Self Attention

Steps in SA:
STEP-2
• Calculate scores for each word input
against the other words. The score
determines how much focus to place on
other parts of the input sentence as we
encode a word at a certain position

• The score is calculated by taking the dot

product of the query vector with the key
vector of the respective word we’re scoring
 Self Attention

Steps in SA:
STEP-3 and 4
• Divide by 8

• Pass the result through a SoftMax

operation. SoftMax normalizes the scores
so they’re all positive and add up to 1

This SoftMax score determines how much each word will be expressed at this position. Clearly the word at its position will
have the highest SoftMax score, but it’s useful to attend to another word that is relevant to the current word
 Self Attention Steps in SA:
STEP-5 and 6
• multiply each value vector by the SoftMax
score. The intuition here is to keep intact
the values of the word(s) we want to focus
on, and drown-out irrelevant words

• Sum up the weighted value vectors. This

produces the output of the self-attention
layer at this position

This SoftMax score determines how much each word will be expressed at a position. Clearly the word at its own position
will have the highest SoftMax score, but it’s useful to attend to another word that is relevant to the current word
Multiheaded attention
Multi-head attention
Transformer: more details
Transformer: positional encoding
● [Link]
[Link]#subsec-positional-encoding

Besides capturing absolute positional information, the

above positional encoding also allows a model to easily
learn to attend by relative positions
Transformer: positional encoding
Masked Multi headed self-attention
Causal mask (autoregressive mask)
Decoder • Use: Autoregressive models (Sequential prediction) One token at a time.

• Need: Capture rich contextual information by mapping dependencies between

[[],[],[],…..[]] past and currently predicted tokens
Masked Self-attention
• Work: Similar to Self-attention with masking to tokens

Training
• The causal mask is applied to ensure that each token can only "see" the previous
tokens since the entire sequence is available during training, This keeps the model
from cheating

Inferencing
• Masking is turned off in the last layer. The other layers still see masking to maintain
consistency between training and inferencing
 Masked Multi headed self-attention
Applying mask:

“It is going to rain heavily today” Token to be trained as a response

“It is” Currently predicted tokens

[[Link], [Link], -108, -108, -108, -108, -108] Masked matrix for predicting the next word (“going”) Q.K output

[0.65, 0.35, 0, 0, 0, 0, 0] Attention scores

 E-D Attention
w1, w2,w3 and w4 are attention scores
that change during each next-word prediction
Query vector for यह
value vectors
H
EDA
Weighted sum of value vectors

[SOS]

  

Encoder Decoder

This a good start

यह
INPUT EMBEDDINGS OF ENGLISH WORDS PREDICTS EMBEDDINGS OF HINDI WORDS
 E-D Attention
w1, w2,w3 and w4 are attention scores
that change during each next-word prediction
Query vector for यह
value vectors
H
EDA
Weighted sum of value vectors

[SOS] यह

  

Encoder Decoder

This a good start यह एक

INPUT EMBEDDINGS OF ENGLISH WORDS PREDICTS EMBEDDINGS OF HINDI WORDS

 E-D Attention
w1, w2,w3 and w4 are attention scores
that change during each next-word prediction
Query vector for एक
value vectors
H
EDA
Weighted sum of value vectors

[SOS] यह एक
  

Encoder Decoder

This a good start यह एक अ छ

INPUT EMBEDDINGS OF ENGLISH WORDS PREDICTS EMBEDDINGS OF HINDI WORDS

 E-D Attention
w1, w2,w3 and w4 are attention scores
that change during each next-word prediction
Query vector for अ छ
value vectors
H
EDA
Weighted sum of value vectors

[SOS] यह एक अ छ

  

Encoder Decoder

This a good start

यह एक अ छ शु आत
INPUT EMBEDDINGS OF ENGLISH WORDS PREDICTS EMBEDDINGS OF HINDI WORDS
BERT: an intro
• Bidirectional Encoder Representations from Transformers

• BERT architecture is a multi-layer bidirectional Transformer encoder. We have two versions of BERT: BERT base and BERT large.
• BERT base has 12 Encoders with 12 self-attention heads and 110 million parameters
• BERT large has 24 Encoders with 24 self-attention heads and 340 million parameters

• When training the BERT model, Masked LM and Next Sentence Prediction are trained together, with the goal of minimizing the
combined loss function of the two strategies

• Masked LM (MLM): Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK]
token. The model then attempts to predict the original value of the masked words, based on the context provided by the other,
non-masked, words in the sequence

• Next Sentence Prediction(NSP): During training, 50% of the inputs are a pair in which the second sentence is the subsequent
sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence.
The assumption is that the random sentence will be disconnected from the first sentence.
 BERT models
Key Differences /
Model Size / Params Architecture Pretraining Objective Use Case Highlights
Improvements
MLM (Masked
General NLP tasks
Base: 110M Encoder-only Bidirectional; trained Language Modeling)
BERT (classification, QA,
Large: 340M Transformer with MLM + NSP NSP (Next Sentence
NER)
Prediction)
Removes NSP, uses
Improved
Base: 125M more data, longer
RoBERTa Encoder-only MLM only performance across
Large: 355M training, dynamic
tasks
masking
Lightweight version of
Encoder-only (6 Faster inference, low-
DistilBERT 66M (40% smaller) BERT, distilled from Distillation + MLM
layers) resource deployment
BERT
Parameter sharing +
MLM + SOP
Base: 12M factorized embedding Efficient training,
ALBERT Encoder-only (Sentence Order
XXL: 235M + sentence-order memory savings
Prediction)
prediction
Distilled BERT with
Encoder-only (4 task-specific Mobile-friendly
TinyBERT ~15M task-specific
layers) distillation deployment
distillation
 BERT models
Key Differences / Pretraining
Model Size / Params Architecture Use Case Highlights
Improvements Objective
Trained to predict
Span Boundary
SpanBERT Similar to BERT Encoder-only entire spans, not NER, QA
Objectives
just tokens
BERT pretrained on
BioBERT BERT-Base (110M) Encoder-only MLM + NSP Biomedical NLP
biomedical corpora
Pretrained on
ClinicalBERT BERT-Base (110M) Encoder-only clinical notes MLM + NSP Healthcare NLP
(MIMIC-III)
Disentangled MLM + R1 Loss SOTA on many
Base: 139M Encoder-only
DeBERTa attention + relative (replaced token GLUE/SuperGLUE
Large: 385M (disentangled)
position encoding loss) tasks
Replaces MLM with
RTD (Discriminator
Encoder-only Replaced Token
ELECTRA Base: 110M trains on corrupted Efficient pretraining
(Discriminator) Detection (RTD);
input)
more efficient

GLUE stands for General Language Understanding Evaluation

BERT models
Key Differences / Pretraining
Model Size / Params Architecture Use Case Highlights
Improvements Objective
Multilingual BERT
mBERT Base: 110M Encoder-only MLM + NSP Cross-lingual tasks
(104 languages)
RoBERTa-based Trained specifically
CamemBERT Base: 110M MLM only French NLP
(French) for French corpus

Application specific recommendations

Goal Recommended BERT Variant

General NLP tasks RoBERTa, BERT
Low-resource / mobile DistilBERT, TinyBERT
Multilingual tasks XLM-R, mBERT
Biomedical or clinical NLP BioBERT, ClinicalBERT
High accuracy (with compute) DeBERTa, ELECTRA
Coreference or span QA SpanBERT