Introduction to Large Language

Models (LLMs)
saaravanangcp@[Link]
R1I8XMOA2V

LARGE LANGUAGE MODELS

Chapter 1
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Is this AI?

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Generative AI
• Traditional AI (TA):
– Focuses on analyzing existing data, performing specific
tasks, and making predictions based on predefined rules
and algorithms. It excels at pattern recognition and
decision-making within a defined scope.
saaravanangcp@[Link]
R1I8XMOA2V
• Generative AI (GenAI):
– Creates new content, like text, images, or code, by
learning patterns from data and generating novel
outputs. It's more adaptable and creative, capable of
producing outputs not explicitly present in the training
data.

This file is meant for personal use by saaravanangcp@[Link] only.

3
Sharing or publishing the contents in part or full is liable for legal action.
saaravanangcp@[Link]
R1I8XMOA2V

Beginning of everything

LANGUAGE MODEL

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Language Model
• At its core, a Language Model (LM) is a statistical tool that
calculates the probability of a sequence of words occurring in
a particular order. More simply, it tries to predict the next
word in a sequence given the words that have come before it.
saaravanangcp@[Link]
R1I8XMOA2V

• What it does:
– Assigns a probability score to a given sequence of words.
– Predicts the most likely next word (or words) in a sentence.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Language Model
• "The patient felt much better after taking the
prescribed..."
saaravanangcp@[Link]
R1I8XMOA2V

medicine
*mountain
pills
*television
dosage

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Why word prediction?
Grammar and Style Correction:
• User Input: "The team will meat on Tuesday."
• LM Correction: "The team will meet on Tuesday."
saaravanangcp@[Link]
R1I8XMOA2V

Predictive Search / Query Expansion:

• User Input: "Best rest..."
• LM Prediction: "Best restaurants near me"

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Why word prediction?
Improving Machine Translation:
• Raw Translation : "She beautiful house living in."
• LM Refinement: "She is living in a beautiful house."
saaravanangcp@[Link]
R1I8XMOA2V

Voice Assistant Understanding:

• User Spoken Input : "Set an alarm for ate o'clock."
• LM Interpretation: "Set an alarm for 8 o'clock.“

• Many more….
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Language Models: A History
Early Beginnings (1940s-1980s):
• Early models were often rule-based, relying on manually crafted
grammatical rules. These were rigid and struggled with ambiguity and
scale.
The Rise of Statistical Models (1980s-Early 2010s):
saaravanangcp@[Link]
• The breakthrough came with the shift towards statistical methods.
R1I8XMOA2V

Instead of hand-coding rules, models learned patterns from large text

corpora (collections of text).
• N-gram models emerged as the dominant approach, leveraging
Markov assumptions to simplify probability calculations. These were
simpler, more robust, and could be trained on vast amounts of data.
• They were the workhorse for applications like speech recognition
throughout thisThis
period.
file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Language Models: A History
The Neural Revolution (Mid-2010s - Present):
• The advent of powerful neural networks (especially Recurrent
Neural Networks like LSTMs and later, Transformers)
revolutionized LMs.
saaravanangcp@[Link]
R1I8XMOA2V

• Neural LMs can capture much longer-range dependencies and

complex linguistic patterns than their statistical predecessors.
• This led to the development of highly sophisticated models
like Word2Vec, ELMo, BERT, GPT, and many more, which have
dramatically improved performance across nearly all NLP
tasks.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Probabilistic Language Models
At their core, all language models are probabilistic. They assign a
probability to a sequence of words.

Let's consider a sequence of words W=w1​,w2​,…,wN​.

saaravanangcp@[Link]
R1I8XMOA2V

A probabilistic language model aims to calculate the probability of

this entire sequence, P(W) or P(w1​,w2​,…,wN​).

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Probabilistic Language Models
Goal: compute the probability of a sentence or
sequence of words W:
• P(W) = P(w1,w2,w3,w4,w5…wn)
saaravanangcp@[Link]
R1I8XMOA2V

Related task: probability of an upcoming word:

• P(w5|w1,w2,w3,w4) or P(wn|w1,w2…wn-1)

An LM computes either of these:

• P(W) or P(wn|w1,w2…wn-1)

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Probabilistic Language Models
Could we just count and divide?
P(sky∣The clouds drifted across the)
• If we tried to calculate this directly by counting:
saaravanangcp@[Link]
R1I8XMOA2V
C(The clouds drifted across the)
C(The clouds drifted across the sky)​

• No! Still too many unique combinations!

• Even for a five-word sequence like "The clouds drifted across
the sky," the likelihood of seeing this exact sequence
repeatedly in any given training corpus is very low.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 How to compute P(W) or P(wn|w1, …wn-1)

How to compute the joint probability P(W):

P(The, cloud, drifted, sky, across)

saaravanangcp@[Link]
R1I8XMOA2V

Intuition: let’s rely on the Chain Rule of

Probability

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Reminder: The Chain Rule
• Recall the definition of conditional probabilities
• P(B|A) = P(A,B)/P(A) Rewriting: P(A,B) = P(A) P(B|A)

saaravanangcp@[Link]
R1I8XMOA2V

• More variables:
• P(A,B,C,D) = P(A) P(B|A) P(C|A,B) P(D|A,B,C)
• The Chain Rule in General
• P(x1,x2,x3,…,xn) = P(x1)P(x2|x1)P(x3|x1,x2)…P(xn|x1,…,xn-1)
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Markov Assumption
The probability of a word appearing depends only on a fixed,
limited number of preceding words, rather than the entire
preceding sequence.
saaravanangcp@[Link]
R1I8XMOA2V

P(sky∣The clouds drifted across the)

≈ P(sky∣the)

P(wn+1|wn) ≈ P(wn|wn—1)
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action. Wikimedia commons
 Markov Assumption
For an N-gram model, this means the probability of the
current word (wi​) depends only on the previous N-1 words.

For a Bigram Model (N=2): The probability of wn​ depends only

on wn—1 ​.
saaravanangcp@[Link]
R1I8XMOA2V

P(wn|wn—1 ) ≈ P(wn|wn—1)

For a Trigram Model (N=3): The probability of wn depends only

on wn—2​ and wn—1.

P(w
This filenis|w n—1
meant for ) ≈ P(w
personal use by n|wn—2, wn—1) only.
saaravanangcp@[Link]
Sharing or publishing the contents in part or full is liable for legal action.
 Limitations of N Gram
N-grams can't handle long-distance dependencies

"The decision to implement these drastic policy changes, which faced

strong opposition from various stakeholders, was ultimately made by
saaravanangcp@[Link]
R1I8XMOA2V
the committee."
• Here, the singular verb "was" needs to agree with the singular
subject "decision."
• An N-gram model, with its limited window, might see "stakeholders
was" or "committee was" and struggle to correctly predict "was"
because the subject "decision" is many words away.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Limitations of N Gram
N-grams don't do well at modeling new sequences with similar
meanings

• If an N-gram model has often seen "The student understood the

saaravanangcp@[Link]
R1I8XMOA2V

lesson," it will assign a high probability to "student understood."

• However, if it has never encountered "The pupil grasped the
concept" in its training data, it might assign a very low or zero
probability to "pupil grasped," even though "student" and
"pupil" are synonyms, and "understood" and "grasped" are
similar in context.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
• Large Language Models (like BERT, GPT, Gemini) were
developed to overcome these fundamental limitations of
N-gram models, primarily due to their underlying architecture
(e.g., the Transformer model) and the way they represent
saaravanangcp@[Link]
R1I8XMOA2V
words.

• It Can
– Handle Much Longer Contexts:
– Model Synonymy & Generate Better Novel Strings
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
• Popular LLMs Today
• GPT-4 (OpenAI), Claude (Anthropic), Gemini
(Google), Mistral.

• Capabilities
saaravanangcp@[Link]
R1I8XMOA2V
• Human-like text generation, content creation,
coding, and reasoning.

• Advancements
• Better context understanding, memory, and
efficiency.

This file is meant for personal use by saaravanangcp@[Link] only.

22
Sharing or publishing the contents in part or full is liable for legal action.
 Multimodal AI (Text, Image, Video, Audio)
• What is Multimodal AI?
AI that processes multiple data
types (text, images, video, speech).

• Latest Developments GPT-4V (vision

saaravanangcp@[Link]
R1I8XMOA2V
capabilities).

• Gemini 1.5 (strong multimodal

processing).

• Sora (AI video generation).

This file is meant for personal use by saaravanangcp@[Link] only.

23
Sharing or publishing the contents in part or full is liable for legal action.
 AI-Generated Images & Art
• Popular Tools
– Midjourney, DALL·E 3, Stable
Diffusion.

• Use Cases
saaravanangcp@[Link]
R1I8XMOA2V

– Digital art, marketing, gaming,

design, and advertising.

• Advancements
– Higher resolution, more control, and
better consistency in outputs.
This file is meant for personal use by saaravanangcp@[Link] only.
24
Sharing or publishing the contents in part or full is liable for legal action.
 AI-Generated Video
• Sora / Veo– Creates high-quality
videos from text prompts.

• Runway AI – Used in filmmaking

and content creation.
saaravanangcp@[Link]
R1I8XMOA2V

• Applications Film production,

advertising, animation, and
education.

This file is meant for personal use by saaravanangcp@[Link] only.

25
Sharing or publishing the contents in part or full is liable for legal action.
 AI in Music & Audio
• AI Music Generation Tools Suno, Stable Audio, AIVA,
Boomy.

• How AI is Changing Music?

saaravanangcp@[Link]

– AI-assisted composition, remixing, and sound

R1I8XMOA2V

enhancement.

• Impact
– Revolutionizing background music for videos, games, and
creative work.
This file is meant for personal use by saaravanangcp@[Link] only.
26
Sharing or publishing the contents in part or full is liable for legal action.
 AI in Gaming & Virtual Worlds
• AI-powered NPCs with dynamic interactions.
• AI-assisted game level and environment design.
• AI-generated textures, animations, and game
scripts.
saaravanangcp@[Link]
R1I8XMOA2V
• Examples
– Unreal Engine’s MetaHumans (realistic
character creation).
– AI-generated narratives in RPGs.

This file is meant for personal use by saaravanangcp@[Link] only.

27
Sharing or publishing the contents in part or full is liable for legal action.
 AI in Healthcare
• AI Applications in Medicine
– AI-assisted drug discovery (reducing
development time).
– AI-enhanced medical imaging (faster
& more accurate diagnosis).
saaravanangcp@[Link]
R1I8XMOA2V – AI-powered personalized treatment
plans.

• Recent Advancements
– AI models predicting diseases more
accurately than doctors in some
cases.

This file is meant for personal use by saaravanangcp@[Link] only.

28
Sharing or publishing the contents in part or full is liable for legal action.
saaravanangcp@[Link]
R1I8XMOA2V

AI AGENTS TAKES THE SHOW

This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Challenges & Ethical Concerns
• Major Concerns Misinformation & Deepfakes:
– AI-generated fake news and altered media.
• Bias in AI Models:
– AI reflecting societal biases.
• Data Privacy Issues:
saaravanangcp@[Link]
R1I8XMOA2V

– AI using personal data.

• Copyright & Ownership:
– Who owns AI-generated content?
• Regulatory Actions
– AI laws and guidelines being developed globally.

This file is meant for personal use by saaravanangcp@[Link] only.

30
Sharing or publishing the contents in part or full is liable for legal action.
 Future of Generative AI
• What’s Next?
– More personalized AI assistants.
– AI reasoning & decision-making improvements.
saaravanangcp@[Link]
R1I8XMOA2V
– AI helping with scientific discoveries.
• Long-Term Impact
– AI becoming an essential part of daily life, work, and
research.

This file is meant for personal use by saaravanangcp@[Link] only.

31
Sharing or publishing the contents in part or full is liable for legal action.
 Prompts

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

32
Sharing or publishing the contents in part or full is liable for legal action.
 Prompt Engineering
• At its heart, prompt engineering is akin to teaching a child
through questions. Just as a well-phrased question can guide
a child's thought process, a well-crafted prompt can steer an
AI model, especially a Large Language Model (LLM), towards a
saaravanangcp@[Link]
R1I8XMOA2V
specific output.

• Prompt engineering is the bridge ensuring effective human-AI

communication. It's not just about getting the right answer;
it's about ensuring AI understands the context, the nuances,
and the intent behind every query.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Art of Prompt engineering- 101
• Start by clearly defining your objective, then be as specific as
possible, providing detailed information and examples where
relevant, breaking down complex tasks into smaller parts, and
iteratively refining your prompts based on the AI's responses,
saaravanangcp@[Link]
R1I8XMOA2V
constantly checking for clarity and adjusting as needed.

• Essentially, treat prompt engineering as a conversation where

you guide the AI towards the desired output through precise
instructions.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Be Specific as possible
• Detailed context: Provide the AI with enough background information to understand the
scenario you’re inquiring about. This includes the subject matter, scope, and any relevant
constraints.
• Desired format: Clearly specify the format in which you want the information to be
presented, whether it’s a list, a detailed report, bullet points, or a summary. Mention any
structural preferences, such as headings, subheadings, or paragraph limits.
• Output length: Detail how long you want the AI’s response, whether “3 paragraphs” or
saaravanangcp@[Link]
R1I8XMOA2V
“250 words.”
• Level of detail: Indicate the level of detail required for the response, from high-level
overviews to in-depth analysis, to ensure the model’s output matches your informational
needs.
• Tone and style: Request the preferred tone and style, whether it’s formal,
conversational, persuasive, or informational, to make sure the output aligns with your
intended audience or purpose.
• Examples and comparisons: Ask the AI to include examples, analogies, or comparisons to
clarify complex concepts or make the information more relatable and easily understood.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Prompt Engineering 101
• Supply the AI with examples
• Specify your desired output
• Provide instructions on what to do instead of
saaravanangcp@[Link]
R1I8XMOA2V

what not to do
• Give the model a persona or frame of reference
• Try chain of thought prompting
• Split complex tasks into simpler ones
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Example
• Please provide an outline for a comprehensive report that
analyzes the current trends in social media marketing for
technology companies, focusing on the developments from
2020 onward.
saaravanangcp@[Link]
R1I8XMOA2V

• The outline should include an introduction, three main

sections addressing different aspects of social media
trends, and a conclusion summarizing the findings. Please
suggest the types of graphs that could illustrate user
engagement trends and list bullet points that summarize
key marketing strategies in each section.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Example

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Example
• I went to the market and bought 10 apples. I gave 2
apples to the neighbor and 2 to the repairman. I then
went and bought 5 more apples and ate 1. How many
apples did I remain with?
saaravanangcp@[Link]
R1I8XMOA2V

• I went to the market and bought 10 apples. I gave 2

apples to the neighbor and 2 to the repairman. I then
went and bought 5 more apples and ate 1. How many
apples did I remain with? Let's think step by step.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Prompt Engineering
• Goal: Create a short story about a robot
experiencing human emotions, specifically
romantic feelings for a human character.
saaravanangcp@[Link]
R1I8XMOA2V

• "Write a story about a robot who

unexpectedly falls in love with a human."

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Prompt Engineering
• Goal: Create a short story about a robot experiencing human
emotions, specifically romantic feelings for a human
character.
saaravanangcp@[Link]
R1I8XMOA2V

• "Write a short story titled 'Circuitry of the Heart' where a

highly advanced domestic robot named 'AX-7' begins to feel a
deep connection with its human owner, a young programmer
named Sarah, while noticing subtle human behaviors and
emotions in her daily routine. Describe the internal conflict
AX-7 experiences as its robotic programming clashes with
these emerging feelings."
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
saaravanangcp@[Link]
R1I8XMOA2V

Challenges faces and Methods used

MODEL EVALUATION

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Need for Model Evaluation

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Need for Model Evaluation

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Train-Test Split

Test
Train
saaravanangcp@[Link]
R1I8XMOA2V

Val

Dataset
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Standard Metrics

Data Ground Truth Prediction

saaravanangcp@[Link]
R1I8XMOA2V
Data 1 Class A Class B
Data 2 Class B Class B
Data 3 Class A Class A
Data 4 Class B Class A

[Link]
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Standard Metrics

Correct Predictions
Accuracy =
saaravanangcp@[Link]
R1I8XMOA2V
Total Predictions

I love this class

Accuracy =
I love this class
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Challenges Faced

saaravanangcp@[Link]
R1I8XMOA2V

Freitag et al. (2020). BLEU might be Guilty but References are not Innocent

There can be many semantically equivalent but syntactically

different ways of expressing the same idea.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Challenges Faced
Semantic Equivalence
Deep really loves Dessert = Deep adores Dessert
saaravanangcp@[Link]
R1I8XMOA2V

Semantic Contrast
He did not pass the exam ≠ He did pass the exam

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Model Evaluation – Standard methods
• Perplexity
• ROUGE (Recall-Oriented Understudy for
Gisting Evaluation)
saaravanangcp@[Link]
R1I8XMOA2V

• BLEU (Bilingual Evaluation Understudy)

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity

saaravanangcp@[Link]
R1I8XMOA2V

Don't confuse it with ChatGPT's "rival" – we're talking about Perplexity!

Perplexity is a crucial metric used to evaluate how effectively a language

model predicts a sequence of words. What's particularly useful is that it
doesn't require ground truth – making it a versatile and powerful tool.
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
"The sky is blue, the grass is green, and the sun is a bright _____“

"Twinkle, twinkle, little _____“

saaravanangcp@[Link]
R1I8XMOA2V

In essence, Perplexity measures how “Perplexed / Surprised" a

language model is by new data.

A model that is less surprised is a better model.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
• A perplexity score of 1 indicates that the model predicted the
sequence with perfect accuracy.

• Conversely, a perplexity of 7 suggests that the model, on

saaravanangcp@[Link]
R1I8XMOA2V

average, considered 7 equally probable alternatives at each

prediction point, signifying a higher degree of uncertainty.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
• Perplexity is fundamentally related to the probability a language
model assigns to a sequence of words.
• The general formula for perplexity (PP) of a sequence of words
W=w1​,w2​,...,wN​ is:
saaravanangcp@[Link]
R1I8XMOA2V

Where:
• P(w1​,w2​,...,wN​) is the probability assigned by the language model
to the entire sequence of N words.
• N is the number ofis meant
This file words in use
for personal the sequence. only.
by saaravanangcp@[Link]
Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
Sentence: "The cat sat" (So, N=3 words)

Assumed Language Model Probabilities

saaravanangcp@[Link]
R1I8XMOA2V
(Hypothetical):
• P(The) = 0.05
• P(cat | The) = 0.40
• P(sat | The cat) = 0.70
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
Step 1 - Calculate the logarithm
For w1​="The": log P(The) =log(0.05) ≈−2.9957
For w2​="cat": log P(cat | The)
saaravanangcp@[Link]
R1I8XMOA2V
=log(0.40) ≈−0.9163
For w3​="sat": log P(sat | The cat) =log(0.70) ≈−0.3567

Step 2 - Sum the log probabilities

Sum =(−2.9957)+(−0.9163)+(−0.3567) ≈−4.2687
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Perplexity
Step 3 - Calculate the logarithm
1
− ​×(−4.2687) ≈1.4229
3
saaravanangcp@[Link]
R1I8XMOA2V

Step 4 - Apply the exponential function

Perplexity = exp(1.4229)
Perplexity ≈ 4.148
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 LLM Evaluation - Metrics - Terminology

n-gram

saaravanangcp@[Link]
R1I8XMOA2V

Deep really loves Dessert

bigram unigram

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE
• Recall-Oriented Understudy for Gisting
Evaluation
saaravanangcp@[Link]
R1I8XMOA2V

• ROUGE-1
• ROUGE-2
• ROUGE-L
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-1
Reference (human):

This dish is good

saaravanangcp@[Link]
R1I8XMOA2V

Generated output:

This dish is very good

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-1
Reference (human):

This dish is good

saaravanangcp@[Link]
R1I8XMOA2V

Generated output:

This dish is not good

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-2
Reference (human):

This dish is good

This dish
saaravanangcp@[Link]
R1I8XMOA2V dish is is good

Generated output:

This dish is very good

This dish dish is is very very good
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-2
Reference (human):

This dish is good

This dish dish is
saaravanangcp@[Link]
R1I8XMOA2V
is good

Generated output:

This dish is very good

This dish dish is

is very very good

This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-L
Reference (human):

This dish is good

saaravanangcp@[Link]

Generated output:
R1I8XMOA2V

This dish is very good

Longest common subsequence (LCS):
This dish
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-L
Reference (human):

He feels very well

saaravanangcp@[Link]

Generated output:
R1I8XMOA2V

He feels quite very well

Longest common subsequence (LCS):
He feels very well
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE-L
Reference (human):

He feels very well

saaravanangcp@[Link]
R1I8XMOA2V

Generated output:

He feels quite very well

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE hacking
Reference (human):

He feels very well

saaravanangcp@[Link]

Generated output:
R1I8XMOA2V

well well well well

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 ROUGE hacking
Reference (human):

He feels very well

saaravanangcp@[Link]
R1I8XMOA2V

Generated output:

well well well well

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 BLEU
• BLEU stands for BiLingual Evaluation Understudy.

• It is a widely used automatic metric for evaluating the quality

of machine-translated text.
saaravanangcp@[Link]
R1I8XMOA2V

• Its core idea is to measure the similarity between a machine-

generated translation (the "candidate") and one or more high-
quality human translations (the "references").

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 BLEU

Where:
• BP is the Brevity Penalty.
saaravanangcp@[Link]
R1I8XMOA2V

• N is the maximum n-gram size considered (e.g., 4 for BLEU-4).

• wn​ are positive weights for each n-gram precision (typically
wn​=1/N for uniform weighting, e.g., 0.25 for BLEU-4).
• Pn​ is the clipped n-gram precision for n-gram length 'n'.

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 BLEU
• N-gram Precision (Pn​):

saaravanangcp@[Link]
R1I8XMOA2V

• Brevity Penalty (BP):

c is the length of the candidate translation.

r is the effectiveThisreference
file is meant forlength
personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 BLEU
BLEU Metric = Avg(precision across range of n-gram sizes) * Brevity Penalty

Reference (human): "The capital city of France is Paris, a beautiful city.“

Generated outputs and their (illustrative) BLEU Scores:

saaravanangcp@[Link]
R1I8XMOA2V

– "The capital of France is beautiful." - BLEU 0.350

(Missing "city," "Paris," "city." Significant information loss.)
– "The capital city of France is Paris." - BLEU 0.680
(Closer, but misses the descriptive "a beautiful city.")
– "The capital city of France is Paris, a beautiful." - BLEU 0.850
(Very close, only missing the final "city.")
– "The capital city of France is Paris, a beautiful city." - BLEU 1.000
(Perfect match to the reference.)
This file is meant for personal use by saaravanangcp@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
 Challenges Faced
Semantic Equivalence
Deep really loves Dessert = Deep adores Dessert
saaravanangcp@[Link]
R1I8XMOA2V

Semantic Contrast
He did indeed pass the exam ≠ He did pass the exam

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
 Need a model for Evaluation

saaravanangcp@[Link]
R1I8XMOA2V

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.
saaravanangcp@[Link]
R1I8XMOA2V

End of Slide

THANK YOU

This file is meant for personal use by saaravanangcp@[Link] only.

Sharing or publishing the contents in part or full is liable for legal action.