IIT Jodhpur — BSc Applied AI & Data Science

Pattern Recognition Principles

Comprehensive Study Notes — Part 2

Weeks 4–7 | Bayesian Decision Theory · Discriminant Functions · Naive Bayes · Evaluation Metrics

Understanding First · Memorization Second · Problem-Solving Third

Wee
k4 Bayesian Decision Theory & Risk

Wee
k5 Discriminant Functions & Parameter Estimation

Wee
k6 Naive Bayes Classification

Wee
k7 Performance Evaluation & Metrics

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 1

 Table of Contents

Table of Contents 2
1. Bayesian Decision Theory & Risk 4
1.1 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
1.2 Bayes Theorem (The Foundation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 The MAP Decision Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Beyond MAP — Cost & Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 The Loss Function λ(ω_i | ω_j) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 The Risk (Expected Loss) R(ω_i | x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.7 Zero-One Loss — The MAP Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.8 Solved Example — Risk Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2. Discriminant Functions & Parameter Estimation 7
2.1 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
2.2 The Discriminant Function g_i(x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Decision Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Parameter Estimation — MLE vs MAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 Solved Example — Discriminant Function Decision . . . . . . . . . . . . . . . . . . . . . . . . 8
3. Naive Bayes Classification 10
3.1 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2 The Curse of Dimensionality Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 The Naive Assumption — The Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.4 The Naive Bayes Decision Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.5 Why 'Naive' Still Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6 Application: Spam Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.7 Solved Example — Naive Bayes Spam Classification . . . . . . . . . . . . . . . . . . . . . . 11
4. Performance Evaluation & Metrics 13
4.1 Key Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 The Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.3 Key Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.4 Precision vs Recall — The Fundamental Trade-off . . . . . . . . . . . . . . . . . . . . . . . . 14
4.5 Why F1-Score? The Imbalanced Data Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.6 Solved Example — Full Metrics Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5. Full Topic Connections & Big Picture 16
5.1 The Decision Rule Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Master Formula Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.3 All Memory Lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
6. Exam-Style Practice Questions & Answers 18
6.1 Section A — Conceptual (MCQ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 2

 6.2 Section B — Numerical Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
6.3 Section C — Key Exam Short Answer Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 3

 1. Bayesian Decision Theory & Risk
Bayesian Decision Theory (BDT) is the gold standard framework for classification — it tells us the
mathematically optimal way to make decisions under uncertainty. Everything else in ML tries to
approximate what BDT achieves exactly.

1.1 Key Terms

Term Meaning

Prior P(ω_i) How likely class ω_i is BEFORE seeing any data. Your initial belief.

Likelihood P(x|ω_i) Probability of seeing data x IF the true class is ω_i. The class model.

Evidence P(x) Overall probability of seeing x. Same for all classes — acts as a normalizer.

Posterior P(ω_i|x) Updated belief: probability of class ω_i AFTER seeing data x. This is what we
maximize.

Loss λ(ω_i|ω_j) Penalty/cost when we decide class ω_i but true class is ω_j.

Risk R(ω_i|x) Expected (average) loss when we assign input x to class ω_i.

MAP Rule Choose class with highest posterior — most common decision rule.

Zero-One Loss All errors cost 1, all correct = 0. Makes MAP the optimal rule.

BDT General Rule Choose class that MINIMIZES expected risk. MAP is a special case.

1.2 Bayes Theorem (The Foundation)

Before we can make a decision, we need to compute the posterior probability for each class using
Bayes' Theorem:

P(ω_i | x) = [ P(x | ω_i) × P(ω_i) ] / P(x)

Formula Intuition: Posterior = (Likelihood × Prior) / Evidence

Think of it like a courtroom: Prior = your initial hunch about guilt. Likelihood = how well the evidence fits that
hypothesis. Posterior = updated verdict after seeing the evidence. Evidence just ensures probabilities sum to
1 — ignore it when comparing classes!

1.3 The MAP Decision Rule

The simplest and most used decision rule: pick the class that has the highest posterior probability.

MAP Rule: Choose ω_i if P(ω_i|x) ≥ P(ω_j|x) for all j ≠ i

Since P(x) is same for all classes, we only compare the numerator:

Choose ω_i that maximizes: P(x|ω_i) × P(ω_i) [Likelihood × Prior]

Why drop P(x)? We are comparing classes for the SAME input x. Since P(x) is identical for all classes,
dividing by it doesn't change which class wins. It's like asking 'who is taller?' — you don't need to know the unit
of measurement if you measure both in the same unit.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 4

 1.4 Beyond MAP — Cost & Risk
The MAP rule assumes all errors are equally bad. Reality disagrees:

Scenario False Negative (FN) False Positive (FP) Which is worse?

Medical Diagnosis Miss disease → patient untreated Healthy flagged → extra tests FN much worse!

Spam Filter Spam reaches inbox Legit email deleted! FP worse for user

Bomb Detection Miss actual bomb Evacuate for nothing FN catastrophically worse

1.5 The Loss Function λ(ω_i | ω_j)

Defines the penalty incurred when the decision is ω_i but the true class is ω_j.

Notation Meaning

λ(ω_i | ω_i) = 0 Correct decision → zero cost (always)

λ(ω_1 | ω_2) = λ_12 Decided Class 1, but true class is Class 2 → Type I Error cost

λ(ω_2 | ω_1) = λ_21 Decided Class 2, but true class is Class 1 → Type II Error cost

1.6 The Risk (Expected Loss) R(ω_i | x)

The risk of assigning input x to class ω_i is the average penalty across all possible true classes, weighted
by how likely each true class actually is:

R(ω_i | x) = Σ_j λ(ω_i | ω_j) × P(ω_j | x)

Formula Intuition: 'If I call this class ω_i, what is the average damage I expect to cause?'

You loop over every possible true class ω_j. For each, you ask: (1) how much does it cost if I'm wrong in this
specific way? [loss] × (2) how likely is this true class? [posterior]. Sum all of these up = total expected damage
= Risk.

Bayes Decision Rule (General): Choose class ω_i that MINIMIZES R(ω_i | x)

1.7 Zero-One Loss — The MAP Connection

When all mistakes are equally bad (cost = 1) and correct decisions cost 0:

Decision True Class Loss λ

ω_i ω_i (correct) 0

ω_i ω_j (wrong) 1

Under Zero-One Loss: R(ω_i | x) = Σ_{j≠i} P(ω_j | x) = 1 - P(ω_i | x)

Minimizing R(ω_i|x) = Minimizing [1 - P(ω_i|x)] = Maximizing P(ω_i|x)

Therefore: MAP is just a special case of BDT when you use Zero-One Loss. The hierarchy is: BDT
General Rule → (with Zero-One Loss) → MAP Rule.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 5

 Memory Line: 'MAP = BDT with fair grading (0 or 1). Real life? Grading is unfair — use full
BDT.'

1.8 Solved Example — Risk Calculation

EXAM-STYLE SOLVED EXAMPLE

Problem:

A medical classifier has two classes: ω_1 = Sick, ω_2 = Healthy. For a new patient x, the posteriors are:
P(ω_1|x) = 0.3, P(ω_2|x) = 0.7. Loss matrix: λ(ω_1|ω_1) = 0, λ(ω_2|ω_1) = 10 (miss disease!), λ(ω_1|ω_2) = 1
(unnecessary alarm), λ(ω_2|ω_2) = 0. Which class should we pick?

Step 1: Calculate risk of deciding ω_1 (Sick)

R(ω_1|x) = λ(ω_1|ω_1)×P(ω_1|x) + λ(ω_1|ω_2)×P(ω_2|x)

R(ω_1|x) = 0×0.3 + 1×0.7 = 0.7

Step 2: Calculate risk of deciding ω_2 (Healthy)

R(ω_2|x) = λ(ω_2|ω_1)×P(ω_1|x) + λ(ω_2|ω_2)×P(ω_2|x)

R(ω_2|x) = 10×0.3 + 0×0.7 = 3.0

Step 3: Decision

R(ω_1|x) = 0.7 < R(ω_2|x) = 3.0 → Choose ω_1 = SICK

Note: MAP would have chosen ω_2 (Healthy) since P(ω_2|x)=0.7 > P(ω_1|x)=0.3! But the asymmetric cost of
missing a disease overrides the raw probability. This is exactly why BDT is more powerful than MAP.

Topic Connection: Week 4 → Week 7

The Loss Function here is directly linked to the metrics you learn in Week 7. High λ for FN → your system is
optimized for high Recall. High λ for FP → your system is optimized for high Precision. BDT is the theoretical
foundation; Precision/Recall are the practical measurement tools.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 6

 2. Discriminant Functions & Parameter Estimation
Week 5 bridges theory and practice: how do we actually implement BDT efficiently? The answer is
Discriminant Functions. And how do we get the class distributions in the first place? Parameter Estimation
— MLE and MAP.

2.1 Key Terms

Term Meaning

Discriminant Function g_i(x) A scoring function for each class. The class with the highest score wins.

Decision Surface The boundary in feature space where two discriminant functions are equal:
g_1(x) = g_2(x).

Hyperplane A flat decision surface (line in 2D, plane in 3D). Result of a LINEAR
discriminant function.

MLE Maximum Likelihood Estimation: finds parameters θ that make observed data
most probable.

MAP Estimation Incorporates a prior belief about θ. Balances data evidence with prior
knowledge.

µ (mu) Mean vector — center of the Gaussian distribution for a class.

Σ (Sigma) Covariance matrix — spread/shape of the Gaussian distribution for a class.

Sample Mean MLE estimate of µ: just the average of all training points for that class.

2.2 The Discriminant Function g_i(x)

Instead of computing full posteriors and comparing, we define a score function g_i(x) for each class.
Classification becomes: compare scores, pick the winner.

Assign x to class ω_i if g_i(x) ≥ g_j(x) for all j ≠ i

Three equivalent choices for g_i(x):

Option Formula for g_i(x) Notes

1 (Posterior) P(ω_i | x) Direct posterior. Most intuitive.

2 (Likelihood × Prior) P(x | ω_i) × P(ω_i) Drop P(x) — doesn't affect ranking.

3 (Log-Likelihood) ln[P(x|ω_i) × P(ω_i)] Preferred! Turns products into sums. Avoids numerical underflow.

WHY LOG?

Why use log? When features are independent (Naive Bayes), the likelihood is a product of many small
probabilities: P(x_1|ω)×P(x_2|ω)×...×P(x_D|ω). With many features, this product becomes astronomically
small (numerical underflow → rounds to 0). Taking log converts the product into a SUM: ln P(x_1|ω) + ln
P(x_2|ω) + ... + ln P(x_D|ω), which is numerically stable.

2.3 Decision Surfaces

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 7

 The decision surface between classes ω_1 and ω_2 is where you are equally uncertain — it is the
boundary that separates the feature space.

Decision Surface: g_1(x) = g_2(x) [the boundary between classes]

Discriminant Type Decision Surface Shape Example

Linear in x Hyperplane (line/plane/flat boundary) LDA, Perceptron

Quadratic in x Non-linear (ellipse, parabola, hyperbola) QDA, Gaussian with different covariances

2.4 Parameter Estimation — MLE vs MAP

When we assume our class distributions are Gaussian (very common), we need to estimate µ and Σ from
training data. Two approaches:

Maximum Likelihood Estimation (MLE)

Find parameters θ that make the observed data as likely as possible. Logic: 'The data we saw is the most
typical data — find the distribution that explains it best.'

θ_MLE = argmax_θ P(Data | θ) → For Gaussian: µ_MLE = (1/n) Σ x_i [sample mean]

Maximum A Posteriori (MAP) Estimation

Incorporate prior belief about θ. Especially useful when you have limited data.

θ_MAP = argmax_θ P(θ | Data) ∝ P(Data | θ) × P(θ) [Likelihood × Prior on θ]

Aspect MLE MAP Estimation

What it maximizes P(Data | θ) — Likelihood only P(θ | Data) ∝ P(Data|θ)×P(θ)

Uses prior on θ? NO YES — P(θ)

With lots of data Very accurate Prior effect shrinks — approaches MLE

With little data Can overfit Prior stabilizes the estimate

Analogy Just trust the data Trust data + your prior experience

Memory Line: 'MLE = trust the data fully. MAP = data + your prior wisdom. When data is
scarce, wisdom matters more.'

2.5 Solved Example — Discriminant Function Decision

SOLVED EXAMPLE

Problem: Two 1D Gaussian classes. Both have equal prior P(ω_1)=P(ω_2)=0.5 and equal variance σ²=1.
Means: µ_1=1, µ_2=4. Classify x=2.5 using the log-discriminant function.

Step 1: Log-discriminant for Gaussian (equal prior, equal σ²) simplifies to:

g_i(x) = -(x - µ_i)² / (2σ²) + ln P(ω_i)

Since priors are equal, they cancel. We just compare -(x - µ_i)²:

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 8

 Step 2: Compute scores for x = 2.5:

g_1(2.5) = -(2.5 - 1)² = -(1.5)² = -2.25

g_2(2.5) = -(2.5 - 4)² = -(1.5)² = -2.25

Result: g_1 = g_2 → x=2.5 is exactly on the decision boundary! (Makes sense: midpoint of µ_1=1 and
µ_2=4 is 2.5.)

Topic Connections:

• Week 4 → Week 5: BDT tells us to maximize posterior. Discriminant functions ARE the posterior (or a
monotone transform of it) — they're the computational tool to apply BDT.

• Week 5 → Week 6: MLE is used inside Naive Bayes to estimate each P(x_j|ω_i).

• Week 5 → Linear Algebra: Equal covariance Gaussians → hyperplane boundary. Different covariances →
quadratic (curved) boundary.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 9

 3. Naive Bayes Classification
Full Bayesian classification is often computationally impossible in high dimensions — the joint likelihood
P(x_1, x_2, ..., x_D | ω_i) requires exponentially more data as D grows. Naive Bayes solves this elegantly
with one bold assumption.

3.1 Key Terms

Term Meaning

Curse of Dimensionality As number of features D grows, the data needed to estimate the joint
distribution grows exponentially.

Joint Likelihood P(x_1, x_2, ..., x_D | ω_i) — the full probability of observing all features
together, given a class.

Conditional Independence Features x_1...x_D are assumed independent GIVEN the class. Key: 'given
the class' is crucial.

Naive Assumption The 'naive' simplification: treat features as conditionally independent given
the class.

Individual Likelihood P(x_j | ω_i) — probability of a single feature value given the class.

Bernoulli NB For binary features (0/1). Used when features represent presence/absence.

Gaussian NB For continuous features. Models each P(x_j | ω_i) as a 1D Gaussian.

Multinomial NB For count features (word frequencies in text). Classic spam filter model.

Laplace Smoothing Add-1 trick: add 1 to all counts to avoid P(word|class) = 0 for unseen words.

3.2 The Curse of Dimensionality Problem

For a classification problem with D features, estimating the joint likelihood P(x_1,...,x_D | ω_i) requires:

If each feature has k discrete values: k^D parameter combinations to estimate!

D=10 features, k=10 values each → 10^10 = 10 BILLION combinations per class!

3.3 The Naive Assumption — The Solution

Assume all features are conditionally independent given the class:

P(x_1, x_2, ..., x_D | ω_i) ≈ P(x_1|ω_i) × P(x_2|ω_i) × ... × P(x_D|ω_i) = Π_j P(x_j|ω_i)

What this saves: Instead of estimating 1 joint distribution over D features, you estimate D separate 1D
distributions. For D=10, k=10: instead of 10^10 parameters, you only need D×k = 10×10 = 100 parameters.
Reduction factor: 10^8 (100 million times fewer!).

3.4 The Naive Bayes Decision Rule

Substitute the independence assumption into the standard MAP rule:

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 10

 Choose ω_i that maximizes: P(ω_i) × Π_{j=1}^{D} P(x_j | ω_i)

In log form (preferred in practice):

Choose ω_i that maximizes: ln P(ω_i) + Σ_{j=1}^{D} ln P(x_j | ω_i)

Formula Intuition (Log form):

Start with your prior (log P(ω_i)) — this is your 'base score'. Then for each feature, add the evidence it
provides (ln P(x_j|ω_i)). Positive features strengthen the score, negative features weaken it. The class with the
best total score wins. It's like a courtroom where each witness adds their vote.

3.5 Why 'Naive' Still Works

The conditional independence assumption is almost always false in the real world. So why does Naive
Bayes work so well?

Key Insight: The goal is CLASSIFICATION, not density estimation.

Even if our probability estimates are wrong, the RANKING of classes (which posterior is highest) is often
preserved. As long as Class A gets a higher score than Class B, the decision is correct — even if the exact
numbers are off. Naive Bayes gets the ordering right surprisingly often.

3.6 Application: Spam Filter

Naive Bayes is the classic spam classifier. Here's how it works:

Component In Spam Filter Example

Classes ω_1 = Spam, ω_2 = Ham —

Features x_j Presence/absence of word j x_1='discount', x_2='meeting'

Prior P(ω_1) Fraction of spam emails in training P(Spam)=0.3

Likelihood P(x_j|ω) How often word j appears in Spam/Ham emails P('discount'|Spam)=0.8, P('discount'|Ham)=0.02

Decision Assign class with higher posterior P(Spam|email) vs P(Ham|email)

Memory Line: 'Naive Bayes = Split the courtroom evidence. Each witness (feature) speaks
independently. Sum up all votes. Most votes wins. Unrealistic? Yes. Effective? Surprisingly
yes.'

3.7 Solved Example — Naive Bayes Spam Classification

EXAM-STYLE SOLVED EXAMPLE

Problem: Classify the email 'free discount meeting' as Spam or Ham.

Given: P(Spam)=0.4, P(Ham)=0.6

P('free'|Spam)=0.9, P('discount'|Spam)=0.8, P('meeting'|Spam)=0.1

P('free'|Ham)=0.05, P('discount'|Ham)=0.02, P('meeting'|Ham)=0.7

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 11

 Step 1: Proportional posteriors (drop P(email) denominator)

Score(Spam) = P(Spam) × P('free'|Spam) × P('discount'|Spam) × P('meeting'|Spam)

= 0.4 × 0.9 × 0.8 × 0.1 = 0.0288

Score(Ham) = P(Ham) × P('free'|Ham) × P('discount'|Ham) × P('meeting'|Ham)

= 0.6 × 0.05 × 0.02 × 0.7 = 0.00042

Decision: Score(Spam) = 0.0288 >> Score(Ham) = 0.00042 → Classify as SPAM ✓

Note: Even though 'meeting' is a Ham word, the strong spam signal from 'free' and 'discount' dominates.

Topic Connections:

• Week 4 → Week 6: Naive Bayes is just MAP rule with the independence assumption. Same posterior
maximization, simpler likelihood.

• Week 5 → Week 6: MLE is used to estimate each P(x_j|ω_i) from training data.

• Week 6 → Week 7: After building the Naive Bayes model, you evaluate it using Confusion Matrix, Precision,
Recall, F1.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 12

 4. Performance Evaluation & Metrics
Building a classifier is only half the job. The other half is measuring how well it actually works — not just
overall, but for each class, each type of error. Week 7 gives you the complete toolkit.

4.1 Key Terms

Term Meaning

Confusion Matrix A 2×2 table showing all four possible outcomes: TP, TN, FP, FN.

True Positive (TP) Model predicted Positive. Actual is Positive. Correct! ✓

True Negative (TN) Model predicted Negative. Actual is Negative. Correct! ✓

False Positive (FP) Model predicted Positive. Actual is Negative. Wrong! = Type I Error = False
Alarm.

False Negative (FN) Model predicted Negative. Actual is Positive. Wrong! = Type II Error = Miss.

Accuracy (TP+TN)/Total — overall correctness. Misleading on imbalanced data.

Precision TP/(TP+FP) — of all predicted positives, how many are real? 'Quality of
positives.'

Recall (Sensitivity) TP/(TP+FN) — of all real positives, how many did we catch? 'Coverage of
positives.'

F1-Score Harmonic mean of Precision and Recall. Best single metric for imbalanced
data.

Specificity TN/(TN+FP) — how good is the model at identifying the negatives?

Imbalanced Dataset When one class has far more samples than the other. Accuracy becomes
meaningless.

4.2 The Confusion Matrix

The confusion matrix is the foundation. Every metric is derived from its four cells.

Predicted: POSITIVE Predicted: NEGATIVE

TRUE POSITIVE (TP) FALSE NEGATIVE (FN)

Actual: POSITIVE
Correctly detected Missed! (Type II)

FALSE POSITIVE (FP) TRUE NEGATIVE (TN)

Actual: NEGATIVE
False alarm (Type I) Correctly rejected

Memory Line: 'True = model was RIGHT. False = model was WRONG. Positive/Negative = what
the model PREDICTED.'

4.3 Key Performance Metrics

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 13

 Metric Formula Answers the Question When to Use

Accuracy TP+TN / Total Overall, how often is the model Balanced datasets only
correct?

Precision TP / (TP+FP) Of all predicted positives, how When FP is costly (spam

many are truly positive? filter)

Recall TP / (TP+FN) Of all actual positives, how many When FN is costly (disease
(Sensitivity) did the model find? detection)

F1-Score 2×(P×R)/(P+R) Balanced metric between Precision Imbalanced datasets

and Recall.

Specificity TN / (TN+FP) Of all actual negatives, how many When TN matters too
were correctly rejected?

4.4 Precision vs Recall — The Fundamental Trade-off

You cannot simultaneously maximize both Precision and Recall. Increasing one typically decreases the
other. This is called the Precision-Recall trade-off:

Goal Prioritize Impact Real Example

Minimize false alarms HIGH Precision Accept more FN (miss some positives)
Spam filter: don't delete real emails

Catch everything real HIGH Recall Accept more FP (more false alarms)Cancer test: catch every sick patient

Balance both HIGH F1-Score Best harmonic mean of P and R General classification

4.5 Why F1-Score? The Imbalanced Data Problem

Suppose you have 990 negative samples and 10 positive samples (1% positive). A lazy model that always
predicts Negative achieves:

Metric Lazy Model (Always Negative) Good Model

Accuracy 990/1000 = 99% ← Deceptive! ~95% (honest)

Precision 0/0 = undefined (0 TP) High

Recall 0/10 = 0% ← Exposed! High

F1-Score ≈0% ← Correctly penalized High

F1 = 2 × (Precision × Recall) / (Precision + Recall) [Harmonic Mean — punishes extreme

imbalance]

WHY HARMONIC MEAN?

Why Harmonic Mean (not Arithmetic Mean)? Arithmetic mean: (1.0 + 0.0) / 2 = 0.50 — makes a model with
100% Precision but 0% Recall look 'decent'. Harmonic mean: 2×(1.0×0.0)/(1.0+0.0) = 0.0 — correctly
penalizes this model to 0. Harmonic mean is dominated by the SMALLER value — it forces both P and R to be
high.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 14

 Memory Line: 'Accuracy lies when classes are unequal. F1 always tells the truth because
Recall exposes lazy models.'

4.6 Solved Example — Full Metrics Calculation

EXAM-STYLE SOLVED EXAMPLE

Problem: A classifier is tested on 1000 samples. Results: TP=100, FP=10, FN=5, TN=885. Calculate all
metrics and comment on model performance.

Given: TP=100, FP=10, FN=5, TN=885, Total=1000

Accuracy = (TP+TN)/Total = (100+885)/1000 = 985/1000 = 98.5%

Precision = TP/(TP+FP) = 100/(100+10) = 100/110 = 90.9%

Recall = TP/(TP+FN) = 100/(100+5) = 100/105 = 95.2%

F1-Score = 2×(0.909×0.952)/(0.909+0.952) = 2×0.865/1.861 = 92.97% ≈ 93.0%

Interpretation: The model is excellent! High Recall (95.2%) means it catches most actual positives. High
Precision (90.9%) means most of its positive predictions are correct. Only 5 positive cases were missed
(FN=5) and only 10 were false alarms (FP=10).

Topic Connections — Full Picture:

• Week 4 BDT → Week 7: Minimizing FN cost (high λ for FN) = maximizing Recall. Minimizing FP cost =
maximizing Precision.

• Week 6 Naive Bayes → Week 7: After training NB on spam, you evaluate with Precision/Recall. High
Precision spam filter = fewer good emails lost.

• Validation Set: NEVER tune hyperparameters on test set — this causes data leakage. Use separate
train/val/test splits.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 15

 5. Full Topic Connections & Big Picture
How all four weeks connect into one coherent framework:

WEEK 4: Theory WEEK 5: Implementation WEEK 6: Scaling Up WEEK 7: Evaluation

BDT Framework Discriminant Functions Naive Bayes (NB) Confusion Matrix

↓ ↓ ↓ ↓
MAP Rule Log-Likelihood Scores Conditional Independence Precision, Recall
↓ ↓ ↓ ↓
Risk Minimization Decision Surfaces Product of Likelihoods F1-Score

'What is the optimal decision?'

'How do we compute it efficiently?'
'How do we handle many features?'
'How good is our classifier?'

5.1 The Decision Rule Hierarchy

Most General (Theory) Bayes Decision Rule: Minimize Expected Risk R(ω_i|x) using any loss function λ

Special Case 1 (Equal costs) MAP Rule: Maximize Posterior P(ω_i|x). Equivalent to Zero-One Loss.

Special Case 2 (No prior) MLE Classification: Maximize Likelihood P(x|ω_i) only. Equal priors assumed.

Practical NB Version Naive Bayes: MAP + Conditional Independence. Scales to high dimensions.

5.2 Master Formula Reference

Formula Name Expression

Bayes Theorem P(ω_i|x) = P(x|ω_i)×P(ω_i) / P(x)

MAP Rule Choose ω_i : max P(x|ω_i)×P(ω_i)

Expected Risk R(ω_i|x) = Σ_j λ(ω_i|ω_j)×P(ω_j|x)

Naive Bayes Choose ω_i : max P(ω_i)×Π_j P(x_j|ω_i)

NB (log form) Choose ω_i : max lnP(ω_i) + Σ_j lnP(x_j|ω_i)

MLE θ_MLE = argmax_θ P(Data|θ)

MAP Estimation θ_MAP = argmax_θ P(Data|θ)×P(θ)

Accuracy (TP+TN) / Total

Precision TP / (TP+FP)

Recall TP / (TP+FN)

F1-Score 2×(Precision×Recall) / (Precision+Recall)

5.3 All Memory Lines

W4: 'MAP = BDT with fair grading. Real life has unfair grading — use full BDT.'

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 16

 W4: 'Risk = average damage you expect. Always pick the action with least expected damage.'

W5: 'MLE = trust the data fully. MAP estimation = data + prior wisdom. Less data = trust prior
more.'

W5: 'Log turns dangerous multiplications into safe additions.'

W6: 'Naive Bayes: each feature votes independently. Add up the votes. Most votes wins.'

W6: 'Naive? Yes. Effective? Surprisingly yes — ranking matters more than exact numbers.'

W7: 'Accuracy lies when classes are unequal. F1 always tells the truth.'

W7: 'TP/TN = correct. FP = false alarm. FN = missed! Type I=FP, Type II=FN.'

W7: 'Precision = quality (don't cry wolf). Recall = coverage (catch all wolves).'

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 17

 6. Exam-Style Practice Questions & Answers
Questions ordered by difficulty. Try each before reading the answer. Focus on the reasoning, not just the
answer.

6.1 Section A — Conceptual (MCQ)

Q1. The MAP rule is a special case of BDT. Under which loss function does MAP emerge as the
optimal decision?

A. Asymmetric loss (λ_12 >> λ_21)

B. Zero-One loss (all errors cost 1)
C. Squared error loss
D. Log-loss

Answer: B — Under Zero-One loss, minimizing R(ω_i|x) = Σ_{j≠i} P(ω_j|x) = 1-P(ω_i|x), which is minimized by
MAXIMIZING P(ω_i|x) — exactly the MAP rule.

Q2. A hospital needs to minimize deaths from undetected cancer. The cost of False Negative is
100x the cost of False Positive. What should the BDT-optimal classifier prioritize?

A. High Accuracy
B. High Precision
C. High Recall
D. High Specificity

Answer: C — High FN cost → system penalizes FN heavily → minimize FN → maximize Recall =

TP/(TP+FN). BDT with asymmetric loss directly translates to maximizing Recall here.

Q3. Naive Bayes solves the Curse of Dimensionality. With D=20 features, each taking 3 discrete
values, how many parameters are saved per class by using Naive Bayes (vs full joint)?

A. 60 saved
B. 3^20 - 60 ≈ 3.5 billion saved
C. 20 saved
D. No parameters are saved

Answer: B — Full joint needs 3^20 ≈ 3.5 billion parameters. NB needs D×k = 20×3 = 60 parameters. Savings
≈ 3,486,784,341 parameters per class — essentially impossible to estimate without NB.

Q4. A classifier's discriminant function g_i(x) is linear in x. What shape is its decision boundary?

A. Ellipse
B. Parabola
C. Hyperplane (straight line/plane)
D. Circle

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 18

 Answer: C — g_1(x)=g_2(x) with linear discriminants gives a linear equation → hyperplane. Quadratic
discriminants (e.g., Gaussian with different covariances) give curved boundaries.

Q5. MLE estimates are obtained by maximizing P(Data|θ). When data is scarce, MAP estimation is
preferred. Why?

A. MAP always gives lower error than MLE

B. MAP incorporates prior P(θ), stabilizing estimates with limited data
C. MAP requires fewer computations
D. MAP doesn't require knowing the likelihood

Answer: B — With few data points, MLE can overfit to noise. MAP adds a prior P(θ) that acts as regularization
— it biases the estimate toward expected values, reducing variance.

6.2 Section B — Numerical Problems

Problem 1 — Risk Calculation: Two classes: ω_1=Fraud, ω_2=Legit. For transaction x: P(ω_1|x)=0.2,
P(ω_2|x)=0.8. Loss: λ(ω_1|ω_1)=0, λ(ω_2|ω_1)=50 (missed fraud!), λ(ω_1|ω_2)=2 (false alarm),
λ(ω_2|ω_2)=0. Which class does MAP choose? Which does BDT choose?

Solution:

MAP: Chooses ω_2 (Legit) since P(ω_2|x)=0.8 > P(ω_1|x)=0.2

BDT — Risk(ω_1=Fraud) = λ(ω_1|ω_1)×P(ω_1|x) + λ(ω_1|ω_2)×P(ω_2|x) = 0×0.2 + 2×0.8 = 1.6

BDT — Risk(ω_2=Legit) = λ(ω_2|ω_1)×P(ω_1|x) + λ(ω_2|ω_2)×P(ω_2|x) = 50×0.2 + 0×0.8 = 10.0

Risk(ω_1)=1.6 < Risk(ω_2)=10.0 → BDT chooses ω_1 = FRAUD

Key: MAP missed the fraud (chose Legit) but BDT correctly flagged it due to high miss-cost!

Problem 2 — Metrics: A model is tested: TP=80, FP=20, FN=10, TN=890. Calculate Accuracy, Precision,
Recall, and F1-Score. Is this model better for precision-critical or recall-critical tasks?

Solution: Total = 80+20+10+890 = 1000

Accuracy = (80+890)/1000 = 970/1000 = 97.0%

Precision = 80/(80+20) = 80/100 = 80.0%

Recall = 80/(80+10) = 80/90 = 88.9%

F1 = 2×(0.80×0.889)/(0.80+0.889) = 2×0.711/1.689 = 84.2%

Recall (88.9%) > Precision (80%) → better at recall-critical tasks (e.g., disease detection where catching all
positives matters more than avoiding false alarms).

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 19

 6.3 Section C — Key Exam Short Answer Tips
Bayes Rule vs MAP Bayes Rule = general (any loss function). MAP = special case (zero-one loss).
Never say they are the same.

Discriminant function Three valid choices: posterior P(ω_i|x), likelihood×prior, or log(likelihood×prior).

options All give same classification.

Why log in discriminant Turns products into sums (numerically stable). Avoids floating-point underflow
with many features.

Naive in Naive Bayes Naive = conditional independence assumption. Features are assumed
independent GIVEN the class. Almost never true, but works.

MAP vs MLE estimation MLE: maximize P(Data|θ). MAP estimation: maximize P(θ|Data) =
P(Data|θ)×P(θ). MAP adds a prior on θ.

F1 vs Accuracy Use F1 when dataset is imbalanced. A lazy 'always predict majority' model gets
high accuracy but F1≈0.

Type I vs Type II Type I = False Positive (false alarm). Type II = False Negative (miss).
Remember: 'crying wolf' is Type I.

Hyperplane condition Decision boundary is a hyperplane ONLY when the discriminant function is
LINEAR in x.

When to use which metric Prioritize Precision: losing good emails (FP) is worse than letting some spam
for spam through (FN).

Validation set importance Never tune on test set! Causes data leakage. Test set must remain unseen
until final evaluation.

Best of luck for your exams! — Understanding first, memorization second,

problem-solving third.

PRP Part 2 — IIT Jodhpur | Pattern Recognition Principles Page 20