Classi cation

Classi cation is a major task in data science, used to automatically make decisions for different
business problems. Examples include identifying whether an email is a phishing attempt, predicting
whether a customer is likely to churn, or checking if a user will click an advertisement. In each of
these situations, the goal is to assign a record to the correct category or class.

Classi cation falls under supervised learning, meaning the model is trained using data where the
outcomes are already known. After learning from this labeled data, the model is applied to new,
unseen data to predict the outcome. Many real-world classi cation problems are binary, where the
output is either:

• 1 or 0

• Yes or No

• Click / No Click

• Churn / Not Churn

However, classi cation is not limited to two categories. Some problems involve multiple classes,
such as Gmail ltering emails into:

• Primary

• Social

• Promotions

• Forums

In several applications, it is not enough to simply know the predicted class. We may also want to
know how likely (the probability) a record belongs to a particular class. This probability is called
the propensity score. Many algorithms, including logistic regression, can provide:

• Class prediction (using predict())

• Class probability (using predict_proba())

To convert these probabilities into nal class decisions, we use a cutoff probability (threshold).
The decision process works as follows:

1. Choose a cutoff probability for the class of interest (e.g., 0.5).

2. Estimate the probability that a record belongs to that class.

3. If the probability is above the cutoff → classify as 1 (positive class).

If below → classify as 0 (negative class).

The choice of cutoff affects how many records are predicted as class 1:

• Higher cutoff → fewer predictions as class 1 (more strict)

fi
fi
fi
fi
fi
fi
fi
 • Lower cutoff → more predictions as class 1 (less strict)

More Than Two Categories

Most classi cation problems involve a binary response, such as yes/no, churn/not-churn, or click/
no-click. However, some problems naturally have more than two possible outcomes, and these
require multiclass classi cation. For example, when a customer’s subscription contract ends, there
may be three possible outcomes:

• Y = 0: Customer signs a new long-term contract

• Y = 1: Customer moves to a month-to-month contract

• Y = 2: Customer leaves (churns)

In such cases, the goal is to predict which category (Y = 0, 1, or 2) the customer will choose.

Most classi cation techniques can handle more than two classes, either directly or with slight
modi cations. However, even when dealing with multiple outcomes, many problems can be
converted into a series of simpler binary classi cation tasks using conditional probabilities. For
example, predicting contract outcomes can be broken down into two steps:

1. Predict whether Y = 0 or Y > 0

(i.e., does the customer stay on a long-term contract or not?)

2. Given that Y > 0, predict whether Y = 1 or Y = 2

(i.e., among those who do not choose long-term, do they go month-to-month or churn?)

Breaking the problem into binary decisions makes the modeling process easier and sometimes more
accurate. This is especially useful when one category is much more common than the others, since
binary classi cation handles imbalance better. Therefore, converting a multiclass problem into
multiple binary problems can be both practical and effective for model building.

Naive Bayes

The Naive Bayes algorithm is a classi cation method based on Bayes’ theorem, which uses
probability to make predictions. The main idea is simple:
It uses the probability of observing certain predictor values (features) given an outcome to estimate
what we really want—the probability of an outcome given the predictor values.

Understanding Naive Bayes Through Exact Bayesian Classi cation

Before understanding the "naive" version, it helps to imagine the ideal or exact Bayesian
classi cation. In this perfect approach, for each new record we want to classify, we would:

1. Find all records with the exact same predictor values

(i.e., same age, same income, same category, etc.)

2. Check which classes those matching records belong to

(for example, how many are “default” vs. “paid off”).
fi
fi
fi
fi
fi
fi
fi
fi
fi
 3. Assign the most common class among those matching records to the new record.

This method directly uses real observed frequencies to estimate the correct class.

Why This Exact Method Is Impractical

In real-world data sets:

• There are many predictor variables,

• Each predictor can take many values, and

• It is extremely rare to nd another record with exactly the same combination of predictor
values.

This problem gets worse as the number of predictors increases. Because of this, exact Bayesian
classi cation becomes impossible for most data sets.

Where Naive Bayes Helps

Naive Bayes solves this problem by making a simple assumption (the "naive" part):
It assumes that the predictor variables are independent of each other given the outcome.

This allows the model to:

• Use all data, not just exact matches

• Compute probabilities even when no exact matching record exists

• Build a fast and ef cient classi cation model

Naive Bayes
The naive Bayes method solves the problem of not nding exact matches by making a simplifying
assumption. Instead of trying to match predictor values exactly, it uses the entire dataset to
estimate probabilities. This allows us to compute class probabilities even when no identical records
exist.

The method is called “naive” because it assumes that all predictor variables are independent of
each other given the class, which is rarely true in real life. However, this assumption greatly
simpli es calculations and still works surprisingly well.

Steps in the Naive Bayes Algorithm

For a binary classi cation problem where
( Y = i ) (i = 0 or 1), the naive Bayes method works as follows:
fi
fi
fi
fi
fi
fi
fi
1. Estimate conditional probabilities for each predictor

◦ Compute ( P(X_j \mid Y = i) )

◦ This is the probability of observing predictor value ( X_j ) given class ( Y = i ).

◦ We estimate this from training data as:

▪ Proportion of records with predictor value ( X_j ) among all records with
class ( Y = i ).

2. Multiply these probabilities together

◦ Multiply all ( P(X_j \mid Y = i) ) values across all predictors.

◦ Then multiply by ( P(Y = i) ), the overall proportion of class ( i ) in the dataset.

3. Repeat for all classes

◦ Perform steps 1 and 2 separately for class 0 and class 1.

4. Normalize to get the nal class probability

◦ For class ( i ), divide the value from step 2 by the sum of the step-2 values for all
classes.

◦ This gives the nal probability that the record belongs to class ( i ).

5. Assign the class

◦ Choose the class with the highest probability.

fi
fi
Why Is It Called “Naive”?
The method is called “naive” because:

• It assumes each predictor variable is independent of every other predictor given the class.

• In real life, predictors are often correlated (e.g., income and education).

• Even though this assumption is unrealistic, the algorithm still works very well in practice.

In short, Naive Bayes simpli es a complex probability estimation problem into a manageable one
by treating predictors as if they do not in uence each other.

Here is an easy, clear, exam-ready example of Naive Bayes that students can understand much
better than the R loan-data example.

Simple Naive Bayes Example

Problem: Predict whether a student will Pass or Fail based on two predictors:

• Studied (Yes / No)

• Attendance (High / Low)

We use a small dataset so the calculations are easy.

✅ Training Data
Studie Attendanc Resul
d e t
Yes High Pass
Yes High Pass
No High Fail
Yes Low Pass
No Low Fail
No High Fail

Step 1: Calculate Prior Probabilities

Count Pass and Fail:
fi
fl
 • Pass = 3

• Fail = 3

• Total = 6

So:

• ( P(Pass) = 3/6 = 0.5 )

• ( P(Fail) = 3/6 = 0.5 )

Step 2: Calculate Conditional Probabilities

A. Probability of Studied = Yes

Among Pass:

• Studied = Yes → 3 out of 3 → ( P(Studied = Yes | Pass) = 3/3 = 1.0 )

Among Fail:

• Studied = Yes → 0 out of 3 → ( P(Studied = Yes | Fail) = 0/3 = 0 )

B. Probability of Attendance = High

Among Pass:

• High attendance → 2 out of 3 → ( P(Attendance=High | Pass) = 2/3 )

Among Fail:

• High attendance → 2 out of 3 → ( P(Attendance = High} | Fail) = 2/3 )

Step 3: Predict for a New Student

New student details:

• Studied = Yes

• Attendance = High

We now compute:
Step 4: Compare Probabilities
• Pass = 0.333

• Fail = 0

Therefore:

✅ Prediction: The student will PASS

Why this example is better for students

• Only two predictors

• Easy categories (Yes/No, High/Low)

• Very simple counts

• Formula becomes easy to apply

• Shows clearly how Naive Bayes works

• Avoids complex tables and long outputs

Numeric Predictor Variables in Naive Bayes

The basic Naive Bayes classi er works naturally with categorical predictors, such as “Yes/No,”
“High/Low,” or “Spam/Not Spam.” This is because the algorithm is based on estimating conditional
probabilities like
( P(X_j \mid Y = i) ),
which are easy to compute for categories (just count and divide).
fi
However, Naive Bayes does not directly handle numerical (continuous) variables, such as age,
income, temperature, marks, height, etc. Numerical values do not repeat exactly, and therefore
conditional probabilities cannot be estimated using simple counting.

To apply Naive Bayes to numerical predictors, we use one of two approaches:

1. Convert Numerical Predictors into Categories (Binning)

In this approach, we convert numerical data into meaningful categories.
Examples:

• Age → Young / Middle-aged / Old

• Income → Low / Medium / High

• Marks → <40 (Fail), 40–70 (Average), >70 (Good)

Once converted, the algorithm can use the same counting-based probability method as categorical
data.

Advantages:

• Very simple

• Easy for students to understand

• Works well when bins are chosen sensibly

2. Use a Probability Model (e.g., Normal Distribution)

Here, we do not convert the numerical values into categories.
Instead, we assume a probability distribution for the numerical variable.

Most commonly, the normal (Gaussian) distribution is used.

For each class ( Y = i ), we estimate:

• Mean of ( X_j ) within that class

• Standard deviation of ( X_j ) within that class

Then we use the normal probability formula to calculate

( P(X_j \mid Y = i) ).

This approach is used in Gaussian Naive Bayes (very common in Python and ML libraries).

Advantages:

• Retains information in the numeric values

• Often gives better accuracy

 • No need to create categories manually

Discriminant Analysis
Discriminant Analysis is one of the earliest statistical methods used for classi cation. It was rst
introduced by R. A. Fisher in 1936 in the Annals of Eugenics journal. Among the different types of
discriminant analysis, the most commonly used method is Linear Discriminant Analysis (LDA).

LDA is a technique that builds a classi cation model by nding a linear combination of predictor
variables that best separates the classes (for example, separating “pass/fail,” “good/bad,” or
“default/not-default”). While Fisher’s original method was slightly different from modern LDA, the
overall idea and mechanics remain the same.

With the development of more advanced models such as decision trees, random forests, logistic
regression, and support vector machines, LDA is used less frequently today. However, it is still
relevant because:

• It is simple and easy to understand

• It works well when class separation is roughly linear

• It performs well when predictors follow a normal distribution

• It has strong connections to other techniques (e.g., Principal Components Analysis (PCA))

You may still see LDA used in applications such as:

• Facial recognition

• Medical diagnosis

• Grouping or separating populations

• Pattern recognition tasks

LDA is also important from a theoretical standpoint and helps in understanding more complex
classi cation models.

Covariance Matrix
Before learning how discriminant analysis works, we must understand the concept of covariance,
because LDA relies heavily on the covariance structure of the predictors.

What is Covariance?

Covariance measures how two variables move together.

fi
fi
fi
fi
fi
 • If both increase together → covariance is positive

• If one increases while the other decreases → covariance is negative

• If they are unrelated → covariance is close to zero

Let the means of variables x and z be:

Where:

• ( n ) = number of records

• ( x_i ) = ith value of variable x

• ( z_i ) = ith value of variable z

• Numerator: multiplies how far each x and z value is from their means

• Denominator: ( n - 1 ) (degrees of freedom)

This formula gives the covariance between two variables.

Covariance Matrix

When dealing with more than two variables, we create a covariance matrix, which includes:

• Variances on the diagonal

• Covariances off the diagonal

The covariance matrix is essential for LDA, because LDA uses it to understand how variables
together differentiate classes.

Understanding Covariance Values

Similar to the correlation coef cient:
fi
 • Positive covariance → variables increase or decrease together

• Negative covariance → one increases while the other decreases

However, covariance is not limited to a range like correlation.

• Correlation lies between –1 and +1

• Covariance depends on the units and scale of the variables

(e.g., height in cm, weight in kg)

So covariance is useful but harder to interpret directly due to scale differences.

Covariance Matrix – Student Notes

When we have more than one variable, we can summarize all covariances and variances in a
single matrix called the covariance matrix.

Below is a clean, simple, exam-ready explanation of Fisher’s Linear Discriminant, written in

easy paragraphs + bullet points so students can understand the concept clearly.

Fisher’s Linear Discriminant – Student Notes

(Simple & Clear)
Fisher’s Linear Discriminant is an early and important statistical classi cation method introduced
by R. A. Fisher in 1936. It is mainly used for binary classi cation when the predictors are
continuous numerical variables. Although modern methods like logistic regression and tree-based
models are more popular today, Fisher’s method is still useful and forms the basis of other
important techniques such as Principal Component Analysis (PCA).

What Fisher’s Linear Discriminant Tries to Do

Suppose we want to predict a binary outcome y (0 or 1) using two numeric variables:

• (x)

• (z)

The main idea of Fisher’s Linear Discriminant is:

✔ Separate the two classes as much as possible

✔ While keeping the spread inside each class as small as possible

So, it tries to nd a line or a linear combination:

that best separates the two groups.

Between-Group vs Within-Group Variation

Fisher’s method compares two types of variation:

1. Between-Group Variation (SS_{between})

• Measures how far apart the means of the two groups are.

• A larger distance between the group means = better separation.

2. Within-Group Variation (SS_{within})

• Measures how spread out the data is inside each group.

• Uses the covariance matrix.

• Smaller within-group spread = better separation.

fi
fi
fi
This ratio becomes large when:

• The groups’ means are far apart

• The variation inside each group is small

So the discriminant function nds weights ( w_x ) and ( w_z ) that achieve maximum separation.

Intuition
Imagine two clouds of points (one for y=0, one for y=1).
We want to draw a line so that:

• The two groups’ projections on the line are far apart

• The points within each group are close together

This ensures the best distinction between the two classes.

Why It Works
Even though the method assumes:

• Predictors follow normal distribution

• Variables have equal covariance within groups

It still performs well even when these assumptions are not perfectly true.

Below is a clean, simple, exam-ready explanation of your last section

“A Simple Example – LDA”, rewritten clearly for students, with bullet points and paragraphs.
fi
 A Simple Example of Linear Discriminant
Analysis (LDA)
To understand linear discriminant analysis (LDA) in practice, The MASS package in R contains an
LDA function applied to data, we specify:

• The outcome variable (the class we want to predict)

• The predictor variables (the features used for classi cation)

In the example given, the LDA model tries to classify loan records using two predictors:

• borrower_score

• payment_inc_ratio

Once the LDA model is tted, it produces linear discriminant weights. These weights determine
the linear combination:

LD1=(w1 ×borrower_score)+(w2 ×payment_inc_ratio)

This linear score helps separate the two classes (paid-off vs. default).

R Example (Simple Explanation)

library(MASS)
loan_lda <- lda(outcome ~ borrower_score + payment_inc_ratio,
data = loan3000)
loan_lda$scaling
This prints the weight values:

• borrower_score has weight 7.1758

• payment_inc_ratio has weight –0.0997

Interpretation

• A higher borrower score strongly increases the likelihood of being in the “paid off” group.

• A higher payment-to-income ratio slightly increases the likelihood of default (negative

in uence).

Python Example
In Python, we use scikit-learn:
fl
fi
fi
from sklearn.discriminant_analysis import
LinearDiscriminantAnalysis

[Link] = [Link]('category')
predictors = ['borrower_score', 'payment_inc_ratio']
X = loan3000[predictors]
y = loan3000[outcome]

loan_lda = LinearDiscriminantAnalysis()
loan_lda.fit(X, y)
[Link](loan_lda.scalings_, index=[Link])

This prints the same weights as the R output.