Search EN Upload Sign in Download free for 30 days

0 ratings · 484 views · 25 pages You might also like

No ratings yet
SMLDS BAD702 Module5 Notes
PDF

Machine
Learning Unit-
2
The document covers Discriminant Analysis and its applications in data science, focusing on classification techniques such as Naive Bayes, Linear Discriminant Analysis (LDA), and
Logistic Re… Full description 89 pages

Uploaded by Shashank R AI-enhanced title and description PDF No ratings yet

Bayesian
Classification,
Download Save Share 0% 0% Print Embed Ask AI Report
Nearest
46 pages

PDF No ratings yet

Data Science
Unit V Notes
SML for DS [BAD702]
17 pages

Module 5 PDF No ratings yet

1 Lect 9-Naive

25
Discriminant Analysis Bayes

12 pages
Syllabus:
PDF No ratings yet
Legal 3 AI
Covariance Matrix, Fisher’s Linear discriminant, Generalized Linear Models, Interpreting the

coefficients and odd ratios, Strategies for Imbalanced Data.

3 pages

Textbook: Chapter 5 PDF No ratings yet

Naive Bayes
Introduction
Data scientists automate business decisions through classification, a type of supervised 9 pages

learning where a model is trained on labeled data (known outcomes) and used to predict PDF No ratings yet

unknown outcomes. CCS - Lec 5

Examples:
33 pages
 Detecting phishing emails (phishing/not phishing)
 Predicting customer churn (churn-stop using a service/not churn) PDF No ratings yet
Module 3 -
 Estimating ad clicks (click/no click) Naive Bayes
Classifier
Types:
23 pages

 Binary classification: Two possible outcomes (1 or 0).

PDF No ratings yet
 Multiclass classification: More than two categories (e.g., Gmail inbox labels – primary, Naive Bayes
Classifier
social, promotions, forums). Guide

Probability Scores: 16 pages

Instead of just assigning classes, models can output probability estimates (propensities) that PDF No ratings yet
Supervised
indicate how likely a record belongs to a particular class. Classification
3601
 Logistic regression outputs log-odds, which are converted to probabilities.
39 pages
 In Python’s scikit-learn, predict() gives classes, while predict_proba() gives
PDF No ratings yet
probabilities. 2.3 Bayes
The General Decision Process: Classification

1. Set a cutoff probability (threshold). 15 pages

2. Estimate the probability of belonging to thetarget class. PDF No ratings yet

3. Classify the record as belonging to that class if the estimated probability exceeds the Classification

cutoff.
20 pages

PDF No ratings yet

Chapter 11
KNN Naive
Bayes and…
Dept. of CSE-DS, RNSIT Smitha B A 1
15 pages

PDF No ratings yet

BIDM Session
07-08
Recommended Download to read ad-free
PDF

· ·
33 pages
From Scribd 89 pages 41 views
No ratings yet
Machine Learning Unit-2 PDF No ratings yet
Classification
2

19 pages

SML for DS [BAD702] PDF No ratings yet

Unit-3 AML
(Bayesian
Concept…
More Than Two Categories?
40 pages
While most classification problems arebinary (two possible outcomes), some involve multiple
PDF No ratings yet
outcomes—known as multiclass classification. Data Mining -
Example: Bayesian
Classification
At a customer’s contract renewal, three outcomes are possible: 6 pages

 Y =0: Signs a new long-term contract PDF No ratings yet

 Y =1: Switches to a month-to-month contract Chapter11
Slides
 Y =2: Leaves or “churns”
20 pages
The goal is to predict which category (Y = j) applies for j = 0, 1, or 2.
Most classification algorithms can handle multiclass problems directly or with slight PDF No ratings yet
Machine
modifications. Learning

Sometimes, the problem can be recast into multiple binary problems using conditional
5 pages

probabilities, for example:

PDF No ratings yet
1. Predict whether Y =0 or Y >0 (churn vs. not churn). 9-Decision
Tree
2. Given Y >0, predict whether Y =1 or Y =2(type of contract). Induction-2…
This stepwise approach simplifies model fitting and is especially helpful whenone class is much 40 pages

more frequent than others. PDF No ratings yet

L25 - Naïve
Bayes

5.1 Naive Bayes

18 pages
The Naive Bayes algorithm is based on Bayes’ theorem, which relates the probability of an
PDF No ratings yet
outcome given some predictors. A5 PDF

It estimates theprobability of a class (Y =i) given a set of predictor values.

Concept: 9 pages

 It uses the probability of observing predictor values given an outcome (P(X | Y)) to PDF No ratings yet
Naïve
compute the probability of the outcome given predictors (P(Y | X)). Bayesian
 This helps determine which class a new record most likely belongs to. Classifier…
48 pages
Exact Bayesian Classification (Conceptual Idea):
PDF No ratings yet
For each new record:
Classification
1. Identify all training records with the same predictor values (predictor profile). Problems
Johari…
2. Observe theclass distribution among these identical records. 39 pages

3. Assign the class that occurs most frequently (most probable class).
PDF No ratings yet
L3 (Week3)
Bayesian
Classifier
21 pages

PDF No ratings yet

Dept. of CSE-DS, RNSIT Smitha B A 2 Classification

vs Prediction
Overview
44 pages

Recommended Download to read ad-free

PDF No ratings yet
PDF
ML Module4
From Scribd · 46 pages · 50 views No ratings yet
Classification

Bayesian Classification, Nearest

79 pages

PDF No ratings yet

Naïve Bayes
Classifier

SML for DS [BAD702]

39 pages

PDF No ratings yet

Limitation: Lecture 6 -
Generative
 In reality, it’s rare to find exact matches of predictor profiles, especially when there are
Models
many predictors. 33 pages

 The Naive Bayes approach overcomes this by assuming that predictors are PDF No ratings yet
ML 05
conditionally independent, allowing probability estimation even when exact matches
Bayesian
don’t exist. Classifier
19 pages

PDF No ratings yet

5.1.1 Why Exact Bayesian Classification Is Impractical
Unit 5-6
When the number of predictor variables increases, finding exact matches for a new record
becomes highly unlikely.
18 pages

Example:
PDF No ratings yet
In predicting voting behaviour using demographic variables, even with a large dataset, it’s rare Bayes
Classifier
to find another record exactly matching a new individual’s detailed profile (e.g., gender,
ethnicity, income, location, voting history, family structure, marital status). 20 pages

This happens because: PDF 100% (1)

U02Lecture0
 Each additional variable increases the number of unique combinations of predictor 7
Classification
values.
56 pages
 As the number of predictors grows, the chance of finding an exact match drops sharply
PDF No ratings yet
— a problem known as thecurse of dimensionality. Pgm5 With
Output
 For instance, adding just one new variable with five categoriesreduces the probability
of an exact match by a factor of 5. 13 pages

Hence, exact Bayesian classification becomes impractical for high-dimensional data, PDF No ratings yet
202109131157
motivating the use of Naive Bayes, which assumes predictors are conditionally independent to
10D3708 -
estimate probabilities efficiently. Session 09-…
30 pages
5.1.2 The Naive Solution
PDF No ratings yet
 TheNaive Bayesalgorithmis a probabilistic classifier based on Bayes’ theorem, which
Unit-3 AML
estimates the probability of a class given a set of predictor values. (Bayesian
Concept…
 It is called “naive” because it assumes that all predictors areindependent of each other 40 pages

given the outcome — an assumption rarely true in real data, but onethat often works well
PDF No ratings yet
in practice. UNIT - IV

 Instead of looking only at records that exactly match a new case (as in exact Bayes
169 pages
classification), Naive Bayes uses the entire dataset to estimate probabilities.
PDF No ratings yet
Bayesian

23 pages
Dept. of CSE-DS, RNSIT Smitha B A 3
PDF No ratings yet
Module 3 -
Classification

Recommended Download to read ad-free

111 pages
PDF

From Scribd · 17 pages · 10 views No ratings yet

PDF No ratings yet
Data Science Unit V Notes Chapter 4
Classification

78 pages

PDF No ratings yet

SML for DS [BAD702] WINSEM2023
-24
MCSE602L…
132 pages

Steps in the Naive Bayes Algorithm PDF No ratings yet

WK 08

10 pages

PDF No ratings yet

K2 Data
Science
Bootcamp:…
11 pages

PDF No ratings yet

KNN and
Baysian
Method
43 pages

PDF No ratings yet

Data Mining
4th Is

In Python (using scikit-learn):

24 pages
from sklearn.naive_bayes import MultinomialNB
PDF No ratings yet
import pandas as pd Module - 4 -
ECE3047 -
predictors =['purpose_', 'home_', 'emp_len_'] Machine…
outcome ='outcome' 81 pages

X =pd.get_dummies(loan_data[predictors], prefix='', prefix_sep='') PDF No ratings yet

Understandin
y =loan_data[outcome] g Bayesian
naive_model =MultinomialNB(alpha=0.01, fit_prior=True) Classificatio…
25 pages
naive_model.fit(X, y)
PDF No ratings yet
 The Naive Bayes classifier provides not only a class prediction (e.g., default or paid off) Lecture 03
but also a posterior probability estimate — the predicted likelihood of an outcome such Bayes
Classifier…
as default (Y =1). 70 pages

 However, these probability estimates are often biased because of the naive independence PDF No ratings yet

assumption among predictors. Datamining

Lect12
 Despite this, Naive Bayes remains effective when the objective is ranking rather than
75 pages
obtaining perfectly accurate probabilities.
PDF No ratings yet
Data Science
5.1.3 Numeric Predictor Variables Lecture:
Classificatio…
27 pages

PDF No ratings yet

Naïve Bayes
Dept. of CSE-DS, RNSIT Smitha B A 4 for Text
Classification
33 pages

PDF No ratings yet

Recommended Download to read ad-free Unit 5
PDF
Classification
From Scribd · 12 pages · 12 views No ratings yet
PDF

Lect 9-Naive Bayes 131 pages

PDF No ratings yet

Business
Analytics
(PEC) -…
90 pages
SML for DS [BAD702]
PDF 100% (1)
Probit Model
The Naive Bayes classifier is designed primarily for categorical predictors, such as in spam for Binary
Outcomes
detection where features represent thepresence or absenceof specific words.
29 pages
When dealing with numerical predictors, Naive Bayes cannot be directly applied. Two main
PDF No ratings yet
approaches are used: Instructions:
Answer Each
1. Discretization (Binning):
of The…
o Convert continuous (numerical) variables into categorical bins or intervals. 3 pages

o Then apply the standard Naive Bayes algorithm for categorical data. PDF No ratings yet
(Ebook) A
o Example: Age can be divided into bins such as <20, 20–40, 40–60, 60+.
Gentle
2. Assume a Probability Distribution: Introduction…
150 pages
o Model each numerical predictor using a known probability distribution
(commonly the normal distribution). PDF No ratings yet
Intro to
o Estimate the conditional probability ( ∣  = )using that distribution’s Classification
& Regression
parameters (mean and variance) computed from the training data. 42 pages

o This approach leads to Gaussian Naive Bayes, often used when predictors are PDF No ratings yet
continuous. Credit Card
Fraud
5.2 Discriminant Analysis Detection…
1 page
 Linear Discriminant Analysis (LDA) is the most common form of discriminant
PDF 0% (1)
analysis.
AZ-900
 Fisher’s original method (1936) differs slightly from modern LDA, but the underlying Latest Exam
Dumps
mechanics are similar.
41 pages

 Usage: Less common today due to more advanced methods like decision trees and
PDF No ratings yet
logistic regression. Six-Sigma
Case Study -
 Relevance: LDA is still used in certain applications and has connections to other Quality…

techniques, such asPrincipal Components Analysis (PCA). 2 pages

PDF No ratings yet

A Comparison
5.2.1 Covariance Matrix of Univariate
Probit and…
 To understand discriminant analysis, it is important to know covariance, which
21 pages
measures how two variables vary together.
PDF No ratings yet
 Covariance between variables x and z is calculated as: Identifing
Software
Bugs or Not…
34 pages

Here, PDF 100% (1)

MACHINE
 ˉ , ˉ: means of x and z
LEARNING 1-5
(Ai &DS)
60 pages

Dept. of CSE-DS, RNSIT Smitha B A 5 PDF 100% (1)

Difference
Between
Logit and…
7 pages
Recommended Download to read ad-free
PDF
PDF No ratings yet
From Scribd · 3 pages · 13 views
No ratings yet
Millet
Legal 3 AI Production
Forecasting …
1 page

PDF No ratings yet

Customer
SML for DS [BAD702] Churn
Prediction…
24 pages
 : number of records
PDF No ratings yet
 The division by (n – 1) adjusts for degrees of freedom. Fraud
Detection
Covariance and CovarianceMatrix Project…
 Covariance measures how two variables change together. 4 pages

o Positive → variables increase/decrease together. PDF No ratings yet

Mini Project
o Negative → one increases while the other decreases.
o Zero → No linear relationship.
59 pages
 Relation to Correlation:
PDF No ratings yet
o Correlation is thescaled (standardized) form of covariance.
5th Module
o Correlation ∈ [–1, +1]; covariance has no fixed range (depends on variable
scale). 15 pages

 Covariance Matrix (Σ): PDF No ratings yet

For variables and : Testing Dick
and Basu's
Loyalty Model
7 pages
Diagonal elements: variances ( ,  )
Off-diagonal elements: covariances (, = , ) PDF No ratings yet
The Effect of
Information
Technology…
5.2.2 Fisher’s Linear Discriminant
48 pages

Goal is to Classify records into two groups (binary outcome y) usingtwocontinuousvariables

PDF No ratings yet
(x, z). W8-
Supervised
Assumptions: Learning…

Predictor variables are continuous and normally distributed (though LDA is fairly robust to 30 pages

mild violations). PDF No ratings yet

Task 2 Model
Fisher’s Linear Discriminant seeks a linear combinationof predictors: Plan

 ′ =     
2 pages
that best separates the two groups.
PDF No ratings yet
And Maximize the ratio: Decision
Authority
between
within
16 pages

where,
PDF No ratings yet
between : variation between group means (distance between groups) Tejas Rastogi
(Final
within : variation within groups (spread around means), adjusted by the covariancematrix Research…

Intuition: 47 pages

Maximize between-group separation PDF No ratings yet

Unino&Find
Algo
Dept. of CSE-DS, RNSIT Smitha B A 6 Approximati…
34 pages

PDF No ratings yet

Youth
Recommended Download to read ad-free Unemployme
PDF
nt in Tanzani…
From Scribd · 9 pages · 9 views No ratings yet
10 pages

Naive Bayes
PDF No ratings yet
Assessment
of Poverty
Situation in…
9 pages

SML for DS [BAD702] PDF No ratings yet

Flood
Prediction
Minimize within-group overlap Using AI…

→ gives the best linear boundary for distinguishing the two classes. 12 pages

5.2.3 A Simple Example PDF No ratings yet

BAI701 DLRL
Predicting Loan Default Assignment

Problem
1 page
We want to predict whether a loan applicant will default (y =1) or pay off (y =0) using two
PDF No ratings yet
numeric predictors: NLP Case
Studynaman
 borrower_score – measure of creditworthiness (0–1 scale)
 payment_inc_ratio – ratio of monthly payment to income 23 pages

borrower_scor payment_inc_ratio outcom PDF No ratings yet

D Project
0.65 5.2 paid off

0.32 10.8 default 35 pages

0.72 4.1 paid off PDF No ratings yet

Corruption in
0.28 12.3 default Indonesia:
Does Greas…
0.81 3.9 paid off 9 pages

0.40 9.7 default PDF No ratings yet

Do Young
Generations
Save For…
import pandas as pd
25 pages
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
PDF No ratings yet
import [Link] as plt The Caring
Under
Pressure
#Sample data 2 pages

data =[Link]({ PDF No ratings yet

Efficient
'borrower_score': [0.65, 0.32, 0.72, 0.28, 0.81, 0.40],
Route
'payment_inc_ratio': [5.2, 10.8, 4.1, 12.3, 3.9, 9.7], Planning in…
1 page
'outcome': ['paid off', 'default', 'paid off', 'default', 'paid off', 'default']
PDF No ratings yet
})
Cyber
Security
ASSIGNMENT
X =data[['borrower_score', 'payment_inc_ratio']]
4 pages

y =data['outcome']
PDF No ratings yet
Boosting
Experiential
#Fit LDA model Loyalty to…
9 pages

PDF No ratings yet

Dept. of CSE-DS, RNSIT Smitha B A 7 Lund-MacKay
System for
Chronic…
8 pages

Recommended Download to read ad-free PDF No ratings yet

Prevalence of
PDF

From Scribd · 33 pages · 35 views

No ratings yet
Internet
CCS - Lec 5 Addiction a…
8 pages

PDF No ratings yet

Modern
Results of
SML for DS [BAD702] Functional…
4 pages

lda =LinearDiscriminantAnalysis()
[Link](X, y)

#Show discriminant weights

weights =[Link](lda.scalings_, index=[Link], columns=['Weight'])
print(weights)

#Predict probabilities
pred =[Link](lda.predict_proba(X), columns=lda.classes_)
print(pred)

LDA creates a straight line that separates “paid off” and “default” regions:
       = 0

 Points left of the line → predicted as default

 Points right of the line → predicted as paid off
Confidence increases as the point moves farther from the boundary.
In Python (matplotlib/seaborn), a scatter plot with color gradient shows:
 Predicted probabilities (e.g., default risk).
 Decision boundary (solid diagonal line).

Key Ideas:
o LDA works for continuous
or categorical predictors with
categorical outcomes.
o Uses thecovariance matrix
to compute a linear discriminant
function.
o Produces scores/weights
that classify each record into a
likely group.
o Core objective: maximize
between-group variance / minimize within-group variance for optimal class
separation.

Dept. of CSE-DS, RNSIT Smitha B A 8

Recommended Download to read ad-free

PDF

From Scribd · 23 pages · 16 views No ratings yet

Module 3 - Naive Bayes Classifier

SML for DS [BAD702]

5.3 Logistic Regression

 Logistic regression is similar to multiple linear regression, but theoutcome variable is
binary (e.g., 0/1, yes/no).
 It uses a transformation (logit function) to model the relationship between predictors
and a binary outcome.
 The model estimates theprobability of an event occurring based on input variables.
 Like discriminant analysis, logistic regression is a structured model-based approach
(not purely data-driven like KNN or naive Bayes).
 It is computationally efficient and provides an interpretable model for predicting new
data quickly.
 Widely used for classification problems and risk prediction.

5.3.1 Logistic ResponseFunction and Logit

In logistic regression, the main components are the logistic response function and the logit
transformation. These help map probabilities (which range from 0 to 1) onto an unbounded
scale suitable for linear modelling.
Rather thandirectly modelling the probability  of the outcome being “1” as a linear combination
of predictors —
 =         ⋯   
which could yield values outside the valid probability range, logistic regression uses the logistic
(inverselogit) function to constrain  between 0 and 1:
1
=
1   −( + + +⋯+ )
This transformation ensures valid probability estimates and forms the basis for interpreting the
relationship between predictors and the likelihood of an event.

Dept. of CSE-DS, RNSIT Smitha B A 9

Recommended Download to read ad-free

PDF

From Scribd · 16 pages · 171 views No ratings yet

Naive Bayes Classifier Guide

SML for DS [BAD702]

To better interpret this relationship, we use odds,

defined as the ratio of the probability of success to
failure:

Odds( = 1) =
1
Fromthis, the probability can be expressed as:
Odds
=
1  Odds
Combining this with the logistic model gives:
Odds ( = 1) =  + + +⋯+ 
Taking the natural logarithm of both sides gives the
logit function (log-odds form):
log (Odds Y =1) =        ⋯   

log( ) =         ⋯   
1
Thus, logistic regression models alinear relationship between the predictors and the log of the
odds of the event.
This allows us to predict probabilities within (0,1) andclassify observations using a chosencutoff
(e.g.,  > 0.5⇒ class 1).

5.3.2 Logistic Regression and the GLM

In logistic regression, theresponse variable represents the log-odds of a binary outcome (e.g.,
success =1, failure =0). Since we only observe the binary outcome itself —not the log-odds—
special estimation methods such asmaximumlikelihood estimation(MLE) areusedto fit the
model.
Logistic regression is aspecial case of the Generalized Linear Model (GLM), which extends
linear regression to handle different types of response variables by using asuitable link function
and error distribution.
 For logistic regression:
o Link function: Logit (log of the odds)
o Error distribution: Binomial
In this logistic regression example, theresponsevariable is outcome, which equals 0 if the loan
is paid off and 1 if the loan defaults.
 The predictors include both numeric and categorical (factor) variables:

Dept. of CSE-DS, RNSIT Smitha B A 10

Recommended Download to read ad-free

PDF

From Scribd · 39 pages · 20 views No ratings yet

Supervised Classification 3601

SML for DS [BAD702]

o purpose_ → purpose of the loan (factor variable)

o home_ → home ownership status (factor variable)
o payment_inc_ratio, emp_len_, and borrower_score → numeric predictors
In Python, the logistic regression model is built using the LogisticRegression class from
sklearn.linear_model:

predictors = ['payment_inc_ratio', 'purpose_', 'home_', 'emp_len_', 'borrower_score']

outcome = 'outcome'

X = pd.get_dummies(loan_data[predictors], prefix='', prefix_sep='', drop_first=True)

y = loan_data[outcome]

logit_reg = LogisticRegression(penalty='l2', C=1e42, solver='liblinear')

logit_reg.fit(X, y)

 pd.get_dummies() creates dummy variables for categorical predictors.

 Regularization (L1 or L2) is enabled by default to prevent overfitting; setting C=1e42
(very large value) effectively removes it.
 solver='liblinear' specifies the optimization algorithm.
 In scikit-learn, classes are ordered alphabetically, so coefficient signs may appear
reversed compared to R.
 predict() gives class labels, while predict_proba() returns predicted probabilities
corresponding to the order in logit_reg.classes_.

5.3.3 Generalized Linear Models

Generalized Linear Models (GLMs) extend traditional linear regression to handle different
types of response variables and relationships. They are defined by two main components:
1. Probability distribution (family): Determines the type of response variable.
o For logistic regression, the family is binomial (binary outcome).
o Other families include Poisson (for count data), negative binomial, and gamma
(for modeling time or duration).
2. Link function: A transformation that connects the linear predictor to the expected value
of the response variable.
o Logistic regression uses the logit link:

logit() = log( )
1

Dept. of CSE-DS, RNSIT Smitha B A 11

Recommended Download to read ad-free

PDF

From Scribd · 15 pages · 24 views No ratings yet

2.3 Bayes Classification

SML for DS [BAD702]

o Sometimes, a log link is used instead of a logit, but results are often similar in
practical cases.
While logistic regression is the most common GLM and widely applicable, using other GLMs
(e.g., Poisson, Gamma) requires deeper statistical understanding, as these models involve more
complex assumptions and are sensitive to data characteristics.
5.3.4 Predicted Values from Logistic Regression
Inlogistic regression, the model predicts thelog-odds of the outcome being 1:
 = log(Odds( = 1))
To convert this linear prediction into aprobability, the logistic response function is applied:
1
=
1   −
This transformation ensures that the predicted probabilities lie between 0 and 1.
InPython (scikit-learn):
 Log-oddsare obtained using predict_log_proba().
 Probabilities are obtained directly using predict_proba():
pred = [Link](logit_reg.predict_proba(X),
columns=loan_data[outcome].[Link])
[Link]()
The predicted probabilities indicate the likelihood of the outcome (e.g., loan default).
Typically, a cutoff of 0.5 is used to classify outcomes (≥0.5 → default, <0.5 → paid off).
However, when identifyingrare events, alower threshold may be chosen to improve detection
of the minority class.

5.3.5 Interpreting theCoefficients and Odds Ratios

One of the main advantages of logistic regression is that it provides a model that can be easily
applied (scored) to new data and is straightforward to interpret. The interpretation centers on
the concept of theoddsratio, which measures how changes in predictor variables affect theodds
of an outcome (e.g., default vs. paid off).
For a binary predictor variable :
Odds ( = 1 ∣  = 1)
Odds Ratio =
Odds ( = 1 ∣  = 0)
If the odds ratio >1, the event  = 1 is more likely when  = 1.
 If the odds ratio <1, the event is less likely when  = 1 .

Dept. of CSE-DS, RNSIT Smitha B A 12

Recommended Download to read ad-free

PDF

From Scribd · 20 pages · 15 views No ratings yet

Classification

SML for DS [BAD702]

The logistic regression coefficient  represents the log of the odds ratio for the variable  :
 = log(odds ratio)
Hence,
odds ratio =  
Example interpretations:
 For a categorical variable purpose_small_business
with coefficient 1.21526,
 .6 ≈ 3.4
Loans to small businesses are about 3.4 times more likely to default than credit card loans (the
reference category).
 For a numeric variable like payment_inc_ratio with coefficient 0.08244,
 .8 ≈ 1.09
Each unit increase in payment-to-income ratio increasesdefault odds by 9%.
 For borrower_score with coefficient –4.61264,
 −.66 ≈ 0.01
Borrowers with excellent creditworthiness have 100 times lower odds of defaulting compared
to those with poor credit.
Because coefficients are expressed on the log scale, a one-unit increase in the coefficient
corresponds to a multiplication of the odds by   ≈ 2.72.

5.3.6 Linear and Logistic Regression: Similaritiesand Differences

Like linear regression, logistic regression uses aparametric linear form to relate predictors to
the response variable. Model exploration, feature selection, and transformations (such assplines)
can be applied in similar ways to improve model fit and flexibility.
However, logistic regression differs from linear regression in two key aspects:
1. Model Fitting:
 Least squares estimation (used in linear regression) is not applicable because the
response variable is binary, not continuous.
 Instead, logistic regression uses maximum likelihood estimation (MLE) to determine
the coefficients that best explain the observed outcomes.
2. Residual Analysis:
 The residuals (differences between observed and predicted values) have a different
distribution and interpretation compared to linear regression.

Dept. of CSE-DS, RNSIT Smitha B A 13

Recommended Download to read ad-free

PDF

From Scribd · 15 pages · 70 views No ratings yet

Chapter 11 KNN Naive Bayes and LDA

SML for DS [BAD702]

 Specialized residual diagnostics (like deviance or Pearson residuals) are used to assess
model fit and identify outliers or influential observations.
Thus, while logistic regression retains the linear relationship in form, its estimation and
diagnostic procedures are fundamentally different from those of ordinary least squares
regression.
Fitting the Model:
Inlinear regression, the model is fit using least squares, and the quality of fit is measured with
metrics like RMSE and R-squared.
Inlogistic regression, however:
 There is no closed-formsolution for the coefficients because the response is binary.
 The model is fit using Maximum Likelihood Estimation (MLE), which finds the
parameter values that make the observed data most probable.
 The logistic regression response is modeled as the log-odds of the outcome being 1,
rather than 0 or 1 directly.
 MLE iteratively updates coefficients using algorithms like quasi-Newton optimization
or Fisher scoring, improving the fit at each step.
For most practitioners, the software handles the optimization, so it is sufficient to understand
that MLE finds thebest-fitting logistic model under certain assumptions.

5.3.7 Assessing the Model

1. Model Evaluation:
Logistic regression is evaluated based on classification accuracy rather than RMSE or R-
squared.
Metrics commonly used include accuracy, precision, recall, F1-score, AUC-ROC, etc.
2. Coefficient Interpretation:
Summary outputs (e.g., in R using summary(logistic_model) or in Python using statsmodels
GLM) provide:
Estimate (β) – regression coefficients
Standard Error (SE) – uncertainty of coefficient estimates
z-value – ratio of estimate to SE
p-value – indicates relative importance (not strict significance)
Example: purpose_small_business coefficient of 1.21526 → odds of default 3.4 times higher
than reference category.
3. Extensions from Linear Regression:

Dept. of CSE-DS, RNSIT Smitha B A 14

Recommended Download to read ad-free

PDF

From Scribd · 33 pages · 14 views No ratings yet

BIDM Session 07-08

SML for DS [BAD702]

Stepwise selection, interaction terms, spline terms, and generalized additive models
(GAMs) are applicable.
In R: gam() function with family='binomial'
In Python: [Link]() supports spline terms using bs() (B-splines).
4. Residual Analysis:
 Residuals differ from linear regression due to the binary nature of the outcome.
 Partial residuals help visualize the effect of a predictor and detect nonlinear
behavior.
 In logistic regression, residuals lie in
two clouds corresponding to 0s and 1s
because the observed outcomes are
binary, while predictions are log-odds.
 Partial residual plots can still identify
influential observations and nonlinear
patterns.
 R supports partial residuals; Python requires custom implementation.
5. Note
 Dispersion parameter in R summary is not relevant for logistic regression.
 Residual devia nce and number of scoring iterations relate to maximum likelihood
fitting.
Key Takeaway: Logistic regression combines interpretability of coefficients, flexibility with
GLM extensions, and classification-based evaluation, but residual analysis and goodness-of-fit
metrics differ fundamentally from linear regression.

5.4 Evaluating ClassificationModels

In predictive modeling, it is standard practice to:
1. Train multiple models on a training dataset.
2. Validate each model on a holdout sample to assess performance.
3. If enough data are available, use athird holdout (test) sample to estimate performance
on completely new data.
o The terms validation and test are often used interchangeably across disciplines.
Measuring Accuracy:
 Accuracy is a simple and common metric for classification performance:

Dept. of CSE-DS, RNSIT Smitha B A 15

Recommended Download to read ad-free

PDF

From Scribd · 19 pages · 11 views No ratings yet

Classification 2

SML for DS [BAD702]

Number of Correct Predictions ΣTrue Positives  ΣTrue Negatives

Accuracy = =
Total Sample Size Sample Size
Decision Cutoff:
 Most classification algorithms predict a probability of being 1.
 The default cutoff is usually 0.5:
o Probability ≥ 0.5 → class 1
o Probability < 0.5 → class 0
 Alternative cutoffs can be used based on the prevalence of 1s in the data or specific
business goals, especially when dealing with imbalanced classes

5.4.1 Confusion Matrix

The confusion matrix is a fundamental tool for evaluating classification models. It summarizes
correct and incorrect predictions for each class in a table format.
 Binary classification convention:
o  = 1 → event of interest (e.g., loan default)
o  = 0 → negative or usual event (e.g ., loan paid off)
The confusion matrix has four key components:

Predicted 1 Predicted 0

Actual 1 True Positive (TP) False Negative (FN)

Actual False Positive (FP) True Negative (TN)

 True Positive (TP): Correctly predicted 1s
 True Negative (TN): Correctly predicted 0s
 FalsePositive (FP): Predicted 1 but actual 0
 False Negative (FN): Predicted 0 but actual 1
Usage:
 Compute by hand or using packages in R (table(), caret::confusionMatrix()) or Python
([Link].confusion_matrix).
 For example, applying it to the logistic_gam model trained on a balanced dataset of
defaulted vs. paid-off loans allows you to see how well the model classifies each
outcome.
From the confusion matrix, you can derive other performance metrics like accuracy, precision,
recall, and F1-score.
In Python:

Dept. of CSE-DS, RNSIT Smitha B A 16

Recommended Download to read ad-free

PDF

From Scribd · 40 pages · 18 views No ratings yet

Unit-3 AML (Bayesian Concept Learning)

SML for DS [BAD702]

pred = logit_reg.predict(X)

pred_y = logit_reg.predict(X) == 'default'

true_y = y == 'default'

true_pos = true_y & pred_y

true_neg = ~true_y & ~pred_y

false_pos = ~true_y & pred_y

false_neg = true_y & ~pred_y

conf_mat = [Link]([[[Link](true_pos), [Link](false_neg)],

[[Link](false_pos), [Link](true_neg)]],

index=['Y = default', 'Y = paid off'],

columns=['Yhat = default', 'Yhat = paid off'])

conf_mat

In a confusion matrix for binary classification:

 Rows correspond to theactual outcomes.
 Columns correspond to thepredicted outcomes.
 Diagonal elements represent correct
predictions:
o Upper-left: True Negatives (TN)
o Lower-right: True Positives (TP)
 Off-diagonal elements represent
incorrect predictions:
o Upper-right: False Positives (FP)
o Lower-left: False Negatives (FN)
Example (Loan Defaults):
 14,295 defaulted loans correctly predicted as defaults (TP)
 8,376 defaulted loans incorrectly predicted as paid off (FN)
Key Metrics Derived fromthe Confusion Matrix:
 Accuracy: Overall proportion of correct predictions
 Precision: TP / (TP + FP) → proportion of predicted positives that are correct
 Recall (Sensitivity): TP / (TP + FN) → proportion of actual positives correctly identified
 Specificity: TN / (TN + FP) → proportion of actual negatives correctly identified
 FalsePositive Rate: FP / (FP + TN) → important when positive events are rare

Dept. of CSE-DS, RNSIT Smitha B A 17

Recommended Download to read ad-free

PDF

From Scribd · 6 pages · 1.3K views No ratings yet

Data Mining - Bayesian Classification

SML for DS [BAD702]

Important Note:
When 1s (positives) are rare, the false positive rate can dominate, making a predicted positive
much more likely to be anegative in reality. This phenomenon occurs in medical screening tests,
such as mammograms, where most positive results are false positives due to the rarity of the
condition.

5.4.2 The Rare Class Problem

In many predictive modelling problems, the classes are imbalanced, meaning one class (e.g.,
legitimate claims, non-purchasers) is much more prevalent thanthe other (e.g., fraudulent claims,
purchasers).
 The rare class is often theclass of interest and is usually labeled 1.
 Misclassifying the rare class (false negative) is often more costly than misclassifying the
common class (false positive).
o Example: Identifying a fraudulent insurance claimsaves significant money, while
misclassifying a legitimate claim as fraudulent incurs smaller costs.
Problemwith Imbalance:
 A naive model that predicts only the majority class (0) can achieve high accuracy but is
practically useless.
o Example: If only 0.1% of website visitors purchase, predicting all as non-
purchasers yields 99.9% accuracy but fails to detect actual purchasers.
Practical Approach:
 Focus on metrics that emphasize the rare class, such as:
o Recall / Sensitivity for the rare class
o Precision for the rare class
o F1-score (harmonic mean of precision and recall)
 Accept some loss in overall accuracy in order to better detect the important rare
events.

5.4.3 Precision, Recall, and Specificity

In addition to overall accuracy, several nuanced metrics are used to evaluate classification
models, especially when dealing with imbalanced classes. These metrics are widely used in
statistics and biostatistics for diagnostic tests.
1. Precision (Positive Predictive Value):
Measures the accuracy of predicted positives:

Dept. of CSE-DS, RNSIT Smitha B A 18

Recommended Download to read ad-free

PDF

From Scribd · 20 pages · 282 views No ratings yet

Chapter11 Slides

SML for DS [BAD702]

True Positives (TP)

Precision =
TP + False Positives (FP)
2. Recall (Sensitivity, True Positive Rate):
Measures the ability to correctly identify actual positives:
True Positives (TP)
Recall =
TP + False Negatives (FN)
Term sensitivity is common in biostatistics; recall is used in machine learning.
3. Specificity (True Negative Rate):
Measures the ability to correctly identify actual negatives:
True Negatives (TN)
Specificity =
TN + False Positives (FP)
In Python:
from [Link] import confusion_matrix, precision_recall_fscore_support

conf_mat = confusion_matrix(y, logit_reg.predict(X))

precision = conf_mat[0,0] / sum(conf_mat[:,0])

recall = conf_mat[0,0] / sum(conf_mat[0,:])

specificity = conf_mat[1,1] / sum(conf_mat[1,:])

# Alternatively, calculate all at once

precision_recall_fscore_support(y, logit_reg.predict(X), labels=['default','paid off'])

Note: These metrics are especially valuable in cases with rare positive events, where overall
accuracy can be misleading.

5.4.5 ROC Curve: Receiver Operating

Characteristic
In classification, there is often a trade-off between
recall (sensitivity) andspecificity:
 Increasing recall (capturing more positives)
generallyincreasesfalsepositives, lowering
specificity.
 Anideal classifier maximizes recall without
sacrificing specificity.
The Receiver Operating Characteristic (ROC)
curve visualizes this trade-off.

Dept. of CSE-DS, RNSIT Smitha B A 19

Recommended Download to read ad-free

PDF

From Scribd · 5 pages · 30 views No ratings yet

Machine Learning

SML for DS [BAD702]

Key Features of theROC Curve:

 Y-axis: Recall / Sensitivity (True Positive Rate)
o X-axis: Either Specificity (1 on left, 0 on right)
o Or 1 – Specificity / False Positive Rate (0 on left, 1 on right)
 Both representations produce identical curves.
Stepsto Compute an ROC Curve:
1. Sort records by predicted probability of being a positive (1), from highest to lowest.
2. Compute cumulative recall and specificity as you move down the sorted list,
generating the ROC points.
The ROC curve allows you to select an appropriate cutoff based on the trade-off between
detecting positives and avoiding false positives.

5.4.6 AUC: Area under the Curve

While the ROC curve visually shows the trade-off
betweenrecall (sensitivity) andspecificity, it does not
provide a single performance metric. The Area Under
the Curve (AUC) is a summary statistic derived from
the ROC curve:
 AUC = 1: Perfect classifier (all 1s correctly
classified, no 0s misclassified).
 AUC = 0.5: Classifier no better than random
guessing (diagonal line).
 Higher AUC → better classifier.
In python:
from [Link] import roc_auc_score

roc_auc_score([1 if yi == 'default' else 0 for yi in y], logit_reg.predict_proba(X)[:,0])

Interpretation:
 The loan model example has AUC ≈ 0.69, indicating a relatively weak classifier.
 AUC provides a single, threshold-independent measure of model discriminative
ability.

Dept. of CSE-DS, RNSIT Smitha B A 20

Recommended Download to read ad-free

PDF

From Scribd · 40 pages · 58 views No ratings yet

9-Decision Tree Induction-23-01-2025

SML for DS [BAD702]

5.4.7 Lift
Using AUC improves model evaluation over simple accuracy because it considers thetrade-off
between identifying positives (1s) and overall accuracy, but it does not fully solve rare-class
problems:
 When positives are rare, a cutoff <0.5 may be necessary to avoid classifying all records
as 0.
o Example: Classifying records with probability ≥ 0.3 as 1 to catch more rare
events.
 Lowering the cutoff increases recall for the rare class but also increases falsepositives.
Lift (or Gains) Metric:
 Measures how much better the model performs in identifying 1s compared to random
selection.
 Example: Top 10% of records by predicted probability may yield 0.3% positive rate vs.
0.1% if selected randomly → lift =3.
 Lift chart / Gains chart:
o X-axis: Cumulative records (or deciles)
o Y-axis: Cumulative recall (percentage of 1s captured)
o Lift curve: Ratio of cumulative gains to the diagonal (random selection)
 Useful for identifying optimal probability cutoff in practice, especially underresource
constraints.
Applications:
 Direct mail marketing: Target top prospects efficiently.
 Tax audits: Select returns most likely to be fraudulent given limited audit resources.
 Marketing / political campaigns: Determine uplift—improvement in outcome due to
treatment A vs. B for individual cases.
Note:
 Lift charts quantify model effectiveness for the rare class and help decide a practical
probability cutoff aligned with business or resource priorities.

Dept. of CSE-DS, RNSIT Smitha B A 21

Recommended Download to read ad-free

PDF

From Scribd · 18 pages · 43 views No ratings yet

L25 - Naïve Bayes

SML for DS [BAD702]

5.5 Strategies for Imbalanced Data

When dealing with imbalanced datasets—where the outcome of interest (e.g., purchase, fraud)
is rare—standard evaluation metrics like accuracy can be misleading. Beyond using metrics such
as precision, recall, specificity, AUC, and lift, additional strategies can improve model
performance for the rare class.

5.5.1 Undersampling
When the dataset is large, oneeffective strategy to handleimbalancedclassesisundersampling
the majority class (0s):
o The dominant class often contains redundant records.
o Removing some of these records creates a more balanced dataset, improving
model performance and simplifying data preparation.
 Benefits:
o Reduces computational burden.
o Makes it easier to explore and pilot models.
o Helps the model better learn patterns for the minority class (1s).
 How much data is enough?
o Depends on the application.

Dept. of CSE-DS, RNSIT Smitha B A 22

Recommended Download to read ad-free

PDF

From Scribd · 9 pages · 59 views No ratings yet

A5 PDF

SML for DS [BAD702]

o Generally, having tens of thousands of records for the less dominant class is
sufficient.
o If the classes are easily distinguishable, less data may suffice.
 Example (Loan Data):
o Training set was balanced: 50% paid off, 50% defaulted.
o Predicted probabilities roughly split around 0.5.
o In the full dataset, only ~19% of loans were in default, illustrating the original
imbalance.
Undersampling is particularly useful when the majority class is overwhelming , allowing the
model to focus on learning the characteristics of the minority class effectively.
In Python:
predictors = ['payment_inc_ratio', 'purpose_', 'home_', 'emp_len_',

'dti', 'revol_bal', 'revol_util']

outcome = 'outcome'

X = pd.get_dummies(full_train_set[predictors], prefix='', prefix_sep='',

drop_first=True)

y = full_train_set[outcome]

full_model = LogisticRegression(penalty='l2', C=1e42, solver='liblinear')

full_model.fit(X, y)

('percentage of loans predicted to default: ',

100 * [Link](full_model.predict(X) == 'default'))

Training on imbalanced data can severely underpredict rare events.

 Example: Only 0.39% of loans predicted as default vs. 19% actual.
Reason: Majority class dominates; defaulting loans may resemble nondefaulting ones.
Balanced sampling (50% defaults, 50% paid off) improves predictions, giving ~50%
predicted defaults.

5.5.2 Oversampling and Up/Down Weighting

Issue with Undersampling: Removes data from the dominant class, risking loss of useful
information, especially in small datasets.
Alternatives:
1. Oversampling: Duplicate or bootstrap rare-class records to balance the dataset.

Dept. of CSE-DS, RNSIT Smitha B A 23

Recommended Download to read ad-free

PDF

From Scribd · 48 pages · 74 views No ratings yet

Naïve Bayesian Classifier Overview

SML for DS [BAD702]

2. Weighting: Assign higher weights to rare-class records during model training.

Example (Loan Data):
 Without weighting: only 0.39% of loans predicted as default.
 With weighting (or upsampling): predictions for defaults increase to ~58%, balancing
influence of rare and common classes.

5.5.3 Data Generation

Create new synthetic records by perturbing existing minority-class records, giving the
algorithm more examples to learn robust classification rules.
SMOTE (Synthetic Minority Oversampling Technique):
 Selects a minority-class record and a similar neighbor.
 Generates a synthetic record as a randomly weighted average of the two.
 Number of synthetic records depends on thedesired oversampling ratio.
Purpose: Improves model learning without simply duplicating records.
Implementation:
 R: unbalanced package or FNN for SMOTE.
 Python: imbalanced-learn package, compatible with scikit-learn, supports SMOTE,
oversampling, undersampling, and integration with ensemble methods like boosting and
bagging.

5.5.4 Cost-Based Classification

Limitation of Accuracy/AUC: They do not account for thedifferent costsofmisclassification.
Cost-Based Decision: Assign costs/returns to outcomes, e.g.,
 C: cost of loan default
 R: return from paid-off loan
 Expected return:
Expected return = ( = 0) ×   ( = 1) × 
Application: Use expected return to decide whether to approve a loan, rather than just
classifying based on probability.
Benefit: Incorporates business value, allowing decisions that maximize profit rather than just
prediction accuracy.

Dept. of CSE-DS, RNSIT Smitha B A 24

Recommended Download to read ad-free

PDF

From Scribd · 39 pages · 14 views No ratings yet

Classification Problems Johari Stanford

SML for DS [BAD702]

5.5.5 Exploring the Predictions

A single metric, such as AUC, cannot evaluate all aspects of the suitability of a model for a
situation. Figure 5-8 displays the decision rules for four different models fit to the loan data
using just two predictor variables: borrower_score and pay ment_inc_ratio.

Example (Loan Data with 2predictors: borrower_score and payment_inc_ratio):

 Linear Discriminant Analysis(LDA) andLogistic Linear Regression: Produce nearly
identical, smooth linear decision boundaries.
 Tree Model: Produces a less regular, stepwise decision boundary.
 Logistic GAM: Provides a compromise—more flexible than linear models, smoother
than tree models.
Note:
sDifferent modeling approaches can yield different decision rules, highlighting the need to
consider interpretability, smoothness, and flexibility alongside metrics like AUC.

Dept. of CSE-DS, RNSIT Smitha B A 25

Share this document

About Support Legal Social Get our free apps

About Scribd, Inc. Help / FAQ Terms Instagram

Slideshare Accessibility Privacy Facebook

Join our team! Purchase help Copyright Pinterest

Contact us AdChoices Cookie Preferences

Documents Language: English Copyright © 2026 Scribd Inc.

We take content rights seriously. Learn more in our FAQs or report infringement here.

We and our 41 IAB TCF partners store and access information on your device for the following purposes: store and/or access information on a device, advertising and content measurement, audience research, and services
development, personalised advertising, and personalised content.
Personal data may be processed to do the following: use precise geolocation data and actively scan device characteristics for identification.
Our third party IAB TCF partners may store and access information on your device such as IP address and device characteristics. Our IAB TCF Partners may process this personal data on the basis of legitimate interest, or with
Customize Your Choices
your consent. You may change or withdraw your preferences at any time by clicking on the cookie icon or link; however, as a consequence, you may not see relevant ads or personalized content.
Our website may use these cookies to:
Measure the audience of the advertising on our website, without profiling Accept All
Display personalized ads based on your navigation and your profile
Personalize our editorial content based on your navigation
Allow you to share content on social networks or platforms present on our website Reject All
Send you advertising based on your location

Privacy Policy
Third Parties