MODULE – 5
Classification
Discriminant Analysis
Discriminant Analysis is a supervised classification technique used in statistics and machine
learning to separate two or more groups (classes) based on their predictor variables
(features).
It creates discriminant functions that best distinguish between categories.
While discriminant analysis encompasses several techniques, the most commonly used is
linear discriminant analysis, or LDA.
The original method proposed by Fisher was actually slightly different from LDA, but the
mechanics are essentially the same. LDA is now less widely used with the advent of more
sophisticated techniques, such as tree models and logistic regression.
Key Terms for Discriminant Analysis :
Covariance - A measure of the extent to which one variable varies in concert
with another (i.e., similar magnitude and direction).
Discriminant function - The function that, when applied to the predictor
variables, maximizes the separation of the classes.
Discriminant weights - The scores that result from the application of the
discriminant function and are used to estimate probabilities of belonging to
one class or another.
Covariance Matrix
A covariance matrix is a square matrix that shows how multiple variables vary
together — that is, how much they change in relation to each other.
To understand discriminant analysis, it is first necessary to introduce the
concept of covariance between two or more variables. The covariance
measures the relationship between two variables x and z.
Denote the mean for each variable by X and Y. The covariance between x
and z is given by:
=
where n is the number of records.
As with the correlation coefficient , positive values indicate a positive
relationship and negative values indicate a negative relationship. Correlation,
however, is constrained to be between –1 and 1,
whereas covariance scale depends on the scale of the variables x and z and
The covariance matrix Σ for x and z , consists of the individual variable ,
variances and on the diagonal (where row and column are the same variable)
and the covariances between variable pairs on the off-diagonals:
Fisher’s Linear Discriminant :
Fisher’s Linear Discriminant (FLD) — also known as Linear Discriminant
Analysis (LDA) in the two-class case — is a dimensionality reduction and
classification technique developed by Ronald A. Fisher (1936).
Fisher’s linear discriminant distinguishes variation between groups, on the
one hand, from variation within groups on the other.
Specifically, seeking to divide the records into two groups, linear discriminant
analysis (LDA) focuses on maximizing the “between” sum of squares
(measuring the variation between the two groups) relative to the “within”
sum of squares (measuring the within-group variation).
In this case, the two groups correspond to the records (, for which y = 0 and
the records for which y = 1.
The method finds the linear combination that maximizes that sum of squares
ratio:
=
Logistic Regression :
Logistic regression is analogous to multiple linear regression , except the
outcome is binary. Various transformations are employed to convert the
problem to one in which a linear model can be fit.
Like discriminant analysis, and unlike K Nearest Neighbor and naive Bayes,
logistic regression is a structured model approach rather than a data-centric
approach.
Due to its fast computational speed and its out put of a model that lends itself
to rapid scoring of new data, it is a popular method.
Key Terms for Logistic Regression :
Logit - The function that maps class membership probability to a range from ±
∞ (instead of 0 to 1).
Odds - The ratio of “success” (1) to “not success” (0).
Log odds - The response in the transformed model (now linear), which gets
mapped back to a probability.
Logistic Response Function and Logit :
The key ingredients for logistic regression are the logistic response function
and the logit, in which we map a probability (which is on a 0–1 scale) to a
more expansive scale suitable for linear modeling.
Naively, we might be tempted to model p as a linear function of the
predictor variables.
p = +++⋯ +
Generalized Linear Models :
Generalized Linear Models (GLMs) are a broad class of models that extend
traditional linear regression to handle non-normal response variables and
nonlinear relationships between predictors and the outcome variable.
Generalized linear models (GLMs) are characterized by two main
components:
• A probability distribution or family (binomial in the case of logistic
regression)
• A link function—i.e., a transformation function that maps the response
to the predictors (logit in the case of logistic regression)
Logistic regression is by far the most common form of GLM.
Sometimes a log link function is used instead of the logit; in practice, use of a
log link is unlikely to lead to very different results for most applications.
In contrast to logistic regression, application of GLMs with these models is
more nuanced and involves greater care. These are best avoided unless you
are familiar with and understand the utility and pitfalls of these methods.
Predicted Values from Logistic
Regression :
Y The predicted value from logistic regression is in terms of the log odds: = log Odds Y =1 .
The predicted probability is given by the logistic response function: