Logistic regression
Introduction
Till now we have seen regression problems where the prediction was all about the value of a
parameter. Logistic regression is used for a different class of problems known as classification
problems. Here the aim is to predict the group to which the current object under observation
belongs to. Classification is all about portioning the data with us into groups based on certain
features. Let us dwell into the most commonly used form of example for this: A tumor has to be
classified malignant or benign based on various features like size, location, etc. Let us look into
simple logistic regression in which is used for binary classification purposes.
Logistic Regression
As discussed above, in linear regression our aim is to achieve binary classification and hence our
hypothesis must be chosen accordingly. Modifying our previous regression models our
hypothesis function, we can write
Y=CTX
where
C=[αβ1β2..βn]
and
X=[1X1X2..Xn]
where Xi is the vector containing ith feature value for all the entries in data set.
A sigmoid function is applied over the general known hypothesis function to get it into a range of
(0,1). This will be more clear once we discuss about boundary estimation. Sigmoid function is as
follows,
sg(y)=11+e−y
and therefore our new hypothesis function becomes
sg(y)=sg(CTx)=11+e−CTx
Boundary estimation
The new hypothesis function gives out a value between 0 & 1 and hence can be interpreted as
the probability that y value will be 1 for that particular x. This interpretation can be formally put
into the following form:
sg(y)=P(y=1|x;C)
and as y can take only 0 & 1, the other value probability is 1 minus the hypothesis value.
With the above interpretation we can safely decide the decision boundary with the following
rule: y=1 if sg(y)≥0.5, else y=0. sg(CTx)≥0.5 implies CTx≥0 and similarly for less than condition.
This constraint will give us the boundary for making a decision. When looked at a linear equation
of two independent variables, it will be quite clear how a line cuts the coordinate plane into two
halves with each class lying on either side of it.
Cost function
With the modified hypothesis function, taking a square error function won't work as it no longer
convex in nature and tedious to minimize. We take up a new form of cost function which is as
follows:
E(sg(C,x),y)=−log(sg(C,x)) if y=1
E(sg(C,x),y)=−log(1−sg(C,x)) if y=0
This can be written in a simpler form as:
E(sg(C,x),y)=−ylog(sg(C,x))−(1−y)log(1−sg(C,x))
and it is quiet evident that it is equivalent to the above cost function. For estimation of
parameters, we take the mean of cost function over all points in the training data. So,
H(C)=1m∑i=1mE(sg(C,xi),yi)
Parameter Estimation
For parameter estimation, we use an iterative method called gradient descent that improves the
parameters over each step and minimizes the cost function H(C) to the most possible value.
Gradient descent needs a convex cost function so that the minimization step doesn't get stuck
over in a local minima. In gradient descent, you start with random parameter values and then
update their values at each step to minimize the cost function by a some amount at each step
until we reach a minimum hopefully or until there is negligible change over certain [Link]
consecutive steps. The steps of gradient descent go as follows:
βi=βi−p∂H(C)∂βi
for each i=1,...n and p is the learning rate at which we move along the slope on the curve to
minimize the cost function.
We need to look at the above equation in computation terms, i.e. the βi on the left hand side is
the new value while the one on right hand side is the one calculated in previous step. Also, the
updating step of all parameters has to be done simultaneously as the value of H(C) is calculated
using the parameters and it changes when anyone of them is updated
Bias
The bias is known as the difference between the prediction of the values by the ML
model and the correct value. Being high in biasing gives a large error in training as
well as testing data. Its recommended that an algorithm should always be low biased
to avoid the problem of underfitting.
By high bias, the data predicted is in a straight line format, thus not fitting accurately
in the data in the data set. Such fitting is known as Underfitting of Data. This
happens when the hypothesis is too simple or linear in nature. Refer to the graph
given below for an example of such a situation.
HighBias
In such a problem, a hypothesis looks like follows.