Amity Institute of Defence Technology
Cost Function:
• The logistic regression cost function, known as Log Loss or
Binary Cross-Entropy, measures the performance of a
classification model by quantifying the error between
predicted probabilities and actual binary labels (0 or 1).
• It uses a logarithmic function to heavily penalize confident,
incorrect predictions, ensuring a convex optimization
surface that helps gradient descent in finding the global
minimum.
• The goal of logistic regression is to minimize Binary Cross-
Entropy cost function J(θ) to determine the best parameters
θ.
1
Amity Institute of Defence Technology
Key Aspects:
• Why Not Mean Squared Error (MSE)? Using MSE with the
non-linear sigmoid function creates a non-convex surface
with many local minima, making optimization difficult. Log
loss ensures a single convex minimum.
• Behavior:
If the actual label y=1 and the model predicts 1, cost is 0. If
y=1 but the model predicts 0, the cost approaches infinity.
• Binary Cases:
2
Amity Institute of Defence Technology
Cost Function Formula:
The cost for training examples is the average of the log loss:
3
Amity Institute of Defence Technology
4
Amity Institute of Defence Technology
5
Amity Institute of Defence Technology
What is the Hypothesis in Logistic Regression?
6
Amity Institute of Defence Technology
If We Use MSE in Logistic Regression:
7
Amity Institute of Defence Technology
• What Happens?
• Because sigmoid is non-linear, the composition:
makes the cost:
• Non-convex in parameters
• Can have multiple local minima
• Gradient descent may get stuck
• Slower convergence
8
Amity Institute of Defence Technology
This is due to the derivative structure:
The gradient becomes complicated and non-linear.
9
Amity Institute of Defence Technology
Why don’t we use `Mean Squared Error as a cost function
in Logistic Regression?
• In Logistic Regression Ŷi is a nonlinear function
(Ŷ=1/1+ e-z), if we put this in the above MSE equation it will
give a non-convex function as shown:
10
Amity Institute of Defence Technology
• When we try to optimize values using gradient descent it
will create complications to find global minima.
• Another reason is in classification problems, we have target
values like 0/1, So (Ŷ-Y)2 will always be in between 0-1
which can make it very difficult to keep track of the errors
and it is difficult to store high precision floating numbers.
11
Amity Institute of Defence Technology
• The Red line represents 1 class. As we can see, when the
predicted probability (x-axis) is close to 1, the loss is less
and when the predicted probability is close to 0, loss
approaches infinity.
• The Black line represents 0 class. As we can see, when the
predicted probability (x-axis) is close to 0, the loss is less
and when the predicted probability is close to 1, loss
approaches infinity.
12
Amity Institute of Defence Technology
Why MSE is Convex in Linear Regression
but Not Here?
13
Amity Institute of Defence Technology
14
Amity Institute of Defence Technology
Decision Boundary in Logistic Regression
• A decision boundary in logistic regression is the line,
curve, or hyperplane that separates different classes in
the feature space.
• It divides the input space into regions belonging to
different classes (e.g., class 0 and class 1).
• For binary classification, it represents the point where the
model is equally likely to assign either class.
15
Amity Institute of Defence Technology
[Link]
in-logistic-regression/#google_vignette
16
Amity Institute of Defence Technology
• In the Logistic regression model the value of the classier lies between
0 to 1.
• So, to establish the hypothesis we also found the Sigmoid function or
Logistic function.
17
Amity Institute of Defence Technology
• In logistic regression, we create a decision boundary. And
this will give us a better seance of, what logistic regression
function is computing.
18
Amity Institute of Defence Technology
• Decision boundary separates different classes in
classification.
• In logistic regression it occurs when probability = 0.5.
• The boundary equation is 𝜃𝑇𝑥+𝑏=0.
• It forms a line, plane, or hyperplane depending on the
number of features.
• Model parameters θ and b determine the position and
orientation of the boundary.
• 1 Feature → Decision boundary is a point.
• 2 Features → Decision boundary is a straight line.
• 3 Features → Decision boundary is a plane.
• Higher dimensions → Decision boundary becomes a
hyperplane.
19
Amity Institute of Defence Technology
• Problem of Overfitting and Underfitting
• 1. Overfitting:
Overfitting occurs when a machine learning model learns the training
data too well, including noise and small fluctuations, resulting in
poor performance on new or unseen data.
Key Points
• Very high accuracy on training data but low accuracy on test data.
• The model memorizes the training dataset instead of learning
general patterns.
• Occurs when the model is too complex (too many parameters or
layers).
• The decision boundary becomes very complex and irregular.
• The model fails to generalize to new data.
20
Amity Institute of Defence Technology
Causes
• Small training dataset
• Very complex model
• Too many features
• Training for too many epochs
Ways to Reduce Overfitting
• Use more training data
• Apply regularization (L1, L2)
• Use dropout in neural networks
• Apply cross-validation
• Reduce model complexity
21
Amity Institute of Defence Technology
[Link]:
Underfitting occurs when a model is too simple to capture
the underlying patterns in the data, leading to poor
performance on both training and testing datasets.
Key Points
• Low accuracy on training data and testing data.
• The model cannot learn important relationships in the
data.
• The decision boundary is too simple.
• The model fails to capture complex patterns in the
dataset.
• Results in high bias.
22
Amity Institute of Defence Technology
Causes
• Very simple model
• Insufficient training time
• Too few features
• Excessive regularization
Ways to Reduce Underfitting
• Increase model complexity
• Add more features
• Train the model for more epochs
• Reduce regularization
• Use more advanced algorithms
23
Amity Institute of Defence Technology
Multi-classification (One vs All, One vs One):
24
Amity Institute of Defence Technology
25