Module 5: Regression and Classification Techniques, Linear Regression, Simple Linear

Regression and Assumptions, Multivariate Linear Regression, Model Assessment and

Variable Importance, Subset Selection, Classification Techniques , Classification using
Logistic Regression.

Regression and Classification Techniques

In data science and machine learning, regression and classification are two primary types of
predictive modeling techniques used to make predictions based on data. While regression is
used for continuous outcome prediction, classification is used for predicting categorical
outcomes. Here's a breakdown of both techniques.

1. Regression Techniques

Regression analysis is used to predict continuous outcomes. The goal of regression is to

model the relationship between one or more independent variables (predictors) and a
dependent variable (target).

a) Simple Linear Regression

• Definition: Simple linear regression involves modeling the relationship between a

single independent variable and a dependent variable using a linear equation.

The equation for simple linear regression is:

 • Assumptions of Simple Linear Regression:

1. Linearity: The relationship between the independent and dependent

variable is linear.

2. Independence: Observations are independent of each other.

3. Homoscedasticity: The variance of residuals is constant across all levels

of the independent variable.

4. Normality: The residuals are normally distributed.

b) Multiple (Multivariate) Linear Regression

• Definition: Multiple linear regression extends simple linear regression to include

multiple independent variables. It allows for the prediction of a dependent
variable using more than one predictor.

The equation for multiple linear regression is:

• Assumptions:

1. Linearity: The relationship between each predictor and the dependent

variable is linear.

2. Multicollinearity: Predictors should not be highly correlated with each

other.

3. Independence of errors: Errors should not be correlated with each other.

4. Homoscedasticity: The residuals should have constant variance.

5. Normality of errors: Errors should be normally distributed.

c) Model Assessment and Variable Importance

• Model Performance Metrics:

o R-squared: Measures the proportion of variance explained by the model.

o Adjusted R-squared: Takes the number of predictors into account.

 o Mean Squared Error (MSE): Measures the average squared difference
between observed and predicted values.

o Root Mean Squared Error (RMSE): The square root of MSE.

• Variable Importance: In multiple regression, determining which variables are

most important in predicting the dependent variable is crucial. Various
techniques such as stepwise regression, LASSO (Least Absolute Shrinkage
and Selection Operator), and ridge regression can be used for variable
selection.

d) Subset Selection

Subset selection involves selecting a subset of predictors that contribute the

most to the model. It helps in reducing overfitting and improving model interpretability.
Methods for subset selection include:

1. Forward Selection: Starts with no predictors and adds one predictor at a time
based on a selection criterion (e.g., AIC).

2. Backward Elimination: Starts with all predictors and removes the least
significant predictor one by one.

3. Stepwise Selection: A combination of forward and backward selection.

2. Classification Techniques

Classification techniques are used to predict categorical outcomes. These

methods assign data points to specific classes or categories based on input features.

a) Logistic Regression

• Definition: Logistic regression is used to predict the probability of a binary

outcome. It is used when the dependent variable is categorical and binary (i.e., 0
or 1).

The equation for logistic regression is:

• Assumptions of Logistic Regression:

1. Linearity in the log-odds: There is a linear relationship between the log-

odds of the outcome and the predictor variables.

2. Independence of observations: The observations are independent of

each other.

3. No multicollinearity: The predictors should not be highly correlated with

each other.

b) Decision Trees

• Definition: Decision trees are non-linear models that recursively partition the
feature space into subregions based on feature values to make predictions.

o Advantages: Easy to interpret, handles both numerical and categorical

data.

o Disadvantages: Prone to overfitting, especially with deep trees.

• Classification Trees: A decision tree that classifies data into distinct categories.
The algorithm splits data based on the best feature that maximizes information
gain (e.g., Gini impurity or entropy).

c) Random Forest

• Definition: Random Forest is an ensemble learning method that creates

multiple decision trees and combines their results to improve classification
performance. It reduces overfitting by averaging multiple trees.

o Advantages: Handles overfitting, handles missing data, and works well

with large datasets.

d) Support Vector Machines (SVM)

 • Definition: SVM is a powerful classifier that finds the hyperplane that best
separates the classes in a high-dimensional space. It works well in high-
dimensional spaces and for non-linear data by using the kernel trick.

• Key Concepts:

1. Margin: The distance between the hyperplane and the closest data points
from each class.

2. Kernel Trick: Transforms data into higher dimensions to make it linearly

separable.

e) K-Nearest Neighbors (KNN)

• Definition: KNN is a simple, non-parametric classification algorithm that assigns

a class to a data point based on the majority class of its k nearest neighbors in
the feature space.

• Advantages: Easy to understand and implement, no explicit model training.

• Disadvantages: Computationally expensive during prediction, especially for

large datasets.

f) Naive Bayes

• Definition: Naive Bayes is a probabilistic classifier based on Bayes' Theorem. It

assumes that features are conditionally independent given the class label, which
is often an oversimplification, but works well in practice.

Conclusion:

• Regression Techniques are primarily used when predicting continuous

variables, such as sales price, temperature, etc.

• Classification Techniques are used when the target variable is categorical (e.g.,
spam vs. non-spam emails, medical diagnosis).

In both regression and classification, the main goal is to build a model that
accurately predicts outcomes, while ensuring that the model is generalizable and
interpretable. Each method has its strengths, weaknesses, and suitable applications
depending on the nature of the data and problem at hand.
Linear Regression

Linear regression is one of the most fundamental and widely used techniques in statistics and
machine learning. It is used to model the relationship between a dependent (target) variable
and one or more independent (predictor) variables.

Below are detailed notes on Linear Regression.

Conclusion

Linear regression is a powerful tool for predicting continuous outcomes based on one or
more independent variables. By understanding the assumptions, methods of
estimation, and how to assess the model, one can effectively apply linear regression to
various real-world problems. However, it is crucial to check the assumptions and be
aware of the model's limitations to ensure reliable predictions.
Simple Linear Regression and Assumptions

Simple Linear Regression (SLR) is one of the most fundamental techniques in statistics used
to model the relationship between two variables. In this technique, we attempt to fit a straight
line through a set of data points that minimizes the error (the difference between the observed
and predicted values). This section will cover the concept of simple linear regression in
detail, along with the assumptions underlying this model.
2. Assumptions of Simple Linear Regression

For the simple linear regression model to produce reliable and valid results, certain
assumptions must be satisfied. These assumptions ensure that the model is properly
specified and the statistical tests (e.g., hypothesis tests on the coefficients) are valid.
Violating these assumptions can lead to biased, inefficient, or misleading results.

1. Linearity

The relationship between the independent variable X and the dependent variable Y
must be linear. In other words, the model assumes that a straight line can best describe
the relationship between the two variables.

• How to check: Plot a scatter plot of the data to visually check for a linear
relationship between X and Y. If the relationship appears curved or non-linear,
simple linear regression may not be appropriate.

2. Independence of Errors

The residuals (errors) should be independent of each other. This means that the error for
one observation should not influence the error for another observation. This
assumption is important for the validity of hypothesis tests and confidence intervals.

• How to check: This can be checked using Durbin-Watson test or by plotting

residuals versus time (if the data is time-series data). If there is a pattern (e.g.,
autocorrelation), then this assumption is violated.
3. Homoscedasticity

Homoscedasticity means that the variance of the residuals (errors) should remain
constant for all levels of the independent variable X. In other words, the spread of the
residuals should not increase or decrease systematically as the value of X changes.

• How to check: Plot the residuals versus the fitted values. If the plot shows a fan
or cone shape (i.e., the spread of residuals increases or decreases with X), the
assumption of homoscedasticity is violated. This issue is called
heteroscedasticity.

4. Normality of Errors

The residuals (errors) should be normally distributed. This assumption allows us to

make statistical inferences, such as hypothesis tests for the regression coefficients. If
the errors are not normally distributed, the confidence intervals and significance tests
may not be valid.

• How to check: You can use a histogram or a Q-Q plot (Quantile-Quantile plot)
to visually assess the normality of the residuals. Additionally, a Shapiro-Wilk
test or Kolmogorov-Smirnov test can be used to formally test for normality.

5. No Multicollinearity

In simple linear regression, this assumption is naturally satisfied because there is only
one independent variable. However, if there were multiple independent variables (in
multiple linear regression), the independent variables should not be highly correlated
with each other. High correlation between predictors can cause instability in the
estimated coefficients, making them unreliable.

• How to check: In simple linear regression, multicollinearity is not an issue.

However, in multiple linear regression, you can check for multicollinearity using
Variance Inflation Factor (VIF).
3. Residual Plot:

A residual plot shows the difference between the actual and predicted values. It is used
to check for homoscedasticity and independence of errors.

• A good residual plot should show no patterns, indicating that the residuals are
randomly distributed and that the model is well specified.
6. Conclusion

Simple Linear Regression is a powerful tool for modeling the relationship between two
variables. However, for the model to be valid, the assumptions must be checked and
satisfied. Understanding and interpreting the coefficients and evaluation metrics are
key to deriving meaningful conclusions from a linear regression model.

By adhering to the assumptions and properly evaluating the model, simple linear
regression can be a valuable technique for prediction and inference.
Multivariate Linear Regression

1. Introduction

Multivariate Linear Regression is a statistical technique used to model the relationship

between two or more features (independent variables) and a dependent variable. It is an
extension of simple linear regression that allows us to handle multiple predictors.

• Linear regression assumes that the dependent variable is a linear combination

of the independent variables.

• The general goal is to model the dependent variable y using multiple predictors
x1,x2,…,xp.
Model Assessment and Variable Importance

1. Introduction to Model Assessment and Variable Importance

In machine learning and statistics, two key aspects of model evaluation and
interpretation are model assessment and variable importance. These concepts help
us understand how well our model is performing and what features (variables) are
driving the model's predictions.

• Model Assessment refers to the process of evaluating how well a predictive

model performs, helping us determine if it is accurate, reliable, and generalizes
well to unseen data.

• Variable Importance refers to understanding which features in the dataset have

the greatest influence on the model's predictions, offering insights into the
relationships between variables.

Both concepts are essential for building effective and interpretable models.

2. Model Assessment

Model assessment is critical to determining the performance of a model. It provides a

way to evaluate how well a model makes predictions and if it can generalize to new,
unseen data. Different assessment methods are used depending on whether the model
is used for regression (predicting continuous variables) or classification (predicting
categorical outcomes).
3. Variable Importance

Variable importance refers to the contribution of each predictor (or feature) to the
model’s predictions. Knowing which features are most influential can help improve the
model by focusing on important variables, simplifying the model, and improving
interpretability.

A. Variable Importance in Regression Models

1. Coefficients:

o In linear regression models, the size of the coefficients reflects the impact
of each feature on the target variable. Larger absolute coefficients
indicate more influential features. For instance, in a model predicting
house prices, the coefficient for the square footage might be larger than
the coefficient for the number of bedrooms, indicating that square
footage is more influential.

2. p-Values:

o p-values assess the statistical significance of each feature. A small p-

value (typically less than 0.05) suggests that the corresponding feature
has a statistically significant relationship with the target variable.
Features with high p-values might be removed from the model to improve
simplicity.

3. Standardized Coefficients:
 o When features are measured on different scales, standardized
coefficients allow for direct comparison of their relative importance.
Standardizing the data (subtracting the mean and dividing by the standard
deviation) puts all features on the same scale, making it easier to
compare their impact.

B. Variable Importance in Classification Models

1. Decision Trees and Random Forests:

o Gini Importance / Information Gain: In tree-based models, features that

produce the most significant splits in the data are considered more
important. The Gini index or Information Gain measures how well each
feature helps reduce impurity (disorder) in the data.

o Feature Importance in Random Forests: Random forests average the

feature importance from many decision trees, providing a robust measure
of variable importance.

2. Permutation Importance:

o Definition: Permutation importance involves randomly shuffling the

values of a feature and measuring how much the model’s performance
degrades. Features that cause a large drop in performance when shuffled
are considered more important.

3. SHAP (Shapley Additive Explanations):

o Definition: SHAP values break down the prediction for a given data point
into contributions from each feature, providing a more interpretable
understanding of variable importance.

o Interpretation: SHAP values can be particularly helpful when working

with complex models like deep learning and ensemble methods.

4. Conclusion

• Model Assessment: It is a process of evaluating how well a model fits the data
and generalizes to new data, using metrics such as R², MSE, RMSE, and various
metrics for classification (accuracy, precision, recall, F1 score, etc.).

• Variable Importance: Understanding which predictors drive the model's

predictions helps in model interpretation and variable selection. It can be
 assessed through coefficients, feature importance in tree-based models, and
advanced methods like SHAP values.

Both model assessment and variable importance are essential for building effective,
interpretable models and improving the model's performance through continuous
refinement.
Subset Selection

1. Introduction to Subset Selection

In machine learning and statistics, subset selection (or feature selection) refers to the
process of selecting a subset of relevant features (or predictors) from a larger set of
features. The goal is to improve model performance, reduce overfitting, and enhance
interpretability by eliminating irrelevant or redundant features.

Subset selection is crucial in scenarios where:

• The dataset contains many features (high-dimensional data), which could lead
to overfitting or unnecessarily complex models.

• The model's interpretability is important (e.g., in business or healthcare

applications).

Subset selection methods are broadly categorized into:

• Filter Methods

• Wrapper Methods

• Embedded Methods

Each method has its strengths, depending on the problem at hand.

2. Motivations for Subset Selection

• Improved model performance: By removing irrelevant or redundant features,

the model becomes simpler and often performs better on unseen data.

• Reduced overfitting: Fewer features reduce the model’s capacity to memorize

noise in the data, thereby improving generalization.

• Faster computation: Smaller models are quicker to train and deploy, especially
when dealing with large datasets.

• Enhanced interpretability: A model with fewer features is easier to understand

and explain, which is often necessary for decision-making in industries like
healthcare, finance, and law.
3. Subset Selection Methods

A. Filter Methods

Filter methods evaluate the relevance of each feature individually, based on its
relationship with the target variable, independently of any model. These methods are
simple, computationally efficient, and often used as a first step in feature selection.

1. Correlation-based Selection:

o Features that are highly correlated with the target variable are selected,
while those that are highly correlated with other features
(multicollinearity) may be removed.

o Example: Pearson correlation coefficient for continuous variables or Chi-

square tests for categorical variables.

2. Statistical Tests:

o ANOVA (Analysis of Variance): Used for comparing means between

groups in the case of categorical independent variables and a continuous
dependent variable.

o Chi-Square Test: Measures the independence of categorical features and

the target variable.

o Mutual Information: Measures the amount of information shared

between a feature and the target variable, used for both categorical and
continuous variables.

3. Univariate Feature Selection:

o In this approach, each feature is assessed independently using statistical

tests (like the ones mentioned above) to evaluate how much each feature
correlates with the target variable.

o Features are ranked, and the top-ranked features are selected.

Advantages of Filter Methods:

• Computationally efficient, especially with large datasets.

• Simple and easy to implement.

Disadvantages of Filter Methods:

• Does not account for interactions between features.

• May exclude features that are useful in combination with others.

B. Wrapper Methods

Wrapper methods evaluate feature subsets by training and testing a model on each
subset. These methods use the predictive performance of a model (such as accuracy or
cross-validation score) as a criterion for feature selection.

1. Forward Selection:

o Starts with no features and adds the best feature at each step based on a
criterion (usually model performance). The process stops when adding
more features does not improve performance.

o Steps:

▪ Begin with an empty set of features.

▪ Add the feature that improves model performance the most (e.g.,
using cross-validation).

▪ Repeat until adding more features no longer improves

performance.

2. Backward Elimination:

o Starts with all features and removes the least useful feature at each step.
The process continues until removing features degrades model
performance.

o Steps:

▪ Begin with all features.

▪ Remove the feature that worsens model performance the least

(e.g., using cross-validation).

▪ Repeat until removing features reduces performance.

3. Stepwise Selection:

o A combination of forward selection and backward elimination. At each

step, it evaluates both adding new features and removing existing
features.

o Steps:

▪ Start with a subset of features (usually empty or full).

▪ Add or remove features based on improvement in model

performance.

▪ Continue until no further improvements are made.

Advantages of Wrapper Methods:

• Takes into account interactions between features.

• Can result in higher-performing models than filter methods, as the model’s

performance directly guides feature selection.

Disadvantages of Wrapper Methods:

• Computationally expensive, especially with large datasets.

• Time-consuming, as it requires multiple model training processes.

C. Embedded Methods

Embedded methods perform feature selection during the model training process. These
methods select features by penalizing the complexity of the model (e.g., using
regularization) and automatically select the most important features as part of the
model fitting.
 1. Decision Tree-based Feature Selection:

o Tree-based algorithms, like Random Forests or Gradient Boosting, can

provide a feature importance score based on how well each feature
improves the model's accuracy. The most important features are those
that contribute the most to the reduction in impurity (e.g., Gini index,
entropy).

Advantages of Embedded Methods:

• More computationally efficient than wrapper methods.

• Features are selected during model training, which often results in better
performance.

Disadvantages of Embedded Methods:

• The feature selection process is tightly coupled with the model, so it may not
generalize well to other models.

• Regularization methods like LASSO may require fine-tuning of hyperparameters

(e.g., λ).
5. Practical Considerations

• Feature scaling: Some subset selection methods (e.g., LASSO) are sensitive to
the scale of features. It is important to standardize or normalize the features
before applying methods like LASSO.

• Dimensionality and Overfitting: While high-dimensional datasets may benefit

from feature selection, the risk of overfitting increases if too many irrelevant
features are included. Subset selection helps mitigate this risk.

• Data type and nature: Some methods, like ANOVA or chi-squared tests, are
more suitable for categorical features, while others like correlation-based
methods are more suited for continuous features.

• Cross-validation: For wrapper methods, it’s often recommended to use cross-

validation to evaluate the performance of different feature subsets to avoid
overfitting to the training set.

6. Conclusion

Subset selection is a powerful tool for improving the performance and interpretability of
predictive models. The choice of method depends on the specific problem, data
characteristics, and computational resources. By reducing the number of irrelevant or
redundant features, subset selection can lead to simpler, faster, and more generalizable
models.
Classification Techniques

1. Introduction to Classification

Classification is a type of supervised learning task where the goal is to assign each input
to one of the predefined categories or classes. In contrast to regression, which deals
with predicting continuous outcomes, classification predicts discrete values or
categories.

For example, in email spam detection, the goal is to classify emails as either spam or
not spam.

Key points in classification:

• Supervised learning: The model is trained on labeled data, i.e., data that
includes both features and corresponding target labels.

• Categorical outcomes: The output variable is categorical (e.g., "yes" vs. "no" or
"spam" vs. "not spam").

2. Common Classification Algorithms

Several classification algorithms exist, each with strengths and weaknesses. Let's
discuss the most popular techniques.

A. Logistic Regression

Logistic regression is one of the simplest and most commonly used classification
algorithms, especially when the relationship between the predictors and the target
variable is approximately linear.

• Concept: Logistic regression models the probability of a binary outcome using

the logistic function (also called the sigmoid function).

• Logistic Function (Sigmoid): The logistic function outputs values between 0 and
1, making it suitable for binary classification. It’s defined as:
 • Advantages:

o Simple and interpretable.

o Works well when the relationship between features and outcome is

roughly linear.

• Disadvantages:

o Assumes linearity between features and log-odds.

o Not suitable for more complex relationships (unless extended to non-

linear models like polynomial logistic regression).

B. Decision Trees

Decision trees are a powerful and easy-to-understand algorithm that can handle both
categorical and continuous variables.

• Concept: Decision trees recursively partition the feature space into subsets
based on feature values, and each subset corresponds to a class label. The goal
is to maximize "information gain" (or equivalently, minimize impurity) at each
split.

o Gini Impurity and Entropy (used in classification) are common metrics to

determine the best feature to split on:

▪ Gini Impurity: Measures the "impurity" of a node. A Gini impurity

of 0 means all the data points belong to a single class.

▪ Entropy: A measure of uncertainty. The goal is to split the dataset

such that the resulting subgroups have low entropy.

• Advantages:

o Easy to visualize and interpret.

o Handles both numerical and categorical data.

o Non-parametric, so it doesn't require assumptions about the underlying

data distribution.

• Disadvantages:

o Prone to overfitting, especially with very deep trees.

o Sensitive to small changes in the data (can lead to different trees being
learned).
 • Pruning: To avoid overfitting, decision trees are often pruned after training by
setting a maximum tree depth or minimum number of samples per leaf.

C. Random Forest

Random Forest is an ensemble learning method that builds multiple decision trees and
merges them together to improve accuracy and control overfitting.

• Concept: Random forests work by constructing multiple decision trees using

random subsets of the training data and random feature selection at each node
(Bootstrap aggregating or "bagging"). Each tree in the forest is trained on a
different subset of the data, and the final prediction is made by averaging the
predictions from all trees (for regression) or using majority voting (for
classification).

• Advantages:

o Reduces overfitting by averaging over multiple trees.

o Handles large datasets with high dimensionality well.

o More robust and accurate than individual decision trees.

• Disadvantages:

o Less interpretable than a single decision tree.

o Computationally expensive, especially for large datasets.

D. k-Nearest Neighbors (k-NN)

k-NN is a simple and intuitive algorithm that classifies new data points based on the
majority class of the nearest neighbors in the feature space.

• Concept: Given a new observation, the k-NN algorithm identifies the 'k' closest
data points in the training dataset and assigns the majority class label among
those neighbors.

o Distance Metric: Commonly used distance metrics include:

▪ Euclidean Distance (for continuous data).

▪ Manhattan Distance (for data with categorical variables).

o Parameter 'k': The number of neighbors to consider. A smaller k can lead

to more noise, while a larger k makes the algorithm more resistant to
noise but may blur the decision boundaries.
 • Advantages:

o Simple and intuitive.

o No explicit model training, which makes it easy to implement.

o Works well for smaller datasets and when the decision boundary is highly
non-linear.

• Disadvantages:

o Computationally expensive for large datasets.

o Performance can degrade with high-dimensional data due to the "curse of

dimensionality."

o Sensitive to the choice of k and distance metric.

E. Support Vector Machines (SVM)

SVM is a powerful classifier that works well for both linear and non-linear classification
tasks by finding the optimal hyperplane that separates data points from different
classes.

• Concept: SVM aims to find the hyperplane that maximizes the margin between
the two classes. In cases where the data is not linearly separable, SVM uses a
technique called the kernel trick to map the data into a higher-dimensional
space where a linear separation is possible.

o Linear SVM: Works when the data can be separated by a straight line or
hyperplane.

o Non-linear SVM: Uses kernels like RBF (Radial Basis Function) or

Polynomial to create non-linear decision boundaries.

• Advantages:

o Effective in high-dimensional spaces.

o Works well with clear margins of separation.

o Robust to overfitting, especially in high-dimensional space.

• Disadvantages:

o Memory and computationally expensive.

o Requires careful tuning of hyperparameters like the regularization

parameter and kernel choice.
F. Naive Bayes

Naive Bayes is a probabilistic classifier based on Bayes' Theorem, assuming

independence among features. It is particularly useful when the dimensionality is high
and when the features are conditionally independent given the class.

• Concept: Naive Bayes calculates the posterior probability for each class and
assigns the class with the highest probability.

• Advantages:

o Simple and computationally efficient.

o Performs well with high-dimensional data and when the independence

assumption holds reasonably well.

• Disadvantages:

o Assumes features are conditionally independent, which is often not the

case in real-world data.

o Not as flexible as other algorithms for capturing complex relationships.

3. Model Evaluation Metrics for Classification

To assess the performance of classification models, several evaluation metrics are

used:

• Accuracy: The proportion of correctly predicted instances among all

predictions.
4. Conclusion

Classification techniques are fundamental in supervised machine learning and are

used in a wide variety of applications, from spam detection to medical diagnosis. The
choice of classification algorithm depends on the nature of the data, the complexity of
the decision boundary, interpretability needs, and computational resources available.

• Logistic Regression: Suitable for linear problems.

• Decision Trees: Good for capturing non-linear relationships and easy to

interpret.

• Random Forest: More robust than decision trees and effective for high-
dimensional data.

• k-NN: Simple and effective for small, non-linear datasets.

• SVM: Powerful for high-dimensional and complex classification tasks.

• Naive Bayes: Efficient for text classification and high-dimensional data.

Each algorithm has its trade-offs in terms of computational complexity, interpretability,

and performance, and it is often necessary to experiment with different methods to find
the most appropriate model for a given task.
Classification using Logistic Regression

Introduction to Logistic Regression

Logistic Regression is one of the most widely used statistical methods for binary
classification tasks. Despite its name, it is not a regression technique but a
classification algorithm. It is used to predict the probability that a given input belongs to
one of two classes (often represented as class 0 or class 1, or "True" vs. "False").

The logistic regression model outputs probabilities, which can then be used to assign a
class label based on a decision threshold (commonly 0.5).
2. Training Logistic Regression Model

The goal of training a logistic regression model is to find the best-fitting values for the
weights w\mathbf{w}w and the bias bbb, so that the predicted probabilities are as close
as possible to the actual class labels in the training data.

Cost Function (Log-Loss or Cross-Entropy Loss)

To evaluate how well the model fits the data, we use the logistic loss function (also
known as binary cross-entropy loss), which penalizes the model for incorrect
predictions. The cost function for logistic regression is:
Conclusion

Logistic Regression is a powerful and interpretable algorithm for binary classification,

capable of being extended to multiclass classification through multinomial logistic
regression. It is widely used for its simplicity and effectiveness. By understanding the
mathematical foundation, training procedure, evaluation metrics, and regularization
techniques, you can leverage logistic regression in a wide variety of machine learning
tasks.