T. Y. B.

Tech (Computer Engineering)

Academic Year -2026-27 (Semester –V)

[CS3201T]:- Machine Learning

UNIT-II Supervised Learning: Regression 08 Hours

Regression Techniques: Linear Regression, Polynomial Regression, Ridge and Lasso Regression,
Logistic Regression; Model Evaluation: Train-test split, cross-validation; Metrics: Accuracy,
precision, recall, F1 score, ROC-AUC, Applications.
Case Study: House Price Prediction using Linear, Ridge and Polynomial Regression

Regression Analysis
Regression Analysis is a fundamental concept in machine learning used to model relationships
between dependent and independent variables. Various regression techniques are tailored to
different data structures and objectives. Below is an exploration of key regression techniques,
their significance, and practical examples.

1. Linear Regression in Machine learning

Linear regression is a type of supervised machine-learning algorithm that learns from the
labelled datasets and maps the data points with most optimized linear functions which can be
used for prediction on new datasets. It assumes that there is a linear relationship between the
input and output, meaning the output changes at a constant rate as the input changes. This
relationship is represented by a straight line.
For example, we want to predict a student's exam score based on how many hours they studied.
We observe that as students study more hours, their scores go up. In the example of predicting
exam scores based on hours studied. Here
• Independent variable (input): Hours studied because it's the factor we control or
observe.
• Dependent variable (output): Exam score because it depends on how many hours
were studied.
Best Fit Line in Linear Regression
In linear regression, the best-fit line is the straight line that most accurately represents the
relationship between the independent variable (input) and the dependent variable (output). It is
the line that minimizes the difference between the actual data points and the predicted values
from the model.
1. Goal of the Best-Fit Line
The goal of linear regression is to find a straight line that minimizes the error (the difference)
between the observed data points and the predicted values. This line helps us predict the
dependent variable for new, unseen data.
 Linear Regression
Here Y is called a dependent or target variable and X is called an independent variable also
known as the predictor of Y. There are many types of functions or modules that can be used for
regression. A linear function is the simplest type of function. Here, X may be a single feature
or multiple features representing the problem.

2. Equation of the Best-Fit Line

For simple linear regression (with one independent variable), the best-fit line is represented by
the equation
y=mx+by
Where:
• y is the predicted value (dependent variable)
• x is the input (independent variable)
• m is the slope of the line (how much y changes when x changes)
• b is the intercept (the value of y when x = 0)
The best-fit line will be the one that optimizes the values of m (slope) and b (intercept) so that
the predicted y values are as close as possible to the actual data points.

3. Minimizing the Error: The Least Squares Method

To find the best-fit line, we use a method called Least Squares. The idea behind this method
is to minimize the sum of squared differences between the actual values (data points) and the
predicted values from the line. These differences are called residuals.
The formula for residuals is:
Residual=yᵢ−y^ᵢ
Where:
• yᵢ is the actual observed value
• y^ᵢ is the predicted value from the line for that xᵢ
The least squares method minimizes the sum of the squared residuals:
Sum of squared error s(SSE)=Σ(yᵢ−y^ᵢ)²
This method ensures that the line best represents the data where the sum of the squared
differences between the predicted values and actual values is as small as possible.
4. Interpretation of the Best-Fit Line
• Slope (m): The slope of the best-fit line indicates how much the dependent variable (y)
changes with each unit change in the independent variable (x). For example if the slope
is 5, it means that for every 1-unit increase in x, the value of y increases by 5 units.
• Intercept (b): The intercept represents the predicted value of y when x = 0. It’s the
point where the line crosses the y-axis.
In linear regression some hypothesis are made to ensure reliability of the model's results.
Limitations
• Assumes Linearity: The method assumes the relationship between the variables is
linear. If the relationship is non-linear, linear regression might not work well.
• Sensitivity to Outliers: Outliers can significantly affect the slope and intercept,
skewing the best-fit line.

For multiple linear regression (with more than one independent variable), the hypothesis
function expands to:
h(x₁,x₂,...,xₖ)=β₀+β₁x₁+β₂x₂+...+βₖxₖh(x₁,x₂,...,xₖ)=β₀+β₁x₁+β₂x₂+...+βₖxₖ
Where:
• x₁,x₂,...,xₖx₁,x₂,...,xₖ are the independent variables.
• β₀β₀ is the intercept.
• β₁,β₂,...,βₖ are the coefficients, representing the influence of each respective
independent variable on the predicted output.

Regression models are very useful to describe relationships between variables by fitting a line
to the observed data. Regression helps us to estimate the change of a dependent variable
according to the independent variable change. Regression analysis is a set of statistical methods
which is used for the estimation of relationships between a dependent variable and one or more
independent variables. It can be also utilized to assess the strength of the relationship between
variables and for modeling the future relationship between them.
There are many types of Regression analysis, like Linear Regression, Logistic Regression,
Multiple Regression, Ridge Regression, Lasso, and many more.

Advantages and Disadvantages of Multiple Regression

The main advantages and disadvantages of Multiple Regression are tabulated below.
Advantages Disadvantages
o It has the ability to determine the o It needs high-level mathematics to analyze the data
relative influence of one or more and is required in the statistical program.
predictor variables to the
criterion value.
o It also has the ability to identify o It is difficult for researchers to interpret the results
outliers, or anomalies. of the multiple regression analysis on the basis of
assumptions as it has a requirement of a large
sample of data to get the effective results.
Difference between Simple Regression and Multiple Regression
The difference between Simple and Multiple Regression is tabulated below.
Simple Regression Multiple Regression
o One dependent variable Y is o One dependent variable Y is predicted from a set of
predicted from one independent independent variables (X1, X2, …, Xk).
variable X.
o There is only one regression o There is one regression coefficient for each
coefficient. independent variable.
o R2: proportion of variation in o R2: proportion of variation in dependent variable
dependent variable Y is Y is predictable by a set of independent variables
predictable from X. (X’s).
o Outcome variable: one o Outcome variable: a set of explanatory variables.
explanatory variable.
o β represents unit change in Y for o βi represents the unit change in Y per unit change
per unit change in X. in Xi.
Polynomial Regression
Polynomial Linear Regression is a type of regression analysis in which the relationship
between the independent variable and the dependent variable is modeled as an n-th degree
polynomial function. Polynomial regression allows for a more complex relationship between
the variables to be captured beyond the linear relationship in simple linear
regression and multiple linear regression. Polynomial regression is an extension of linear
regression where higher-degree terms are added to model non-linear relationships.

Why Polynomial Regression?

In machine learning (ML) and data science, choosing between a linear regression or
polynomial regression depends upon the characteristics of the dataset. A non-linear dataset can't
be fitted with a linear regression. If we apply linear regression to a nonlinear dataset, it will not
be able to capture the non-linear patterns in the data.
Look at the below diagram to understand why we need polynomial regression for non-linear
data.

The above diagram shows the simple linear model hardly fits the data points whereas the
polynomial model fits most of the data points.
Equation of Polynomial Regression Model
In machine learning, the general formula for polynomial regression of degree n is as follows

Where
• y is the dependent variable (output).
• x is the independent variable (input).
• w0,w1,w2,…,wn are the coefficients (parameters) of the model.
• n is the degree of the polynomial (the highest power of x).
 • ϵ is the error term or residual, representing the difference between the observed value
and the model's prediction.
For a quadratic (second-degree) polynomial regression, the formula would be:

How does Polynomial Regression Work?

In machine learning, the polynomial regression actually works in a similar way as linear
regression works. It is modeled as multiple linear regression. The input feature is transformed
into polynomial features of higher degrees (x2,x3,...,xn). These features are now treated as
separate independent variables as in multiple linear regression. Now, a multiple linear regressor
is trained on these transformed polynomial features.
The polynomial regression is a special case of multiple linear regression but there is a
difference that multiple linear regression assumes linearity of input features. Here, in
polynomial regression, the transformed polynomial features are dependent on the original input
feature.
While the regression function is linear in terms of the unknown coefficients 𝑏0,𝑏1,…,𝑏𝑛 the
model itself captures non-linear patterns in the data. The coefficients are estimated using
techniques like Least Square technique to minimize the error between predicted and actual
values.
Choosing the right polynomial degree n is important: a higher degree may fit the data more
closely but it can lead to overfitting. The degree should be selected based on the complexity of
the data. Once the model is trained, it can be used to make predictions on new data, capturing
non-linear relationships and providing a more accurate model for real-world applications.
Logistic Regression
Logistic Regression is a supervised machine learning algorithm used for classification
problems. Unlike linear regression which predicts continuous values it predicts the probability
that an input belongs to a specific class. It is used for binary classification where the output can
be one of two possible categories such as Yes/No, True/False or 0/1. It uses sigmoid function
to convert inputs into a probability value between 0 and 1.
Logistic regression is a supervised learning classification algorithm used to predict the
probability of a target variable. The nature of target or dependent variable is dichotomous,
which means there would be only two possible classes.
In simple words, the dependent variable is binary in nature having data coded as either 1 (stands
for success/yes) or 0 (stands for failure/no).
Mathematically, a logistic regression model predicts P(Y=1) as a function of X. It is one of the
simplest ML algorithms that can be used for various classification problems such as spam
detection, Diabetes prediction, cancer detection etc.

Types of Logistic Regression

Generally, logistic regression means binary logistic regression having binary target variables,
but there can be two more categories of target variables that can be predicted by it. Based on
those number of categories, Logistic regression can be divided into following types −
Binary or Binomial
In such a kind of classification, a dependent variable will have only two possible types either
1 and 0. For example, these variables may represent success or failure, yes or no, win or loss
etc.
Multinomial
In such a kind of classification, dependent variable can have 3 or more possible unordered
types or the types having no quantitative significance. For example, these variables may
represent "Type A" or "Type B" or "Type C".
Ordinal
In such a kind of classification, dependent variable can have 3 or more possible ordered types
or the types having a quantitative significance. For example, these variables may represent
"poor" or "good", "very good", "Excellent" and each category can have the scores like 0,1,2,3.

Logistic Regression Assumptions

Before diving into the implementation of logistic regression, we must be aware of the following
assumptions about the same −
• In case of binary logistic regression, the target variables must be binary always and the
desired outcome is represented by the factor level 1.
• There should not be any multi-collinearity in the model, which means the independent
variables must be independent of each other .
• We must include meaningful variables in our model.
• We should choose a large sample size for logistic regression.

Understanding Sigmoid Function

1. The sigmoid function is a important part of logistic regression which is used to convert the
raw output of the model into a probability value between 0 and 1.
2. This function takes any real number and maps it into the range 0 to 1 forming an "S" shaped
curve called the sigmoid curve or logistic curve. Because probabilities must lie between 0 and
1, the sigmoid function is perfect for this purpose.
3. In logistic regression, we use a threshold value usually 0.5 to decide the class label.
• If the sigmoid output is same or above the threshold, the input is classified as Class 1.
• If it is below the threshold, the input is classified as Class 0.
This approach helps to transform continuous input values into meaningful class predictions.
How does Logistic Regression work?
Logistic regression model transforms the linear regression function continuous value output
into categorical value output using a sigmoid function which maps any real-valued set of
independent variables input into a value between 0 and 1. This function is known as the logistic
function.
Suppose we have input features represented as a matrix:

and the dependent variable is Y having only binary value i.e 0 or 1.

then, apply the multi-linear function to the input variables X.

Here xi is the ith observation of X, wi=[w1,w2,w3,⋯,wm] is the weights or Coefficient and b

is the bias term also known as intercept. Simply this can be represented as the dot product of
weight and bias.

At this stage, z is a continuous value from the linear regression. Logistic regression then applies
the sigmoid function to z to convert it into a probability between 0 and 1 which can be used to
predict the class.
Now we use the sigmoid function where the input will be z and we find the probability between
0 and 1. i.e. predicted y.
As shown above the sigmoid function converts the continuous variable data into the probability
i.e between 0 and 1.

where the probability of being a class can be measured as:

Cost function for Linear Regression
As we have discussed earlier about best fit line in linear regression, its not easy to get it easily
in real life cases so we need to calculate errors that affects it. These errors need to be calculated
to mitigate them. The difference between the predicted value Y^ and the true value Y and it is
called cost function or the loss function.
In Linear Regression, the Mean Squared Error (MSE) cost function is employed, which
calculates the average of the squared errors between the predicted values y^i and the actual
values yi. The purpose is to determine the optimal values for the intercept θ1 and the coefficient
of the input feature θ2 providing the best-fit line for the given data points. The linear equation
expressing this relationship is y^i=θ1+θ2xi.
MSE function can be calculated as:

Utilizing the MSE function, the iterative process of gradient descent is applied to update the
values of θ1&θ2. This ensures that the MSE value converges to the global minima, signifying
the most accurate fit of the linear regression line to the dataset.
This process involves continuously adjusting the parameters θ1&θ2 based on the gradients
calculated from the MSE. The final result is a linear regression line that minimizes the overall
squared differences between the predicted and actual values, providing an optimal
representation of the underlying relationship in the data.
Now we have calculated loss function we need to optimize model to mitigate this error and it
is done through gradient descent.

Gradient Descent for Linear Regression

Gradient descent is an optimization technique used to train a linear regression model by

minimizing the prediction error. It works by starting with random model parameters and
repeatedly adjusting them to reduce the difference between predicted and actual values.

Gradient Descent
How it works:
• Start with random values for slope and intercept.
• Calculate the error between predicted and actual values.
• Find how much each parameter contributes to the error (gradient).
• Update the parameters in the direction that reduces the error.
• Repeat until the error is as small as possible.
This helps the model find the best-fit line for the data.

Ridge and Lasso Regression

Ridge and Lasso Regression are two popular techniques in machine learning used
for regularizing linear models to avoid overfitting and improve predictive performance. Both
methods add a penalty term to the model’s cost function to constrain the coefficients, but they
differ in how they apply this penalty.
 • Ridge Regression, also known as L2 regularization, adds the squared magnitude of the
coefficients as a penalty.
• On the other hand, Lasso Regression, or L1 regularization, introduces a penalty based
on the absolute value of the coefficients.
What is Ridge Regression or (L2 Regularization) Method?
Ridge regression, also known as L2 regularization, is a technique used in linear regression to
prevent overfitting by adding a penalty term to the loss function. This penalty is proportional
to the square of the magnitude of the coefficients (weights).
Ridge Regression is a version of linear regression that includes a penalty to prevent the model
from overfitting, especially when there are many predictors or not enough data.
The standard loss function (mean squared error) is modified to include a regularization term:

Here, λ is the regularization parameter that controls the strength of the penalty, and wi are the
coefficients.

Ridge regression, also known as L2 regularization, is a technique used in linear regression to

address the problem of multicollinearity among predictor variables. Multicollinearity
occurs when independent variables in a regression model are highly correlated, which can lead
to unreliable and unstable estimates of regression coefficients.
Ridge regression mitigates this issue by adding a regularization term to the ordinary least
squares (OLS) objective function, which penalizes large coefficients and thus reduces their
variance.

What is Lasso Regression or (L1 Regularization) Method?

Lasso regression, also known as L1 regularization, is a linear regression technique that adds a
penalty to the loss function to prevent overfitting. This penalty is based on the absolute
values of the coefficients.
Lasso regression is a version of linear regression including a penalty equal to the absolute value
of the coefficient magnitude. By encouraging sparsity, this L1 regularization term reduces
overfitting and helps some coefficients to be absolutely zero, hence facilitating feature
selection.
The standard loss function (mean squared error) is modified to include a regularization term:

Here, λ is the regularization parameter that controls the strength of the penalty, and wi are the
coefficients.

Lasso Regression is a regression method based on Least Absolute Shrinkage and Selection
Operator and is used in regression analysis for variable selection and regularization. It helps
remove irrelevant data features and prevents overfitting. This allows features with weak
influence to be clearly identified as the coefficients of less important variables are shrunk
toward zero.
Difference between Ridge Regression and Lasso Regression
The key differences between ridge and lasso regression are discussed below:
Characteristic Ridge Regression Lasso Regression

Applies L2 regularization, Applies L1 regularization, adding a

Regularization adding a penalty term penalty term proportional to
Type proportional to the square of the the absolute value of the
coefficients coefficients.

Does not perform feature

Performs automatic feature
selection. All predictors are
Feature selection. Less important predictors
retained, although their
Selection are completely excluded by setting
coefficients are reduced in size to
their coefficients to zero.
minimize overfitting

Best suited for situations

Ideal when you suspect that only
where all predictors are
a subset of predictors is important,
When to use potentially relevant, and the goal
and the model should focus on those
is to reduce overfitting rather
while ignoring the irrelevant ones.
than eliminate features

Produces a model that Produces a model that is simpler,

includes all features, but their retaining only the most significant
Output model
coefficients are smaller in features and ignoring the rest by
magnitude to prevent overfitting setting their coefficients to zero.

Reduces the magnitude of

Shrinks some coefficients to exactly
coefficients, shrinking them
zero, effectively removing their
Impact on towards zero, but does not set
influence from the model. This leads
Prediction any coefficients exactly to zero.
to a simpler model with fewer
All predictors remain in the
features
model

Generally faster as it doesn’t May be slower due to the feature

Computation
involve feature selection selection process

Use when you have many

Use when you believe only some
predictors, all contributing to the
Example Use predictors are truly important (e.g.,
outcome (e.g., predicting house
Case genetic studies where only a few
prices where all features like
genes out of thousands are relevant).
size, location, etc., matter)

When to Use Ridge Regression?

Ridge Regression is most suitable when all predictors are expected to contribute to the outcome
and none should be excluded from the model. It reduces overfitting by shrinking the
coefficients, ensuring they don’t become too large, while still keeping all the predictors in the
model.
For example, when predicting house prices, features like size, number of bedrooms, location,
and year built are all likely relevant. Ridge Regression ensures these features remain in the
model but with reduced influence to create a balanced and robust prediction.
When to Use Lasso Regression?
Lasso Regression is ideal when you suspect that only a few predictors are truly important, and
the rest may add noise or redundancy. It performs automatic feature selection by shrinking the
coefficients of less important predictors to zero, effectively removing them from the model.
For example, in genetic research, where thousands of genes are analyzed for their effect on a
disease, Lasso Regression helps by identifying only the most impactful genes and ignoring the
irrelevant ones, leading to a simpler and more interpretable model.

Regression Metrics
In the regression task, we are supposed to predict the target variable which is in the form of
continuous values. To evaluate the performance of such a model below metrics are used:

1. Mean Absolute Error (MAE)

MAE calculates the average of the absolute differences between the predicted and actual
values. It gives a clear view of the model’s prediction accuracy but it doesn't shows whether
the errors are due to over- or under-prediction. It is simple to calculate and interpret helps in
making it a good starting point for model evaluation.

2. Mean Squared Error (MSE)

MSE calculates the average of the squared differences between the predicted and actual values.
Squaring the differences ensures that larger errors are penalized more heavily helps in making
it sensitive to outliers. This is useful when large errors are undesirable but it can be problematic
when outliers are not relevant to the model’s purpose.
Formula:

3. Root Mean Squared Error (RMSE)

RMSE is the square root of MSE, bringing the metric back to the original scale of the data.
Like MSE, it heavily penalizes larger errors but is easier to interpret as it’s in the same units as
the target variable. It’s useful when we want to know how much our predictions deviate from
the actual values in terms of the same scale.
Formula:
4. R² (R-squared)
R2 score represents the proportion of the variance in the dependent variable that is predictable
from the independent variables. An R² value close to 1 shows a model that explains most of the
variance while a value close to 0 shows that the model does not explain much of the variability
in the data. R² is used to assess the goodness-of-fit of regression models.
Formula:

Classification Metrics

Confusion Matrix:
A confusion matrix or error matrix is a table that shows the number of correct and incorrect
predictions made by the model compared with the actual classifications in the test set or what
type of errors are being made.
This matrix describes the performance of a classification model on test data for which true
values are known. It is a n*n matrix, where n is the number of classes. This matrix can be
generated after making predictions on the test data.

Here, columns represent the count of actual classifications in the test data while rows represent
the count of predicted classifications made by the model.
Let’s take an example of a classification problem where we are predicting whether a person is
having diabetes or not. Let’s give a label to our target variable:
1: A person is having diabetes | 0: A person is not having diabetes

Four possible outcomes could occur while performing classification predictions:

• True Positives (TP): Number of outcomes that are actually positive and are predicted
positive.
For example: In this case, a person is actually having diabetes(1) and the model predicted that
the person has diabetes(1).
• True Negatives (TN): Number of outcomes that are actually negative and are predicted
negative.
For example: In this case, a person actually doesn’t have diabetes(0) and the model predicted
that the person doesn’t have diabetes(0).
• False Positives (FP): Number of outcomes that are actually negative but predicted
positive. These errors are also called Type 1 Errors.
For example: In this case, a person actually doesn’t have diabetes(0) but the model predicted
that the person has diabetes(1).
• False Negatives (FN): Number of outcomes that are actually positive but predicted
negative. These errors are also called Type 2 Errors.

for example: In this case, a person actually has diabetes(1) but the model predicted that the
person doesn’t have diabetes(0).

Positive and Negatives refers to the prediction itself. True and False refers to the correctness
of the prediction.

In short
There are 4 terms we should keep in mind:
1. True Positives: It is the case where we predicted Yes and the real output was also
Yes.
2. True Negatives: It is the case where we predicted No and the real output was also
No.
3. False Positives: It is the case where we predicted Yes but it was actually No.
4. False Negatives: It is the case where we predicted No but it was actually Yes.

Classification problems aim to predict discrete categories. To evaluate the performance of

classification models, we use the following metrics:

1. Accuracy
Accuracy is a fundamental metric used for evaluating the performance of a classification
model. It tells us the proportion of correct predictions made by the model out of all predictions.

However, accuracy may be misleading when dealing with imbalanced datasets, where one class
significantly outweighs the other.
Strengths of Accuracy:
• Easy Interpretation: Accuracy is easy to understand and interpret. It is expressed as a
percentage, making it accessible to both technical and non-technical stakeholders.
 • Suitable for Balanced Datasets: Accuracy is a reliable metric when dealing with
balanced datasets, where each class has roughly equal representation. In such cases, it
provides an accurate reflection of model performance.
Limitations of Accuracy:
• Imbalanced Datasets: Accuracy can be misleading when dealing with imbalanced
datasets, where one class significantly outnumbers the other. In imbalanced scenarios,
a model that predicts the majority class for all instances can achieve a high accuracy
simply because it correctly predicts the dominant class. This can lead to a false sense
of model effectiveness.
• Misleading in Critical Applications: In some applications, the cost of false positives and
false negatives may vary significantly. Accuracy treats all types of errors equally, which
may not be suitable for situations where the consequences of different types of errors
differ. For instance, in medical diagnostics, a false negative (missed disease) could be
life-threatening, whereas a false positive (unnecessary treatment) might have less
severe consequences.
When to Use Accuracy
Accuracy is a valuable metric in scenarios where class balance is not a concern and the cost of
misclassification errors is relatively equal for all classes. It is commonly used as a starting point
for evaluating models but should be complemented with other metrics, such as precision, recall,
F1-score, and the analysis of a confusion matrix, to gain a more comprehensive understanding
of a model's performance, especially in imbalanced or critical applications.

2. Precision
It measures how many of the positive predictions made by the model are actually correct. It's
useful when the cost of false positives is high such as in medical diagnoses where predicting a
disease when it’s not present can have serious consequences.

Where:
• TP = True Positives
• FP = False Positives
Precision helps ensure that when the model predicts a positive outcome, it’s likely to be
correct.

3. Recall/ True Positive Rate (TPR)

Recall or Sensitivity measures how many of the actual positive cases were correctly identified
by the model. It is important when missing a positive case (false negative) is more costly than
false positives.

Where:
• FN = False Negatives
In scenarios where catching all positive cases is important (like disease detection), recall is a
key metric.

4. F1 Score
The F1 Score is the harmonic mean of precision and recall. It is useful when we need a balance
between precision and recall as it combines both into a single number. A high F1 score means
the model performs well on both metrics. Its range is [0,1].
Lower recall and higher precision gives us great accuracy but then it misses a large number of
instances. More the F1 score better will be performance. It can be expressed mathematically in
this way:

5. False Positive Rate (FPR): False Positive Rate corresponds to the proportion of negative
data points that are mistakenly considered as positive, for all negative data points.

ROC Curve
Receiver Operating Characteristic (ROC) Curve:
The Receiver Operating Characteristic (ROC) curve is a graphical representation of a
classification model's ability to distinguish between positive and negative classes at various
classification thresholds. It plots the True Positive Rate (TPR), also known as recall or
sensitivity, against the False Positive Rate (FPR), which is calculated as 1−Specificity.
The ROC curve visually illustrates how the model's performance changes as the threshold for
classifying an instance as positive varies.
In the ROC curve:
• The x-axis represents the False Positive Rate (FPR), which measures the proportion of
negative instances incorrectly classified as positive.
• The y-axis represents the True Positive Rate (TPR), which measures the proportion of
positive instances correctly classified as positive.\
• A typical ROC curve looks like an ascending curve, moving from the bottom-left corner
to the top-right corner of the plot. The ideal ROC curve would be a right-angle (90-
degree) curve from the bottom-left corner to the top-left corner, indicating perfect
discrimination between positive and negative instances at all thresholds.
Area Under the ROC Curve (AUC):
The Area Under the ROC Curve (AUC) quantifies the overall performance of a classification
model. It measures the area under the ROC curve, ranging from 0 to 1, where:
• An AUC of 0.5 indicates that the model's performance is equivalent to random
guessing.
• An AUC of 1.0 indicates perfect discrimination, where the model can perfectly
distinguish between positive and negative instances at all thresholds.
• The AUC provides a single scalar value that summarizes the model's ability to rank
positive instances higher than negative instances, regardless of the specific threshold
chosen for classification. Higher AUC values indicate better model performance.
ROC Curve
It is a graphical representation of the True Positive Rate (TPR) vs the False Positive Rate (FPR)
at different classification thresholds. The curve helps us visualize the trade-offs between
sensitivity (TPR) and specificity (1 - FPR) across various thresholds.
Area Under Curve (AUC) quantifies the overall ability of the model to distinguish between
positive and negative classes.
• AUC = 1: Perfect model (always correctly classifies positives and negatives).
• AUC = 0.5: Model performs no better than random guessing.
• AUC < 0.5: Model performs worse than random guessing (showing that the model is
inverted).

ROC Curve for Evaluation of Classification Models

Significance of ROC Curve and AUC
• Model Comparison: ROC- AUC enable the comparison of multiple classification
models to determine which one performs better. A model with a higher AUC is
generally more effective at distinguishing between classes.
• Threshold Selection: ROC curves help in choosing an appropriate classification
threshold based on the specific application's requirements. You can select a threshold
that balances TPR and FPR according to the desired trade-off between true positives
and false positives.
• Imbalanced Datasets: ROC curves and AUC are particularly useful when dealing with
imbalanced datasets, where one class significantly outnumbers the other. These metrics
provide a more comprehensive evaluation of model performance beyond accuracy.

Actual Y=[3, −1, 2, 7]

Predicted Y^=[2.5, 0.0, 2, 8]