What is Machine Learning?

Machine Learning (ML) is a branch of Artificial Intelligence (AI) that works on

algorithm developments and statistical models that allow computers to learn from
data and make predictions or decisions.

How does Machine Learning Work?

Machine Learning algorithm is trained using a training data set to create a model.
When new input data is introduced to the ML algorithm, it makes a prediction on the
basis of the model. The prediction is evaluated for accuracy and if the accuracy is
acceptable, the Machine Learning algorithm is deployed.

If the accuracy is not acceptable, the Machine Learning algorithm is trained again
and again with an augmented raining data set.

Types of machine Learning

Supervised Machine Learning (SVM):

Unsupervised Machine Learning:
Semi-supervised Machine Learning:
Reinforcement Machine Learning:

VTU, CPGS, Mysuru

VTU, CPGS, Mysuru
VTU, CPGS, Mysuru
 Regression

Regression is a statistical method that helps to understand and predict the

relationship between variables.
Describes how one variable (dependent variable) changes as another variable
(independent variable) changes.
Dependent Variable: We are trying to predict or explain (Y).
Independent Variable: that are used to predict or explain the changes in the
dependent variable (X).

For example: predicting salary based on years of experience, predicting exam score
based on study hours, predicting resale price based on vehicle age.

1. Linear Regression
Linear regression is one of the most fundamental algorithms in the field of supervised
machine learning and statistics. Its primary purpose is to model the relationship
between two variables by fitting a linear equation to observed data. It is extensively
used for predictive analysis where the target variable is continuous, and the goal is to
predict or estimate future outcomes based on the relationships learned from historical
data.

In its simplest form, known as simple linear regression, the model analyzes the
relationship between a single independent variable (X) and a dependent variable (Y).
The linear equation for this model is:

VTU, CPGS, Mysuru

Y = mX + b

Here, 'Y' is the dependent variable we want to predict, 'X' is the independent variable
used for prediction, 'm' is the slope of the line (which quantifies the change in Y for a
one-unit change in X), and 'b' is the y-intercept (the predicted value of Y when X = 0).

The objective of linear regression is to determine the best-fitting line through the data
points, which minimizes the sum of squared differences between the observed values
and the values predicted by the model. This method is known as the least squares
method. The formulas for calculating the slope (m) and intercept (b) are derived using
this approach:

m = [n(∑XY) - (∑X)(∑Y)] / [n(∑X²) - (∑X)²] b = [∑Y - m(∑X)] / n

Where:

• ∑XY = sum of the product of corresponding X and Y values

• ∑X = sum of the X values

• ∑Y = sum of the Y values

• ∑X² = sum of the squares of X values

• n = number of observations

Let us consider a practical example. Suppose we want to predict a student's exam

score based on the number of hours studied. The dataset might look like this:

Hours Studied Exam Score

1 50

2 55

3 65

4 70

5 75

Using the least squares method, we can calculate the slope and intercept, fit the
regression line, and use it to predict the exam score for any number of study hours.

Graphically, the data points are plotted on a two-dimensional plane. A straight line is
drawn such that the distance (error) from each point to the line is minimized. The
quality of the linear regression model is often evaluated using metrics like Mean
Squared Error (MSE), Root Mean Squared Error (RMSE), and the Coefficient of
Determination (R²).

VTU, CPGS, Mysuru

Linear regression assumes a few key properties:

1. Linearity – The relationship between X and Y is linear.

2. Independence – The observations are independent of each other.

3. Homoscedasticity – The residuals (errors) have constant variance.

4. Normality – The residuals are normally distributed.

Violations of these assumptions can affect the accuracy and reliability of the model.
Despite these limitations, linear regression is a powerful tool when used with care. It
forms the foundation for many other advanced techniques.

Linear regression can also be extended to multiple linear regression, where more than
one independent variable is used to predict a dependent variable. For instance,
predicting a house price based on area, number of bedrooms, and location. The
equation becomes:

Y = b0 + b1X1 + b2X2 + ... + bnXn

In practical applications, linear regression is used in finance to predict stock prices, in

marketing to estimate sales, in agriculture to forecast crop yields, and in medical
research to understand relationships between risk factors and diseases.

In Python, implementing linear regression is straightforward using the scikit-learn

library:

from sklearn.linear_model import LinearRegression

import numpy as np

X = [Link]([[1], [2], [3], [4], [5]])

y = [Link]([50, 55, 65, 70, 75])

model = LinearRegression()

[Link](X, y)

print([Link]([[6]]))

In conclusion, linear regression is not only a key technique for predictive modeling
but also a stepping stone to understanding more complex models in machine learning
and data science.

VTU, CPGS, Mysuru

2. Polynomial Regression
Polynomial regression is a type of regression analysis that models the relationship
between the independent variable X and the dependent variable Y as an nth-degree
polynomial. It is particularly useful when the data shows a non-linear trend, which
simple linear regression cannot effectively capture.

The general form of a polynomial regression equation is:

y = β0 + β1x + β2x² + β3x³ + ... + βnxⁿ + ε

Where:

• y is the dependent variable

• x is the independent variable

• β0 to βn are the model coefficients

• n is the degree of the polynomial

• ε is the error term

The addition of polynomial terms (x², x³, etc.) enables the regression curve to better fit
complex, non-linear data patterns. For example, a quadratic model (n = 2) can capture
U-shaped relationships.

Let’s consider the problem of predicting pizza prices based on diameter. The dataset
might look like this:

VTU, CPGS, Mysuru

Diameter (inches) Price (Rs)

6 150

8 200

10 275

12 370

Plotting this data reveals a curve rather than a straight line. Linear regression would
not fit this data well, but a polynomial regression with a degree of 2 or 3 could model
the price growth more accurately.

In implementing polynomial regression, the original feature vector is transformed by

adding new features representing higher-order terms. For instance, if the input feature
is x, then the transformed feature vector would include x, x², x³, ..., xn.

In Python, this can be done using the PolynomialFeatures class from

[Link]:

from sklearn.linear_model import LinearRegression

from [Link] import PolynomialFeatures

from [Link] import make_pipeline

X = [[6], [8], [10], [12]]

y = [150, 200, 275, 370]

model = make_pipeline(PolynomialFeatures(degree=2), LinearRegression())

[Link](X, y)

print([Link]([[14]]))

However, while polynomial regression improves model flexibility, it also increases

the risk of overfitting, especially when the degree of the polynomial is too high.
Overfitting occurs when the model captures noise in the training data as if it were a
valid pattern, leading to poor generalization on new data.

To mitigate overfitting, it’s essential to use cross-validation and select the optimal
degree for the polynomial. Also, multicollinearity can be an issue, where polynomial
terms are highly correlated with each other. Regularization techniques like Ridge and
Lasso regression can help address this problem.

VTU, CPGS, Mysuru

Evaluation metrics for polynomial regression remain the same as for linear regression,
including R², MSE, and RMSE. Visualization is particularly useful for polynomial
regression. Plotting the predicted polynomial curve against actual data points can
visually validate how well the model fits.

Real-world applications of polynomial regression include modeling growth curves,

analyzing investment returns, optimizing pricing strategies, and forecasting sales
trends. In the physical sciences, it’s used to model natural phenomena like
temperature changes, sound waves, and radioactive decay.

Polynomial regression is both intuitive and powerful, offering a simple yet effective
way to model complex relationships. While it is not suitable for all types of data, it
provides a practical middle ground between underfitting and overfitting when
applied judiciously.

In summary, polynomial regression extends the capabilities of linear regression by

enabling it to model non-linear data patterns. It is particularly useful when the
relationship between variables is not strictly linear and when a curved fit provides
more accurate predictions. With proper tuning and validation, polynomial regression
can be an indispensable tool in any data scientist’s arsenal.

Classification
What is Classification in Machine Learning?
Classification is a supervised learning technique used to categorize data into
predefined classes or labels. The model learns from a labeled dataset and then
assigns labels to new, unseen data based on what it has learned.
Real-World Examples of Classification

Application Description

Email Spam Detection Classify emails as spam or not spam.

Predict whether a patient has a disease or is healthy

Medical Diagnosis
based on test results.

Image Recognition Identify whether an image contains a cat, dog, or car.

VTU, CPGS, Mysuru

1. Logistic Regression
Logistic regression is one of the most important classification techniques used in
supervised learning. Unlike linear regression, which is used to predict continuous
outcomes, logistic regression is applied when the dependent variable is categorical.
Typically, it is used for binary classification tasks where the output variable has only
two possible classes, such as Yes/No, True/False, or 0/1.

The central idea behind logistic regression is to estimate the probability that a given
input point belongs to a certain class. To achieve this, logistic regression applies the
logistic (or sigmoid) function to a linear combination of input features.

The logistic (sigmoid) function is defined as:

P(Y = 1) = 1 / (1 + e^-(β0 + β1x))

Where:

• P(Y = 1) is the probability that the output is class 1.

• β0 is the intercept.

• β1 is the coefficient (slope).

• x is the input feature.

• e is the base of the natural logarithm.

The output of the sigmoid function lies between 0 and 1, making it a perfect choice for
probability prediction. Based on a threshold value (commonly 0.5), the probability is
converted into a class label. If the probability is greater than or equal to 0.5, the output
is considered class 1; otherwise, it is class 0.
The decision boundary in logistic regression is a straight line in two dimensions or a
hyperplane in higher dimensions. The model adjusts the weights β0 and β1 during
training to minimize the difference between predicted and actual class labels. This is
done using a loss function called the log loss or binary cross-entropy loss, defined as:

Loss = - [y * log(p) + (1 - y) * log(1 - p)]

Where y is the actual class label, and p is the predicted probability.

Logistic regression is trained using optimization techniques like Gradient Descent,

which updates the model parameters in the direction that reduces the loss.

A practical example of logistic regression is email spam detection. Emails are

represented using features such as the presence of certain keywords, number of
capital letters, etc. The model then predicts whether an email is spam (1) or not spam
(0).
Logistic regression can also be extended to handle multi-class problems through
techniques like One-vs-Rest (OvR) or Multinomial Logistic Regression. In OvR, one

VTU, CPGS, Mysuru

classifier is trained per class, with the class being treated as one class versus all others.
The final prediction is made based on the classifier with the highest probability score.

In the healthcare domain, logistic regression is used to predict the presence or absence
of diseases based on patient data. For example, predicting whether a person has
diabetes based on BMI, age, and glucose levels.

Some of the key advantages of logistic regression include its simplicity, efficiency, and
interpretability. The coefficients β indicate the impact of each feature on the log-odds
of the outcome. However, logistic regression assumes a linear relationship between
input features and the log-odds of the target, which may not hold for all datasets.

It is also important to standardize the features before training the model to ensure
faster and more stable convergence. Logistic regression performs best when there is
little multicollinearity among the input features and when the classes are separable.

In Python, logistic regression can be implemented using libraries such as Scikit-learn

as follows:

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import train_test_split

from [Link] import accuracy_score

X = [[1.5], [2.0], [3.6], [4.5]]

y = [0, 0, 1, 1]

model = LogisticRegression()

[Link](X, y)

print([Link]([[2.5]]))

To summarize, logistic regression is a robust and widely used classification algorithm

that is suitable for binary and multiclass classification tasks. It is computationally
efficient and interpretable, making it a good starting point for many machine learning
problems. Its performance can be enhanced with techniques such as feature scaling,
regularization, and careful preprocessing of data.

VTU, CPGS, Mysuru

2.K-Nearest Neighbors (KNN)

The K-Nearest Neighbors (KNN) algorithm is a simple, intuitive, and powerful non-
parametric supervised learning method used for both classification and regression
tasks. It is based on the idea that similar instances exist in close proximity in the feature
space. KNN works by finding the 'k' training samples that are closest in distance to a
new point and predicting the label of the new point based on those neighbors.

Unlike most machine learning algorithms, KNN does not build an internal model or
perform explicit training. Instead, it stores all training data and makes decisions at the
time of prediction. This characteristic makes it a lazy learner. Despite its simplicity,
KNN often performs surprisingly well in practical scenarios.

Working Principle

To classify a new data point using KNN, follow these steps:

1. Choose the number of neighbors 'k'.

2. Calculate the distance between the new data point and all the points in the
training set using a distance metric like Euclidean distance.

3. Identify the 'k' nearest neighbors to the new point.

4. Count the number of data points in each class among these neighbors.

5. Assign the class label with the majority vote among the k neighbors.

The most common distance metric used is Euclidean distance, given by:
d = √((x1 - x2)² + (y1 - y2)²)

VTU, CPGS, Mysuru

However, other distance metrics such as Manhattan, Minkowski, or Hamming
distance may be used depending on the problem and data types.

Example
Suppose we want to classify a fruit based on its color and size. If we know that among
the three nearest fruits, two are apples and one is a mango, the KNN model (with k =
3) would classify the new fruit as an apple.

Choosing the Right 'k'

• If k = 1, the model becomes highly sensitive to noise and may overfit.

• If k is too large, the model may underfit by averaging too many neighbors from
possibly different classes.

• Typically, an odd value is chosen for binary classification to avoid ties.

To select the optimal value of k, one can use cross-validation, where different values
of k are tested, and the one that gives the best performance on validation data is
selected.

Applications
• Recommender systems: Suggesting products based on similar users' behavior.

• Medical diagnosis: Predicting diseases based on symptoms and patient

history.

• Pattern recognition: Handwriting, face recognition, and biometric verification.

Advantages

• Simple and Intuitive: Easy to understand and implement.

• No Training Phase: All computations are done at the time of prediction.

• Adaptability: Works well for multi-class classification.

Disadvantages
• Computationally Expensive: Especially for large datasets, as it requires storing
and comparing all training samples.

• Sensitive to Irrelevant Features: Features should be properly scaled and

selected.

• Not Ideal for High-Dimensional Data: Suffers from the curse of

dimensionality, which makes distance measures less meaningful in high-
dimensional spaces.

VTU, CPGS, Mysuru

Feature Scaling
Because KNN relies on distance calculations, feature scaling is crucial. If features have
different scales (e.g., age in years vs. income in lakhs), the feature with the larger scale
may dominate the distance computation. Techniques like Min-Max scaling or
Standardization (Z-score) are applied to bring all features to a common scale.
Implementation in Python
from [Link] import KNeighborsClassifier

from [Link] import StandardScaler

from sklearn.model_selection import train_test_split

from [Link] import accuracy_score

# Example data

X = [[1.2], [2.4], [3.5], [5.0], [6.3]]

y = [0, 0, 1, 1, 1]

# Scaling the data

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

# Splitting the data

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

# Training the model

knn = KNeighborsClassifier(n_neighbors=3)

[Link](X_train, y_train)

# Predicting

y_pred = [Link](X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))

Real-World Use Case

In medical diagnostics, KNN can be used to identify whether a tumor is malignant or
benign based on features like size, shape, and texture. The new tumor is compared
with past cases, and its class is predicted based on its closest neighbors.

VTU, CPGS, Mysuru

KNN models are evaluated using standard classification metrics such as accuracy,
precision, recall, F1-score, and confusion matrix.
Summary

K-Nearest Neighbors is a highly flexible and intuitive algorithm that is particularly

well-suited to classification problems. Although it has limitations in terms of
computational efficiency and sensitivity to feature scaling, it remains a reliable
algorithm for many real-world problems. Its performance can be significantly
improved through proper preprocessing, feature selection, and by choosing an
optimal value of k. KNN provides a solid foundation for understanding the principles
of instance-based learning in machine learning.

3. Decision Tree
A Decision Tree is a flowchart-like tree structure used for classification and regression
tasks in machine learning. Each internal node represents a decision based on the value
of a feature, each branch represents the outcome of a decision, and each leaf node
represents a final output or class label. It is a popular and intuitive method for
decision-making and data analysis.

Decision trees are a type of supervised learning algorithm. In the context of

classification, they help predict the class label of an instance by learning simple
decision rules inferred from the data features. They are capable of handling both
categorical and numerical data.

VTU, CPGS, Mysuru

Structure of a Decision Tree
1. Root Node – Represents the entire dataset, and splitting begins from here.

2. Decision Nodes – Nodes where data is split based on certain criteria.

3. Leaf Nodes – Terminal nodes that carry the final output or class.

4. Branches – Arrows connecting nodes representing the decision outcomes.

Example
Consider a simple decision-making scenario: determining whether a person will buy
a computer.

• Features: Age, Income, Student Status, Credit Rating

• Target: Buys Computer (Yes/No)

A decision tree might ask the following:

• Is age < 30?

o Yes → Is the person a student?

▪ Yes → Buy

▪ No → Don’t Buy

o No → Move to other conditions

Splitting Criteria
A decision tree builds itself by recursively splitting the dataset based on certain
criteria. Common splitting criteria include:

1. Gini Index:

o Measures the impurity of a dataset.

o Formula: Gini(D) = 1 – ∑(p_i)^2 where p_i is the probability of class i.

o Lower values indicate better splits.

2. Information Gain:

o Based on entropy; it measures the effectiveness of an attribute in

classifying data.

o Entropy: H(D) = – ∑(p_i * log2(p_i))

o Information Gain = Entropy(Parent) – Weighted Entropy(Children)

VTU, CPGS, Mysuru

 3. Gain Ratio:

o Modification of Information Gain that takes intrinsic information into

account.

The algorithm chooses the attribute with the highest information gain (or lowest Gini
Index) to split the data.
Types of Decision Trees

• Classification Trees – Predict categorical values (e.g., Yes/No, Red/Green).

• Regression Trees – Predict continuous values (e.g., prices, ratings).

Tree Building Algorithms

• ID3 (Iterative Dichotomiser 3) – Uses Information Gain.

• C4.5 – Extension of ID3 using Gain Ratio.

• CART (Classification and Regression Trees) – Uses Gini Index for

classification and variance reduction for regression.
Advantages of Decision Trees
• Easy to Understand and Interpret – Can be visualized.

• Handles Both Numerical and Categorical Data

• Requires Little Data Preparation – No need for normalization or scaling.

• Non-linear Relationships Captured

Disadvantages
• Overfitting – Trees can grow deep and fit noise in training data.

• Unstable – Small changes in data can result in different trees.

• Biased with Imbalanced Data – May favor classes with more instances.
Avoiding Overfitting

To avoid overfitting, techniques like pruning are applied:

• Pre-pruning – Stop tree growth early based on a condition (e.g., minimum

samples per leaf).

• Post-pruning – Allow the tree to grow fully and then remove branches that
have little importance.
Feature Importance

Decision Trees naturally compute feature importance based on the improvement in

the splitting criterion. This can be used to select and rank features.

VTU, CPGS, Mysuru

Real-World Applications
• Medical Diagnosis – Determine illness based on symptoms.

• Loan Approval – Predict risk based on applicant data.

• Marketing – Identify target customers.

• Fraud Detection – Classify transactions as legitimate or fraudulent.

Implementation in Python

from [Link] import DecisionTreeClassifier

from sklearn import datasets

from sklearn.model_selection import train_test_split

from [Link] import accuracy_score

# Load dataset

iris = datasets.load_iris()

X = [Link]

y = [Link]

# Split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Train
model = DecisionTreeClassifier(criterion='gini')

[Link](X_train, y_train)

# Predict

y_pred = [Link](X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))

Visualization

Decision Trees can be visualized using tools like graphviz or plot_tree() in matplotlib.
Visual representations help explain the model decisions effectively.
Summary
Decision Trees are powerful and versatile models that provide clear rules for
classification and regression problems. While they are prone to overfitting, their

VTU, CPGS, Mysuru

interpretability and effectiveness make them a key component in many real-world
applications. They also form the base for more complex ensemble methods like
Random Forests and Gradient Boosting Machines.

6. Random Forest
Random Forest is an ensemble learning method used for both classification and
regression tasks. It builds multiple decision trees and merges their outputs to produce
a more accurate and stable prediction than any individual tree. The key idea behind
Random Forest is to reduce the variance of decision trees by averaging their
predictions (for regression) or taking the majority vote (for classification).

A Random Forest is essentially a collection (or "forest") of decision trees, hence the
name. Each tree is trained on a random subset of the data using a technique called
bootstrap sampling, and at each split in the tree, a random subset of features is
considered. This combination of randomness in both the data and feature selection
helps make Random Forest robust against overfitting and capable of handling large
datasets with high dimensionality.

VTU, CPGS, Mysuru

How It Works
1. Bootstrap Sampling: From the original dataset of size N, generate multiple
datasets by randomly selecting samples with replacement.

2. Build Trees: For each bootstrapped dataset, construct a decision tree by

selecting a random subset of features at each node.

3. Aggregate Results:

o For classification: Use majority voting to determine the predicted class.

o For regression: Take the average of the predicted values.

This process introduces diversity among the trees, which when combined, leads to
better generalization performance on unseen data.

Example
Suppose we want to classify whether a patient has a certain disease based on features
such as age, weight, and blood pressure. A Random Forest would build multiple trees
using different subsets of the data and features, and each tree might give a prediction.
The final prediction is the majority decision across all trees.

Key Parameters
• n_estimators: Number of trees in the forest (more trees generally improve
performance but increase computation time).

• max_features: Number of features to consider when looking for the best split.

• max_depth: Maximum depth of each tree (can prevent overfitting).

• min_samples_split: Minimum number of samples required to split an internal

node.

• criterion: Metric used to measure the quality of a split (Gini or Entropy for
classification).
Advantages

• High Accuracy: Combines results from multiple models.

• Reduces Overfitting: More robust than individual decision trees.

• Handles Large Data: Efficient with large datasets and high feature dimensions.

• Works with Missing Values: Can maintain performance even if data has some
missing entries.

• Estimates Feature Importance: Provides insights into the most important

variables.

VTU, CPGS, Mysuru

Disadvantages
• Computational Cost: Requires more memory and processing power.

• Less Interpretability: Harder to visualize and explain than a single decision

tree.

• Slower Predictions: Especially with a large number of trees.

Feature Importance
Random Forest provides a mechanism to measure feature importance based on how
much each feature decreases the impurity across all trees. This can help in feature
selection and understanding the model.
Real-World Applications

• Healthcare: Disease diagnosis and prognosis.

• Finance: Credit scoring and fraud detection.

• Marketing: Customer segmentation and behavior prediction.

• E-commerce: Recommendation systems and user profiling.

Implementation in Python

from [Link] import RandomForestClassifier

from [Link] import load_iris

from sklearn.model_selection import train_test_split

from [Link] import accuracy_score

# Load dataset

iris = load_iris()

X = [Link]
y = [Link]

# Split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Train

model = RandomForestClassifier(n_estimators=100, random_state=42)

[Link](X_train, y_train)

# Predict

VTU, CPGS, Mysuru

y_pred = [Link](X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))

Evaluation Metrics
• Accuracy: Overall correctness.

• Precision/Recall/F1-Score: For imbalanced data.

• Confusion Matrix: For visualizing predictions.

• ROC-AUC: For evaluating probabilistic predictions.

Summary
Random Forest is a highly effective and versatile machine learning model that
leverages the power of multiple decision trees. It offers a good trade-off between
accuracy and generalization, making it suitable for many practical applications.
Though it may not be as interpretable as a single decision tree, its performance and
resilience to overfitting make it a go-to algorithm for structured data tasks.

1. Mean Absolute Error (MAE)

Mean Absolute Error (MAE) is a fundamental evaluation metric used in regression
tasks to measure the average magnitude of errors between predicted values and actual
observed values. It is calculated by taking the absolute differences between the
predicted and actual values and then computing their average. MAE gives an intuitive
sense of how far off predictions are, on average, from actual results.
Formula:

MAE treats all errors equally without considering whether the model underestimates
or overestimates. Unlike squared errors, it does not penalize large deviations more
harshly.
Example:

Suppose a model predicts house prices as follows:

• Actual prices (in ₹ lakhs): [60, 75, 90]

• Predicted prices: [65, 70, 85]

VTU, CPGS, Mysuru

The absolute errors are:

• |60 − 65| = 5

• |75 − 70| = 5

• |90 − 85| = 5

MAE=5+5+53=153=5MAE = \frac{5 + 5 + 5}{3} = \frac{15}{3} = 5MAE=35+5+5=315=5

This means that, on average, the model's predictions are ₹5 lakhs off from the actual
values.
When to Use:

• When all errors are equally important

• When the model needs to be interpretable (easy to explain to stakeholders)

Limitations:
• Does not heavily penalize large errors

• Not differentiable at all points (which may affect optimization in some models)

In summary, MAE is a robust and interpretable error metric. It is particularly useful

when simplicity and equal treatment of errors are desired. However, when large
errors should be penalized more, MSE or RMSE may be more appropriate.

2. Mean Squared Error (MSE)

Mean Squared Error (MSE) is one of the most commonly used metrics in regression
analysis. It evaluates the average of the squares of the errors — that is, the average
squared difference between predicted values and actual values. Because errors are
squared, MSE places a larger penalty on larger errors, making it sensitive to outliers.
Formula:

Example:
Using the same example:

• Actual prices: [60, 75, 90]

• Predicted prices: [65, 70, 85]

VTU, CPGS, Mysuru

The squared errors are:

• (60 − 65)² = 25

• (75 − 70)² = 25

• (90 − 85)² = 25

MSE=25+25+253=753=25MSE = \frac{25 + 25 + 25}{3} = \frac{75}{3} =

25MSE=325+25+25=375=25

This value represents the average squared error in predictions.

Advantages:

• Differentiable (suitable for gradient-based optimization)

• Penalizes larger errors more (suitable in high-risk applications)

Disadvantages:
• Not in the same unit as the predicted variable (squared units)

• Sensitive to outliers (a few large errors can distort the metric)

Interpretation:
A lower MSE means better model performance. However, due to squaring, the value
tends to be disproportionately large compared to MAE. MSE is widely used in
algorithms like Linear Regression and during model training to optimize parameters
using techniques like gradient descent.

In practice, MSE is ideal when we want to strongly penalize large deviations and are
less concerned with interpretability in natural units.

3. Root Mean Squared Error (RMSE)

Root Mean Squared Error (RMSE) is the square root of the Mean Squared Error and
is widely used for evaluating regression models. It provides a measure of how well
predicted values match actual values in the same unit as the original output variable.
RMSE combines the advantages of MSE’s sensitivity to large errors with the
interpretability of a metric that has the same unit as the target variable.
Formula:

VTU, CPGS, Mysuru

Example:
Using the MSE of 25 from earlier:

RMSE=25=5RMSE = \sqrt{25} = 5RMSE=25=5

This means the model predictions are off by ₹5 lakhs on average, similar in magnitude
to the MAE, but derived via a different penalty mechanism.

Benefits:
• Same unit as target variable (more interpretable than MSE)

• Strongly penalizes large errors

Limitations:
• Sensitive to outliers (due to squaring)

• May mask the distribution of individual errors

Use Cases:
RMSE is preferred in domains like:

• Forecasting (e.g., weather, sales)

• Financial modeling

• Real-estate pricing

In these cases, minimizing large errors is more important than minimizing average
errors. For example, in predicting rent prices, a few large mistakes could lead to
significant financial losses.

In conclusion, RMSE is a balanced metric that combines mathematical robustness with

practical interpretability. While it shares the limitations of MSE regarding sensitivity
to outliers, its unit consistency with the target variable makes it widely preferred in
applied regression analysis.

1. Accuracy

Accuracy is the most intuitive and widely used metric for evaluating classification
models. It measures the ratio of correctly predicted instances to the total number of
predictions made. In other words, it tells us how often the classifier is correct.
Formula:

VTU, CPGS, Mysuru

Where:

• TP (True Positives): Correctly predicted positives

• TN (True Negatives): Correctly predicted negatives

• FP (False Positives): Incorrectly predicted as positive

• FN (False Negatives): Incorrectly predicted as negative

Example:
Let’s say we are building a spam detection system and we tested it on 100 emails. Out
of them:

• 40 were correctly predicted as spam (TP)

• 50 were correctly predicted as not spam (TN)

• 5 were non-spam but predicted as spam (FP)

• 5 were spam but predicted as non-spam (FN)

Accuracy=40+5040+50+5+5=90100=90%

Accuracy = \frac{40 + 50}{40 + 50 + 5 + 5} = \frac{90}{100} =

90\%Accuracy=40+50+5+540+50=10090=90%
Advantages:

• Simple and intuitive

• Gives a quick idea of overall performance

Limitations:

Accuracy can be misleading in imbalanced datasets. For example, in a disease

detection model where only 5 out of 100 patients have the disease, predicting “no
disease” for everyone gives 95% accuracy — but the model is useless because it failed
to detect the disease.

When to Use:
• Balanced datasets

• Equal cost of false positives and false negatives

In summary, while accuracy is a good starting point, it must be interpreted in context,

especially for problems where different types of misclassifications carry different
consequences.

VTU, CPGS, Mysuru

2. Precision
Precision is a metric that measures the proportion of true positive predictions among
all predicted positives. In other words, it answers the question: Out of all the instances
that were predicted as positive, how many were actually positive?

Formula:

Example:

Continuing with the spam detection system:

• TP = 40 (actual spam correctly predicted)

• FP = 10 (non-spam wrongly predicted as spam)

Precision=4040+10=4050=0.80=80%Precision = \frac{40}{40 + 10} = \frac{40}{50} =

0.80 = 80\%Precision=40+1040=5040=0.80=80%
Importance:

Precision is critical in scenarios where false positives are costly. For example:

• In spam detection, falsely labeling important emails as spam can result in lost
messages.

• In fraud detection, wrongly flagging legitimate transactions could

inconvenience users.
Advantages:
• Focuses on the correctness of positive predictions

• Useful when the cost of false positives is high

Limitations:

• Ignores false negatives; a model can have high precision but low recall

• Not reliable alone in imbalanced datasets

When to Use:

• Email spam filters

• Fraud detection

• Search engines (returning only relevant results)

VTU, CPGS, Mysuru

In summary, precision is essential for problems where accuracy isn’t sufficient and
where minimizing false positives is a high priority. It is best used in combination with
recall for a more holistic view.

3. Recall (Sensitivity or True Positive Rate)

Recall is the metric that measures the ability of a classification model to identify all
relevant instances. It is the ratio of true positives to all actual positives and answers
the question: Out of all the actual positive cases, how many did the model successfully
identify?

Formula:

Example:

Using the same spam filter example:

• TP = 40 (spam correctly predicted)

• FN = 10 (spam missed and predicted as not spam)

Recall=4040+10=4050=0.80=80%Recall = \frac{40}{40 + 10} = \frac{40}{50} = 0.80 =

80\%Recall=40+1040=5040=0.80=80%
Importance:

Recall is vital when missing a positive case has a high cost, such as:

• In cancer detection, failing to identify a patient who actually has cancer (false
negative) can be life-threatening.

• In fraud detection, missing a fraudulent transaction can lead to financial loss.

Advantages:
• Measures the model’s ability to find all relevant data points

• Useful for imbalanced datasets with rare positive cases

Limitations:

• High recall may come at the cost of low precision

• A model predicting everything as positive will have perfect recall but be useless

VTU, CPGS, Mysuru

When to Use:
• Medical diagnoses

• Security and surveillance

• Information retrieval systems

To summarize, recall is critical in scenarios where missing a positive case has more
severe consequences than incorrectly identifying a negative one. It works best when
balanced with precision.

4. F1-Score

F1-Score is the harmonic mean of precision and recall. It provides a single metric that
balances both concerns, especially in scenarios where there's a trade-off between
precision and recall. It is particularly useful when dealing with imbalanced datasets
where neither accuracy nor precision alone provides the full picture.
Formula:

Example:
Let’s say:

• Precision = 0.75

• Recall = 0.60

F1=2⋅0.75⋅0.600.75+0.60=2⋅0.451.35=0.666≈66.6%F1 = 2 \cdot \frac{0.75 \cdot

0.60}{0.75 + 0.60} = 2 \cdot \frac{0.45}{1.35} = 0.666 \approx
66.6\%F1=2⋅0.75+0.600.75⋅0.60=2⋅1.350.45=0.666≈66.6%

Importance:
F1-score provides a balanced view of a classifier's performance when both false
positives and false negatives are important. It penalizes extreme values of precision
and recall.

VTU, CPGS, Mysuru

Advantages:
• Balances precision and recall

• Good indicator of overall model performance, especially in skewed datasets

Limitations:

• Does not differentiate between types of errors

• May not reflect the business cost associated with each type of misclassification
When to Use:

• When both false positives and false negatives carry costs

• Text classification (e.g., sentiment analysis)

• Imbalanced classification tasks

[Link]-Test Split
We divide the dataset into two parts:

• Training Set – used to train the model.

• Testing Set – used to evaluate how well the model performs on unseen data.

Typical split: 70% training and 30% testing, or 80%-20%, depending on dataset size.

• Why it's necessary:

To check whether the model can generalize to new data rather than just
memorizing the training data.

[Link]-Validation

Instead of just one train-test split, the data is split into k folds (e.g., 5 or 10). The model
is trained k times, each time using a different fold as the test set and the remaining
folds as the training set.

VTU, CPGS, Mysuru

Why use it:
Gives a more reliable estimate of model performance by reducing dependency on a
single split.

7. Hyperparameter Tuning using GridSearchCV

Models have parameters (weights) and hyperparameters (e.g., number of trees in a

random forest, learning rate in gradient boosting). Hyperparameters are manually
set, and GridSearchCV helps find the best combination.

How it works:

• A grid of possible hyperparameter values is created.

• Cross-validation is performed for each combination.

• The best combination is selected based on a performance metric (e.g., accuracy).

• Why use it:

To optimize model performance by fine-tuning settings.
[Link] and Underfitting

VTU, CPGS, Mysuru

 Term Description Effect

Model learns the training High accuracy on training

Overfitting data too well, including data, poor accuracy on test
noise data

Model is too simple to learn Poor accuracy on both

Underfitting
the patterns in the data training and test data

VTU, CPGS, Mysuru

 Question Bank
1. Polynomial Regression

a) Explain polynomial regression. How is it different from linear regression? Describe

its mathematical formulation, advantages, and limitations. Provide an illustrative
example and discuss how to select the optimal degree of the polynomial. (10 marks)

2. Logistic Regression

a) Discuss logistic regression in detail. Derive the logistic (sigmoid) function and
explain how the model converts linear outputs into probabilities. Describe the training
process (loss function and optimization) and its application in a real dataset scenario.
(10 marks)

3. K-Nearest Neighbors (KNN)

a) Explain the working of the K-Nearest Neighbors algorithm. Discuss how to choose
an appropriate value of k, the role of distance metrics and feature scaling, and evaluate
the model using relevant metrics. Illustrate using a practical example. (10 marks)

4. Decision Tree Learning

a) Explain the Decision Tree algorithm, including the concepts of Gini Index, Entropy,
and Information Gain. Show, with an example dataset, how a Decision Tree is
constructed step-by-step using ID3 or CART. Discuss pruning methods and
challenges such as overfitting. (10 marks)

5. Random Forest

a) Describe Random Forest in detail. Discuss the mechanisms of bagging and random
feature selection, how it overcomes the limitations of single Decision Trees, and how
feature importance is computed. Support your answer with a suitable example. (10
marks)

6. Comparative Analysis
a) Compare and contrast Linear Regression, Logistic Regression, KNN, Decision
Trees, and Random Forests. Cover aspects such as learning paradigms, types of
outputs, model interpretability, computational complexity, sensitivity to overfitting,
and best-use scenarios. (10 marks)

VTU, CPGS, Mysuru

7. Applied Case Study – Module Integration

a) A dataset contains two features (x₁, x₂) and a binary label (0/1). As a data scientist,
describe how you would apply logistic regression, KNN, and decision trees to solve
this classification problem. Explain your methodology, preprocessing steps, and
model evaluation process. (10 marks)

VTU, CPGS, Mysuru