INTRODUCTION OF MACHINE LEARNING

(CSE3008)

Dr. G V Sivanarayana
Assistant Professor Senior

SCOPE, VIT-AP GVSN

Course Objectives:
❖ Introduce the fundamental problems of machine learning.

❖ Provide understanding of techniques, mathematical concepts, and algorithms used in machine learning
to facilitate further study in this area.

❖ Provide understanding of the limitations of various machine learning algorithms and the way to evaluate
performance of machine learning algorithms.

❖ Practice software implementation of different concepts and algorithms covered in the course.

SCOPE, VIT-AP GVSN 2

 SYLLABUS
Module - I
Introduction - Regression: 7 hours

Gradient Descent, Convex optimization. Basics of Statistical Decision Theory - Regression, Linear
Regression, Multivariate Regression.

Module - II
Dimensionality Reduction and Introduction to Classification: 7 hours

Subset Selection, Shrinkage Methods-Ridge, Lasso, PCA, LDA, Classification, Linear Classification, Logistic
Regression, K-Nearest Neighbor learning, Bias- Variance, Tradeoff.

SCOPE, VIT-AP GVSN 3

 Module - III
Decision trees and Ensemble Methods: 9 hours

Decision Trees, Regression Tree, Decision Trees - Stopping Criterion & Pruning, Loss functions, Decision
Trees Multiway Splits, Missing Values, Decision Trees - Instability, Example, Evaluation Measures,
Bootstrapping & Cross Validation, Class Evaluation Measures, ROC curve, MDL, Ensemble Methods -
Bagging, Committee Machines and Stacking, Ensemble Methods – Boosting- Gradient Boosting, Random
Forests..
Module - IV
SVM and Bayesian Network & Naïve Bayes: 9 hours

Single & Multiclass classification - SVM- formulation, Interpretation & Analysis, SVM for Linear Separable
Data, SVM Kernels, Hinge Loss Functions; Naive Bayes, Bayesian Networks.

SCOPE, VIT-AP GVSN 4

 Module - V
Artificial Neural Networks: 6 hours

ANN- Introduction, Neural Network Representations, Problems for Neural Network Learning, Perceptron’s,
Multilayer Networks and Backpropagation Algorithm.

Module - VI
Unsupervised Learning: 7 hours

Unsupervised Learning - agglomerative clustering, hierarchical clustering, k-means clustering, limitations.

Introduction to Gaussian Mixture Models, Maximum Likelihood Estimation, parameter estimation for a
mixture of gaussians, Expectation Maximization.

SCOPE, VIT-AP GVSN 5

Text Book(s) :
1. T. Hastie, R. Tibshirani, J. H. Friedman, Introduction to Statistical Machine Learning 1/e, Springer, July
2023.
2. Ethem Alpaydin, Introduction to Machine Learning (Adaptive Computation and Machine Learning), The
MIT Press, 2020.
3. Chirag Shah, A Hands-On Introduction to Machine Learning,2022, Cambridge University Press.

References :
1. Aurelien Geron, Hands-On Machine Learning With Scikit-Learn and Tensorflow, O’Really publication 2019.
2. Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning, Cambridge University Press. 2017.
3. Tom M. Mitchell, Machine Learning, McGraw Hill, 2017.
4. M Narasimha Murty, Introduction to Pattern Recognition and Machine Learning, World Scientific Publishing Company,
2015.

SCOPE, VIT-AP GVSN 6

Course Outcomes:

After successful completion of the course the student will be able to:

❖ Understand the basic mathematics required for the machine learning concepts to solve various problems.
Distinguish classification and regression problems and apply to real world problems.

❖ Be able to use various decision trees.

❖ Understand and apply SVM and NB.

❖ Understand the concept behind neural networks for learning non-linear functions.

❖ Implement Unsupervised Learning models.

SCOPE, VIT-AP GVSN 7

 Continuous Assessment Test-1 15

Continuous Assessment Test-2 15

Theory 75%
Mode of
Evaluation
Digital Assignments/Quizes (Min) 30

Final Assessment Test 40

Laboratory -- -- 25%

SCOPE, VIT-AP GVSN 8

 SYLLABUS

Module - I
Introduction - Regression: 7 hours

Gradient Descent, Convex optimization. Basics of Statistical Decision Theory - Regression, Linear
Regression, Multivariate Regression.

SCOPE, VIT-AP GVSN 9

 Introduction to Machine Learning
What is Machine Learning?

➢ “Learning is any process by which a system improves performance from experience.” -Herbert Simon
➢ Definition by Tom Mitchell (1998):
Machine Learning is the study of algorithms that
-- improve their performance P
-- at some task T
-- with experience E
A well-defined learning task is given by < P, T, E >.
➢ Arthur Samuel, a pioneer in the field of artificial intelligence and computer gaming, coined the term “Machine
Learning”. He defined machine learning as – “Field of study that gives computers the capability to learn without
being explicitly programmed”.

SCOPE, VIT-AP GVSN 10

Definitions of Machine Learning

Machine learning is a branch of artificial intelligence

based on the idea that systems can learn from data,
identify patterns and make decisions with minimal
human intervention.

A computer program that can learn from experience E

with respect to some class of tasks T and performance
measure P, so that its performance at tasks in T, as
measured by P, improves with experience E.

SCOPE, VIT-AP GVSN 11

“A Machine Learning process begins by feeding the machine lots of data; by using this data the machine is
trained to detect hidden insights and trends. These insights are then used to build a Machine Learning
Model by using an algorithm in order to solve a problem.”

SCOPE, VIT-AP GVSN 12

How does Machine Learning work

✓ A Machine Learning system learns from historical data, builds the prediction models, and whenever it receives
new data, predicts the output for it. The accuracy of predicted output depends upon the amount of data, as the
huge amount of data helps to build a better model which predicts the output more accurately.

SCOPE, VIT-AP GVSN 13

Evolution of Machine Learning:

SCOPE, VIT-AP GVSN 14

SCOPE, VIT-AP GVSN 15
Machine Learning Vs Deep Learning

SCOPE, VIT-AP GVSN 16

✓ What are Features?

✓ What are Labels?

✓ Datasets

▪ Labelled Dataset → Input + Output

▪ Unlabelled Dataset → Inputs

SCOPE, VIT-AP GVSN 17

How we split data in Machine Learning?

✓ Training Data: Data used to train model. your model actually sees(both input and output) and learn from.

✓ Validation Data: The part of data which is used to do a frequent evaluation of model, fit on training dataset along
with improving involved hyper-parameters.

✓ Testing Data: Once our model is completely trained, testing data provides the unbiased evaluation. After
prediction, we evaluate our model by comparing it with actual output present in the testing data.

SCOPE, VIT-AP GVSN 18

What is training data?

✓ In machine learning, training data is the data you use to train a machine learning algorithm or model.

✓ Training data requires some human involvement to analyze or process the data for machine learning use.

✓ Training datasets can include text (words and numbers), images, video, or audio. And they can be available to you
in many formats, such as a spreadsheet, PDF, HTML, or JSON.

✓ When labeled appropriately, your data can serve as ground truth for developing an evolving, efficient machine-
learning formula.

✓ The features in your training data and the quality of labeled training data will determine how accurately the
machine learns to identify the outcome, or the answer you want your machine learning model to predict.

✓ The quality and quantity of your training data determine the accuracy and performance of your machine learning
model.

SCOPE, VIT-AP GVSN 19

What is Validation Dataset?

✓ The validation set is used to evaluate a given model, but this is for frequent evaluation.

✓ We, as machine learning engineers, use this data to fine-tune the model hyper-parameters. Hence the model
occasionally sees this data, but never does it “Learn” from this.

✓ We use the validation set results, and update higher level hyper-parameters. So the validation set affects a
model, but only indirectly.

✓ The validation set is also known as the Dev set or the Development set. This makes sense since this dataset
helps during the “development” stage of the model.

SCOPE, VIT-AP GVSN 20

Types of Learning:

✓ Supervised (inductive) learning (Gmail Spam Detection)

– Given: training data + desired outputs (labels)
✓ Unsupervised learning (Google Photos)
– Given: training data (without desired outputs)
✓ Semi-supervised learning
– Given: training data + a few desired outputs
✓ Reinforcement learning (Chess game)
– Rewards from sequence of actions

Data Information Knowledge

SCOPE, VIT-AP GVSN 21

 Supervised Learning
✓ Supervised learning is a type of machine learning method in which we provide sample labeled data to the
machine learning system in order to train it, and on that basis, it predicts the output.

✓ The system creates a model using labeled data to understand the datasets and learn about each data, once the
training and processing are done then we test the model by providing a sample data to check whether it is
predicting the exact output or not.

✓ Supervised learning can be grouped further in two categories of algorithms:

➢ Classification

➢ Regression

SCOPE, VIT-AP GVSN 22

How Supervised Learning Works
✓ Supervised learning is a process of providing input data as well as correct output data to the machine learning model.

✓ The aim of a supervised learning algorithm is to find a mapping function to map the input variable(x) with the output
variable(y). Y = f(x)

SCOPE, VIT-AP GVSN 23

Steps Involved in Supervised Learning

➢ First Determine the type of training dataset

➢ Collect/Gather the labelled training data.
➢ Split the training dataset into training dataset, test dataset, and validation dataset.
➢ Input features of the training dataset, which should have enough knowledge so that the model can accurately
predict the output.
➢ Determine the suitable algorithm for the model.
➢ Execute the algorithm on the training dataset.
➢ Evaluate the accuracy of the model by providing the test set. If the model predicts the correct output, which
means our model is accurate.

SCOPE, VIT-AP GVSN 24

Advantages of Supervised learning:

✓ With the help of supervised learning, the model can predict the output on the basis of prior experiences.

✓ In supervised learning, we can have an exact idea about the classes of objects.

✓ Supervised learning model helps us to solve various real-world problems such as fraud detection, spam
filtering, etc.

Disadvantages of Supervised learning:

✓ Supervised learning models are not suitable for handling the complex tasks.

✓ Supervised learning cannot predict the correct output if the test data is different from the training dataset.

✓ Training required lots of computation times.

✓ In supervised learning, we need enough knowledge about the classes of object.

SCOPE, VIT-AP GVSN 25

 Regression O/P is Continuous. Ex: Price prediction

Supervised Learning
Classification O/P is Discrete. Ex: Spam detection

SCOPE, VIT-AP GVSN 26

 Unsupervised Learning
✓ “Unsupervised learning is a type of machine learning in which models are trained using unlabeled dataset and
are allowed to act on that data without any supervision”

✓ Unsupervised learning uses unlabeled data to find patterns in the data, such as inferences or clustering of data
points.

✓ The goal of unsupervised learning is to find the underlying structure of dataset, group that data according to
similarities, and represent that dataset in a compressed format.

✓ The system doesn’t figure out the right output, but it explores the data and can draw inferences from datasets to
describe hidden structures from un -labelled data.

SCOPE, VIT-AP GVSN 27

Working of Unsupervised Learning : Example cat and dog

SCOPE, VIT-AP GVSN 28

Advantages of Unsupervised learning:

✓ Unsupervised learning is used for more complex tasks as compared to supervised learning because, in
unsupervised learning, we don't have labeled input data.
✓ Unsupervised learning is preferable as it is easy to get unlabeled data in comparison to labeled data.

Disadvantages of Unsupervised learning:

✓ Unsupervised learning is intrinsically more difficult than supervised learning as it does not have corresponding
output.
✓ The result of the unsupervised learning algorithm might be less accurate as input data is not labeled, and
algorithms do not know the exact output in advance.

SCOPE, VIT-AP GVSN 29

 Clustering

Unsupervised Learning

Association

SCOPE, VIT-AP GVSN 30

 Semi-Supervised Machine Learning

✓ Problems where you have a large amount of input data (X) and only some of the data is labeled (Y) are called
semi-supervised learning problems.

✓ The most basic disadvantage of any Supervised Learning algorithm is that the dataset has to be hand-labeled
either by a Machine Learning Engineer or a Data Scientist. This is a very costly process, especially when dealing
with large volumes of data.

✓ The most basic disadvantage of any Unsupervised Learning is that it’s application spectrum is limited.

✓ To counter these disadvantages, the concept of Semi-Supervised Learning was introduced.

✓ In this type of learning, the algorithm is trained upon a combination of labeled and unlabeled data.

SCOPE, VIT-AP GVSN 31

 Reinforcement Learning
✓ Reinforcement Learning is a part of Machine learning where an agent is put in an environment and he learns to
behave in this environment by performing certain actions and observing the rewards which it gets from those
actions.

✓ It is a Machine Learning algorithm that allows software agents and machines to automatically determine the
ideal behavior within a specific context to maximize its performance.

✓ It does not have labelled dataset or results associated with data so the only way to perform a given task is to
learn from experience.

✓ Reinforcement learning can, therefore, help in industrial automation as well as the gaming sector primarily.

SCOPE, VIT-AP GVSN 32

✓ Reinforcement Learning is a feedback-based Machine learning technique in which an agent learns to behave in
an environment by performing the actions and seeing the results of actions. For each good action, the agent gets
positive feedback, and for each bad action, the agent gets negative feedback or penalty.

✓ In Reinforcement Learning, the agent learns automatically using feedbacks without any labeled data, unlike
supervised learning.

✓ Since there is no labeled data, so the agent is bound to learn by its experience only.

SCOPE, VIT-AP GVSN 33

Example:

✓ Imagine that you were dropped off at an isolated island! What would you do?

✓ Panic? Yes, of course, initially we all would. But as time passes by, you will learn how to live on the island.
You will explore the environment, understand the climate condition, the type of food that grows there, the
dangers of the island, etc.

✓ This is exactly how Reinforcement Learning works, it involves an Agent (you, stuck on the island) that is put
in an unknown environment (island), where he/she must learn by observing and performing actions that
result in rewards.

SCOPE, VIT-AP GVSN 34

SCOPE, VIT-AP GVSN 35
SCOPE, VIT-AP GVSN 36
SCOPE, VIT-AP GVSN 37
▪ Health Care
▪ Social networks
▪ Web search
▪ Recommendations
▪ Finance / Trading
▪ NLP / Digital Assistants
▪ Marketing
▪ ‘Self Driving’ Cars
▪ Fraud and Security
▪ [Your favorite area]
▪ E-commerce
▪ Robotics

SCOPE, VIT-AP GVSN 38

Tuning Hyperparameters:

▪ Learning Rate ▪ Optimizer

▪ Number of layers ▪ Weight Initialization
▪ Number of Neurons per Layer ▪ Dropout Rate
▪ Batch Size ▪ L1/L2 Regularization (Weight Decay)
▪ Epochs ▪ Loss Function
▪ Activation Function ▪ Early Stopping

SCOPE, VIT-AP GVSN 39

✓ Loss function: Measures the error on a single training example (e.g., squared difference for one data point in
regression).

✓ Cost function: The average (or total) loss over the entire dataset (or mini-batch), which is what we actually
minimize during training (e.g., Mean Squared Error as the average of individual losses).

✓ Error rate (or misclassification rate) is usually the proportion of incorrect predictions in classification tasks
(e.g., 1 - accuracy, or percentage of wrong labels).

SCOPE, VIT-AP GVSN 40

 Regression
What is Regression?

➢ Regression is a fundamental supervised learning technique in machine learning used to model and analyze the
relationship between a dependent variable (target) and one or more independent variables (features).
➢ The main goal of regression analysis is to plot a line or curve that best fit the data and to estimate how one variable
affects another.
➢ Its primary goal is to predict continuous numerical values.

Examples:
• What will the house price be based on its size and location?
• What will be the temperature tomorrow?

SCOPE, VIT-AP GVSN 41

 Gradient Descent
➢ When we say Gradient, it refers to gradient of loss function with respect to weights in a network.
➢ The gradient gives the direction and rate of steepest ascent (or descent) at a given point, but it represents an
infinitesimally small change.
➢ In optimization, particularly in methods like gradient descent, we use the gradient to determine the direction to
move, but we multiply it by a step size (or learning rate) to make a finite change.
➢ Minimize the cost function by updating the model parameters (weights) step by step.
➢ A cost function in machine learning is a mathematical method that measures the difference between a model's
predicted outputs and the actual, true values.
➢ It quantifies the model's error, with the goal of the training process being to minimize this cost function to improve
the model's accuracy.
➢ There are different types of errors that can be used as a cost function. The most common cost functions for
regression problems are mean absolute error (MAE) and mean squared error (MSE).

SCOPE, VIT-AP GVSN 42

➢ To reduce the cost function (no loss), weights should be adjusted.
➢ To understand how weights are affecting the inputs, derivative of cost function is calculated i.e. rate of change of
loss w.r.t weight (𝝏J / 𝝏w). Thus, came Gradient Descent.
➢ Back-propagation is used to compute partial derivative of cost function J(w) whose value will be used in Gradient
Descent algorithm.
➢ Term “backward” means that gradient computation starts from backwards through the network. Gradient of
weights of last layer is computed first while first layer at the last.
➢ Again, feed-forward the activation outputs, get the loss and repeat until satisfactory result is obtained.
➢ Gradient is a slope (derivative w.r.t weights) of convex curve. It is calculated during back-propagation after which
parameters(weights) got updated.

SCOPE, VIT-AP GVSN 43

Gradient Descent Algorithm:

1. Randomly initialize weights w.

2. Compute gradient G using derivative of cost function w.r.t weights J(w).

𝜕𝐽
𝐺=
𝜕𝜃
3. Weight update equation: w = w – η . G
Here, η is a learn_rate which should not be too high or low to skip or not at all converging to min point.
4. Repeat steps 2 to 3 until it becomes a constant change.

SCOPE, VIT-AP GVSN 44

▪ Epoch: An epoch is one full pass through the entire training dataset.
▪ Iteration: An iteration is one update step of model parameters.
▪ Learning Rate: The learning rate controls how big a step the model takes while updating weights during
gradient descent.

Example:

• Dataset size = 1000 samples

1000
• Batch size = 100 Number of iterations per epoch: = 10
100

Then:
1 Epoch = model sees all 1000 samples once So, if you train for 10 epochs,
• Within one epoch, there will be: total iterations = 10 epochs × 10 iterations
1000 samples ÷ 100 batch size = 10 = 100 updates
So, one epoch = 10 iterations

SCOPE, VIT-AP GVSN 45

The Loss Function:

✓ A model uses a loss function that quantifies the error between the predicted output and the ground truth for a given
training sample.

✓ When model is trained in practice, many images are used to compute a loss before the network weights are
updated. Therefore, the Mean Squared Error (MSE) for a number of training images.

✓ The number of images used to update the weights is referred to as the batch size (better batch size of 32 is
default). The processing of a batch of images is referred to as an “iteration.”

SCOPE, VIT-AP GVSN 46

Gradient Descent (Optimization):

✓ Optimization process is used to update the weights in a neural network. Fortunately, there is a principled way to
tune the weights of a model called gradient descent.

✓ Referring to the plot on the left, If we compute the slope

of the loss function at the point corresponding to the current

weight estimate We1, we can see that the slope (gradient)

is negative. In this situation, we would need to increase the

weight to get closer to the optimum value indicated by Wo. So we would need to move in a direction opposite from the sign of the

gradient.

✓ On the other hand, if our current weight estimate, We1 > Wo (as shown in the plot to the right), the gradient would be positive,
and we would need to reduce the value of the current weight to get closer to the optimum value of Wo.

SCOPE, VIT-AP GVSN 47

The best way to think about this is that the sign of the gradient determines the direction we need to move in. But the
amount that we need to move needs to be tempered with a parameter called the learning rate, which is often a small
number much less than 1.

SCOPE, VIT-AP GVSN 48

Weight Update Sample Calculation:

✓ let’s assume that the current weight is We1, which has a value of 0.38, learning rate of .01 and that the slope of the loss function at
the point We1 is equal to -.55. Using the update equation above, we can easily compute a new estimate for the weight which we will
refer to as We2.

SCOPE, VIT-AP GVSN 49

 𝜕𝐿
Types of Gradient Descent: 𝜃 𝑛𝑒𝑤 = 𝜃 𝑜𝑙𝑑 − 𝛼
𝜕𝜃

▪ Batch Gradient Descent:

▪ Uses the entire dataset for each update.

1
➢ Where L = 𝑀𝑆𝐸 = 𝑛 σ𝑛𝑖=1(𝑦 − 𝑦)
ො 2

▪ Stochastic Gradient Descent (SGD):

▪ Updates parameters for each training example.

➢ Where L = 𝑀𝑆𝐸 = 𝑦 − 𝑦ො 2
▪ Fast and Good for large datasets

▪ Mini-Batch Gradient Descent

▪ Uses small subsets of data (e.g., 32, 64 samples). Here n = B (B = Batch Size)
1
➢ Where L = 𝑀𝑆𝐸 = 𝑛 σ𝑛𝑖=1(𝑦 − 𝑦)
ො 2

SCOPE, VIT-AP GVSN 50

Mini-Batch Stochastic Gradient Descent

✓ Most machine learning problems can be recast as optimization problems over specific objective functions.

✓ The loss function of a neural network in the following form:

✓ Here, Li is the loss contributed by the ith training point.

✓ In gradient descent, one tries to minimize the loss function of the neural network by moving the parameters
along the negative direction of the gradient.

ഥ whereas for
✓ For single-layer networks like the perceptron, gradient-descent is done only with respect to W,
larger networks, all parameters in the network need to be updated with backpropagation.

SCOPE, VIT-AP GVSN 51

✓ In large-scale applications, training neural networks with millions of parameters is challenging due to the high
memory and computational demands.

✓ Running all examples at once to compute gradients is impractical. However, at the start of training, even small
samples can give a good estimate of the gradient direction, making stochastic gradient descent (SGD) and its
variants effective for optimization.

✓ Since the loss function of most optimization problems can be expressed as a linear sum of the losses with respect
to individual points [ ], it is easy to show the following:

✓ In this case, updating the full gradient with respect to all the points sums up the individual point-specific effects.

SCOPE, VIT-AP GVSN 52

✓ Machine learning datasets often contain redundant information, where many data points share similarities.

✓ Stochastic Gradient Descent (SGD) speeds up the learning process by updating the model based on a single
or small batch of data points at a time, rather than the whole dataset.

✓ Stochastic Gradient Descent (SGD) updates the model quickly but with less precise gradients, making it faster
but less accurate than traditional gradient descent.

✓ Mini-batch Stochastic Gradient Descent (SGD) strikes a balance between stability, speed, and memory usage.

✓ In this method, computations involve matrices instead of vectors, both during forward and backward
propagation.

✓ If the batch size is too small, it leads to constant overhead and inefficient computation. Larger batch sizes
(usually in the range of a few hundred) don’t significantly improve gradient accuracy.

✓ Powers of 2 (like 32, 64, 128, or 256) are commonly chosen because they often provide the best
computational efficiency on most hardware. Mini-batch SGD is widely used in neural network training for this
reason. SCOPE, VIT-AP GVSN 53
Gradient Problems

Vanishing Gradient

✓ Vanishing gradient is a scenario in the learning process of neural networks where model doesn’t learn at all.

✓ It is due to when gradient becomes too small, almost vanishes leads to weights got stuck and never reach the
optimal value for minimal loss(global minima).

✓ Thus network not able to learn and converge, especially during chain rule differentiation, back-propagating
from last to initial layer may lead to no updates of weights at all.

Exploding Gradient
✓ Exactly opposite to vanishing gradient when model keeps on learning, weights keep on updating large but
model never gets converged.

✓ Computes gradient (loss) with respect to weights which becomes extremely large in the earlier layers in such a
way that it explodes.
SCOPE, VIT-AP GVSN 54
Figure:A Figure:B

Figure:C

SCOPE, VIT-AP GVSN 55

SCOPE, VIT-AP GVSN 56
Gradient Descent – Numerical Example
x y
1 1.5
Model: 𝑦ො = 𝑚𝑥 + 𝑐
2 2.3
3 3.2
Step 1: Initialize Parameters
𝑚 = 0, 𝑐 = 0
Learning rate: 𝛼 = 0.1 Step 4: Gradient with Respect to 𝑐 ∶

𝑛
Step 2: Predicted Values 1 𝜕𝐽 1
= −1.5 − 2.3 − 3.2 = ෍(𝑦ෝ𝑖 − 𝑦𝑖)
3 𝜕𝑐 𝑛
𝑦ො = 0𝑥 + 0 = 0 𝑖=1

So predictions: 𝑦ො = 0, 0, 0
−7
= = −2.333
Step 3: Errors 3
𝑦ො − 𝑦 = 0 − 1.5, 0 − 2.3, 0 − 3.2
= −1.5, −2.3, −3.2
SCOPE, VIT-AP GVSN 57
Step 5: Gradient with Respect to 𝑚 :
𝜕𝐽 1
= 𝑛 σ𝑛𝑖=1(𝑦ෝ𝑖 − 𝑦𝑖 ) xi
𝜕𝜃

1
= 3 −1.5 1 + −2.3 2 + −3.2 3

1
= −1.5 − 4.6 − 9.6
3
Parameter Updates(After One Iteration)
−15.7
= = −5.233 𝜕𝐽
3 𝑐new = 𝑐𝑜𝑙𝑑 − 𝛼 𝜕𝑐

= 0 − 0.1 −2.333 = 0.2333

𝜕𝐽
𝑚new = 𝑚𝑜𝑙𝑑 − 𝛼 𝜕𝑚

= 0 − 0.1 −5.233 = 0.5233

SCOPE, VIT-AP GVSN 58

Problem:
Consider the cost function: J(x, y) = x² + y²

The initial values are x₀ = 2 and y₀ = 4.

The learning rate α = 0.1.
Apply the Gradient Descent algorithm for two iterations to minimize the function.

Step 1: Find the partial derivatives

The partial derivative of J with respect to x is:
∂J/∂x = 2x
The partial derivative of J with respect to y is:
∂J/∂y = 2y
These together form the gradient of the function.

SCOPE, VIT-AP GVSN 59

Step 2: Write the gradient descent update equations
x(new) = x(old) − α · (∂J/∂x)
y(new) = y(old) − α · (∂J/∂y) Step 4: Iteration 2
Gradient at point (1.6, 3.2):
Step 3: Iteration 1 ∂J/∂x = 2 × 1.6 = 3.2
Initial point: (x₀, y₀) = (2, 4) ∂J/∂y = 2 × 3.2 = 6.4
Gradient at this point: Update the values:
∂J/∂x = 2 × 2 = 4 [∂J/∂x = 2x] x₂ = 1.6 − 0.1 × 3.2 = 1.28
∂J/∂y = 2 × 4 = 8 [∂J/∂y = 2y] y₂ = 3.2 − 0.1 × 6.4 = 2.56
Update the values:
x₁ = 2 − 0.1 × 4 = 1.6 After two iterations of Gradient Descent,
x = 1.28
y₁ = 4 − 0.1 × 8 = 3.2 y = 2.56
So, after the first iteration, the point becomes (1.6, 3.2). The values are moving closer to the global
minimum at (0, 0).
SCOPE, VIT-AP GVSN 60
Practice…

x y
Cost Function: 𝐽(𝑥, 𝑦) = 𝑥2 + 2𝑦2 + 2𝑥𝑦 1 2
2 3
o Initial point: 𝑥0 𝑦0 = 2 1
3 5
o Learning rate: 𝛼 = 0.1
o Perform 2 iterations Initial values: 𝑤=0, 𝑏=0
Learning rate: 𝛼=0.05
Perform two iteration

SCOPE, VIT-AP GVSN 61

Practice…
Learning Rate Instability

➢ A cost function: 𝐽 𝑤 = 𝑤 2

➢ Two learning rates tested:

o 𝛼 = 0.01
o 𝛼 = 1.2
o Initial value: 𝑤 = 5
Tasks:
❖ Predict behavior for both learning rates
❖ Identify which converges and why

SCOPE, VIT-AP GVSN 62

 Convex optimization
✓ Convex optimization plays a central role in many regression problems in machine learning and statistics,
primarily because common loss functions and regularization terms are convex.

✓ This ensures that local minima are global minima, making optimization reliable and efficient (e.g., via
gradient descent or specialized solvers).

✓ Optimization means finding the best parameters that make the error as small as possible. It's like
searching for the lowest point in a valley.

✓ A convex shape is like a smooth bowl or U-shape. No matter where you start, if you roll downhill (using
math tricks like gradient descent), you always end up at the very bottom—the absolute best spot.

✓ In non-convex cases (wavy hills and valleys), you might get stuck in a small dip that's not the deepest.

SCOPE, VIT-AP GVSN 63

SCOPE, VIT-AP GVSN 64
Convex Function – Verification

A function 𝑓 𝑥 is convex if for any 𝑥1 , 𝑥2 and 𝜆 ∈ 0 1 ,

𝑓(𝜆𝑥1 + (1 − 𝜆)𝑥2) ≤ 𝜆𝑓(𝑥1) + (1 − 𝜆)𝑓(𝑥2)

Example: f(x)=3x+5
Right-Hand Side (RHS)
Left-Hand Side (LHS):
RHS = 𝜆𝑓 𝑥1 + 1 − 𝜆 𝑓 𝑥2
LHS = f (λx1+(1−λ)x2)
= 𝜆 3𝑥1 + 5 + 1 − 𝜆 3𝑥2 + 5
= 3 𝜆𝑥1 + 1 − 𝜆 𝑥2 + 5
= 3𝜆𝑥1 + 5𝜆 + 3 1 − 𝜆 𝑥2 + 5 1 − 𝜆
= 3𝜆𝑥1 + 3 1 − 𝜆 𝑥2 + 5
= 3𝜆𝑥1 + 3 1 − 𝜆 𝑥2 + 5

LHS = RHS
Hence: 𝑓 𝜆𝑥1 + 1 − 𝜆 𝑥2 = 𝜆𝑓 𝑥1 + 1 − 𝜆 𝑓 𝑥2
So, 𝑓 𝑥 = 3𝑥 + 5 is a convex function.

SCOPE, VIT-AP GVSN 65

Convexity Using Second Derivative Test

𝑓(𝑥) = 𝑥2 + 3𝑥 + 5

𝑑
First Derivative: 𝑓 ′ 𝑥 = 𝑑𝑥 𝑥 2 + 3𝑥 + 5 = 2𝑥 + 3

𝑑
Second Derivative: 𝑓 ′′ 𝑥 = 𝑑𝑥 2𝑥 + 3 = 2

Conclusion
𝑓 ′′ 𝑥 = 2 > 0 for all 𝑥
Hence,
𝑓 𝑥 is a convex function

SCOPE, VIT-AP GVSN 66

✓ Ordinary Linear Regression (least squares): Measures error by squaring the differences. The error surface
is a perfect bowl—super convex

𝑦: Actual target values (vector)

𝑋: Input features (matrix)
𝛽: Parameters to optimize
∥⋅∥22 :Squared L2 norm (sum of squared errors)

✓ This is convex because it's a quadratic function with a positive semi-definite shape (like a bowl).

SCOPE, VIT-AP GVSN 67

✓ Ridge Regression (L2 Regularization): Like regular regression, but adds a penalty for overly large
parameters (to avoid overfitting). Still a smooth bowl.

𝜆 > 0: Regularization strength

∥ 𝛽 ∥22 = ෍𝛽𝑗2 :Penalty on size of coefficients

✓ Still fully convex (strictly convex).

SCOPE, VIT-AP GVSN 68

✓ Lasso Regression (L1 Regularization): Adds a different penalty that can force some unimportant
parameters to exactly zero (great for selecting features). It's convex, but the shape has sharp edges (like a
diamond), still with a clear best point.

∥ 𝛽 ∥1 = σ ∣ 𝛽𝑗 ∣ :L1 norm

✓ Convex (but not strictly—can have multiple solutions).

SCOPE, VIT-AP GVSN 69

 Basics of Statistical Decision Theory

✓ Loss Function: It is a mathematical measure of error between predicted and actual values.

✓ Decision Rule: it is a mechanism or algorithm that is used to minimize the loss function.

✓ Risk: It is the expected value of the loss function over the entire data distribution that we assume exists due
to pre-existing bias in the dataset.

✓ Bias: Error from simplistic assumptions. High bias models underfit the data.

✓ Variance: Error from sensitivity to training data. High variance models overfit the data.

✓ Irreducible error: Noise inherent in the data that no model can eliminate.

SCOPE, VIT-AP GVSN 70

✓ Bias: This is the error caused by a model being too simple.
▪ High bias leads to underfitting: The model performs poorly on both training and test data because it
oversimplifies things.
✓ Variance: This is the error caused by a model being too complex and sensitive.
▪ It fits the training data (including random noise) too closely, so small changes in the training data cause
big changes in predictions.
▪ High variance leads to overfitting: The model does great on training data but poorly on new or test data
because it's memorized quirks instead of learning the real pattern.

SCOPE, VIT-AP GVSN 71

Variance:
✓ Variance is the amount by which the estimate of the target function changes if different training data were
used. The target function f establishes the relation between the input (properties) and the output variables
(predicted temperature).
✓ When a different dataset is used the target function needs to remain stable with little variance because, for
any given type of data, the model should be generic.
✓ To avoid false predictions, we need to make sure the variance is low.

Bias:
✓ Bias is the algorithm’s tendency to consistently learn the wrong thing by not taking into account all the
information in the data. For the model to be accurate, bias needs to be low.
✓ If there are inconsistencies in the dataset like missing values, less number of data tuples or errors in the
input data, the bias will be high.

For a model to be ideal, it’s expected to have low variance, low bias and low error.
SCOPE, VIT-AP GVSN 72
✓ The bias-variance trade-off is fundamental to understanding how to optimize model performance.
✓ The bias-variance trade-off is a key concept in machine learning that explains the balance between a model’s
complexity and its ability to generalize to new data.
✓ It’s about finding the perfect spot where a model performs well on both the training data and unseen data
(test data).

Fig(a):Under Fitting Fig(b):Appropriate Fitting Fig(b):Over Fitting

SCOPE, VIT-AP GVSN 73

 Regression

✓ Regression analysis is a statistical method to model the relationship between a dependent (target) and
independent (predictor) variables with one or more independent variables.

✓ More specifically, Regression analysis helps us to understand how the value of the dependent variable is
changing corresponding to an independent variable when other independent variables are held fixed.

✓ Regression is a supervised learning technique which helps in finding the correlation between variables and
enables us to predict the continuous output variable based on the one or more predictor variables.

✓ In Regression, we plot a graph between the variables which best fits the given datapoints, using this plot, the
machine learning model can make predictions about the data.

✓ In simple words, "Regression shows a line or curve that passes through all the datapoints on target-predictor
graph in such a way that the vertical distance between the datapoints and the regression line is minimum."

SCOPE, VIT-AP GVSN 74

✓ Some examples of regression can be as:

▪ Prediction of rain using temperature and other factors

▪ Determining Market trends

▪ Prediction of road accidents due to rash driving.

✓ Dependent Variable: The main factor in Regression analysis which we want to predict or understand is called
the dependent variable. It is also called target variable.

✓ Independent Variable: The factors which affect the dependent variables or which are used to predict the
values of the dependent variables are called independent variable, also called as a predictor.

SCOPE, VIT-AP GVSN 75

✓ Multicollinearity: If the independent variables are highly correlated with each other than other variables, then
such condition is called Multicollinearity. It should not be present in the dataset, because it creates problem
while ranking the most affecting variable.

✓ Underfitting and Overfitting: If our algorithm works well with the training dataset but not well with test
dataset, then such problem is called Overfitting. And if our algorithm does not perform well even with training
dataset, then such problem is called underfitting

✓ It is used to find the trends in data.

✓ It helps to predict real or continuous values.
✓ By performing the regression, we can confidently determine the most important factor, the least important
factor, and how each factor is affecting the other factors

SCOPE, VIT-AP GVSN 76

 Linear Regression

➢ What is Linear Regression

➢ Types of linear regression

➢ Linear regression Line

➢ Finding the best Fit Line

➢ Cost Function

➢ Gradient Descent

➢ Model Performance

SCOPE, VIT-AP GVSN 77

What is Linear Regression?

✓ Linear regression is a statistical regression method which is used for predictive analysis.
✓ It is one of the very simple and easy algorithms which works on regression and shows the relationship
between the continuous variables.
✓ Linear regression shows the linear relationship between the independent variable (X-axis) and the
dependent variable (Y-axis), hence called linear regression.
✓ Predictive modelling method to investigate the mathematical relationship between an independent
variable(predictor –x) and a continuous (float value) dependent variable (outcome-y), E.g., y = mx + c.
✓ Predictor shows the changes in the Dependent variable (y-axis) when the changes in explanatory variables
occur on the x-axis. It uses current information about a phenomenon to predict its future behavior.

SCOPE, VIT-AP GVSN 78

▪ The relationship between variables in the linear regression model can be explained using the below
image. Here we are predicting the salary of an employee on the basis of the year of experience

SCOPE, VIT-AP GVSN 79

✓ Below is the mathematical equation for Linear regression

Some popular applications of linear regression are:

▪ Analyzing trends and sales estimates
▪ Salary forecasting
▪ Real estate prediction
▪ Arriving at ETAs in traffic.

SCOPE, VIT-AP GVSN 80

Types of Linear Regression

✓ Simple Linear Regression: If a single independent variable is used to predict the value of a numerical
dependent variable, then such a linear regression algorithm is called Simple Linear regression.
✓ Multiple Linear Regression: If more than one independent variable is used to predict the value of a
numerical dependent variable , then such a linear regression algorithm is called “Multiple Linear Regression”.

Findig the Best Fit Line

✓ Our Main goal is to find the best fit line.
✓ It is the error between Predicted Values and Actual Values should be minimized.
✓ The best fit line will have the least error.
✓ The different values for weights or the coefficient of lines, gives a different line of regression, So we need to
calculate the best fit lines, so to calculate this we use cost function.

SCOPE, VIT-AP GVSN 81

Types of Relationships:

SCOPE, VIT-AP GVSN 82

✓ Positive Linear Relationship: If the dependent variable increases on the Y-axis and independent variable
increases on X-axis , then such a relationship is termed as a Positive linear relationship.
✓ Negative Linear Relationship: If the dependent variable decreases on the Y-axis and independent variable
increases on the X-axis , then such a relationship is called a negative relationship.

Cost Function:
✓ The Cost Function is used to find the accuracy of the mapping function, Which maps the input variable to the
output variable. This mapping function is also known as “Hypothesis Function”.
✓ Mapping Function: Mapping to the input variables to output variables.
✓ For Linear Regression, we use the MSE cost function, which is the average of squared error occurred between
the predicted values and actual values.

SCOPE, VIT-AP GVSN 83

✓ The distance between the actual value and observed value is called Residuals.
✓ If the observed points are far from the regression line, then the residual will be high , and so Cost function will
high.
✓ If the Scatter points are close to the regression line, then the residual will be small and hence the cost function.

Gradient Descent:
✓ Gradient Descent used to minimize the MSE by calculating the gradient of the cost function.
✓ A regression Model uses gradient descent to update the coefficients of the line by reducing the cost function.
✓ It is done by a random selection of Values of Coefficient and then iteratively update the values to reach the
minimum cost function.
✓ The goodness of Fit determines how the line of regression fits the set of observations.
✓ The process of finding the best model out of various models is called optimization.

SCOPE, VIT-AP GVSN 84

Evaluation Metrics for Regression:

✓ Mean Absolute error (MAE) − It is the average of the absolute difference between predicted values and true
values.
✓ Mean Squared error (MSE) − It is the average of the square of the difference between actual and estimated
values.
✓ Median Absolute error − It is the median value of the absolute difference between predicted values and true
values.
✓ Root mean square error (RMSE) − It is the square root value of the mean squared error (MSE).
✓ R2 (coefficient of determination) Score − the best possible score is 1.0, and it can be negative (because the
model can be arbitrarily worse).
✓ Mean absolute percentage error(MAPE) − It is the percentage equivalent of mean absolute error (MAE).

SCOPE, VIT-AP GVSN 85

R-square Method:

✓ R-suared is a statistical method that determines the goodness of fit.

✓ It measures the strength of the relationship between the dependent and independent variables on a scale of 0-
100%.
✓ The High value of R-squared determines the less difference between the predicted values and actual values
and hence represents a good model.
✓ It is also called a coefficient of determination or coefficient of multiple determination for multiple regression.
✓ R-Squared=explained Variation/Total Variation.

SCOPE, VIT-AP GVSN 86

Small (or) Multicollinearity between the features:

✓ Multicollinearity means high-Correlation between the independent variables.

✓ Due to multicollinearity , it may difficult to find the true relationship between the predictors and target
variables.
✓ Or we can say. It is difficult to determine which predictor variable is affecting the target variable and which is
not.
✓ So , the model assumes either little or Multicollinearity between the features or independent variables.

SCOPE, VIT-AP GVSN 87

➢ Hypothesis: ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥

➢ Parameters: 𝜃0 , 𝜃1

1 2
➢ Cost function: 𝐽 𝜃0 , 𝜃1 = σ𝑚 ℎ𝜃 𝑥 𝑖
−𝑦 𝑖
2𝑚 𝑖=1

➢ Goal: minimize 𝐽 𝜃0 , 𝜃1
SCOPE, VIT-AP GVSN 88
We adjust the line using:

The line in linear regression is → 𝑦 = 𝑚𝑥 + 𝑐

1. Cost Function
Measures how far predictions are from actual values.

2. Gradient Descent
This optimization algorithm updates the slope and intercept:

SCOPE, VIT-AP GVSN 89

Examples: Study Hours (x) Marks (y)
1 35
Study Hours vs Marks 2 45
Problem: Predict marks based on number of hours studied. 3 55
4 65
Mapping to 𝑦 = 𝑚𝑥 + 𝑐
Model: Marks = 10 × Hours + 25 Linear regression form Example meaning
(y) Marks
𝑦 −𝑦
Find slope (m): 𝑚 = 𝑥2−𝑥1 (x) Study Hours
2 1
(m) 10 (slope)
(c) 25 (intercept)
Find intercept (c): Use 𝑦 = 𝑚𝑥 + 𝑐
Take any point (1, 35): 35 = 10 1 + 𝑐
Final equation: Marks = 10 × Hours + 25
𝑐 = 35 − 10 = 25

SCOPE, VIT-AP GVSN 90

Examples: Size ([Link]) Price (₹ lakhs)
500 20
House Size vs Price 700 28
Problem: Predict house price based on size. 900 36

Mapping to 𝑦 = 𝑚𝑥 + 𝑐
Model: Price = 0.04 × 𝑆𝑖𝑧𝑒 + 0 Linear regression form Example meaning
(y) Price
𝑦 −𝑦 28−20 8
Find slope (m): 𝑚 = 𝑥2−𝑥1 𝑚 = 700−500 = 200 = 0.04 (x) Size
2 1
(m) 0.04 (slope)
(c) 0 (intercept)
Find intercept (c): Use 𝑦 = 𝑚𝑥 + 𝑐
Take any point (500, 20): 20 = 0.04 500 + 𝑐
Final equation: Price = 0.04 × 𝑆𝑖𝑧𝑒 + 0
𝑐 = 20 − 20 = 0

SCOPE, VIT-AP GVSN 91

Examples: Temperature (°C) Sales (units)
20 100
Temperature vs Ice Cream Sales 25 150
Problem: Predict ice cream sales based on temperature. 30 200

Mapping to 𝑦 = 𝑚𝑥 + 𝑐
Model: Sales = 10 × (Temp) − 100 Linear regression form Example meaning
(y) Sales
𝑦 −𝑦 150−100 50
Find slope (m): 𝑚 = 𝑥2−𝑥1 𝑚= = = 10 (x) Temperature
2 1 25−20 5
(m) 10 (slope)
(c) -100 (intercept)
Find intercept (c): Use 𝑦 = 𝑚𝑥 + 𝑐
Take any point (20, 100): 100 = 10 20 + 𝑐
Final equation: Sales = 10 × (Temp) − 100
𝑐 = 100 − 200 = −100

SCOPE, VIT-AP GVSN 92

 Multivariate Regression
✓ Multivariate regression is a statistical technique used to model the relationship between multiple independent
variables (predictors) and a single dependent variable (outcome) and making it a powerful tool for
understanding complex data.
✓ Multivariate regression typically refers to regression models where there are multiple independent variables
(predictors/features) used to predict one dependent variable (target).
✓ Multivariate regression models relationships between multiple predictors and a single outcome.
✓ Multivariate linear regression (often called multiple linear regression) predicts a continuous outcome using
multiple predictors.

Example: let's say a person decides to buy a car. Then before coming up with a decision, we will look at few
aspects such as price, looks, maintenance, size, color, engine model, fuel capacity, mileage capacity, etc..

SCOPE, VIT-AP GVSN 93

SCOPE, VIT-AP GVSN 94
➢ Accurate Predictions: Multivariate regression makes better predictions by using multiple factors at once. It
shows how each factor affects the outcome, giving a clearer and more accurate picture, especially in
complex situations where one factor alone isn't enough.
➢ Understand Relationships: Multivariate regression shows how multiple factors affect an outcome at the
same time. It's great for complex situations with interacting influences, helping uncover hidden patterns and
how the factors work together.
➢ Control Confounding Variables: Multivariate regression controls for confounding variables by including
multiple predictors. This isolates the true effects of key factors, making results more reliable and accurate.
▪ Confounding variables are outside factors that mix up the true relationship between your main
predictor (like exercise) and outcome (like weight loss). For example, if you don't account for diet, it
could seem like exercise alone causes weight loss when both do.

SCOPE, VIT-AP GVSN 95

➢ Improved Decision Making: Multivariate regression shows which factors most strongly influence an
outcome. This helps organizations focus on what matters, make smarter decisions, and allocate resources
better.
➢ Model Complex Scenarios: Multivariate regression accurately models outcomes influenced by multiple
variables. It's ideal for complex situations where simple linear models fall short.
➢ Assess the Impact of Multiple Factors: Multivariate regression lets you assess the individual and
combined effects of multiple predictors on an outcome. It's key for understanding how factors interact in
real-world contexts.

SCOPE, VIT-AP GVSN 96

Cost Function of Multivariate Regression:

✓ The cost function in multivariate regression measures how well the prediction of the model matches the
actual values (observed data). By minimizing this error during model training, you can ultimately
improve the model’s accuracy.

✓ Mean Squared Error (MSE) is one of the most common cost functions for regression tasks. By
penalizing large errors more significantly than smaller ones, it encourages the model to make precise
predictions.

✓ By using techniques like parameter tuning, you can reduce MSE, thereby improving the model’s
accuracy over iterations.

SCOPE, VIT-AP GVSN 97

Steps to Implement Multivariate Regression

✓ Selection of Features: It is the process of choosing the most relevant variables that contribute to
predicting the outcome. Through feature selection, you can avoid redundant features that can degrade the
model’s performance.

✓ Feature Normalizing: Features in your dataset may have different units or scales (e.g., age and marital
status), which can affect the regression analysis. Use techniques like min-max scaling or standardization
to scale the features so that they are all on a similar scale.

✓ Selecting Loss Function and Hypothesis: The loss function measures how well the model’s predictions
align with the actual outcomes. A common loss function is Mean Squared Error (MSE), which penalizes
larger errors more heavily.

SCOPE, VIT-AP GVSN 98

✓ Fixing Hypothesis Parameter: The parameters (0,1 etc) in the hypothesis are initially set randomly. The
model must learn to adjust these parameters based on the training data in order to minimize the error
(loss function). This process is usually done through optimization techniques like Gradient Descent.

✓ Reducing the Loss Function and Analyzing the Hypothesis Function: The loss function is reduced
using optimization techniques like Gradient Descent. After training, you have to analyze the model to see
if it makes sense logically and aligns with expectations. Gradient Descent iteratively minimizes the loss
function by updating parameters in the direction of the steepest descent.

SCOPE, VIT-AP GVSN 99

Multivariate Linear Regression
✓ A multivariate linear regression approach is used when the relationship between a dependent variable
(target) and multiple independent variables (features) is assumed to be linear.

✓ The objective is to model the target variable as a weighted sum of the input features, allowing for
prediction based on these relationships. It is widely used for continuous outcome variables.

✓ Multivariate linear regression in machine learning is calculated using the following formula.

𝒚 = 𝜷𝟎 + 𝜷𝟏 ∙ 𝒙𝟏 + 𝜷𝟐 ∙ 𝒙𝟐 + 𝜷𝟑 ∙ 𝒙𝟑 +∙∙∙∙∙ +𝜷𝒏 ∙ 𝒙𝒏 + 𝜺

▪ y is the predicted dependent variable.

▪ 𝒙𝟏 , 𝒙𝟐, 𝒙𝟑 ∙∙∙∙∙ 𝒙𝒏 ∙ are the independent variables (features)
▪ 𝜷𝟎 is the intercept.
▪ 𝜷𝟏 , 𝜷𝟐 , 𝜷𝟑, ∙∙∙∙∙ 𝜷𝒏 are the coefficients for each feature.
▪ 𝜺 is the error term
SCOPE, VIT-AP GVSN 100
✓ OLS (Ordinary Least Squares) is the one method to compute coefficients and intercept in linear
regression — it's the most common and default for simple cases, but other methods are often better
depending on the data and goals.

✓ When OLS is Ideal (and Usually Sufficient)

▪ Small to moderate datasets with few predictors.

▪ No multicollinearity (predictors not highly correlated).

▪ No overfitting risk (more data points than predictors).

SCOPE, VIT-AP GVSN 101

How to Apply OLS to Get Coefficients and Intercept
✓ Ordinary Least Squares (OLS) is the standard method to find the best-fitting linear regression line by
minimizing the sum of squared residuals. It has a closed-form solution (exact math formula) for linear
regression.
✓ The Mathematical Formula
▪ For a model: ŷ = β₀ + β₁ x₁ + β₂ x₂ + ... + βₚ xₚ
▪ The OLS estimates are:
o β = (Xᵀ . X)⁻¹ Xᵀ y
▪ Where:
o X is the design matrix: rows = observations, columns = predictors (add a column of 1s for the
intercept).
o y is the vector of target values.
o β = [β₀, β₁, ..., βₚ] (intercept first).
o This matrix calculation gives exact coefficients and intercept.
SCOPE, VIT-AP GVSN 102
Step-by-Step Application:

1. Prepare the data: Collect your predictors (X) and target (y).
2. Add intercept column: Column of 1s to X.
3. Compute matrices:
▪ Xᵀ . X (dot product of transpose and X)
▪ Invert it: (Xᵀ . X)⁻¹
▪ Xᵀ . y
▪ Multiply: β = (Xᵀ . X)⁻¹ (Xᵀ . y)
4. Result: First value = intercept (β₀), rest = coefficients.

SCOPE, VIT-AP GVSN 103

Examples:
Student Hours Studied Attendance (%) Sleep (hours/night) Final Score

Predicting Exam Score 1 10 90 8 85

2 5 70 6 60
3 15 95 9 95
4 8 80 7 75

Score ≈ 5 + 5×Hours + 3×Attendance - 30×Sleep

➢ Hours studied has the strongest positive effect (+5 points per hour, holding others constant).
➢ Attendance helps (+3 points per %).
➢ Sleep has a strong negative coefficient (data artifact—more sleep correlates with fewer hours studied here).

SCOPE, VIT-AP GVSN 104

Examples: Person Height(cm) Age(years) Weight(Kg)
1 160 20 55

Predicting Weight from Height and Age 2 165 25 60

3 170 22 65
4 175 30 72
5 180 28 78

Weight ≈ -127 + 1.12×Height + 0.09×Age

➢ Height is the main driver (+1.12 kg per cm, holding age constant).
➢ Age has a small positive effect (+0.09 kg per year).

SCOPE, VIT-AP GVSN 105

Examples: Car Engine Size (L) Mileage (000s km) Age (years) Price ($000s)
1 2.0 50 5 15

Car Price Prediction 2 1.6 80 8 10

3 2.5 30 3 22
4 1.8 60 6 13
5 3.0 40 4 20

Price ≈ 20.45 + 2.49×Engine - 0.19×Mileage - 0.02×Age

➢ Engine size most positively affects price (+$2,490 per liter).

➢ Mileage most negatively affects price (-$190 per 1,000 km, holding others constant).
➢ Age has a small negative effect.

SCOPE, VIT-AP GVSN 106

Practice…
Housing Price Prediction

➢ A real-estate company builds a linear regression model: 𝑦 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑏

Where:
o 𝑥1 =house size ([Link])
o 𝑥2 =number of rooms
o 𝑦= price
➢ After training:
𝑤1 = 150, 𝑤2 = 30000, 𝑏 = −200000
Tasks:
❖ Predict the price of a 1200 [Link], 3-room house
❖ Explain why the model may overfit
❖ Suggest one solution

SCOPE, VIT-AP GVSN 107

THANK YOU

SCOPE, VIT-AP GVSN 108