Module – 2

Regression Models

[Link] Linear Regression

• It is a Supervised Machine Learning Technique.
• Each Data Point/Example has a single feature and one label.
• The model is a line best fitting the given data points

Example1 : Data Points/Examples

Salary 12000 35000 50000 23450 36450 43750
Loan 1100 1300
350 1000 1400 450
Provided

Example-2
Marks in IAT
25 37 39 17 34 29 29 31 28 37
(40)
Marks in Final
30 53 55 20 50 42 43 41 43 54
Exam (60)
Model
The Equation of the model is :
Y=W*X +b

Where Y is the Predicted Label

X is the Feature of the Example or Data Point

W is the Weight of the feature

b is the bias or intercept of the straight line that fits the

data
Example: The following graphs show the plotting of data points and
these graphs are called Scatter Plots.

In Simple Linear Regression we try to formulate the Simple

Linear model that best fits the given Training Data.
The following plot shows that Line – 3 fits the data best as
compared to the other three lines
How to find Weight and Bias
To find the weight and bias of the best line fitting the data point
we use the following formula:

W=

Where n is the number of examples or Data Points

X is the feature of each example or Data Point

Y is the label of each Example or Data Point

 b= W

Example (Simple Linear Regression)

Data Points are:
X 2 4 6 8
Y 2 4 3 6

W=

b=

Try the following Exercises:

Exercise – 1

Answer :

Equation of the line :

Expenditure of the 6 Month : 53.6 thousand
th

Exercise – 2
Answer:

Equation of the Line :

Exam Score of the Student who has 86 marks in the Mid Term : 81.02

[Link] Linear Regression

1. It is also a Supervised Machine Learning Technique.
2. Each Data Point/Example has more than one feature and
one label.
3. The model is a plane best fitting the given data points

Example1 : Data Points/Examples

Humidity Wind Temp
(x1) (x2) (Y)
14 7 26
30 6 32
12 13 19
25 14 18
26 20 18
15 7 27
8 5 35

Model
The Equation of the model is :
Y = W1 * X1 + W2 * X2 +………….. + Wm * Xm + b
Where xi are the features
Wi are the weights
B is the bias
Y is the predicted Label
Data Format

Parameters

How to find Weights and Bias

To find the weights and bias of the best line fitting the data point
we use the following formula:

Parameters =
Example -1
Consider the following data with two features and a label for each Data
Point:

Carry out Multiple Regression Analysis and find out the values of the
parameters of the Linear Model.

Answer:
Each data point has two features (viz. X1 and X2). So, the there are two
weights i.e. W1 and W2 for the features X1 and X2 respectively. Also, there
is Bias (b). Therefore, there are three parameters for the Multiple Linear
Model for the given data.
Model : Y = W1*X1 + W2*X2+ b
Matrices for the Given Data

1 1 1 1 1 1 1
XT = 7 1 11 11 7 11 3
2.6 2.9 5.6 3.1 5.2 5.5 7.1
 1 7 2.6
1 1 1 1 1 1 1 1 1 2.9
1 11 5.6 7 51 32
XT *X= 7 1 11 11 7 11 3 * 51 471 235
2.6 2.9 5.6 3.1 5.2 5.5 7.1 1 11 3.1
1 7 5.2 32 235 163.84
1 11 5.5
1 3 7.1

1.7996 -0.0685 -0.2532

Inverse of (XT * X) = (XT * X)-1 -0.0685 0.0101 -0.0011
-0.2532 -0.0011 0.0571

1 1 1 1 1 1 1
1.7996 -0.0685 -0.2532
7 1 11 11 7 11 3
-0.0685 0.0101 -0.0011
= 2.6 2.9 5.6 3.1 5.2 5.5 7.1
-0.2532 -0.0011 0.0571

0.6618 0.9968 -0.3718 0.2612 0.0034 -0.3465 -0.2036

-0.0007 -0.0616 -0.0364 0.0392 -0.0035 0.0365 -0.0763
-0.1124 -0.0887 0.0545 -0.0883 0.0360 0.0487 0.1489

0.6618 0.9968 -0.3718 0.2612 0.0034 -0.3465 -0.2036

-0.0007 -0.0616 -0.0364 0.0392 -0.0035 0.0365 -0.0763
-0.1124 -0.0887 0.0545 -0.0883 0.0360 0.0487 0.1489
3. Multivariate Linear Regression
• It is a Supervised Machine Learning Technique.
• Each Data Point/Example has one or more than one feature
and more than one label.
Example:
IAT-1 (X) 15 17 17 7 17 15 15 16 14 18
IAT-2 (Y1) 14 16 16 10 18 13 16 13 16 16
Final Exam
30 53 55 20 50 42 43 41 43 54
(Y2)

Model
The Equation of the model is :
Yi = Wi * X + bi

Where Yi is a vector of k Predicted Labels where k >1

X is the Feature of the Example or Data Point
Wi is the Weight of the feature to produce Yi
bi is the bias or intercept of the straight line that fits the
data

(Try This → Given the format of the Data Points, learn how to
decide the number of parameters of the model. To be explained
in the class)
Data Format

Parameters

How to find Weights and Bias

To find the weights and bias of the best line fitting the data point
we use the following formula:

Parameters =
Example
Given the following Data, Carryout Multivariate Regression.

Y=

X=
Equations are : y1 = x -1

Y2 = 2x +1
4. Logistic Regression
• It is a Supervised Machine Learning (ML) Technique.
• Each Data Point/Example will have one or more than one
feature and a label.
• This ML technique is used for Classification of Data
Points/Examples.
• Here the label values are discrete (or integer values).
• Logistic Regression is used for classification of Data Points.
Example :
Income Other Loan
(X1) Loans Sanctioned
(X2) (Y)
2000 1200 0
12000 1500 1
15000 2500 1
15000 25000 0
10000 8000 0
12000 2500 1
15000 4500 1

Types of Logistic Regression

Depending on the type label there are three types of Logistic
Regression

a) Binary Logistic Regression

Here the label of each data point (or example) has only two
classes (either 0 or 1).
The data points shown above have binary labels.
Here we classify data points only two classes.
b) Nominal Logistic Regression
• Here the data points (or examples) are classified into three or
more classes
• The classes do not have any specific order

c) Ordinal Logistic Regression

• Here the data points (or examples) are classified into three or
more classes
• Here the classes have a specific order
Logistic Regression Modelling Technique
We will learn in our syllabus about binary logistic regression.
The method is as follows:

a) Use Sigmoid Function to predict Label

We use Sigmoid function to decide predict the output. The
characteristics of the Sigmoid function are as follows:
The following formula is used to predict the Label

Example:
The following example shows how to predict a label of a given
Data Point
Compute the value of zi, f(zi) and then predict the label (yi). The
value of zi is a linear summation of features. The value of f(zi)
needs to be computed using Sigmoid function. The table given
below the computed values of zi and f(zi).

b) Train the Model Using Gradient Descent

Technique
• The logistic Classification Model is trained using the given
data points.
• Training the model means deciding the Weights of the
features.
• We use Gradient Descent Method to train the model.

Gradient Descent Method

The gradient Descent Method is used to find out the Optimal
Values of the parameters. The GD method is as follows:
1. Calculate Gradient of each parameter for a given data point
using the following formula:

2. Update each parameter using the following formula:

3. Repeat step-1 and 2 for all data points. This is the first cycle
of updates

4. Repeat step-3 for several cycles until two successive cycles

result in same updated values of parameters.
5. The final values of the parameters that you get after
carrying out the above mentioned process are the Best
Values.