MODULE-2

Learning with
Regression and Trees
Linear Regression
⚫ Linear regression analysis is used to
predict the value of a variable based on
the value of another variable. The variable
you want to predict is called the
dependent variable/Criterion Variable ie
Y. The variable you are using to predict
the other variable's value is called the
independent variable/Predictor Variable ie
X.
TYPES
1) SIMPLE
⚫ Linear
⚫ Non-linear

2) MULTIPLE
⚫ Linear
⚫ Non-linear
SIMPLE LINEAR REGRESSION
⚫ Linear regression is a statistical method
determining the relationship between two
or more variables. It involves using an
independent or known variable to predict
the dependent or response [Link]
can demonstrate the relationship by
drawing a straight line on a graph, where
the independent variable is on the x-axis
and the dependent on the y-axis.
⚫ Example

Determining the effects of different doses

of medication, whereby the doses are the
independent variable, and the effects are
the dependent variable.
Linear and Non Linear
Prediction and the Error in the
Prediction
Let’s take an example,
Sr. No. X Y Y’ Y-Y’ (Y-Y’)2

1 2 2 2.2 -0.2 0.04

2 4 4 3.2 0.8 0.64

3 6 3 4.3 -1.3 1.69

4 8 6 5.4 0.6 0.36

⚫ The simple linear regression is represented
by equation,
Y’ = a X + b + u or
Y’= a X + b / b + a X
Where:
Y’ = dependent variable
X = independent variable
a = intercept (where the line intercepts the X
or Y axis)
b = slope of the graph
u = regression residual/error (vertical distance
between a data point and the regression line)
Examples/Numericals of linear
regression
The expenditure of an organization (in
thousand) for every month is shown in table
below:
X(month) 1 2 3 4 5
Y (Expenditure) 12 19 29 37 45

Find regression line,Y=aX+b using least

square method
Estimate the expenditure of company in 6th
month using line as a model
Solution-
Sr. No. X Y XY X^2
1 1 12 12 1
2 2 19 38 4
3 3 29 87 9
4 4 37 148 16
5 5 45 225 25
Total 15 142 510 55

(A) Y’ = a X+b
a= n ∑X Y - ∑X ∑ Y / n ∑ X ^2 –(∑ X) ^2
= 5* 510-15*142/ 5*55-225 = 8.4

b= 1/n (∑ Y – a ∑ X) =1/5(142 - 8.4*15)=3.2

(B) Y’ = 8.4 X +3.2

In 6th month Y’= 8.4*6+3.2
Y’= 53.6 thousand
 2) Following table shows the midterm and
final grades for students in a database
course. Use the method of least squares
using regression to predict the final exam
grades of a student who received 86 in the
mid-term exam.
Midterm 72 50 81 74 94 86 59 83 86 33 88 81
exam(X)
Final exam(Y) 84 53 77 78 90 75 49 79 77 52 74 90
Practice Problems
solution
Note-? Is ∑
Problem
BENEFITS
⚫ Predicting outcomes

⚫ Preventing mistakes

⚫ Increasing efficiency
Applications
⚫ Market analysis

⚫ Financial analysis

⚫ Sports analysis

⚫ Medicine
MULTIPLE LINEAR REGRESSION
⚫ Multiple Linear Regression is one of the
important regression algorithms which
models the linear relationship between a
single dependent continuous variable and
more than one independent variable.

● E.g. Travel time to deliver parcel depend on

miles travelled, no of deliveries
● CO2 emission based on engine size, no of
cylinder in the car
⚫Represented by equation
Y’ = β0 + β1 X1 + β2 X2 + ……+ βn Xn + e
Where,Y’=predicted value
X1 , X2, …Xn = predictors
e = random error
β0, β1, β2, βn = regression parameters
 Method -2
⚫ Predict the value of Y for 6th subject from
the given dataset that contain values for
X1,X2 and Y by using a multiple regression
model
Subject Y X1 X2
1 -3.7 3 8
2 3.5 4 5
3 2.5 5 7
4 11.5 6 3
5 5.7 2 1
6 ? 3 2
⚫Multiple linear regression model is given by
Y = a+b1X1+b2X2+b3X3+……..+bnXn+u
So,Y can be predicted by finding --a, b1,b2
a= Y’-b1(X1’)-b2(X2’)
b1= (∑X22)(∑X1Y)-(∑X1X2)(∑X2Y)
(∑ X12 ) (∑ X22 )- (∑ X1X2)2

(∑X12)(∑X2Y)-(∑X1X2)(∑X1Y)
b2=
(∑ X12 ) (∑ X22 )- (∑ X1X2)2
∑X12 = ∑X1 X1 –(∑X1 )(∑X1 )/N

∑ X22 = ∑X2 X2 –(∑X2 )(∑X2 )/N

∑ X1 Y= ∑X1Y –(∑X1 )(∑Y )/N

∑ X2 Y= ∑X2Y –(∑X2 )(∑Y )/N

∑ X1 X2 = ∑X1X2 –(∑X1 )(∑x2 )/N

Y’=∑ Y/N , X1’= ∑ X1/N , X2’= ∑ X2 /N

Subject Y X1 X2 X1X2 X1X1 X2X2 X1Y X2Y
1 -3.7 3 8 24 9 64 -11.1 -29.6
2 3.5 4 5 20 16 25 14 17.5
3 2.5 5 7 35 25 49 12.5 17.5
4 11.5 6 3 18 36 9 69 34.5
5 5.7 2 1 2 4 1 11.4 5.7
SUM 19.5 20 24 99 90 148 95.8 45.6

∑X12 = 10

∑X22 = 32.8

∑X1Y = 17.8

∑X2 Y = -48

∑X1X2 = 3

b1=2.2816

b2= -1.672
a= 2.796

Y= 2.796+2.28X1-1.67X2

Y=2.796+(2.28*3)-(1.67*2)= 6.296
Multivariable Linear Regression
● Multivariate Linear Regression involves
multiple data variables for analysis.
● For example: let's say a person decides to buy a
car. Then before coming up with a decision, he
shall look at few aspects such as price, looks,
maintenance friendliness, size, color, engine
model, fuel capacity, mileage capacity, etc., and
his personal preferences are also taken into
consideration. Based on this information,
Multivariate regression can help to predict the
outcome by co-relating all of these aspects.
 here we consider the relationship between
more than one dependent variable and one
independent variable
Yi = β0i + β1i.x1
where i=1,2,..m

generalized equation shall be,

y = β0i + β1ixi + β[Link] +….. + β[Link]
● More than one dependent variable(y1,y2)
and one independent variable(x).
● Eg The researchers want to see how
messiness of a room might influence a
persons mood. The messiness of a room
would be independent varaible and the
study would have two dependent
variable -level of creativity, mood
NUMERICALS

Given the following data in table, find multivariate

regression equations
MULTIVARIATE MULTIPLE LINEAR
REGRESSION
⚫ Multivariate multiple regression (MMR)
is used to model the linear relationship
between more than one independent
variable (IV) and more than one dependent
variable (DV). MMR is multiple because
there is more than one IV. MMR is
multivariate because there is more than
one DV.
⚫ Represented by equation

Y1 = β01 + β11 X1 + β21 X2 +….+ βk1 Xk + ε1

Y2 = β02 + β12 X1 + β22 X2 +….+ βk2 Xk + ε2

:
:
Ym= β0m + β1m X1 + β2m X2 +..+ βkm Xk + εm
Suppose we had six sample observations, as shown
in table on 2 independent variable (palatability and
texture) and 2 dependent variable (purchase intent
and overall quality) for some product. Use the data
to estimate the multivariate multiple linear
regression model.

Palatability(X1) Texture(X2) Overall Quality(Y1) Purchase

intent(Y2)
65 71 63 67
72 77 70 70
77 73 72 70
68 78 75 72
81 76 89 88
73 87 76 77
Y1 = β01 + β11 X1 + β21 X2
Y2 = β02 + β12 X1 + β22 X2

Y=

1 65 71
1 72 77
1 77 73
X= 1 68 78
1 81 76
1 73 87
XT = 1 1 1 1 1 1
65 72 77 68 81 73
71 77 73 78 76 87

6 436 462
T
XX = 436 31852 33591
462 33591 35728

65.5606 -0.3783 -0.4553

(X XT ) -1 = -0.3783 0.060 -0.0007
0.4533 -0.0007 0.0066
β = (X XT ) -1 XT Y
-37.5012 -21.4323
1.1346 0.9409
0.3795 0.3514

Y1 = -37.5012+1.1346X1+0.3795X2
Y2 = -21.4323+0.9409X1+ 0.3514X2
Find Y1 & Y2 if X1=68 & X2=78
LOGISTIC REGRESSION
⚫ Logistic regression estimates the
probability of an event occurring, such as
voted or didn't vote, based on a given
dataset of independent variables.

⚫ This type of statistical model (also known

as logist model) is often used for
classification and predictive analytics.
⚫ Y=1/ 1+e-x or Y=1/1+e-(bo+b1x1)
X= bo+b1x1+b2x2+----+bnxn
LOGISTIC OR SIGMOID
FUNCTION
GRADIENT ASCENT METHOD
⚫ In gradient ascent method we move in the
direction of the gradient to find
maximum point on a function.
The gradient ascent method operator will
always point towards the direction of gradient
increase.

This step is repeated until we reach the

stopping criteria.
GRADIENT DESCENT METHOD
● Gradient Descent Method- We move function in the opposite
direction of the gradient to find the minimum point on a function.

● The gradient operator will always point opposite to the direction of

gradient increase.
● This step is repeated until we reach stopping criteria
Type of Logistic Regression:
⚫ Binary : In binomial Logistic regression,
there can be only two possible types of the
dependent variables, such as 0 or 1, Pass or
Fail, etc.
⚫ Nominal: In nomial Logistic regression,
there can be 3 or more possible without any
specific order.
Eg- “computer , IT , Mechanical, Civil”
departments
⚫ Ordinal: In ordinal Logistic regression,
there can be 3 or more possible classes with
specific orders.
Eg-“Excellent”, “Good”, or “Bad”.
Regression Problem
solution
Sr. No Linear Regresssion Logistic Regression

Linear regression is used to predict the continuous Logistic regression is used to predict the categorical
1 dependent variable using a given set of independent dependent variable using a given set of independent
variables. variables.

Linear regression is used for solving Regression

2 It is used for solving classification problems.
problem.

3 In this we predict the value of continuous variables In this we predict values of categorical varibles

4 In this we find best fit line. In this we find S-Curve .

Least square estimation method is used for Maximum likelihood estimation method is used for
5
estimation of accuracy. Estimation of accuracy.

The output must be continuous value,such as Output is must be categorical value such as 0 or 1,
6
price,age,etc. Yes or no, etc.

It required linear relationship between dependent

7 It not required linear relationship.
and independent variables.

There may be collinearity between the independent There should not be collinearity between
8
variables. independent varible.
 DECISION TREE
⚫ A decision tree is a tree-like structure that
represents a series of decisions and their
possible consequences. It is used in machine
learning for classification and regression
tasks.
 Example of a decision tree

Root Node

Splitting

Decision
Node

Leaf/
Terminal
Node

Parent and
Child
Node

Branch/Sub
Tree
Attribute selection Measures
⚫
⚫
* I(Pi,ni)
 Strength of decision tree
⚫ Able to generate understandable rules
⚫ Performs classification without requiring
much computation
⚫ Decision tree clearly indicates which fields
are most important for prediction or
classification
Weakness of decision tree
⚫ Perform poorly with many class and small
class
⚫ expensive to train
⚫ Decision tree model generally overfits. It
means it does not perform well on
validation sample
Issues in Decision Tree

⚫ Determine how deeply to grow the

decision tree.
⚫ Handling continuous attributes.
⚫ Choosing an appropriate attribute
selection measure.
⚫ Handling training data with missing
attribute values.
⚫ Handling attributes with differing values.
⚫ Improving computational efficiency.
⚫ Example of Decision Tree
using ID3(Iterative
Dichotomiser3)
1) Suppose we want ID3 to evaluate car
database as whether the car is acceptable
or not. The target classification is “should
we accept car?” which can be acceptable
or unacceptable.
Decision Tree Example using Gini Index
Performance Metrics
⚫ Performance Metrics for Regression
1) Mean Absolute Error (MAE)

2) Mean Squared Error(MSE)

 3) RMSE-Root mean square error is use for
evaluating the quality of predictions.

4) R-Squared- It shows how well the model

predicts the outcome of the dependent
variable.

R² = 1 - RSS/TSS or 1- ∑ (Yi-Y’i)2 /∑(Yi-ˉY)2

RSS = sum of squares of residuals

TSS = total sum of squares
⚫
Performance Metrics for
Classification
Accuracy
⚫ The accuracy metric is one of the
simplest Classification metrics to
implement, and it can be determined as
the number of correct predictions to the
total number of predictions.
⚫ It can be formulated as:
Confusion Matrix

1. Accuracy
Accuracy is used to measure the performance of the model. It is the
ratio of Total correct instances to the total instances.

Accuracy = (5+3)/(5+3+1+1) = 8/10 = 0.8

2. Precision
Precision is a measure of how accurate a model’s
positive predictions are. It is defined as the ratio
of true positive predictions to the total number of
positive predictions made by the model.

Precision = 5/(5+1) =5/6 = 0.8333

3. Recall
Recall measures the effectiveness of a classification
model in identifying all relevant instances from a
dataset. It is the ratio of the number of true
positive (TP) instances to the sum of true positive
and false negative (FN) instances.

Recall = 5/(5+1) =5/6 = 0.8333

4. F1-Score
F1-score is used to evaluate the overall
performance of a classification model. It is the
harmonic mean of precision and recall

F1-Score: = (2* 0.8333* 0.8333)/( 0.8333+ 0.8333) =

0.8333
5. Specificity
Specificity is another important metric in the evaluation of
classification models, particularly in binary classification. It
measures the ability of a model to correctly identify
negative instances. Specificity is also known as the True
Negative Rate. Formula is given by

Specificity=3/(1+3)=3/4=0.75
AUC(Area under Curve)-ROC(Receiver
Operating Characteristics) Curve

⚫ The ROC curve is a graphical representation

of the trade-off between true positive rate
and false positive rate at various thresholds.
It shows the performance of a classification
model at different classification thresholds.
The AUC (Area Under the Curve) is a
summary measure of the ROC curve
performance.
⚫ True Positive Rate (TPR):
Proportion of actual positives correctly
identified by the classifier is

False Positive Rate (FPR):

Proportion of actual negatives incorrectly
classified as positives.