ITDX45- Big Data

Analytics
Module 3- Advanced Analytics
Regression

• Regression analysis attempts to explain the influence

that a set of variables has on the outcome of another
variable of interest.
E.g:
What is a person’s expected income?
What is the probability that an applicant will default on a
loan?
• Linear regression is a useful tool for answering
the first question, and logistic regression is a
popular method for addressing the second.
Linear regression
• Linear regression models are useful in physical and
social science applications where there may be
considerable variation in a particular outcome based on
a given set of input values.
1. Simple Linear regression:
X -> Y
2. Multiple Linear regression:
X1,X2,… -> Y
Simple Linear regression
• This is the simplest form of linear regression, and it
involves only one independent variable and one
dependent variable. The equation for simple linear
regression is:
y=β0+β1 X
where:
• y is the dependent variable
• X is the independent variable
• β0 ​is the intercept
• β1 is the slope
Simple linear regression
• The equation is given by y = a+bx
Step 1 : Find mean and covariance
Variance of x is given by,

Covariance of x and y is given by,

Step2: Find value for a & b
Intercept a = ȳ - bx̄
Slope b = cov(x,y) / var(x)
X Y
Problem: 1 1
Find the regression equation for the given dataset
2 2
using simple linear regression.
x̄ = (1+2+3+4+5) / 5 = 3 3 1.3

ȳ = (1+2+1.3+3.75+2.25) / 5 = 10.30 /5 =42.063.75

5 2.25
X Y
1 1

2 2

3 1.3

4 3.75

5 2.25
=
=
 X Y
1 1 -2 4 -1.06 2.12

2 2 -1 1 -0.06 0.06

3 1.3 0 0 -0.76 0

4 3.75 1 1 1.69 1.69

5 2.25 2 4 0.19 0.38

= 10
= 4.25
Slope, b = cov(x,y) / var(x)

Cov(x,y) = 1/4 (4.25) = 1.0625

Var(x) = ¼ (10) = 2.5

Slope, b = 1.625 / 2.5 = 0.65
Intercept, a = ȳ - bx̄
a= 2.06 – 0.65(3) = 0.11
The regression equation is given by, y = 0.11+ 0.65 x
Multiple Linear regression
• This involves more than one independent variable and
one dependent variable. The equation for multiple linear
regression is:
y=β0+β1X1+β2X2+………βnXn
where:
• Y is the dependent variable
• X1, X2, …, Xn are the independent variables
• β0 is the intercept
• β1, β2, …, βn are the slopes
Muliple linear regression problem
• Check the video shared
Logistic regression
• Logistic regression is used for binary classification
where we use sigmoid function, that takes input as
independent variables and produces a probability value
between 0 and 1.
• For example, we have two classes Class 0 and Class 1 if
the value of the logistic function for an input is greater
than 0.5 (threshold value) then it belongs to Class 1
otherwise it belongs to Class 0.
• It’s referred to as regression because it is the extension
of linear regression but is mainly used for classification
problems.
Key Points:
• Logistic regression predicts the output of a categorical
dependent variable. Therefore, the outcome must be a
categorical or discrete value.
• It can be either Yes or No, 0 or 1, true or False, etc. but
instead of giving the exact value as 0 and 1, it gives the
probabilistic values which lie between 0 and 1.
• In Logistic regression, instead of fitting a regression line,
we fit an “S” shaped logistic function, which predicts
two maximum values (0 or 1).
 Logistic regression problem
The dataset of pass or fail in an exam of 5
students is given in the table. Use logistic
regression as classifier to answer the
Hours study Pass(1) / Fail following questions.
(0) 1. Calculate the probability of pass for
29 0 the student who studied 33 hours.
15 0 2. At least how many hours students
should study that makes he will pass
33 1 the course with the probability of more
28 1 than 95%
Assume the model suggested by the
39 1
optimizer for odds of passing the course
is,
log(odds)= -64+2*hours study
1. Calculate probability for 33 hours
of study
P=
Where z= log(odds)= -64+2*hours study
Substitute the hours of study,
Z= -64+2*33
=2
Probability =
= 0.88
2. Hours of study for 95% probability
• P= = 0.95
Solving the equation,
0.95 * ()=1
0.95 * = 1-0.95
= 0.05 / 0.95 = 0.0526
log
-z=-2.94 (since, log = x)
Z=2.94
• z= log(odds)= -64+2*hours study
2.94 = -64+2 *hours study
Hours study = 66.94 / 2
= 33.47 hours

Hence the student should study atleast 33.47 hours, so

that he will pass the exam with more than 95%
probability.
Decision tree
• A decision tree (also called prediction tree) uses a tree
structure to specify sequences of decisions and
consequences.
• The input values of a decision tree can be categorical or
continuous.
• A decision tree employs a structure of test points (called
nodes) and branches, which represent the decision being
made.
• A node without further branches is called a leaf node.
• The leaf nodes return class labels and, in some
implementations, they return the probability scores.
• Decision trees have two varieties:
1. Classification trees
2. Regression trees
• Internal nodes are the decision or test points.
• Each internal node refers to an input variable or an
attribute.
• The top internal node is called the root.
• The branching of a node is referred to as a split.
• The depth of a node is the minimum number of steps
required to reach the node from the root.
• By limiting the number of splits, a short tree can be
created. Short trees are often used as components (also
called weak learners or base learners) in ensemble
methods.
• The simplest short tree is called a decision stump, which is
a decision tree with the root immediately connected to the
leaf nodes.
• Most commonly used decision tree algorithm is ID3,
which is an entrophy based method.
• Entropy, which measures the impurity of an attribute.
• Information gain, which measures the purity of an
attribute.
Construct the decision tree for the following
training dataset using ID3 algorithm
Age Income Student Credit_rating Buys_Computer
<=30 High No Fair No
<=30 High No excellent No Attributes/
31..40 High No Fair Yes
Features:
>40 Medium No Fair Yes
>40 Low Yes Fair Yes 1. Age
>40 Low Yes excellent No
31..40 Low Yes excellent Yes
2. Income
<=30 Medium No Fair No 3. Student
<=30 Low Yes Fair Yes
>40 Medium Yes Fair Yes 4. Credit-rating
<=30 Medium Yes excellent Yes
31..40 Medium No excellent Yes
5. Buys-
31..40 High Yes Fair Yes Computer
>40 Medium No excellent No
1. Calculate Entrophy which is formulated as,

2. Calculate Information Gain as,

• Calculate overall Entrophy: (Entrophyparent)

Entrophy (E0)=
Step I: Finding Root node
Age Income Student Credit_rating Buys_Computer
<=30 High No Fair No
<=30 High No excellent No
<=30 Medium No Fair No
Age: (3 categories) <=30 Low Yes Fair Yes
a. <=30 (E1) 2 yes 3 no (tot:5) <=30 Medium Yes excellent Yes

b. 31..40 (E2) 4 yes(tot:4) Age Income Student Credit_rating Buys_Computer

31..40 High No Fair Yes
c. >40 (E3) 3 yes 2 no (tot:5) 31..40 Low Yes excellent Yes
31..40 Medium No excellent Yes
31..40 High Yes Fair Yes

Age Income Student Credit_rating Buys_Computer

>40 Medium No Fair Yes
>40 Low Yes Fair Yes
>40 Low Yes excellent No
>40 Medium Yes Fair Yes
>40 Medium No excellent No
Entrophy for <=30: Age Income Student Credit_rating Buys_Computer
<=30 High No Fair No
<=30 High No excellent No
E1= <=30 Medium No Fair No
<=30 Low Yes Fair Yes
<=30 Medium Yes excellent Yes
Entrophy for 31..40:
Age Income Student Credit_rating Buys_Computer
31..40 High No Fair Yes

E2= 31..40 Low Yes excellent Yes

31..40 Medium No excellent Yes
31..40 High Yes Fair Yes

Entrophy for >40: Age Income Student Credit_rating Buys_Computer

>40 Medium No Fair Yes
>40 Low Yes Fair Yes
E3= >40 Low Yes excellent No
>40 Medium Yes Fair Yes
>40 Medium No excellent No
• Average Entrophy =

• Information Gain(age) =
Income: (3 categories)
a. High (2 yes 2 no, tot:4) Age Income Student Credit_rating Buys_Computer
<=30 High No Fair No
b. Medium (4 yes 2 no, tot:6) <=30 High No excellent No
c. Low (3 yes 1 no, tot:4) 31..40 High No Fair Yes
31..40 High Yes Fair Yes

E1=-2/4log22/4-2/4log22/4 >40 Medium No Fair Yes

<=30 Medium No Fair No
>40 Medium Yes Fair Yes
<=30 Medium Yes excellent Yes
E2=-4/6log24/6-2/6log22/631..40 Medium No excellent Yes
>40 Medium No excellent No

>40 Low Yes Fair Yes

E3=-3/4log23/4-1/4log21/4 >40 Low Yes excellent No
31..40 Low Yes excellent Yes
<=30 Low Yes Fair Yes
• Average Entrophy =
4/14(-2/4log22/4-2/4log22/4) + 6/14(-4/6log24/6-2/6log22/6) + 4/14(-
3/4log23/4-1/4log21/4)

• Information gain(income) = 0.94 – Average Entrophy = 0.03

student: (2 categories)
a. Yes (6 yes 1 no, tot:7) Age Income Student Credit_rating Buys_Computer
>40 Low Yes Fair Yes
b. No (3 yes 4 no, tot:7) >40 Low Yes excellent No

E1=-6/7log26/7-1/7log21/731..40 Low Yes excellent Yes

<=30 Low Yes Fair Yes
E2=-3/7log23/7-4/7log24/7 >40 Medium Yes Fair Yes
<=30 Medium Yes excellent Yes
31..40 High Yes Fair Yes

Average Entropy=
<=30 High No Fair No
(7/14)E1 + (7/14) E2 <=30 High No excellent No
31..40 High No Fair Yes
>40 Medium No Fair Yes

I.G(student) = [Link] <=30 Medium No Fair No

31..40 Medium No excellent Yes
= 0.152 >40 Medium No excellent No
Credit-rating: (2 categories)
Age Income Student Credit_rating Buys_Computer
<=30 High No Fair No
a. fair (6 yes 2 no, tot:8) 31..40 High No Fair Yes
b. excellent (3 yes 3 no, tot:6) >40 Medium No Fair Yes
>40 Low Yes Fair Yes
E1=-6/8log26/8-2/8log22/8 <=30 Medium No Fair No
<=30 Low Yes Fair Yes
E2=-3/6log23/6-3/6log23/6 >40 Medium Yes Fair Yes
31..40 High Yes Fair Yes

Average Entropy= <=30 High No excellent No

>40 Low Yes excellent No
(8/14)E1 + (6/14) E2 31..40 Low Yes excellent Yes
<=30 Medium Yes excellent Yes
31..40 Medium No excellent Yes

I.G(credit-rating) = [Link]
>40 Medium No excellent No

= 0.048
Select the attribute having highest information gain for
root node

<= 31..4 >4

• I.G(age)=0.246 30 0 0
• I.G(income)=0.03
• I.G(student)=0.152
• I.G(credit-rating)=0.048
Step II: Now Calculate I.G for the attributes in the subset
of data to find the next decision node
• Overall Entropy =
-(2/5log22/5 + 3/5log22/5)=0.97
• Attributes:
Income,student,credit-rating
Income:
1. High (2 nos tot :2)E1=0
2. Low (1 yes tot:1)E2=0
3. Medium (1 yes 1 no tot:2)E3=-(1/2log21/2 + 1/2log21/2)
I.G(income)=0.97-(2/5 E1+1/5 E2+2/5 E3)=0.57
student:
1. Yes (2 yes)E1=0
2. No (3 nos)E2=0
I.G(student)=0.97-(0)=0.97
Credit-rating:
1. Fair (1 yes 2 no)
E1=-(1/3log21/3+2/3log22/3)
2. Excellent (1 yes 1no)
E2=-(1/2log21/2+1/2log21/2)
I.G(credit-rating)= 0.97-(3/5 E1+2/5 E2)= 0.02
Step III: Now Calculate I.G for the attributes in
the subset of data to find the next decision node
Overall Entropy =
-(3/5log23/5 + 2/5log22/5)=0.97
Attributes:
Income,student,credit-rating
Income:
1. Low (1 yes 1no tot:2)E1=-(1/2log21/2+
1/2log21/2)
2. Medium (1 yes 2 no tot:3)
E2=-(1/3log21/3 + 2/3log22/3)
I.G(income)=0.97-(3/5 E1+2/5 E2)=0.02
student:
1. Yes (2 yes 1 no)E1=-(2/3log22/3 + 1/3log21/3)
2. No (1yes 1no)E2=-(1/2log21/2 + 1/2log21/2)
I.G(student)=0.97-(3/5 E1 + 2/5 E2)=0.02
Credit-rating:
1. Fair (3 yes) E1=0
2. Excellent (2 no) E2=0
I.G(credit-rating)= 0.97-(0)= 0.97
<=3 31..40 >40
0
Naïve Bayes
• Naïve Bayes is a probabilistic classification method
based on Bayes’ theorem.
• Bayes’ theorem gives the relationship between the
probabilities of two events and their conditional
probabilities.
• naïve Bayes classifiers are easy to implement and can
execute efficiently even without prior knowledge of the
data.
• Spam filtering and fraud detection are the classic use
cases of naïve Bayes text classification.
Bayes theorem
• The conditional probability of event C occurring, given
that event A has already occurred is denoted as
which can be found using the formula

• where C is the class label and A is the observed

attributes.
Naïve Bayes Classifier
Consider the below dataset, identify the species of
an entity having attributes like (color:Green, Legs:2,
height:Tall, Smelly:No) using naïve bayes classifier.
Step 1: Find Total Probability
S.N Colo Leg Heigh Smell Specie P(M) = 4/8 = 0.5
o r s t y s
P(H) = 4/8 = 0.5
Whit
1 3 Short Yes M Step 2: Find Conditional
e
Gree
Probability
2 2 Tall No M
n For Color:
Gree Color M H
3 3 Short Yes M
n
Whit 2/4 = 3/4
4 3 Short Yes M White
e 0.5 =0.75
Gree 2/4 = 1/4
5 2 Short No H Green
n 0.5 =0.25
Whit
6 2 Tall No H
e
Whit
For Legs:
Legs M H

¼ =
2 4/4 = 1
0.25 Leg
[Link] Color Height Smelly Species
¾= s
3 0
0.75 1 White 3 Short Yes M
For height: 2 Green 2 Tall No M
Height M H 3 Green 3 Short Yes M
¼=
Tall 2/4=0.5 4 White 3 Short Yes M
0.25
5 Green 2 Short No H
Short ¾=0.75 2/4=0.5
6 White 2 Tall No H
For Smelly:
7 White 2 Tall No H
Smelly M H
8 White 2 Short Yes H
Yes ¾=0.75 ¼=0.25

No ¼=0.25 ¾=0.75
 Step 3: Calculate the probability of new attribute given

Vj ϵ
{M,H}

Vj ϵ
{M,H}
New attributes are (color:Green, Legs:2, height:Tall, Smelly:No)
• P(color=green| ),P(Legs=2| ),P(Height=Tall| ),P(Smelly=No|
)
• (M) = P(M)*P(green|M)* P(2|M)*P( Tall|M) * P(No|M)
= 0.5 * 0.75 * 0.25 * 0.25 = 0.0234
(H) = P(H)*P(green|H)* P(2|H)*P( Tall|H) * P(No|H)
= 0.5 * 0.25 * 1 * 0.5 * 0.75 = 0.0469
The new attribute belongs to the outcome having higher value of .
Hence (color:Green, Legs:2, height:Tall, Smelly:No) belongs to the species
H.
Diagnostics of classifiers
• A few tools have been designed to evaluate the
performance of a classifier.
• Confusion matrix
• The accuracy (overall success rate)
• Precision
(% instances marked positive that
really are positive)
• Recall
(% of positive instances that were
correctly identified)
• Accuracy = (TP+TN) / (TP+TN+FP+FN)

• Precision = TP / (TP +FP)

• Recall = TP / (TP+FN)
Watch the Finding ROC video shared
in blog
Working of AUC
Index Class Probability Step 1:
P1 1 0.95
Split the points based on class
labels
P2 1 0.90 (p1,p2,p5) – class 1
P3 0 0.85 (p3,p4,p6) – class 0
P4 0 0.81

P5 1 0.78

P6 0 0.70
 Pair

Step 2: (P1,P3)

Form pairs of points such that, each (P1,P4)

point belongs to separate class. (P1,P6)
(i.e., here, one point belongs to class
(P2,P3)
1 and another point belongs to class
0) (P2,P4)

(P2,P6)

(p1,p2,p5) – class 1 (P3,P5)

(p3,p4,p6) – class 0 (P4,P5)

(P5,P6)
 Pair isCorrect

(P1,P3) Yes
Step 3:
Check the mentioned pair is correctly
(P1,P4) rank-ordered
Yes based
(P1,P6) Yes
(p1,p2,p5) – class 1 (P2,P3) Yes
(p3,p4,p6) – class 0
(P2,P4) Yes

(P2,P6) Yes

(P3,P5) No

(P4,P5) No

(P5,P6) Yes
Step 4:
Calculate AUC = 7/9 = 0.78
Hence the given data points having AUC as 78%