BIG DATA AND BUSINESS

ANALYTICS
Module: 3

PREDICTIVE ANALYTICS SUPERVISED

3.1 Simple Linear Regression. Multiple Linear Regression (MLR) –

Assumptions of MLR, Interpretation of MLR Coefficients, Statistical
significance of Individual Predictors (t-Test), Overall model Validation (F-
Test), MLR metric – R-Squared and RMSE

3.2 Residual Analysis in MLR. Multi-Collinearity and Variance 68 Inflation

Factor (VIF) (Theory only)

3.3 Classification – Logistic Regression, Decision trees – CART and CHAID

SIMPLE LINEAR REGRESSION – MULTIPLE LINEAR

REGRESSION(MLR) –ASSUMPTIONS OF MLR - INTERPRETATION
OF MLR CO-EFFICIENTS

SLR

 A statistical method that allows us to summarize and study relationships

between two continuous (quantitative) variables:
 One variable, denoted , is regarded as the predictor, explanatory, or
independent variable.
 The other variable, denoted , is regarded as the response, outcome, or
dependent variable.
 Simple linear regression gets its adjective "simple," because it concerns
the study of only one predictor variable.

Module :03 1
  A linear regression line has an equation of Y = a + bX,
 where X is the explanatory variable and Y is the dependent variable. The
slope of the line is b, and a is the intercept (the value of y when x = 0).

MULTIPLE LINEAR REGRESSION(MLR)

Multiple linear regression is used to estimate the relationship between two or

more independent variables and one dependent variable. You can use multiple
linear regression when you want to know:

■ How strong the relationship is between two or more independent

variables and one dependent variable (e.g. how rainfall, temperature, and
amount of fertilizer added affect crop growth).
■ The value of the dependent variable at a certain value of the independent
variables (e.g. the expected yield of a crop at certain levels of rainfall,
temperature, and fertilizer addition).

ASSUMPTIONS OF MLR

Multiple linear regression makes all the same assumptions as simple linear
regression:

Module :03 2
 ■ Homogeneity of variance : the size of the error in our prediction doesn’t
change significantly across the values of the independent variable.
■ Independence of observations: the observations in the dataset were
collected using statistically valid methods, and there are no hidden
relationships among variables.
■ Normality: The data follows a normal distribution.
■ Linearity: the line of best fit through the data points is a straight line,
rather than a curve or some sort of grouping factor.

INTERPRETATION OF MLR CO-EFFICIENTS

■ In simple or multiple linear regression, the size of the coefficient for each
independent variable gives you the size of the effect that variable is
having on your dependent variable, and the sign on the coefficient
(positive or negative) gives you the direction of the effect.
■ In regression with a single independent variable, the coefficient tells you
how much the dependent variable is expected to increase (if the
coefficient is positive) or decrease (if the coefficient is negative) when
that independent variable increases by one.
■ In regression with multiple independent variables, the coefficient tells
you how much the dependent variable is expected to increase when that
independent variable increases by one, holding all the other independent
variables constant. Remember to keep in mind the units which your
variables are measured in.

RESIDUAL ANALYSIS IN MLR MULTI – COLLINEARITY

VARIANCE 68 INFLATION FACTOR

RESIDUAL ANALYSIS

Module :03 3
 ■ Residual analysis is used to assess the appropriateness of a linear
regression model by defining residuals and examining the residual plot
graphs.

RESIDUALS

■ The difference between the observed value of the dependent variable (y)
and the predicted value (ŷ) is called the residual (e). Each data point has
one residual.
■ Residual = Observed value - Predicted value
e=y-ŷ
■ Both the sum and the mean of the residuals are equal to zero. That is,
Σ e = 0 and e = 0.

RESIDUAL PLOTS

■ A residual plot is a graph that shows the residuals on the vertical axis
and the independent variable on the horizontal axis.
■ If the points in a residual plot are randomly dispersed around the
horizontal axis, a linear regression model is appropriate for the data;
otherwise, a nonlinear model is more appropriate.

TYPES OF RESIDUAL PLOT

Following example shows few patterns in residual plots.

Module :03 4
 ■ In first case, dots are randomly dispersed. So linear regression model is
preferred. In Second and third case, dots are non-randomly dispersed and
suggests that a non-linear regression method is preferred.

Multi collinearity

■ Multi collinearity occurs when independent ariables in a regression model

are correlated. This correlation is a problem because independent
variables should be independent. If the degree of correlation between
variables is high enough, it can cause problems when you fit the model
and interpret the results.

There are two basic kinds of multi collinearity:

■ Structural multi collinearity: This type occurs when we create a model

term using other terms. In other words, it’s a byproduct of the model that
we specify rather than being present in the data itself. For example, if you

Module :03 5
 square term X to model curvature, clearly there is a correlation between
X and X2.
■ Data multi collinearity: This type of multi collinearity is present in the
data itself rather than being an artifact of our model. Observational
experiments are more likely to exhibit this kind of multi collinearity.

What Problems Do Multicollinearity Cause?

■ The coefficient estimates can swing wildly based on which other

independent variables are in the model. The coefficients become very
sensitive to small changes in the model.

■ Multi collinearity reduces the precision of the estimate coefficients,

which weakens the statistical power of your regression model.

Testing for Multi collinearity with Variance Inflation

Factors (VIF)
The variance inflation factor (VIF) identifies correlation between independent
variables and the strength of that correlation.

■ Statistical software calculates a VIF for each independent variable. VIFs

start at 1 and have no upper limit.
■ A value of 1 indicates that there is no correlation between this
independent variable and any others.
■ VIFs between 1 and 5 suggest that there is a moderate correlation, but it
is not severe enough to warrant corrective measures.
■ VIFs greater than 5 represent critical levels of multi collinearity where
the coefficients are poorly estimated, and the p-values are questionable.
■ Assessing VIFs is particularly important for observational studies because
these studies are more prone to having multi collinearity

Module :03 6
How to Deal with Multi collinearity

The potential solutions include the following:

■ Remove some of the highly correlated independent variables.

■ Linearly combine the independent variables, such as adding them
together.
■ Perform an analysis designed for highly correlated variables, such as
principal components analysis or partial least squares regression.
■ LASSO and Ridge regression are advanced forms of regression analysis
that can handle multicollinearity. If you know how to perform linear least
squares regression, you’ll be able to handle these analyses with just a
little additional study.

Module :03 7
 2 types of Decision Trees: CHAID &
CART
Tree-based learning algorithms are one of the best and
mostly used supervised learning methods as they empower
predictive models with high accuracy, stability, and ease of
interpretation. Unlike linear models, they map non-linear
relationships quite well. They are adaptable at solving any kind
of problem at hand (classification or regression).

CHAID and CART are the two oldest types of Decision

trees. They are also the most common types of Decision trees
used in the industry today as they are super easy to understand
while being quite different from each other. In this post, we’ll
learn about all the fundamental information required to
understand these two types of decision trees.

CHAID- Chi-Squared Automatic

Interaction Detection
This algorithm was originally proposed by Kass in 1980. As is evident
from the name of this algorithm, it is based on the chi-square statistic. A Chi-
square test yields a probability value as a result lying anywhere between 0 and 1.
A chi-square value closer to 0 indicates that there is a significant difference
between the two classes which are being compared. Similarly, a value closer to 1
indicates that there is not any significant difference between the 2 classes.

Module :03 8
 CART- Classification & Regression
Tree
This algorithm was introduced to the world by Breiman et al. in 1984.

This algorithm works on twin pillars of “Purity” and “Balance”. Purity is

indicated by Gini as explained above and balance is indicated by the same
population distribution among the left and the right partitions of the decision
tree. An optimal CART tree will not sway in on either one pillar.

1. Forcing “balance” on a CART tree can lead to many impure leaf

nodes which defeat the purpose of decision making using a decision
tree.
2. Forcing “purity” on a CART tree can give us very less population
distribution in one segment, again, defeating the purpose of a healthy
Decision tree.

Reference : [Link]
decision-trees-chaid-cart-8695e441e73e

LOGISTIC REGRESSION, DECISION TREES, CHAID,

CART

LOGISTIC REGRESSION

 Logistic regression is a statistical analysis method used to

predict a data value based on prior observations of a data
set.

Module :03 9
  Logistic regression has become an important tool in the
discipline of machine learning.
 Logistic regression is one of the most commonly used
machine learning algorithms for binary classification
problems, which are problems with two class values,
including predictions such as “this or that,” “yes or no”
and “A or B.”

EXAMPLES

 One example of this is predicting if a student will pass or

fail an exam when the number of hours spent studying is
provided as a feature and the variable for the response
has two values: pass and fail.
 An e-commerce company that mails expensive
promotional offers to customers would like to know
whether a particular customer is likely to respond to the
offers or not. For example, they’ll want to know whether
that consumer will be a “responder” or a “non-responder.”
In marketing, this is called propensity to respond
modeling.

PURPOSE OF LOGISTIC REGRESSION

The purpose of logistic regression is to estimate the

probabilities of events, including determining a relationship
between features and the probabilities of outcomes.

USES OF LOGISTIC REGRESSION

Module :03 10
Logistic regression has become particularly popular in online
advertising, enabling marketers to predict the likelihood of
specific website users who will click on particular
advertisements as a yes or no percentage.

 Logistic regression can also be used in:

• Healthcare to identify risk factors for diseases and plan

preventive measures.
• Weather forecasting apps to predict snowfall and weather
conditions.
• Voting apps to determine if voters will vote for a particular
candidate.
• Insurance to predict the chances that a policy holder will
die before the term of the policy expires based on certain
criteria, such as gender, age and physical examination.
• Banking to predict the chances that a loan applicant will
default on a loan or not, based on annual income, past
defaults and past debts.

DECISION TREES

 A decision tree is a very specific type of probability tree

that enables you to make a decision about some kind of
process.
 For example, you might want to choose between
manufacturing item A or item B, or investing in choice 1,
choice 2, or choice 3.

Module :03 11
  Trees are an excellent way to deal with these types of
complex decisions, which always involve many different
factors and usually involve some degree of uncertainty.
 Although they can be drawn by hand, software is often
used as the trees can become complex very quickly.

COMPONENTS OF DECISION TREES

There are three broad areas usually displayed in a tree:

1. The Decision: displayed as a square node with two or

more arcs (called “decision branches”) pointing to the
options.
2. The Event sequence: displayed as a circle node with two
or more arcs pointing out the events. Probabilities may be
displayed with the circle nodes, which are sometimes
called “chance nodes”.
3. The Consequences: the costs or utilities associated with
different pathways of the decision tree. The endpoint is
called a “Terminal” and is represented by a triangle or bar
on a computer.

ADVANTAGES OF DECISION TREE

• A decision tree is easy to understand and interpret.

• Expert opinion and preferences can be included, as well as
hard data.
• Can be used with other decision techniques.
• New scenarios can easily be added.

DISADVANTAGES OF DECISION TREES

Module :03 12
 • If a decision tree is used for categorical variables with
multiple levels, those variables with more levels will have
more information gain.
• Calculations can quickly become very complex, although
this is usually only a problem if the tree is being created
by hand.

CHI-SQUARE AUTOMATED INTERACTION

DETECTION (CHAID)

• CHAID (Chi-Square Automatic Interaction Detection) is one

of the oldest tree-building techniques
• CHAID works only with categorical predictors and targets.
The algorithm computes a chi-square test between the
target variable and each available predictor and then uses
the best predictor to partition the sample. It then
proceeds, in turn, with each segment and repeats the
process until no significant splits remain. The standard
CHAID algorithm does not prune or cross-validate the tree.
• Software implementations of CHAID vary; typically, the
user can specify a minimum significance of the chi-square
test, a minimum cell size, and a maximum depth for the
tree.
• The principal advantages of CHAID are its use of the chi-
square test (which is familiar to most statisticians) and its
ability to perform multiway splits. The main weakness of
CHAID is its limitation to categorical data.

CLASSIFICATION AND REGRESSION TREES (CART)

Module :03 13
 • CART, or Classification and Regression Trees. CART is a
nonparametric algorithm that learns and validates
decision tree models.
• A Classification and Regression Tree(CART) is a predictive
algorithm used in machine learning. It explains how a
target variable’s values can be predicted based on other
values.
• Like CHAID, the algorithm proceeds recursively,
successively splitting the data set into smaller segments.
However, there are key differences between the CHAID
and CART algorithms:
• CHAID uses the chi-square measure to identify split
candidates, whereas CART uses the Gini rule.
• CHAID supports multiway splits for predictors with more
than two levels; CART supports binary splits only and
identifies the best binary split for complex categorical or
continuous predictors.
• CART prunes the tree by testing it against an independent
(validation) data set or through n-fold cross-validation;
CHAID does not prune the tree.
• CART works with either categorical targets (classification
trees) or continuous targets (regression trees) as well as
either categorical or continuous predictors.

Module :03 14