Multiple
Linear
Regression
VA N D I TA B A N S A L
What is Multiple Linear
Regression?
Regression is used to study the relationship between one dependent variable
and two or more independent variables.
Just as in single regression, we need the dependent variable to be numerical.
The independent variables can be numerical or categorical.
Why do we use MLR?
1. Single regression (i.e., with one IV) allows us to study the relationship between two variables
only.
2. However, we do not believe that only a single variable explains all the variation of the
dependent variable.
3. For example, in the scenario of IQ and income, we do not expect IQ only to explain income, but
we expect that there are also other variables, such as years of education, to explain income.
4. Hence, to make the model more realistic, it makes sense to include multiple independent
variables in the regression.
How is MLR different from SLR?
1. We now have more than 1 variable, there is also a problem that out of so many
variables which one should be selected?
2. Now, we also need to check for multicollinearity assumption.
3. N-number of Betas must be estimated, so to obtain a unique solution for Betas,
n>>p (Number of observations should be greater than the number of
parameters)
4. Data is treated as a matrix,
Can we plot the data for MLR in a
graph?
As opposed to the simple linear
regression case, we do not have a
way to plot all the variables at the
same time.
Hence, the scatterplot can be
performed only for each continuous
independent variable independently.
Estimation of Betas in MLR
1. This equation identifies k-1 explanatory variables, namely X1, X2,……Xk.
The y-vector is thus expressed as a linear combination of the x vectors +
the disturbances.
Linear Equation as a matrix
Estimation of Betas
Betas are estimated using Linear Least squares Method, where sum of square of
error can be minimized.
Estimation of Betas
Example
Example- Derive Beta
Other methods of estimating
Betas
Maximum Likelihood Method (MLE)
Method of Moments
In Linear regression: OLS estimate of Beta is same as MLE estimate of Beta
Interpretation of Multiple Linear
Regression
As in the two-variable case, multiple regression analysis is regression analysis
conditional upon the fixed values of the regressors, and what we obtain is the
average or mean value of Y or the mean response of Y for the given values of the
regressors.
the regression coefficients β2 and β3 are known as partial regression or partial
slope coefficients. The meaning of partial regression coefficient is as follows: β2
measures the change in the mean value of Y, E(Y), per unit change in X2, holding the
value of X3 constant. Put differently, it gives the “direct” or the “net” effect of a unit
change in X2 on the mean value of Y, net of any effect that X3 may have on mean Y.
Likewise, β3 measures the change in the mean value of Y per unit change in X3,
holding the value of X2 constant.4 That is, it gives the “direct” or “net” effect of a unit
change in X3 on the mean value of Y, net of any effect that X2 may have on mean Y.
Goodness of Fit in MLR
An important property of R2 is that it is a
nondecreasing function of the number of
explanatory variables or regressors present in the
model, unless the added variable is perfectly
collinear with the other regressors; as the number
of regressors increases, R2 almost invariably
increases and never decreases. Stated differently,
an additional X variable will not decrease R2.
To compare two R2 terms, one must consider the
number of X variables present in the model. This
can be done readily if we consider an alternative
coefficient of determination, which is as follows:
Example 1
Regress Child Mortality (CM) against PGNP (per capita GNP) and FLR (Female
Literacy Rate). Interpret the results
Example 2
Run regression on Log of Y against Log of X1 and X2. Interpret the results.
Regression on Standardized
Variables