ICS422 Applied Predictive Analytics [3- 0-0-3]

Linear Regression In R
Class 18
Presented by
Dr. Selvi C
Assistant
Professor
IIIT Kottayam
Multiple Linear Regression in R
• Multiple linear regression is an extension of
simple linear regression used to predict an outcome variable
(y) on the basis of multiple distinct predictor variables (x).
• With three predictor variables (x), the prediction of y is
expressed by the following equation:
y = b0 + b1*x1 + b2*x2 + b3*x3
• The “b” values are called the regression weights (or beta
coefficients). They measure the association between the
predictor variable and the outcome. “b_j” can be interpreted as
the average effect on y of a one unit increase in “x_j”, holding
all other predictors fixed.
2
Loading required R packages
• tidyverse for data manipulation and visualization
Examples of data:

3
Building model
We want to build a model for
estimating sales based on the
advertising budget invested
in youtube, facebook and
newspaper, as follow:
• sales = b0 + b1*youtube +
b2*facebook +
b3*newspaper
• You can compute the model
coefficients in R as follow:

4
Contd...
• Estimate: the intercept (b0) and the beta coefficient
estimates associated to each predictor variable
• [Link]: the standard error of the coefficient
estimates. This represents the accuracy of the
coefficients. The larger the standard error, the less
confident we are about the estimate.
• t value: the t-statistic, which is the coefficient estimate
(column 2) divided by the standard error of the estimate
(column 3)
• Pr(>|t|): The p-value corresponding to the t-statistic.
The smaller the p-value, the more significant the
estimate is. 5
Interpretation
• The first step in interpreting the multiple regression analysis is
to examine the F-statistic and the associated p-value, at the
bottom of model summary.
• In our example, it can be seen that p-value of the F-statistic is
< 2.2e-16, which is highly significant. This means that, at least,
one of the predictor variables is significantly related to the
outcome variable.
• To see which predictor variables are significant, you can
examine the coefficients table, which shows the estimate of
regression beta coefficients and the associated t-statitic p-
values:
6
Contd...

• For a given the predictor, the t-statistic evaluates whether or not there is significant association between the
predictor and the outcome variable, that is whether the beta coefficient of the predictor is significantly
different from zero.
• It can be seen that, changing in youtube and facebook advertising budget are significantly associated to
changes in sales while changes in newspaper budget is not significantly associated with sales.
• For a given predictor variable, the coefficient (b) can be interpreted as the average effect on y of a one unit
increase in predictor, holding all other predictors fixed.
• For example, for a fixed amount of youtube and newspaper advertising budget, spending an additional 1 000
dollars on facebook advertising leads to an increase in sales by approximately 0.1885*1000 = 189 sale units,
on average.
• The youtube coefficient suggests that for every 1 000 dollars increase in youtube advertising budget, holding
all other predictors constant, we can expect an increase of 0.045*1000 = 45 sales units, on average.
• We found that newspaper is not significant in the multiple regression model. This means that, for a fixed
amount of youtube and newspaper advertising budget, changes in the newspaper advertising budget will not
significantly affect sales units.

7
 As the newspaper variable is not
significant, it is possible to remove it
from the model:
• Finally, our model equation can
be written as follow: sales = 3.5
+ 0.045*youtube +
0.187*facebook.

• The confidence interval of the

8
R-Squared vs Adjusted R- Squared
• R-Squared:
• R-squared measures the proportion of the variance in the dependent variable explained by
the independent variables in the model.
• It ranges from 0 to 1, where 0 indicates that the model does not explain any variability, and
one indicates that it explains all the variability.
• Higher R-squared values suggest a better fit, but it doesn’t necessarily mean the model is a
good predictor in an absolute sense.
• Adjusted R-Squared:
• Adjusted R-squared addresses a limitation of R-squared, especially in multiple regression
(models with more than one independent variable).
• While R-squared tends to increase as more variables are added to the model (even if they
don’t improve the model significantly), Adjusted R-squared penalizes the addition of
unnecessary variables.
• It considers the number of predictors in the model and adjusts R-squared accordingly. This
adjustment helps to avoid overfitting, providing a more accurate measure of the model’s
goodness of fit.

9
Problems with R-squared statistic
• Its value never decreases no matter the number of variables
we add to our regression model. That is, even if we are
adding redundant variables to the data, the value of R-
squared does not decrease. It either remains the same or
increases with the addition of new independent variables.
This clearly does not make sense because some of the
independent variables might not be useful in determining the
target variable. Adjusted R-squared deals with this issue.

10
Comparison:
• R-squared will stay the same when adding more predictors, even if they are not contributing
meaningfully. It may give a falsely optimistic view of the model.
• Adjusted R-squared is more conservative and will decrease if additional variables do not
contribute to the model’s explanatory power.
• As a rule of thumb, a higher R-squared or Adjusted R-squared is desirable, but it’s crucial to
consider the context of the specific analysis and the trade-off between model complexity and
explanatory power

• R-squared = (TSS-RSS)/TSS
= Explained variation/ Total variation
= 1 – Unexplained variation/ Total variation

11
What is Adjusted R-squared?
• Adjusted R-squared is a statistical measure used to evaluate the goodness of fit of a
regression model. It provides insights into how well the model explains the variability
in the data.
• Unlike the standard R-squared, which simply tells you the proportion of variance
explained by the model, Adjusted R-squared takes into account the number of
predictors (independent variables) in the model.
• The advantage of Adjusted R-squared is that it penalizes the inclusion of unnecessary
variables. This means that as you add more predictors to the model, the Adjusted R-
squared value will only increase if the new variables significantly improve the model’s
performance.
• In summary, a higher Adjusted R-squared value indicates that more of the variation in
the dependent variable is explained by the model, while also considering the model’s
simplicity. It’s a valuable tool for model selection, helping you strike a balance
between explanatory power and complexity.

12
• Here,
• n represents the number of data points in our dataset
• k represents the number of independent variables, and
• R represents the R-squared values determined by the model.
• So, if R-squared does not increase significantly on the addition of a new independent
variable, then the value of Adjusted R-squared will actually decrease.

13
Model accuracy assessment
R-squared:
• In multiple linear regression, the R2 represents the correlation coefficient between the observed
values of the outcome variable (y) and the fitted (i.e., predicted) values of y. For this reason, the
value of R will always be positive and will range from zero to one.
• R2 represents the proportion of variance, in the outcome variable y, that may be predicted by
knowing the value of the x variables. An R2 value close to 1 indicates that the model explains a
large portion of the variance in the outcome variable.
• A problem with the R2, is that, it will always increase when more variables are added to the
model, even if those variables are only weakly associated with the response (James et al. 2014).
A solution is to adjust the R2 by taking into account the number of predictor variables.
• The adjustment in the “Adjusted R Square” value in the summary output is a correction for the
number of x variables included in the prediction model.
• In our example, with youtube and facebook predictor variables, the adjusted R2 = 0.89, meaning
that “89% of the variance in the measure of sales can be predicted by youtube and facebook
advertising budgets.
• Thi model is better than the simple linear model with only youtube (Chapter simple-linear-
regression), which had an adjusted R2 of 0.61.
14
Residual Standard Error (RSE)
• The RSE estimate gives a measure of error of prediction. The
lower the RSE, the more accurate the model (on the data in
hand).
• The error rate can be estimated by dividing the RSE by the
mean outcome variable:
• sigma(model)/mean(marketing$sales) ## [1] 0.12
• In our multiple regression example, the RSE is 2.023
corresponding to 12% error rate.
• Again, this is better than the simple model, with only youtube
variable, where the RSE was 3.9 (~23% error rate)

15
Any
Queries?
Thank you