1 Regression and Correlation Analysis

1.1 Introduction
In the previous units, two aspects of statistics were introduced, namely, descriptive
statistics and inferential statistics. This unit introduces another aspect of statistics,
statistical modelling that involves the investigation of a linear relationship between
two numerical (quantitative) random variables. For example:

• A businessperson may wish to know whether the volume of sales is related to the
amount of advertising the company did in a month.

• Educators are interested in determining whether the amount of hours a student

spent studying is related to the exam marks.

• Medical researchers are interested is questions such as:

– Is caffeine related to heart damage?

– Does a relationship exists between a person’s age and their blood pressure?

These questions and more can be answered using regression and correlation analysis
methods.

1.2 Scatterplots
The first step to identify relationships and correlations between two variables is through
the use of basic graphical depictions of the data. The most useful in this case is the scatter
plot.

Suppose two continuous random variables, X and Y , are considered. Observations of

these random variables are made in pairs. This means that for item 1 the observed value
of the random variable X is x1 and the observed value of the same item for the random
variable Y is y1 , this is recorded as the bivariate data point (x1 , y1 ). That is, the first
observation is now recorded as a pair. This is continued for all n observations in the
sample. The observation i in the sample is denoted (xi , yi ) for each i = 1, . . . , n.

The observations are depicted in a scatter plot as a set of bivariate pairs with the xi
observations indicating the position of point i on the x (horizontal) axis and the yi
observations indicating the position of point i on the y (vertical) axis. Consider the
following example regarding the screen brightness and battery duration for a sample of 6
smart phones.

1
 Phone Brightness (%) Battery Life (hrs)

Phone 1 32 12.5
Phone 2 54 8.7
Phone 3 100 5.4
Phone 4 67 8.0
Phone 5 28 13.3
Phone 6 82 6.2

The table above indicates that the first phone has a observation of (x1 , y1 ) = (32, 12.5),
where X is the screen brightness (in %) and Y is the battery life (in hours). To determine
whether there is some relationship between these two variables these are plotted on a
scatter plot.

12
Battery_Life

40 60 80 100
Brightness

It is clear from the plot that there is some sort of relationship between the screen brightness
and the battery life. In fact, as screen brightness increases, the battery life seems to
decrease. To confirm this suppose 50 phones are tested at various brightness settings. The
results are plotted in the following figure.

2
 Battery Life (hrs) 16

12

25 50 75 100
Brightness (%)

Clearly there is a relationship that exists here. For the purposes of this module we will
focus on linear relationships between two variables. These can be broadly classified as
positive linear relationships, negative linear relationships and no linear relationship. These
are depicted in the figure below.
Positive Linear Relationship No Linear Relationship Negative Linear Relationship

3
1.3 Linear Regression
The method that is used to determine the nature of a relationship that exists between
variables is called the regression analysis, and for the purposes of this course, we shall
use the method of least squares. When there are two variables only, this method is
refereed to as simple regression analysis. When the relationship between the dependent
and independent variables is linear, this method is called simple linear regression.

1.3.1 Simple Linear Regression

The objective of simple linear regression analysis (which we shall call regression analysis)
is to determine the nature of relationship between the values of X and Y with a model of
the form: ŷ = a + bx. Once again, the process of finding this equation is called statistical
modelling (which is the third component of statistics).

Formulae for finding the regression line: ŷ = a + bx

For the following population simple linear regression model: Y = β0 + β1 X1 + e

SXX = x2 − n1 ( x)2
P P

SY Y = y 2 − n1 ( y)2
P P

SXY = xy − n1 ( x) ( y)
P P P

SXY
b= SXX
a = ȳ − bx̄
where
a is the y intercept and b is the slope of the line.

[Link] Example 1 Consider the screen brightness and battery duration data defined
earlier. Find the equation of the regression line for the data, and graph the line on the
scatter plot.

Solution
P P P
The values needed for the equation are: n = 6, x = 363, y = 54.1, xy = 2826.6
and x2 = 25937, x̄ = 60.5, ȳ = 9.0167. Thus,
P

1 X 2 1
x2 − x = 25937 − (363)2 = 3975.5
X
SXX =
n n
X 1 X   X  1
SXY = xy − x y = 2826.6 − (363) (54.1) = −446.45
n n
SXY −446.45
b= = = −0.1123
SXX 3975.5
a = ȳ − bx̄ = 9.01667 − (−0.1123)(60.5) = 15.8108

4
Thus, the equation of the regression line, ŷ = a + bx, is

ŷ = 15.8108 − 0.1123x

For completeness, below is a graph of the original data in blue, with the regression line
drawn in black. As you can see, this is a graph which best represents represents our data.
Battery Life

12
8
4

30 40 50 60 70 80 90 100

Brightness

1.3.2 The slope of the regression line

The slope of the regression line, denoted by b above, depending on it’s sign, can tell a
story on it’s own:

• If it is positive, then there is a linear positive relationship between the variables.

• If it is negative, then there is a linear negative relationship between the variables.

It should be noted that for our example above, b = −0.1123, indicating a negative linear
relation between the variables. Indeed a quick look at the graph, one can see the negative
relationship.

1.3.3 Predictions using the Simple Linear Regression Model

Predictions using the simple linear regression model are easily made by substituting the
predictor value x into the estimated model ŷ. That is, the predicted value of the response
variable Y for a given value of the predictor variable X = x∗ is given as

ŷ ∗ = a + bx∗ .

5
A condition required for prediction using the fitted model is that the value x must fall
within the range of the data which was used to fit the model, that is

min{x1 , . . . , xn } ≤ x∗ ≤ max{x1 , . . . , xn }.

Using the model to make predictions for values of x∗ which are outside of this boundary
is known as extrapolation. Extrapolation should always be avoided, since while we can be
confident about our model within the range of the values of the predictor variable which
was used to fit the model, we can not be sure what the model will look like outside of this
range.

[Link] Example 2 Using the equation of the regression line found in the previous
example, predict the battery life for brightness of 45%.

Solution

Substituting for x in the regression line ŷ = 15.8108 − 0.1123x, yields,

ŷ = 15.8108 − 0.1123x = 15.8108 − 0.1123(45) = 10.757

Therefore, a person who uses brightness of 45% in their phone can expect battery life of
10.757 hours.

6
1.4 Correlation Analysis
While the relationships are visible in scatter plots in order to assess and provide an
indication of the nature and strength of these linear relationships, the correlation is used.
The correlation between two variables is typically measured using Pearson’s correlation
coefficient. The population correlation coefficient is denoted ρ and is estimated using a
collected sample (xi , yi ) i = 1, . . . , n as

SXY
r=√ .
SXX SY Y

Values for the correlation coefficient ρ, and indeed the sample estimate r are indicative
of the nature, direction and strength of the relationship between the variables being
compared. These are broadly shown in the figure below.

r=1 r = 0.81 r = 0.39

r=0 r = −0.71 r = −1

From the above, there are a few observations to be made regarding the sign of r (or ρ):

• when r (and similarly ρ) is positive there is a linear relationship between X and Y

in which as X increases, Y increases (and vice versa).
• when r (and similarly ρ) is negative there is a linear relationship between X and Y
in which as X increases, Y decreases (and vice versa).

7
 • when r (and similarly ρ) is zero there is a no linear relationship between X and Y .

From the specific values of r (or ρ) we have

• −1 ≤ r ≤ 1
• when r approaches either -1 or 1, the linear relationship between X and Y is strong.
The closer r is to these extremes the stronger (and more evident) is the linear
relationship between X and Y .
• when r approaches 0, the weaker the relationship is between X and Y and the
relationship (if it exists) is more difficult to determine.

A simple suggestion for the interpretation of the absolute value of a calculated correlation
coefficient (i.e. after the direction of the association is determined) is

|r| Interpretation

(0.00, 0.35) Weak correlation

[0.35, 0.70) Moderate correlation
[0.70, 1) Strong correlation

1.4.1 Testing the Correlation Coefficient

To determine whether there is sufficient evidence in a sample to suggest that there is a

significant correlation between two random variables X and Y we would wish to test the
null hypothesis that there is no correlation (or no linear relationship) H0 : ρ = 0, where ρ
is the true, population correlation coefficient, against one of the alternative hypotheses:

H1 : ρ > 0 (Positive Linear Correlation)

H1 : ρ < 0 (Negative Linear Correlation)
H1 : ρ 6= 0 (Either Positive or Negative Linear Correlation)

As you should be quite comfortable with hypothesis testing now - you should know that
we need a test statistic that follows a specific distribution. It can be shown that the
statistic √
r n−2
tr = √
1 − r2
has a t-distribution with n − 2 degrees of freedom. Knowing this it is straight forward to
perform the above tests, by allocating the significance level α into the appropriate tail
and determining the critical regions.

[Link] Strength of Association The coefficient of determination (denoted by

r2 ) is a key output of regression analysis. It is interpreted as the proportion of the variance

8
in Y that is predictable from the X. The coefficient of determination is the square of the
correlation (r) and thus it ranges from 0 to 1, and is always expresses as a percentage. r2
is measure of the total variation in y that can be explained by the variation in x.

The formula is given as

(ŷi − ȳ)2
Pn
SSreg
r = Pni=1
2
2 =
i=1 (yi − ŷi )
SSy

[Link] Testing the Regression Coefficient

To examine the significance of the linear relationship between X and Y , a significance
test on the population regression coefficient β1 can be conducted as,

State the hypotheses

The hypotheses
Right-tailed test Left-tailed test Two-tailed test
H0 : β1 = 0 H0 : β1 = 0 H0 : β1 = 0
H1 : β1 > 0 H1 : β1 < 0 H1 : β1 6= 0
This is equivalent to testing for the correlation coefficient in simple linear regression.

Test Statistic The t statistic is given by:

b
t=
SE(b)

where SE(b) is the standard for the slope and given by

se
SE(b) = qP
n
i=1 (xi − x̄)2
rP
n
i=1 i
(y −ŷ)
and se = n−2
is the Root Mean Square Error - the average magnitude of difference
between predicted and actual values in a model.

Decision Rule Reject H0 if p − value < α.

[Link] Example The managers of a local resort want to investigate whether there is
a relationship between marketing expenditure and visitors to the resort. The monthly
marketing expenditure (in R 000’s) and number of visitors to the resort are collected for
a random sample of 10 months. The data are given in the following table:

9
 Marketing Expenditure Visitors
19.2 384
12.5 211
16.4 200
32.1 654
22.0 521
10.5 184
9.6 172
11.5 212
24.6 598
20.1 204
1. Determine the equation of the estimated linear regression line that will describe the
relationship between marketing expenditure and visitors.
2. Calculate and interpret the value of the correlation coefficient.
3. How many visitors would the resort expect if they spend R 18 000 on marketing?
4. Calculate and interpret that coefficient of determination.
5. Test at a 5% level of significance whether a significant correlation exists between
the marketing expenditure and visitors.

Solution

10
 1. The model equation is given by

V isitors = −85.907 + (0.024 × Expense)

2. The correlation coefficient is 0.893. This means that, an increase in marketing

expenditure, also increases the number of visitors.

3. V isitors = −85.907 + (0.024 × 18000) = 346.093. If R18 000 is spent on marketing

activities, the resort should expect 346 visitors.

4. r2 = 0.798. This shows that, 79.8% in the variability of visitors is explained by the
marketing spend.

5. Under the Coefficient table, Marketing Expenditure statistically contributes to the

number of expected visitors, shown by a p − value(Sig.) < 0.001 < 0.005.

1.4.2 Multiple Regression

Involves a single dependent variable and 2 or more independent variables.

THe general form of the multiple regression model is as follows:

Y = β0 + β1 X1 + β2 X2 + · · · + βk Xk + e

which is estimate by

ŷ = a + b1 X1 + b2 X2 + · · · + bk Xk

The least squares criterion estimates the parameters in such a way as to minimise the
total error,
n
(yi − ŷi )2
X
SSres =
i=1
.

The interpretation of the multiple regression is that, a regression coefficient, b1 , represents

the expected change in Y when X1 is changed by a single unit and X2 to Xk are held
constant. It can also be observed that the combined effects of X1 , . . . , Xk on Y are
additive. That is, if X1 , . . . , Xk are changed simultaneously by a single unit, the expected
change in Y would be kj=1 bj = b1 + · · · + bk .
P

Regression coefficients cannot be obtained if

11
 1. the sample size, n, is less than or equal to the number of independent variables, k,
or

2. a single independent variabe is perfectly correlated with another independent vari-

able.

[Link] Strength of association The strength of the relationship stipulated by the

regression equation can be determined by using the coefficient of multiple determi-
nation, R2 :

Pn 2
2 i=1 (ŷi − ȳ) SSreg
M ultiple R = Pn 2 =
i=1 (yi − ŷi )
SSy

Therefore, R2 is a measure of how much total variability in the dependent variable can be
explained by the independent variables.

R2 cannot decrease as more independent variables are added to the regression equation.
However, after the first few variables, the additional independent variables do not make
much of a contribution. For this reason, R2 is adjusted for the number of independent
variables and the sample size by using

k (1 − R2 )
Adjusted R2 = R2 −
n − (k + 1)

[Link] Significance Testing

Involves testing the significance of the overall regression equation as well as specific
regression coefficients.

To test for the overall model utility, we use the F -test, specified as follows

State the hypotheses

H0 : β1 = β2 = · · · = βk = 0

H1 : At least one βi differs, for i = 1, . . . , k

Test Statistic The overall test can be conducted by the using an F statistic:

R2 /k
F =
(1 − R2 ) /(n − (k + 1))

Decision Rule Reject H0 if p − value < α.

If the overall null hypothesis is rejected, one or more of the population regression coefficients
(βi ) are nonzero.

12
To determine which specific coefficients (βi ) are non-zero, test the significance of the
individual regression coefficients using t tests.

State the hypotheses

The hypotheses
Right-tailed test Left-tailed test Two-tailed test
H0 : βi = 0 H0 : βi = 0 H0 : βi = 0
H1 : βi > 0 H1 : βi < 0 H1 : βi 6= 0
for i = 1, . . . , k

Test Statistic The t statistic is given by:

bi
t=
SE(bi )

where SE(bi ) is the standard error of the estimate of the βi coefficient.

for i = 1, . . . , k

Decision Rule Reject H0 if p − value < α.

1.5 Example
Imagine you are the Marketing Manager for a local coffee chain. Over the last 10 weeks, you
have been experimenting with two specific growth levers: Instagram Ad Spend (how much
you pay for “Swipe Up” ads) and Local Event Sponsorships (the number of community
events you support, like 5km runs or school fairs). You want to see if these two activities
are actually driving Weekly Revenue.
Instagram Ads Event Sponsorships Weekly Revenue
200 1 12500
350 2 18200
150 1 10800
500 3 25100
420 2 21000
280 2 16500
600 4 30200
100 0 7500
480 3 24500
300 1 15800

13
 1. Determine the estimate model that describes the relationship between Weekly
Revenue and Instagram Ads and Event Sponsorships.

2. How strong is the association between the Weekly Revenue and the growth levers.

3. Is the entire model that describes the relationship between Weekly Revenue and the
growth levers useful? Test at 5% significance level.

4. Which of the 2 growth levers contributes significantly to the Weekly Revenue? Test
at 5% significance level.

5. Determine the estimate model that describes the relationship between Weekly
Revenue and Instagram Ads and Event Sponsorships.

Solution

A very high R2 of 99% indicates that the growth levers: Instagram Ads and Event
Sponsorships both explain a large amount of variability in the weekly revenue.

Since the p − value (Sig.) < 0.05 of the F (2, 7) = 3480.837 statistic under the ANOVA
table. This suggests that both Ads and Events have a statistically significant contribution
towards weekly revenue.

14
At the 5% significance level, both growth levers statistically significantly contribute
towards weekly revenue. This is indicated by the Sig. column of both variables. That
is, the p − value of each growth lever is less than 0.05. Where the t-statistic for Ads is
19.013 and for Events is 6.651.

The model equation is given by

Revenue = 4294.478 + (32.495 × Ads) + (1543.316 × Events)

Initially, without any money spent on any of the growth levers, the sales are worth
R4294.48. For every R100 spent on Instagram Ads, the business gets back R3249.50 back
in sales. Whereas, for each event sponsored by the business, it gets back R1543.32 in
weekly revenue.

15