REGRESSION AND CORRELATION ANALYSIS

6.1 Introduction
Correlation and regression are the two most commonly used techniques for investigating the
relationship between quantitative variables. Here regression refers to linear regression.
Correlation is used to give the relationship between the variables whereas linear regression uses
an equation to express this relationship. Correlation and regression are used to define some form
of association between quantitative variables that are assumed to have a linear relationship.

Definition of Correlation: Correlation can be defined as a measurement that is used to quantify

the relationship between variables. If an increase (or decrease) in one variable causes a
corresponding increase (or decrease) in another then the two variables are said to be directly
correlated. Similarly, if an increase in one causes a decrease in another or vice versa, then the
variables are said to be indirectly correlated. If a change in an independent variable does not
cause a change in the dependent variable then they are uncorrelated. Thus, correlation can be
positive (direct correlation), negative (indirect correlation), or zero. This relationship is given by
the correlation coefficient.

Definition of Regression: Regression can be defined as a measurement that is used to quantify

how the change in one variable will affect another variable. Regression is used to find the cause
and effect between two variables. Linear regression is the most commonly used type of
regression because it is easier to analyze as compared to the rest. Linear regression is used to find
the line that is the best fit to establish a relationship between variables. The change is given by
the regression coefficient.

6.2 Correlation Analysis

Correlation analysis is done so as to determine whether there is a relationship between the
variables that are being tested. Furthermore, a correlation coefficient such as Pearson's
correlation coefficient is used to give a signed numeric value that depicts the strength as well as
the direction of the correlation. The scatter plot gives the correlation between two variables x and
y for individual data points as shown below.
A correlation coefficient is a number between -1 and 1 that tells you the strength and direction
of a relationship between variables.

Correlation Coefficient Correlation Strength Correlation Type

-0.7 to -1 Very Strong Negative
-0.5 to -0.7 Strong Negative
-0.3 to -0.5 Moderate Negative
0 to -3 Weak Negative
0 None Zero
0 to 0.3 Weak Positive
0.3 to 0.5 Moderate Positive
0.5 to 0.7 Strong Positive
0.7 to 1 Very Strong Positive

If all points are perfectly on this line, you have a perfect correlation.
If all points are close to this line, the absolute value of your correlation coefficient is high.

If these points are spread far from this line, the absolute value of your correlation coefficient
is low.

The formula for the Pearson’s r is given as:

Where:

 = strength of the correlation between variables x and y

 = sample size
 = sum of what follows…
 = every x-variable value
 = every y-variable value
 = the product of each x-variable score and the corresponding y-variable score
Illustration

X Y XY
1 3 1 9 3
3 6 9 36 18
5 9 25 81 45
8 11 64 121 88
9 5 81 25 45
SUM = 26 Sum = 34 Sum = 180 Sum = 272 Sum = 199

6.3 Regression Analysis

Regression analysis is the art and science of fitting straight lines to patterns of data. In a linear
regression model, the variable of interest (the so-called ―dependent‖ variable) is predicted from k
other variables (the so-called ―independent‖ variables) using a linear equation. If Y denotes the
dependent variable, and X1, …, Xk, are the independent variables, then the assumption is that
the value of Y at time t (or row t) in the data sample is determined by the linear equation.

β0 is the so-called intercept of the model—the expected value of Y when all the X’s are zero—
and βi is the coefficient (multiplier) of the variable Xi. The corresponding equation for predicting
Yt from the corresponding values of the X’s is therefore:

Where the b’s are estimates of the betas obtained by Ordinary Least Squares (OLS) method i.e.,
minimizing the squared prediction error within the sample. This is about the simplest possible
model for predicting one variable from a group of others.

Assumptions of OLS

1. The regression model has linearity in its error term and coefficients
The first OLS regression assumption refers to the estimator's linear regression model. It's the
only assumption that refers to both the OLS estimator and the linear regression model, while all
the other assumptions refer to only the OLS estimator. It's linear because all terms follow one of
two conditions. The first condition is that the terms are constant numbers. The second condition
is that the terms are the multiplication of an independent variable and a parameter. The formula
looks like this:

Linear regression formula: Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε

The parameters that the OLS estimator predicts are the beta (β) symbols. The random error that
the linear regression model produces is the epsilon (ε) symbol. The Xs are the independent
variables that the statistician can alter.
2. The error term’s population mean is zero
The error term is the number that considers any variation in the "Y," or the dependent variable,
that the independent variables fail to show. In ideal circumstances, random chance determines
the error term's value.

3. There are no correlations between the independent variables and error term
Another important OLS regression assumption is that there are no correlations between the
independent variables and the error term. If there are correlations, it's possible to predict the error
term by using the independent variable. This would mean that the error term represents
predictable random error, which violates the second assumption in this list. Some statisticians
refer to this assumption as exogeneity. Several factors may cause the opposite of exogeneity,
which is called endogeneity, to occur.

4. Each observation of the error term is independent of others

When you implement the OLS estimator in a linear regression model, you can make independent
observations of the error term. A statistician may have to implement corrective measures if they
notice correlations between different observations of the error term. For example, consider a
linear regression model that has a positive error for one observation. If a statistician can assume
that the subsequent error is also positive, this implies a positive correlation.

Similarly, if a linear regression model has a positive error for one observation and an individual
can assume a negative error for the next observation, there's a negative correlation present. If
these circumstances are present, the linear regression model fails to meet this fourth OLS
regression assumption.

5. The error term’s variance is constant

Some statisticians refer to this OLS assumption as homoscedasticity. It states that the error term
of the OLS estimator has constant variance. This means that the variance remains the same
across a single observation or a range of observations. You can confirm that this assumption is
true by plotting the true values versus the residuals. If you find that the spread of the residuals
continues to get larger in one direction, the model fails to meet the assumption of
homoscedasticity.

6. There are no independent variables that are perfect linear functions of other variables
(Multicollinearity)
If two variables have a coefficient of negative one or positive one, perfect correlation is present.
Some models can account for perfect correlation, but OLS regression cannot. A linear regression
model with an OLS estimator that has two independent variables with perfect correlation cannot
display properly. It becomes necessary to remove one of the independent variables to
successfully continue with the graphing process.

7. The error term adheres to a normal distribution pattern

This is the only OLS regression assumption that's optional. Ideally, the error term adheres to a
normal distribution pattern. This allows statisticians to produce reliable prediction intervals,
generate accurate confidence intervals and conduct informative hypothesis testing.
6.4 Simple Linear Regression Analysis
We consider the modeling between the dependent and one independent variable. When there is
only one independent variable in the linear regression model, the model is generally termed as a
simple linear regression model. Consider a simple linear regression model given as:

Where y is termed as the dependent or study variable and X is termed as the independent or
explanatory variable. The terms 0 and 1 are the parameters of the model. The parameter 0 is
termed as an intercept term, and the parameter 1 is termed as the slope parameter. These
parameters are usually called regression coefficients. The unobservable error component 
accounts for the failure of data to lie on a straight line and represents the difference between the
true and observed realization of y. There can be several reasons for such difference, e.g., the
effect of all deleted variables in the model, variables may be qualitative, inherent randomness in
the observations etc. We assume that  is observed as an independent and identically distributed
random variable with mean zero and constant variance  . We will additionally assume that  is
normally distributed.

The ordinary least squares estimates are calculated as:

̅ ̅

Where:

Predicted values are values of y predicted by the least squares regression line obtained by
plugging in x1,x2,…,xn into the estimated regression line: Determined as:

Residuals are the deviations of observed and predicted values:

Illustration

X Y
1 3
3 6
5 9
8 11
9 5

Calculate the regression coefficients and interpret.

6.5 Multiple Regression Model

When there is more than one independent variable in the model, then the linear model is termed
as the multiple linear regression model. It is an extension of the simple linear regression model to
two or more independent variables.

Can be given as:

5.6 Coefficient of Determination and Standard Error of Estimate and Interval Estimation
in Regression
1. Standard Error of Estimate (ESS): Measures how well a given linear regression model
captures the relationship between the dependent and independent variables

The lower the SEE, the better the fit of the regression line. SSE is the Sum of the Squared Error
terms
2. Regression Inference and Confidence Interval for a Regression Coefficient
To estimate the parameters of the regression, we calculate the standard errors for the estimated
regression coefficients.
The standard error of the least-squares slope β1 is:
The standard error of the intercept β0 is

Degrees of freedom = n-k-1, where n is the sample size, k is the number of slope coefficients
Example
Given the regression results as:

Calculate the confidence interval for age coefficient for a sample size of 16.
Solution
Degrees of freedom = n-k-1 = 16-2-1 = 13

At 13 degrees of freedom and 95% confidence interval, the tc will be 2.160.

Upper Limit = -1027.8+410.44 = -617.36

Lower Limit = -1027.8-410.44 = -1438.24

Confidence Interval = -1438.24-------617.36

Confidence Interval for a Sample

Given that the sample mean is 996.21, standard deviation is 5.65 and the sample size is 14.
Calculate the confidence interval for a sample at 95% confidence interval.
̅
√

√
= 1.51

Degrees of freedom = n-1 = 14 – 1 = 13

From the table at 13 degrees of freedom and 95% confidence interval, the tc will be 2.160.
̅

Upper Limit = 996.21+3.26 = 999.49

Lower Limit = 996.21 – 3.26 = 992.95
Confidence Interval = 992.95 ------999.49

3. Coefficient of Determination : Measures the fraction of the total variation in the

dependent variable that is explained by the independent variable. It is the percentage of the
variance in y (vertical scatter from the regression line) that can be explained by changes in x.
Total Variation = Unexplained Variation + Explained Variation
It can also be determined by squaring the value of the coefficient of correlation.
Example
Given the values of X and Y as:
X 1 2 3 4 5 6
Y 2 4 7 9 12 14
Calculate the coefficient of determination.
Solution

r = 0.998
Coefficient of Determination = = 0.998*0.998 = 0.9960