From: Flatt, C. and Jacobs, R.L., 2019.

Principle assumptions of regression

analysis: Testing, techniques, and statistical reporting of imperfect data sets.
Advances in Developing Human Resources, 21(4), pp.484-502. Tests for
Statistical Independence of Error Terms. Type Methods Description Graphical
Scatterplot of the residuals against any time or spatial variables Time and
spatial variables are common sources of dependence. If the plots on the
graph are not random, then the assumption of independence is likely
violated. Numerical Durbin–Watson coefficient The Durbin–Watson coefficient
is a measurement of the residual differences over time. Its value is between
0 and 4. Values less than 1 imply that successive error terms are positively
related. Conversely, a Durbin–Watson coefficient greater than 3 implies
successive error terms are negatively related. The test is available in SAS,
Stata, SPSS, and Python. Assumption 1: Error Terms are Statistically
Independent The independence assumption requires observations to be
independent of each other (Nimon, 2012). Violation of this assumption often
occurs in time series regression models. In essence, serial correlation
(autocorrelation) is the result of time series data that are influenced by past
values. A violation of statistical independence indicates that the model could
be improved. In extreme cases, this violation signals that the model is mis-
specified. In non–time series models, a violation of statistical independence
can be present if the model systematically underpredicts or overpredicts the
coefficient estimates (Nau, 2018). Table 1 provides the graphical and
numerical types of tests for statistical independence of error terms. The
graphical analysis consists of a scatterplot of the residuals against time (or a
variable characterized by patterned spatial points). The scatterplot of
residuals versus time should take on a rectangular shape as an indication of
randomness. The Durbin–Watson statistic is a test for significant residual
autocorrelation at Lag 1. Ideally, the statistic should be close to 2. mall
adjustments to the model. A small adjustment could be the addition of
lagged independent or dependent variables. Some statistical software
packages provide an ARIMA+ regressor procedure which provides the option
of adding lagged variable and error terms. A Durbin–Watson value between
1.2 and 1.6 is an example of a minor violation. A major violation of statistical
independence of error terms (Durbin–Watson < 1) Type Methods Description
Graphical Plot residuals versus predicted values A scatterplot of the residuals
versus predicted values provides insight on homoscedasticity at a glance.
The residuals (and the variance of the residuals) should be the same for all
predicted values (Tabachnick & Fidell, 2007). Numerical White test The White
test is a test for heteroscedasticity, model mis-specifications, or both. The
White test is a special case of the Breusch–Pagan test by relaxing the
assumption that errors are normally distributed. The test is available in SAS,
Stata, and Python Assumption 2: Error Terms Have Constant Variance
(Homoscedasticity) Nonconstant variances (heteroscedasticity) can originate
from violations of the other assumptions. Given all other assumptions are
met, coefficients from heteroscedastic regression results are not BLUE (best
linear unbiased estimators) under OLS. When heteroscedasticity is present,
OLS gives equal weight to all observations regardless of the magnitude of
the variance. The standard errors in the presence of heteroscedasticity are
biased, which leads to biased test statistic and confidence intervals
(Williams, 2015). Slight violations of the homoscedasticity assumption are
generally acceptable, but violations to the assumption can result in an
increased risk of Type I error (Osborne & Waters, 2002). Table 2 provides
common graphical and numerical tests for the constant variance of error
terms. The points on the scatterplot of residuals versus predicted values
should take on a rectangular shape, with the most points concentrated
around zero. The mean residual (zero) is constant for all values of the
predicted value. The White test is available in SAS, Stata, and Python. The
null hypothesis is that the error terms have a constant variance. Although
the test is not readily available in SPSS, IBM (2016) provides instructions on
SPSS code to produce the White test. Although severe heteroscedasticity is
problematic, the parameter estimates are not biased. In other words, OLS
estimation can still be used without the risk of distortion. When
heteroscedasticity is a concern, there are two ways to address the problem.
First, heteroscedasticity is often due to a violation of another assumption.
There are several tests available to determine if a model is mis-specified (see
Assumption 4). A model re-specification or transformation of variables may
eliminate heteroscedasticity. Second, robust standard errors can be used to
obtain unbiased standard errors. Robust standard errors relax the
assumption that errors are independent and identically distributed (Williams,
2015). Assumption 3: Error Terms are Normally Distributed Park (2008)
describes normality testing and provides the associated codes in SAS, Stata,
and SPSS. A common misconception is that the variables should be normally
distributed (Osborne & Waters, 2002); however, the correct assumption is
that the error terms are normally distributed. This particular assumption
needs to be met for the p-values of the t-tests to be valid (Chen, Ender,
Mitchell, & Wells, 2003a; 2003b; 2003c). According to Nau (2018), a violation
of normality can distort confidence intervals for forecasts and cause
difficulties in determining the significance of model coefficients. A violation of
normally distributed error terms can signal the existence of unusual data
points or that the model can be improved. Table 3 provides common tests for
normality. Smith (2012) cautions against using stem-and-leaf plots and
histograms. The P-P and Q-Q plots of the residuals are the preferred
graphical tests for normality. These plots provide a visual comparison
between the error distribution and a normal distribution with the same mean
and variance. A bow-shaped pattern indicates skewness, and an S-shaped
pattern indicates kurtosis. Normality tests vary by statistical package. Nau
(2018) states that the Anderson–Darling test is considered to be superior
because it takes into consideration the entire normal distribution rather than
only skewness and kurtosis. All normal tests are considered overly “picky” (p.
6). In reality, error terms seldom exhibit a perfectly normal distribution.
Violations of normality can stem from a violation of linearity (Assumption 4).
A transformation of variables may fix the problem. Outliers are another
possible source of this particular violation. Finally, the dependent and
independent variables may produce error terms that are not normally
distributed (Nau, 2018). Assumption 4: The Relationship Between Dependent
and Independent Variables are Linear and Additive The linearity assumption
requires a straight-line relationship between two variables (Nimon, 2012).
Nonlinear or nonadditive data fitted to a linear model results in Type Methods
Description Graphical Stem-and-leaf plot A stem-and-leaf plot presents the
shape of the data by displaying the numbers (i.e., error terms) in two
columns. The left column (stem) contains the largest place value with a
different value in each row. The right column (leaf) contains the remaining
portion of each number. This graph is useful for small sample sizes.
Histogram A histogram displays the observations (i.e., error terms)
proportionally by intervals or categories. According to Smith (2012), a
histogram is not a good way to check for normality because different
intervals influence the shape of the graph. Probability– Probability plot (P-P
plot) The P-P plot is a comparison of an empirical distribution function with a
theoretical cumulative distribution function (i.e., normal distribution
function). Quantile–Quantile plot (Q-Q plot) The Q-Q plot is a quantile
comparison of a probability distribution with a specific theoretical distribution
(i.e., normal distribution). Numerical Skewness The right tail is longer for a
positively skewed probability distribution. The left tail is longer for a
negatively skewed probability distribution. Skewness is a descriptive statistic
that captures the direction and magnitude a probability distribution deviates
from normality. Kurtosis Kurtosis measures the peakedness of the
distribution. A normal distribution has a kurtosis of 3. A distribution with
kurtosis greater than 3 is leptokurtic. A distribution with kurtosis less than 3
is platykurtic. Normality tests The Shapiro–Wilk test can be performed for
sample sizes ≥ 7 and ≤ 2,000. It is the ratio of the best estimator for the
sum of squares estimator of the variance. The value of 1 indicates normality.
The test is available in SAS, Stata, and SPSS. The Shapiro–Francis test is a
modification of the Shapiro–Wilk test. It can be used for sample sizes ≥ 5 and
≤ 5,000. The test is available in Stata. The Kolmogorov–Smirnov test
compares the empirical distribution function of a sample (i.e., error terms)
with a cumulative distribution function of a specific distribution (i.e., normal
distribution). The test is available in SAS, SPSS, and Python. This test is
useful when the sample size is greater than 2,000. The Anderson–Darling
test is considered one of the most powerful tools for determining if the error
term is normally distributed. Compared with the Cramer–von Mises test, the
Anderson–Darling test places greater weight on the distribution tails. Both of
these tests belong to the class of quadratic empirical distribution functions
and are available in SAS. These tests are useful when the sample size is
greater than 2,000. The Jarque–Bera test determines if the skewness and
kurtosis of the error term match a normal distribution. This test is a good
alternative for large sample sizes. Other tests tend to reject the null
hypothesis (the error term is normally distributed) with large sample sizes.
The Jarque–Bera test can be performed in Stata using the .sktest command
or Python using jarque_bera. incorrect estimations or predictions. Violations
of linearity or additivity are considered extremely serious (Nau, 2018). Table
4 describes several methods that can be used to detect nonlinearity or
nonadditivity. Although tests on the relationships between variables are
insightful, the model specification should primarily be based on theory. A
violation of this assumption can be addressed through transformations or
additional variables. When independent variables are nonadditive, the
inclusion of an interaction term can solve the problem. For example, if the
relationship of X on Y is dependent on gender, then including a term
X*gender is a good option. A nonlinear relationship can be addressed by
transforming an existing variable into a new variable that is linearly related
to the dependent variable. Common transformations include reciprocals,
logarithms, cube root, square root, and squares (Williams, 2015). Type
Methods Description Graphical Scatterplots with smoother curves (such as
lowess) Scatterplots of the dependent versus independent variables provide
an indication of the type of relationships and the potential problems that
might be encountered in the regression analysis. Plot standardized residuals
versus predicted values A scatterplot of the residuals versus predicted values
provides insight on linearity at a glance. In a linear relationship, the line
constructed from joining residual means at each predicted value will be a
horizontal line through zero. If the mean residual varies depending on the
predicted value, then the relationship between dependent and independent
variables is nonlinear. Numerical Correlations between dependent and
independent variables The Pearson correlation measures the linear
relationship between two variables. A value close to zero indicates either no
relationship or a nonlinear relationship. The variables must be measured in
interval scales. Other correlation measurements include Spearman’s
correlation that can be used for ordinal variables and Hoeffding’s D
correlation that can measure monotonic (one-directional curve),
nonmonotonic (curve with hills and valleys), and linear relationships (Bhalla,
2015). Specification error tests The incremental F test is not the easiest
approach, but it is available in most statistical software. This test estimates
both a full model (including the nonlinear terms) and a constrained model
(excluding the nonlinear terms). The R squares of the two models are used to
calculate the incremental F statistic (Williams, 2015). The Wald test uses
estimated coefficients and variances/ covariances to determine specification
errors. For a linearity test, the coefficients of nonlinear terms in the model
would be equal to zero in the null hypothesis. If the null hypothesis cannot be
rejected, then the variables can be removed from the model. The Wald test is
an easier alternative to the incremental F test, which only requires one
estimation model through the test command in Stata (Williams, 2015). The
Ramsey RESET test can be used to determine if a nonlinear combination of
terms can help explain a model. The test uses an original model with no
nonlinear terms and an expanded model with nonlinear terms. The null
hypothesis is that the coefficients of the nonlinear terms are equal to zero. If
the null hypothesis is rejected, the RESET test offers no further guidance. It is
purely a functional form test (Wooldridge, 2013). The Application of the
Regression Development System to an Imperfect Data Set With the
availability of free statistical software, such as R or Python, practitioners from
any size organization can run regression analyses. Perktold, Seabold, and
Taylor (2018) provide an overview of the regression diagnostic tests and
codes available in Python. Python developers freely share code and advice to
novice users. An example of a study with an imperfect data set found in all
organizations is capturing the change in job performance resulting from HRD
interventions (e.g., S-OJT training, coaching, or apprenticeships). The
measurement of the same individual’s job performance before and after an
HRD intervention indicates that the data most likely violate Assumptions 1
(independence of error terms) and 2 (homogeneity of error terms). In
essence, performance is influenced by past performance. In addition, the
dependent variable representing job performance could be the probability
that the individual improved performance. The probability distribution
violates Assumption 3 (normal distribution of error terms). The Regression
Development System in Figures 1 to 4 reveals information that is needed to
determine the limitations of the study under OLS and construct the most
appropriate model. The application of the Regression Development System is
exemplified in Table 5 using SAS. The coding and results of a similar model
are available in SAS, SPSS, Stata, and Python (Flatt, 2019). For simplicity, the
codes and examples are limited to dependent and independent variables. In
practice, the full model—including controls—should be evaluated. From Table
5, the error terms are statistically independent. A graph of the residual over
time shows a random, rectangular shape. The Durbin–Watson D value of
1.824 confirms that the error terms are statistically independent. Second, the
error terms do not have a constant variance. The heteroscedasticity can be
seen through the negative shape of the residual versus predicted value
graph. In comparison, the graph of a homoscedastic model would show a
random, rectangular shape. The White test confirms the result. The null
hypothesis—the error terms have a constant variance—is rejected at the
alpha level of .05. Third, the Q-Q plot shows an S-shape, which indicates
kurtosis. The normality tests in SAS confirm that the error terms are not
normally distributed. The null hypothesis—the error terms are normally
distributed—is rejected at the alpha level of .05. Finally, the tests for linear
and additive relationships find that the relationships between dependent and
independent variables are not linear. The intent of the study and the
magnitude of the violations determine the next course of action. Minor
violations of the OLS assumptions that do not interfere with the integrity of
the results are acceptable. With minor violations, the study should include an
acknowledgment of the limitations as well as a justification of the results
under the limitations. For this specific example, three out of the four
assumptions of OLS are violated. The limitations under OLS include the
following: (a) the regression results are not BLUE, (b) the significance of the
coefficients is in question, and (c) the coefficient estimates are likely
incorrect. Due to the seriousness of the violations, a different model could be
chosen rather than acquiescing to the constraints of OLS. Choosing a LMM
would address the lack of independence and homogeneity of the error terms.
Extending the model to a GLMM can accommodate models suspected to
have nonlinear relationships or residuals with nonnormal distributions—such
as binomial or gamma distributions (Anderson, Verkuilen, & Johnson, 2012).
GLMM accommodates multi-level data and is particularly useful with
longitudinal data sets. When job performance is measured before and after
an HRD intervention, each training participant is associated with multiple
measurements—creating two levels of [Link], the results of GLMM
produce a unique line for each training participant through the random
effects (random intercept and random slope). The construction of a GLMM
involves choosing a link function. The link function is chosen based on theory
and the distribution of the data. The distribution of the random component of
the dependent variable determines the type of GLMM and the link function.
Common link functions include logit, probit, logarithm, and multinomial logit
(Liao, 1994). For this particular example, a logistic link function and a
binomial distribution would be a good option to accommodate the dependent
variable (the probability of good job performance From: Dixon, P. & Woolner,
P. (2012) ‘Quantitative Data Analysis: Using SPSS’, in Marianne Coleman et
al. (eds.) Research Methods in Educational Leadership & Management.
[Online]. United Kingdom: SAGE Publications. pp. 340–362. Crosstabs and Chi
Squared Tests Where the dependent and independent variables are either
nominal or ordinal, cross-tabulation and the chi squared test can be used to
determine whether membership in one category has a bearing on
membership in another. A cross-tabulation is a table that sets out how cases
are distributed between each combination of category. The chi squared test
is based on comparing the expected with the observed frequency in each
cell. Table 23.2 is an example of cross-tabulation from Data Set I. By looking
at the table that has been produced, it would seem that there are some
differences between male and female leaders and the way they answer this
question regarding their ability as a manager. However, we wish to calculate
the significance level of any difference that may exist. The chi squared test
tells us if there is a statistically significant difference between how the
different genders answer this question. It tests the hypothesis that the row
and column variables are not related to one another and are independent. In
order to carry out the chi squared test and obtain the significance level in
SPSS the following steps need to be undertaken The tables produced by this
procedure are the cross-tabulation table and another table which has the chi
squared test results. There are two pieces of information we require here.
First of all, we need to look at the ‘Pearson Chi-Square’ row and the final
number in the third column labelled ‘Asymp. Sig. (two-sided)’ (where
[Link] is asymptotic significance). This shows the test statistic or p-value.
In statistics, it is typically accepted that a p-value of less than 0.05 reveals a
statistically significant difference and this difference has a 95 per cent
confidence level. Where the p-value is less than 0.01, this indicates a 99 per
cent confidence level and less than 0.01 99.9 per cent confidence level. In
this case, as shown in our ‘Chi Squared Tests’ table, in the output the p-value
is 0.000. We can therefore state that there is a statistically significant
difference between the way men and women answered this question.
However, the second important piece of information is that given below the
table in this case: ‘a. 0 cells (.0 per cent) have expected count less than 5.’
The cut-off point for this statement is 20 per cent in the brackets. If such a
case occurs, you will need to carry out an exact test in order to ascertain
whether the statistical significant difference is held, or combine columns in
order to eradicate smaller values. This may sometimes occur if your data set
is small. In order to report the above finding, one could use the following
sentence: ‘carrying out a chi squared test shows that there is a statistically
significant difference between the way men and women respond to the item
“considered to be a good manager”’ (chi square = 19.149, df = 3, p =
0.000).

From: Hair, J. F. et al. (2013) Multivariate data analysis. Pearson new international edition.
Harlow, England: Pearson.

coefficient of determination (R 2 ) Measure of the proportion of the variance

of the dependent variable about its mean that is explained by the
independent, or predictor, variables. The coefficient can vary between 0 and
1. If the regression model is properly applied and estimated, the researcher
can assume that the higher the value of R 2 , the greater the explanatory
power of the regression equation, and therefore the better the prediction of
the dependent variable

Adjusted coefficient of determination (adjusted R 2 ) Modified measure of the

coefficient of determination that takes into account the number of
independent variables included in the regression equation and the sample
size. Although the addition of independent variables will always cause the
coefficient of determination to rise, the adjusted coefficient of determination
may fall if the added independent variables have little explanatory power or
if the degrees of freedom become too small. This statistic is quite useful for
comparison between equations with different numbers of independent
variables, differing sample sizes, or both.

Partial F (or t) values The partial F test is simply a statistical test for the
additional contribution to prediction accuracy of a variable above that of the
variables already in the equation. When a variable 1Xa 2 is added to a
regression equation after other variables are already in the equation, its
contribution may be small even though it has a high correlation with the
dependent variable. The reason is that Xa is highly correlated with the
variables already in the equation. The partial F value is calculated for all
variables by simply pretending that each, in turn, is the last to enter the
equation. It gives the additional contribution of each variable above all
others in the equation. A low or insignificant partial F value for a variable not
in the equation indicates its low or insignificant contribution to the model as
already specified. A t value may be calculated instead of F values in all
instances, with the t value being approximately the square root of the F
value.

Variance inflation factor (ViF) Indicator of the effect that the other
independent variables have on the standard error of a regression coefficient.
The variance inflation factor is directly related to the tolerance value 1VIFi 5
1/TOLi 2. Large VIF values also indicate a high degree of collinearity or
multicollinearity among the independent variables.

tHe neW VARiAte To improve further our prediction of credit card holdings, let
us use additional data obtained from our eight families. The second
independent variable to include in the regression model is family income 1V2
2, which has the next highest correlation with the dependent variable.
Although V2 does have a fair degree of correlation with V1 already in the
equation, it is still the next best variable to enter because V3 has a much
lower correlation with the dependent variable. We simply expand our simple
regression model to include two independent variables as follows: Predicted
number of credit cards used 5 b0 1 b1V1 1 b2V2 1 e where: b0 5 constant
number of credit cards independent of family size and income b1 5 change in
credit card usage associated with unit change in family size b2 5 change in
credit card usage associated with unit change in family income V1 5 family
size V2 5 family income e 5 prediction error (residual) The multiple
regression model with two independent variables, when estimated with the
least squares procedure, provides a constant of .482 with regression
coefficients of .63 and .216 for V1 and V2 , respectively. PReDictiVe
AccURAcY We can again find our residuals by predicting Y and subtracting
the prediction from the actual value. We then square the resulting prediction
error, as in Figure 5.4. The sum of squared errors for the multiple regression
model with family size and family income is 3.04. This result can be
compared to the simple regression model value of 5.50 (Figure 5.3), which
uses only family size for prediction. When family income is added to the
regression analysis, R 2 also increases to .86. The inclusion of family income
in the regression analysis increases the prediction by 11 percent 1.86 2 .752,
all due to the unique incremental predictive power of family income.
Adding a Third Independent Variable We have seen an increase in prediction
accuracy gained in moving from the simple to multiple regression equation,
but we must also note that at some point the addition of independent
variables will become less advantageous and even in some instances
counterproductive. The addition of more independent variables is based on
trade-offs between increased predictive power versus overly complex and
even potentially misleading regression models. The survey of credit card
usage provides one more possible addition to the multiple regression
equation, the number of automobiles owned 1V3 2. If we now specify the
regression equation to include all three independent variables, we can see
some improvement in the regression equation, but not nearly of the
magnitude seen earlier. The R 2 value increases to .87, only a .01 increase
over the previous multiple regression model. Moreover, as we discuss in a
later section, the regression coefficient for V3 is not statistically significant.
Therefore, in this instance, the researcher is best served by employing the
multiple regression model with two independent variables (family size and
income) and not employing the third independent variable (number of
automobiles owned) in making predictions.

ADDinG tHe MoDeRAtoR eFFect The moderator effect is represented in

multiple regression by a term quite similar to the polynomials described
earlier to represent nonlinear effects. The moderator term is a compound
variable formed by multiplying X1 by the moderator X2 , which is entered
into the regression equation. In fact, the nonlinear term can be viewed as a
form of interaction, where the independent variable “moderates” itself, thus
the squared term 1XiXi 2. The moderated relationship is represented as: Y 5
b0 1 b1X1 1 b2X2 1 b3X1X2 where: b0 5 intercept b1X1 5 linear effect of X1
b2X2 5 linear effect of X2 b3X1X2 5 moderator effect of X2 on X1 Because of
the multicollinearity among the old and new variables, an approach similar to
testing for the significance of polynomial (nonlinear) effects is employed. To
determine whether the moderator effect is significant, the researcher follows
a three-step process: 1 Estimate the original (unmoderated) equation. 2
Estimate the moderated relationship (original equation plus moderator
variable). 3 Assess the change in R 2 : If it is statistically significant, then a
significant moderator effect is present. Only the incremental effect is
assessed, not the significance of individual variables.