PROJECT: ADVANCED

STATISTICS
Adlin Jenice Jesuraj
 Advanced Statistics

Table of Contents
Problem Statement 1A:........................................................................................................................................2
I. State the null and the alternate hypothesis for conducting one-way ANOVA for both Education and
Occupation individually..............................................................................................................................................2
II. Perform a one-way ANOVA on Salary with respect to Education. State whether the null hypothesis is
accepted or rejected based on the ANOVA results.....................................................................................................2
III. Perform a one-way ANOVA on Salary with respect to Occupation. State whether the null hypothesis is
accepted or rejected based on the ANOVA results.....................................................................................................2
IV. If the null hypothesis is rejected in either (2) or in (3), find out which class means are significantly
different. Interpret the result......................................................................................................................................3
Problem Statement 1B:........................................................................................................................................3
V. What is the interaction between two treatments? Analyse the effects of one variable on the other
(Education and Occupation) with the help of an interaction plot.[hint: use the ‘pointplot’ function from the
‘seaborn’ function].......................................................................................................................................................3
VI. Perform a two-way ANOVA based on Salary with respect to both Education and Occupation (along
with their interaction Education*Occupation). State the null and alternative hypotheses and state your results.
How will you interpret this result?.............................................................................................................................4
VII. Explain the business implications of performing ANOVA for this particular case study..........................5
Problem Statement 2:..........................................................................................................................................7
I. Perform Exploratory Data Analysis [both univariate and multivariate analysis to be performed]. What
insight do you draw from the EDA?...........................................................................................................................7
II. Is scaling necessary for PCA in this case? Give justification and perform scaling.......................................14
III. Comment on the comparison between the covariance and the correlation matrices from this data......15
IV. Check the dataset for outliers before and after scaling. What insight do you derive here?....................17
V. Perform PCA and export the data of the Principal Component scores into a data frame...........................17
VI. Extract the eigenvalues and eigenvectors.[print both]...............................................................................19
VII. Write down the explicit form of the first PC (in terms of the eigenvectors. Use values with two places of
decimals only).............................................................................................................................................................21
VIII. Consider the cumulative values of the eigenvalues. How does it help you to decide on the optimum
number of principal components? What do the eigenvectors indicate?................................................................22
IX. Explain the business implication of using the Principal Component Analysis for this case study. How
may PCs help in the further analysis? [Hint: Write Interpretations of the Principal Components Obtained]. .24
 Advanced Statistics

Problem Statement 1A:

Salary is hypothesized to depend on educational qualification and occupation. To understand the
dependency, the salaries of 40 individuals [[Link]] are collected and each person’s educational
qualification and occupation are noted. Educational qualification is at three levels, High school graduate,
Bachelor, and Doctorate. Occupation is at four levels, Administrative and clerical, Sales, Professional or
specialty, and Executive or managerial. A different number of observations are in each level of education –
occupation combination.

I. State the null and the alternate hypothesis for conducting one-way ANOVA for
both Education and Occupation individually.

Null and alternate hypothesis for Education:

H0 = Salary at different levels of education is equal.

Ha = At least at one of the Education levels, Salary is different from the other.

Null and alternate hypothesis for Occupation:

H0 = Salary at different levels of Occupation is equal.

Ha = At least at one of the Occupation levels, Salary is different from the other.

II. Perform a one-way ANOVA on Salary with respect to Education. State whether the
null hypothesis is accepted or rejected based on the ANOVA results.

One-way ANOVA - Education

Source of Variation df sum_sq mean_sq F PR(>F)
Education 2 1.026955e+11 5.134773e+10 30.9562 1.257709e-08
8
Residual 37.0 6.137256e+10 1.658718e+09 NaN Nan

The calculated value of F is 30.95628 and the p value is 0.0000000126 which is less than
the significance level 0.05.
We shall therefore reject the null hypothesis and state that there is an impact on the Salary
based on the levels of Education. (At least one of the Means of the three levels of Education are
not the same)

III. Perform a one-way ANOVA on Salary with respect to Occupation. State whether the
null hypothesis is accepted or rejected based on the ANOVA results.

One-way ANOVA - Occupation

Source of Variation df sum_sq mean_sq F PR(>F)
Occupation 3.0 1.125878e+10 3.752928e+09 0.88414 0.458508
4
Residual 36.0 1.528092e+11 4.244701e+09 NaN Nan

The calculated value of F is 0.884144 and the p value is 0.458508 which is more than the
significance level 0.05.
 Advanced Statistics
Since the p-value (0.4585) is greater than the significance level (0.05), we fail to reject the null
hypothesis.
Therefore, we cannot conclude that Salary depends on occupation.

IV. If the null hypothesis is rejected in either (2) or in (3), find out which class means
are significantly different. Interpret the result.

Problem Statement 1B:

V. What is the interaction between two treatments? Analyse the effects of one variable
on the other (Education and Occupation) with the help of an interaction plot.[hint: use
the ‘pointplot’ function from the ‘seaborn’ function]

Point plot 1: X= Education, Y = Salary, Hue = Occupation

Point plot 2: X= Occupation, Y = Salary, Hue = Education

Following are the observations:

• From the above 2 plots we see that all of the Exec-managerial occupation levels are
either from a Doctorate or Bachelor education level
• In the above point plots we can state that education level Doctorate has an
interaction with all four levels of Occupation
 Advanced Statistics
• From the above plots it seems that people with education level ‘Bachelor’ will be
drawing the same salary in occupation levels ‘Exec Managerial’ and ‘Sales’.
• Education levels ‘Doctorates’ and ‘Bachelors’ are almost paid equally in Occupation
levels ‘Sales’ and ‘Adm-clerical’
• In the first point plot we can see that mostly High School graduates are paid
lower compared to the other Education levels.
• In the second point plot we see that High School graduates are more likely to find a job
in Sales, prof-speciality and Adm-clerical and not in Exec-managerial occupation level.

As seen from the above plots, HS-grad is paid lowest in all occupational levels. Education level
‘Doctorate’ can find a job in all Occupation levels. To find a job in ‘Exec-managerial’ occupation
level, one must have an education level of ‘Bachelor’ or ‘Doctorate’, although, a ‘Doctorate’ might
pay more salary.

VI. Perform a two-way ANOVA based on Salary with respect to both Education and
Occupation (along with their interaction Education*Occupation). State the null
and alternative hypotheses and state your results. How will you interpret this
result?

Two-way ANOVA
Source of Variation df sum_sq mean_sq F PR(>F)
Education 2.0 1.026955e+11 5.134773e+10 31.257677 1.981539e-08
Occupation 3.0 5.519946e+09 1.839982e+09 1.120080 3.545825e-01
Residual 34.0 5.585261e+10 1.642724e+09 NaN Nan

Null hypothesis:
H0 = Levels of both Education and Occupation have an impact on Salary.

Alternate hypothesis:
Ha = Levels of at least one of the variables (either Occupation or Education) do not have an
impact on Salary.

• Education - The calculated value of F is 31.257677 and the p value is

0.00000001981539 which is less than the significance level 0.05.
• Occupation - The calculated value of F is 31.257677 and the p value is 0.3545825 which
is more than the significance level 0.05.
• We fail to reject the null hypothesis as the p value of one of the variables (Occupation) is
greater than the significance level 0.05. Therefore, we cannot conclude that levels of
one among Education and Occupation does not have an impact on Salary.

Interaction between Education & Occupation

Source of Variation df sum_sq mean_sq F PR(>F)
Education 2.0 1.026955e+11 5.134773e+10 72.211958 5.466264e-12
Occupation 3.0 5.519946e+09 1.839982e+09 2.587626 7.211580e-02
Education:Occupation 6.0 3.634909e+10 6.058182e+09 8.519815 2.232500e-05
Residual 29.0 2.062102e+10 7.110697e+08 NaN Nan

Null hypothesis:
H0 = Interaction between the levels of both Education and Occupation have an impact on Salary.

Alternate hypothesis:
Ha = Interaction between Occupation or Education does not have an impact on Salary.
 Advanced Statistics

• Education - The calculated value of F is 72.211958 and the p value is 5.466264e-12

which is less than the significance level 0.05.
• Occupation - The calculated value of F is 2.587626 and the p value is 0.07211580 which
is more than the significance level 0.05.
• Education:Occupation - The calculated value of F is 8.519815 and the p value
is 2.232500e-05 which is less than the significance level 0.05.
• Even though, p value of Occupation is higher than significance level, the interaction of
Education with Occupation has p value less than significance level. Hence, we reject
the null hypothesis and conclude that the interaction between the levels of both
Education
and Occupation does not have an impact on Salary.

VII. Explain the business implications of performing ANOVA for this particular case study.

ANOVA is performed in this case study to analyse if there is any difference in the Salary drawn by
an individual based on the combination of Education at 3 different levels and Occupation at 4
different levels. Salary being the dependant variable and Education and Occupation being
independent variables.
It is important to see if both Education and Occupation are significant in raising the Salary drawn
by an individual. This can be better understood from the output of ANOVA results by regarding
the significance level of Education and Occupation levels. We proceed with samples from 40
individuals.

One-way ANOVA helped us identify the impact of Education levels and Occupation levels
individually on the Salary drawn by an individual. We found that Education levels have an impact
on the Salary drawn by an individual, however, we could not prove that Occupation levels have an
impact on Salary.

We then performed a Two-way ANOVA and studied the impact on Salary by both Education levels
and Occupation levels together. We also studied the interaction between them variables and
their effect on Salary. From Two-way ANOVA it is still unclear that there is any impact on Salary by
both levels, however, the interaction between Education and Occupation levels show that there is
indeed an influence in the Salary drawn by an individual.

The Researchers can now be sure the interaction between Education levels and Occupation levels
are influencing the Salary drawn by an individual.

Below is the methodology used in performing ANOVA for this case study and the corresponding
results explained:

Assumptions:

• Population where samples are drawn follow a normal distribution

• The samples have been selected randomly and independently
• Each group has common variance

Degrees of Freedom:

• One way ANOVA for Education with respect to Salary: k-1, n-k: 2, 37
• One way ANOVA for Occupation with respect to Salary: k-1, n-k: 3, 36
• Two way ANOVA: r-1, c-1,rcn-1: 2,3,34
 Advanced Statistics
• Interaction effect: (r-1)(c-1),rcn-1:2,3,6,29

Alpha (⍺) is fixed at 0.05

Table value of Decision

Rule:

One Way ANOVA for Education Level:

The calculated value of F is 30.95628 and the p value is 0.0000000126 which is less than the
significance level 0.05. H0 will be rejected and we shall state that there is an impact of Education
level on Salary.

One Way ANOVA for Occupation:

The calculated value of F is 0.884144 and the p value is 0.458508 which is more than the
significance level 0.05. H0 is not rejected and we state that there is no significant impact of
Occupation level on Salary.

Two-way ANOVA:

• Education - The calculated value of F is 31.257677 and the p value is

0.00000001981539 which is less than the significance level 0.05.
• Occupation - The calculated value of F is 31.257677 and the p value is 0.3545825 which
is more than the significance level 0.05.
• We fail to reject the null hypothesis as the p value of one of the variables (Occupation) is
greater than the significance level 0.05. Therefore, we cannot conclude that levels of
one among Education and Occupation does not have an impact on Salary.

Interaction effect ANOVA:

• Education - The calculated value of F is 72.211958 and the p value is 5.466264e-12

which is less than the significance level 0.05.
• Occupation - The calculated value of F is 2.587626 and the p value is 0.07211580 which
is more than the significance level 0.05.
• Education:Occupation - The calculated value of F is 8.519815 and the p value is
2.232500e- 05 which is less than the significance level 0.05.
• Even though, p value of Occupation is higher than significance level, the interaction of
Education with Occupation has p value less than significance level. Hence, we reject
the null hypothesis and conclude that the interaction between the levels of both
Education and Occupation does not have an impact on Salary.
 Advanced Statistics

Problem Statement 2:
The dataset Education - Post 12th [Link] contains information on various colleges. You are expected
to do a Principal Component Analysis for this case study according to the instructions given. The data
dictionary of the 'Education - Post 12th [Link]' can be found in the following file: Data [Link].

I. Perform Exploratory Data Analysis [both univariate and multivariate analysis to

be performed]. What insight do you draw from the EDA?

(1) The dataset comprises of 777 rows or datapoints and 18 columns or features.

(2) Snippet below shows the head of the dataset.

(3) Snippet below are the variables and their data types in the dataset. The dataset has 17
float and integer type variables and 1 object type variable(Names)

(4) Snippet below shows there are no null values in the dataset.
 Advanced Statistics

(5) Snippet below shows there are no duplicated values in the dataset.

(6) Snippet below is the Descriptive Statistics of the dataset.

 Advanced Statistics

• We observe that Mean is greater than median (50th percentile) for most of the variables. The
variables where Mode is lower than median and median is lesser than the mean – Apps,
Accept, Enroll, Top 10 perc, [Link], [Link], Outstate, [Link], Books,
Personal, [Link], [Link] and Expend are positively/right skewed.
• Mean and median are same for Student Faculty Ration and Graduation Rate (symmetric)
while mean is lesser than median for Phd faculty and Terminal Grad. faculty (left skewed)
• We also see that the difference between 75th percentile value and max value for all the
variables except for the variables which are provided as percentage (Top10perc,
Top25perc, Phd, Terminal, [Link], [Link] and [Link])
• We thereby know the dataset has outliers.

(7) Univariate Analysis using Histogram and Boxplot

Advanced Statistics
Advanced Statistics
Advanced Statistics
 Advanced Statistics

(8) Snippet below shows the skewness of the dataset.

 Advanced Statistics
(9) Multivariate Analysis:

Below snippet is of the Correlation Matrix

Below is the heatmap

From the above heat map, we observe the following:

• We can see that some columns are co-related to each other and the highest is
between Full time Undergraduates and New enrolment.
• Higher positive correlation is seen between Applications and Acceptance, New
enrolment and Acceptance.
• Negatively correlation is seen between [Link] and Outstate, Expend and [Link].

II. Is scaling necessary for PCA in this case? Give justification and perform scaling.

The main objective of scaling or standardization to normalize a data within a particular range. We
have observed that dataset variables are of different scales, while some are actual numbers
some are percentage measures/ratios. Hence it is important to apply scaling function and obtain
data in common scales as step of pre-processing. Another importance of scaling is it helps in
speeding up the calculations in an algorithm.
 Advanced Statistics
Before standardizing, lets remove the outliers in the given data. To do so, we must drop all non-
float and non-int variables.

Below snippet shows that all outliers are removed and column name is dropped.

Below snippet shows the head of the data before scaling and the head of the data after scaling.

III. Comment on the comparison between the covariance and the correlation matrices
from this data.

Correlation is a scaled version of covariance; note that the two parameters always have the same
sign (positive, negative, or 0). When the sign is positive, the variables are said to be positively
correlated; when the sign is negative, the variables are said to be negatively correlated; and when
the sign is 0, the variables are said to be uncorrelated.

In simple sense,

Covariance is a measure used to determine how much two variables change in tandem. It
indicates the direction of the linear relationship between variables.
Below is a snippet of covariance matrix
 Advanced Statistics

Correlation measures both the strength and direction of the linear relationship between two
variables
Below is a snippet of correlation matrix
 Advanced Statistics
IV. Check the dataset for outliers before and after scaling. What insight do you derive here?

Below is the snippet of dataset before scaling in the form of boxplot:

Below is the snippet of dataset after scaling in the form of boxplot:

Observation:
• There is a significant shift in the lower quartile, median and upper quartile of variable
Outstate and Expend
• Variable Top25perc looks the same before and after scaling
• After scaling, it is noticed that most of the variable’s median has shifted. Eg:
Applications. Acceptance, [Link] etc
• Most variables upper quartile has also shifted. Eg: [Link], Apps.

V. Perform PCA and export the data of the Principal Component scores into a data frame.
 Advanced Statistics
Principle component scores obtained are as follows:

We are considering first 7 PCA which explains about 87.6% of the variance and thereby have our
dimensions reduced from 17 to 6.

The explained variance ratio is as follows:

Cumulation of explained variance ratio is as follows:

The Cumulative % gives the percentage of variance accounted for by the n components. For
example, the cumulative percentage for the second component is the sum of the percentage of
variance for the first and second components. It helps in deciding the number of components by
selecting the components which explained the high variance
In the above array we see that the first feature explains 33.3% of the variance within our data set
while the first two explain 62.1 and so on. If we employ 7 features we capture ~ 87.6% of the
variance within the dataset.

Plot to show cumulated explained variance ratio

 Advanced Statistics

PCA component scores in to a Data frame:

This heatmap and the colour bar basically represent the correlation between the various feature
and the principal component itself

Component 2 and 5 look more related to Books. Similarly component 6 looks more related to
Personal.

VI. Extract the eigenvalues and eigenvectors.[print both]

 Advanced Statistics
Below is the snippet of Eigen Values

Below is snippet of Eigen Vectors

 Advanced Statistics

VII. Write down the explicit form of the first PC (in terms of the eigenvectors. Use
values with two places of decimals only).

We have taken the Education - Post 12th std data collected from various Institutions. The dataset
contains 777 observations and 17 variables.

In our attempt to find the Principal components, we standardise the data using z-score,
derived the covariance matrix and did the single vector decomposing to find the eigen values
and eigen vectors.

We get 17 eigenvalues and 17 eigenvectors corresponding to a principal component.

To explain the explicit form of the first PC we elaborate the procedure on finding the Principal
component 1.

EV0 - Eigen vector

 Advanced Statistics

Scaled Data set value first row

We derive the Principal component 1 by doing sum product of the first row of the data with the
first Eigen vector values = (EV0 - Eigen vector) *Scaled Data set value first row

PC1 = 0.37

VIII. Consider the cumulative values of the eigenvalues. How does it help you to decide
on the optimum number of principal components? What do the eigenvectors
indicate?

To decide how many eigenvalues/eigenvectors to keep, we should clearly define the objective
first for doing PCA in the first place. Are we doing it for reducing storage requirements, to reduce
dimensionality for a classification algorithm, or for some other reason.

If we don't have any strict constraints, then we should plot the cumulative sum of eigenvalues.
If we divide each value by the total sum of eigenvalues prior to plotting, then your plot will show
the fraction of total variance retained vs. number of eigenvalues. The plot will then provide a
good indication of when you hit the point of diminishing returns.

Eigen Values:

Variance Explained:

Cumulative Variance Explained:

 Advanced Statistics

From the above array the first cumulative variance explains up to 33.26% of the variance but if we
choose up to 7 cumulative variances then it explains up to 87.6% of the variance.

Scree plot: A scree plot helps the analyst visualize the relative importance of the factors, a sharp
drop in the plot signals that subsequent factors are ignorable. Below is the snippet of scree plot.

Plot explaining both individual and cumulative variance:

Eigen values are the true replica of the variance. It is the percentage of variance being explained.

Eigen Vectors indicates there is a linear transformation in the data and it is a nonzero vector that
changes at most by a scalar factor when that linear transformation is applied.
 Advanced Statistics
IX. Explain the business implication of using the Principal Component Analysis for this
case study. How may PCs help in the further analysis? [Hint: Write Interpretations of
the Principal Components Obtained]

PCA is a statistical technique and uses orthogonal transformation to convert a set of observations
of possibly correlated variables into a set of values of linearly uncorrelated variables. PCA also is a
tool to reduce multidimensional data to lower dimensions while retaining most of the
information. Principal Component Analysis (PCA) is a well-established mathematical technique for
reducing the dimensionality of data, while keeping as much variation as possible. This PCA can
only be done on continuous variables

• In the current dataset we had 777 datapoints and 17 variables. Proceeding with all 17
variables would involve more time, storage and cost associated. PCA comes handy in
helping us reducing the variables in mathematical approach
• We have reduced the noise by removing the correlated features and retaining the
features that are not correlated
• We have applied PCA and successfully reduced the variables from 17 to 7 (more than half
the size)
• We have ensured to capture 87.6% of the variance despite reducing the dimension
of variables by more than half
• This pre-processing step is of great help as it reduces the size of the data and improves
the efficiency of the processes as they are applied on smaller dimensions.