Normal Distribution
Plotting the Distribution
It will help us analyze the shape of the data and visualize the PDF of normal distribution using
the parameters (mean (mu) and Standard deviation (sigma)) from the data.
The [Link](x, loc, scale) function will be used to calculate the probability density.
The [Link]() function takes three parameters
x : Scalar or Array of numbers
loc : Sample mean
scale : standard deviation
from [Link] import norm
# calculate the pdf of SAT scores using [Link]()
density = [Link]() # create an empty DataFrame
density["x"] = [Link](
sat_score["score"].min(), sat_score["score"].max(), 100
) # create an array of 100 numbers in between the min and max score range and store it in the
first column of the empty DataFrame
density["pdf"] = [Link](density["x"], mu, sigma) # calculate the pdf() of the created numbers
and store it in another column named 'pdf'
fig, ax = [Link]() # create the subplot
[Link](sat_score["score"], ax=ax, kde=True, stat="density") # plot the distribution of data
using histogram
[Link](density["x"], density["pdf"], color="red") # plot the pdf of the normal distribution
[Link]("Normal Distribution") # set the title
[Link]() # display the plot
find the cumulative probability
# [Link]() calculates the cumulative probability
prob_less_than_800 = [Link](800, mu, sigma)
find the cumulative probability and subtract it from 1 to calculate the probability that a student
will score more than 1300
prob_greater_than_1300 = 1 - [Link](1300, mu, sigma)
Here, [Link](x, loc, scale)) function is used to calculate thpppppe probability density.
The [Link]() function takes three parameters
x : Scalar or Array of numbers
loc : Sample mean
scale : standard deviation
# calculate the 90th percentile score using ppf() function
# [Link]() calculates the percentile point
score_90th_percentile = [Link](0.90, mu, sigma)
Discrete Uniform Distribution
# import the required function
from [Link] import uniform
use the [Link]() function to generate the probability distribution
k = [Link](90, 101)
probs = [Link](k, loc=90, scale=11)
1 - [Link](96, loc=90, scale=11)
[Link](93, loc=90, scale=11)
Tests from [Link] library (Alias: stats)
1. ttest_1samp(): Test to compare a sample mean with a population mean
when the population standard deviation is unknown.
2. ttest_ind(): Test to compare two sample means from two independent
populations when the population standard deviations are unknown.
3. chi2_contingency(): Test to check the dependence(relationship)
between two categorical variables.
Note: In '[Link]' test functions, the 'alternative' argument is used to
define the alternative hypothesis. The following options are available (the
default is 'two-sided’):
‘two-sided’: to perform the test for a two-tailed alternative
hypothesis (containing ≠ sign)
‘less’: to perform the test for a one-tailed alternative hypothesis
(containing < sign)
‘greater’: to perform the test for a one-tailed alternative hypothesis
(containing > sign)
One sample testing
In one sample test, we compare the population parameter such as mean of a single sample of data
collected from a single population
import math
from scipy import stats
from [Link] import ttest_1samp
# one sample t-test
# null hypothesis: expected value = 144
t_statistic, p_value = ttest_1samp(mydata, 144)
print('One sample t test \nt statistic: {0} p value: {1} '.format(t_statistic, p_value))
One sample t test
t statistic: [1.22467437] p value: [0.23055327]
# p_value < 0.05 => alternative hypothesis:
alpha_value = 0.05 # Level of significance
print('Level of significance: %.2f' %alpha_value)
if p_value < alpha_value:
print('We have evidence to reject the null hypothesis since p value < Level of significance')
else:
print('We have no evidence to reject the null hypothesis since p value > Level of significance')
print ("Our one-sample t-test p-value=", p_value)
Level of significance: 0.05
We have no evidence to reject the null hypothesis since p value > Level of significance
Our one-sample t-test p-value= [0.23055327]
In this example, p value is 0.23055327 and it is greater than 5% level of significance
So the statistical decision is failing to reject the null hypothesis at 5% level of significance.
2 sample t test
from [Link] import ttest_1samp, ttest_ind
import [Link] as stats
import [Link] as sm
t_statistic, p_value = ttest_ind(mydata['WingA'],mydata['WingB'])
print('tstat',t_statistic)
print('P Value',p_value)
p_value < 0.05 => alternative hypothesis:
# they don't have the same mean at the 5% significance level
print ("two-sample t-test p-value=", p_value)
alpha_level = 0.05
if p_value < alpha_level:
print('We have enough evidence to reject the null hypothesis in favour of alternative
hypothesis')
print('We conclude that the mean time to deliver luggages in of both the wings of the hotel are
not same.')
else:
print('We do not have enough evidence to reject the null hypothesis in favour of alternative
hypothesis')
print('We conclude that mean time to deliver luggages in of both the wings of the hotel are
same.')
use the [Link].ttest_rel to calculate the T-test on TWO RELATED samples of scores.
This is a two-sided test for the null hypothesis that 2 related or repeated samples have
identical average (expected) values. Here we give the two sample observations as input.
This function returns t statistic and two-tailed p value.
# paired t-test: doing two measurments on the same experimental unit
# e.g., before and after a treatment
t_statistic, p_value = stats.ttest_rel(mydata['Two Days'],mydata['Seven Days'])
print('tstat %1.3f' % t_statistic)
print("p-value for one-tail:", p_value/2)
Chi square
from [Link] import chi2_contingency
Calculate the p - value and test statistic
chi2, pval, dof, exp_freq = chi2_contingency(df, correction = False)
pval
0.41943105261448455
ANOVA
One way anova
Hypothesis Testing
Step 1: Define null and alternative hypotheses
The null and alternative hypotheses can be formulated as:
H0 : The mean weight losses with respect to each diet category is equal.
Ha : At least one of the mean weight losses with respect to the three diet category is different.
Step 2: Select Appropriate test
This is a problem, concerning three population means. One-way ANOVA is an appropriate test
here provided normality and equality of variance assumptions are verified.
One-way ANOVA test
In a one-way ANOVA test, we compare the means from several populations to test if there is any
significance difference between them. The results from an ANOVA test are most reliable when
the assumptions of normality and equality of variances are satisfied.
For testing of normality, Shapiro-Wilk’s test is applied to the response variable.
For equality of variance, Levene test is applied to the response variable
Shapiro-Wilk’s test
We will test the null hypothesis
H0: The weight losses follow a normal distribution
against the alternative hypothesis
Ha: The weight losses do not not follow a normal distribution
Assumption 1: Normality
# Use the shapiro function for the [Link] library for this test
# find the p-value
w, p_value = [Link](df['weightloss'])
print('The p-value is', p_value)
Levene’s test
We will test the null hypothesis
H0: All the population variances are equal
against the alternative hypothesis
Ha: At least one variance is different from the rest
#Assumption 2: Homogeneity of Variance
# use levene function from [Link] library for this test
# find the p-value
statistic, p_value = [Link](df[df['diet']=='A']['weightloss'],
df[df['diet']=='B']['weightloss'],
df[df['diet']=='C']['weightloss'])
print('The p-value is', p_value)
The p-value is 0.5376731304274011
Since the p-value is large than the 5% significance level, we fail to reject the null hypothesis of
homogeneity of variances.
Collect and prepare data
# create separate variables to store the weightlosses with respect to the three diet-plans
weightloss_diet_A = df[df['diet']=='A']['weightloss']
weightloss_diet_B = df[df['diet']=='B']['weightloss']
weightloss_diet_C = df[df['diet']=='C']['weightloss']
Calculate the p-value
We will use the f_oneway() function from the [Link] library to perform a one-way
ANOVA test.
The f_oneway() function takes the sample observations from the different groups and
returns the test statistic and the p-value for the test.
The sample observations are the values of weight losses with respect to the three
diet-plans.
import the required function
from [Link] import f_oneway
# find the p-value
test_stat, p_value = f_oneway(weightloss_diet_A, weightloss_diet_B,
weightloss_diet_C)
print('The p-value is ', p_value)
The p-value is 0.0032290142385893524
# print the conclusion based on p-value
if p_value < 0.05:
print(f'As the p-value {p_value} is less than the level of significance, we reject the null
hypothesis.')
else:
print(f'As the p-value {p_value} is greater than the level of significance, we fail to
reject the null hypothesis.')
Since the p-value is less than the level of significance (5%), we reject the null hypothesis. Hence,
we have enough statistical evidence to say that at least one of the mean weight losses with
respect to the three diet-plans is different.
Multiple Comparison test (Tukey HSD)
In order to identify for which fuel type mean carbon emission is different from other groups, the
null hypothesis is
H0:μ1=μ2 and μ1=μ3 and μ2=μ3
against the alternative hypothesis
Ha:μ1≠μ2 or μ1≠μ3 or μ2≠μ3
The pairwise_tukeyhsd() function of Statsmodels will be used to compute the test statistic and p-
value.
#import the required function
from [Link] import pairwise_tukeyhsd
# perform multiple pairwise comparison (Tukey HSD)
m_comp = pairwise_tukeyhsd(endog = aovdata['co_emissions'], groups = aovdata['fuel_type'],
alpha = 0.05)
print(m_comp)
Multiple Comparison of Means - Tukey HSD, FWER=0.05
====================================================
group1 group2 meandiff p-adj lower upper reject
----------------------------------------------------
E85 LPG 25.6199 0.0012 8.6843 42.5554 True
E85 Petrol 33.5984 0.0 16.8712 50.3256 True
LPG Petrol 7.9785 0.4916 -8.5139 24.471 False
----------------------------------------------------
Insight
As the p-values (refer to the p-adj column) for comparing the mean carbon emissions for the pair
E85-LPG and E85-Petrol is less than the significance level, the null hypothesis of equality of all
population means can be rejected.
Thus, we can say that the mean carbon emission for Petrol and LPG is similar but emission for
fuel type E85 is significantly different from LPG and Petrol.