Non Parametric Methods
A Gentle Introduction to Nonparametric Statistics
A large portion of the field of statistics and statistical methods is
dedicated to data where the distribution is known.
Samples of data where we already know or can easily identify the
distribution of are called parametric data.
Often, parametric is used to refer to data that was drawn from a
Gaussian distribution in common usage.
Data in which the distribution is unknown or cannot be easily identified
is called nonparametric.
Parametric Data
Parametric data is a sample of data drawn from a known data
distribution.
This means that we already know the distribution
If we have parametric data, we can harness the entire suite of statistical
methods developed for data assuming a Gaussian distribution, such as:
Summary statistics.
Correlation between variables.
Significance tests for comparing means.
Non-Parametric Methods
Data that does not fit a known or well-understood distribution is referred
to as nonparametric data.
Data could be non-parametric for many reasons, such as:
Data is not real-valued, but instead is ordinal, intervals, or some other form.
Data is real-valued but does not fit a well understood shape.
Data is almost parametric but contains outliers, multiple peaks, a shift, or
some other feature.
In the case of ordinal or interval data, nonparametric statistics are the
only type of statistics that can be used.
For real-valued data, nonparametric statistical methods are required in
applied machine learning when you are trying to make claims on data
that does not fit the familiar Gaussian distribution.
Ranking Data
The SciPy library provides the rankdata() function to rank numerical
data, which supports a number of variations on ranking.
The example below demonstrates how to rank a numerical dataset.
R from [Link] import rand
from [Link] import seed
a from [Link] import rankdata
n # seed random number generator
k seed(1)
i # generate dataset
data = rand(1000)
n # review first 10 samples
g print(data[:10])
# rank data
ranked = rankdata(data)
# review first 10 ranked samples
print(ranked[:10])
Working with Ranked Data
For example, if we take nonparametric data as data that does not look
Gaussian.
then you can use statistical methods that quantify how Gaussian a
sample of data is and use nonparametric methods if the data fails those
tests.
Introduction to Normality Tests
Normality Assumption:
A large fraction of the field of statistics is concerned with data that
assumes that it was drawn from a Gaussian distribution.
If Data Is Gaussian:
Use Parametric Statistical Methods
Else:
Use Nonparametric Statistical Methods
Sample data
# generate gaussian data
from [Link] import seed
from [Link] import randn
from numpy import mean
from numpy import std
# seed the random number generator
seed(1)
# generate univariate observations
data = 5 * randn(100) + 50
# summarize
print('mean=%.3f stdv=%.3f' % (mean(data), std(data))
Mean=50.303 and Std= 4.426
Visual Normality Checks
We can create plots of the data to check whether it is Gaussian
Histogram Plot:
A simple and commonly used plot to quickly check the distribution of a
sample of data is the histogram
A sample of data has a Gaussian distribution of the histogram plot, showing
the familiar bell shape.
# histogram plot
from [Link] import seed
from [Link] import randn
from matplotlib import pyplot
# seed the random number generator
seed(1)
# generate univariate observations
data = 5 * randn(100) + 50
# histogram plot
[Link](data)
[Link]()
Quantile-Quantile Plot
Another popular plot for checking the distribution of a data sample is the
quantile-quantile plot, Q-Q plot, or QQ plot for short
A perfect match for the distribution will be shown by a line of dots on a
45-degree angle from the bottom left of the plot to the top right. Often a
line is drawn on the plot to help make this expectation clear. Deviations
by the dots from the line shows a deviation from the expected
distribution.
Statistical Normality Tests
There are many statistical tests that we can use to quantify whether a
sample of data looks as though it was drawn from a Gaussian
distribution.
Each test will return at least two things:
Statistic: A quantity calculated by the test that can be interpreted in
the context of the test via comparing it to critical values from the
distribution of the test statistic.
p-value: Used to interpret the test, in this case whether the sample was
drawn from a Gaussian distribution.
Use of Hypothesis test
The tests assume that that the sample was drawn from a Gaussian
distribution. Technically this is called the null hypothesis, or H0. A
threshold level is chosen called alpha, typically 5% (or 0.05), that is used
to interpret the p-value.
In the SciPy implementation of these tests, you can interpret the p value
as follows.
p <= alpha: reject H0, not normal.
p > alpha: fail to reject H0, normal.
This means that, in general, we are seeking results with a larger p-value
to confirm that our sample was likely drawn from a Gaussian distribution.
A result above 5% does not mean that the null hypothesis is true. It
means that it is very likely true given available evidence.
Shapiro-Wilk Test
The Shapiro-Wilk test evaluates a data sample and quantifies how likely
it is that the data was drawn from a Gaussian distribution.
# Shapiro-Wilk Test
from [Link] import seed
from [Link] import randn
from [Link] import shapiro
# seed the random number generator
seed(1)
# generate univariate observations
data = 5 * randn(100) + 50
# normality test
stat, p = shapiro(data)
print('Statistics=%.3f, p=%.3f' % (stat, p))
# interpret
alpha = 0.05
if p > alpha:
print('Sample looks Gaussian (fail to reject H0)')
else:
print('Sample does not look Gaussian (reject H0)')
D’Agostino’s K^2 Test
The D’Agostino’s K^2 test calculates summary statistics from the data,
namely kurtosis and skewness, to determine if the data distribution
departs from the normal distribution.
Skew is a quantification of how much a distribution is pushed left or
right, a measure of asymmetry in the distribution.
Kurtosis quantifies how much of the distribution is in the tail. It is a
simple and commonly used statistical test for normality
The D’Agostino’s K^2 test is available via the normaltest() SciPy
function and returns the test statistic and the p-value.
Code for R^2 test
Sample code:
# D'Agostino and Pearson's Test
from [Link] import seed
from [Link] import randn
from [Link] import normaltest
# seed the random number generator
seed(1)
# generate univariate observations
data = 5 * randn(100) + 50
# normality test
stat, p = normaltest(data)
print('Statistics=%.3f, p=%.3f' % (stat, p))
# interpret
alpha = 0.05
if p > alpha:
print('Sample looks Gaussian (fail to reject H0)')
else:
print('Sample does not look Gaussian (reject H0)')
Anderson-Darling Test
Anderson-Darling Test is a statistical test that can be used to evaluate
whether a data sample comes from one of among many known data
samples.
It can be used to check whether a data sample is normal. The test is a
modified version of a more sophisticated nonparametric goodness-of-fit
statistical test called the Kolmogorov-Smirnov test.
A feature of the Anderson-Darling test is that it returns a list of critical
values rather than a single p-value. This can provide the basis for a
more thorough interpretation of the result.
Code for A-D Test
# Anderson-Darling Test
from [Link] import seed
from [Link] import randn
from [Link] import anderson
# seed the random number generator
seed(1)
# generate univariate observations
data = 5 * randn(100) + 50
# normality test
result = anderson(data)
print('Statistic: %.3f' % [Link])
p = 0
for i in range(len(result.critical_values)):
sl, cv = result.significance_level[i], result.critical_values[i]
if [Link] < result.critical_values[i]:
print('%.3f: %.3f, data looks normal (fail to reject H0)' % (sl, cv))
else:
print('%.3f: %.3f, data does not look normal (reject H0)' % (sl, cv))
Statistic: 0.220
15.000: 0.555, data looks normal (fail to reject H0)
10.000: 0.632, data looks normal (fail to reject H0)
5.000: 0.759, data looks normal (fail to reject H0)
2.500: 0.885, data looks normal (fail to reject H0)
1.000: 1.053, data looks normal (fail to reject H0)
Make data Normal