DATA EXPLORATION MODULE 2
MODULE 2 : DATA EXPLORATION
Types of data, Properties of data
Descriptive Statistics: Univariate Exploration: Measure of Central Tendency,
Measure of Spread, Symmetry, Skewness: Karl Pearson Coefficient of
skewness, Bowley‘s Coefficient, Kurtosis Multivariate Exploration: Central
Data Point, Correlation, Different forms of correlation, Karl Pearson
Correlation Coefficient for bivariate distribution.
Inferential Statistics: Overview of Various forms of distributions: Normal,
Poisson, Test Hypothesis, Central limit theorem, Confidence Interval, Z-test, t-
test, Type-I, Type-II Errors, ANOVA.
INFERENTIAL STATISTICS
INFERENTIAL STATISTICS
Inferential statistics is a branch of statistics that makes the use of various
analytical tools to draw inferences about the population data from sample
data.
Inferential statistics help to draw conclusions about the population while
descriptive statistics summarizes the features of the data set.
There are two main types of inferential statistics - hypothesis testing and
regression analysis
TYPES OF INFERENTIAL STATISTICS
Inferential statistics can be classified into hypothesis testing and
regression analysis.
Hypothesis testing also includes the use of confidence intervals to
test the parameters of a population.
INFERENTIAL STATISTICS
Inferential statistics helps to develop a good understanding of
the population data by analyzing the samples obtained from it.
It helps in making generalizations about the population by using
various analytical tests and tools.
In order to pick out random samples that will represent the
population accurately many sampling techniques are used.
Some of the important methods are simple random sampling,
stratified sampling, cluster sampling, and systematic sampling
techniques.
OVERVIEW OF VARIOUS FORMS OF
DISTRIBUTIONS: NORMAL, POISSON
WHAT IS DISTRIBUTION IN STATISTICS
The distribution of a statistical dataset is the spread of the data which
shows all possible values or intervals of the data and how they occur.
A distribution is simply a collection of data or scores on a variable. Usually,
these scores are arranged in order from ascending to descending and then
they can be presented graphically.
The distribution provides a parameterized mathematical function which will
calculate the probability of any individual observation from the sample
space.
WHY ARE DISTRIBUTIONS IMPORTANT?
Sampling distributions are important for statistics because we need to collect the
sample and estimate the parameters of the population distribution. Hence
distribution is necessary to make inferences about the overall population.
For example, the most common measures of how sample differs from each other
is the standard deviation and standard error of the mean.
TYPES OF DISTRIBUTIONS
Uniform Distribution
Binomial Distribution
Normal Distribution
Poisson Distribution
Exponential Distribution
NORMAL DISTRIBUTION
Normal distribution, also known as the Gaussian distribution, is
a probability distribution that is symmetric about the mean, showing that
data near the mean are more frequent in occurrence than data far from the
mean.
The normal distribution is the proper term for a probability bell curve.
In a normal distribution the mean is zero and the standard deviation is 1. It
has zero skew and a kurtosis of 3.
Normal distributions are symmetrical, but not all symmetrical distributions
are normal
STANDARD NORMAL DISTRIBUTION
Its mean (average), median (midpoint), and mode (most frequent
observation) are all equal to one another.
Moreover, these values all represent the peak, or highest point, of
the distribution.
The distribution then falls symmetrically around the mean, the
width of which is defined by the standard deviation.
EMPIRICAL RULE
If distribution is bell shape then it
is assumed that
68% of the elements have a z-score
between -1 and 1;
95% have a z-score between -2 and 2;
99.7 % have a z-score between -3 and
3.
POISSON DISTRIBUTION
In statistics, a Poisson distribution is a probability distribution used to show
how often an event is likely to occur over a specified period. In other
words, it is a count distribution.
Poisson distributions are often used to understand independent events that
occur at a constant rate within a given interval of time.
It was named after French mathematician Siméon Denis Poisson.
Poisson distributions are used when the variable of interest is a discrete
count variable.
A Poisson distribution can be used to estimate how likely it is that
something will happen "X" number of times.
For example, if the average number of people who buy cheeseburgers
from a fast-food chain on a Friday night at a single restaurant location is
200, a Poisson distribution can answer questions such as, "What is the
probability that more than 300 people will buy burgers?“
The application of the Poisson distribution thereby enables managers to
introduce optimal scheduling systems that would not work with, say,
a normal distribution.
POISSON DISTRIBUTION
HYPOTHESIS TESTING
HYPOTHESIS TESTING
FUNDAMENTALS OF HYPOTHESIS TESTING
FUNDAMENTALS OF HYPOTHESIS TESTING
A person is on trial for a criminal offense, and the judge needs to provide a
verdict on his case.
Now, there are four possible combinations in such a case:
•First Case: The person is innocent, and the judge identifies the person as
innocent
•Second Case: The person is innocent, and the judge identifies the person as
guilty
•Third Case: The person is guilty, and the judge identifies the person as innocent
•Fourth Case: The person is guilty, and the judge identifies the person as guilty
There can be two types of error in the judgment –
Type 1 error, when the verdict is against the person
while he was innocent, and Type 2 error when the
verdict is in favor of the person while he was guilty.
According to the Presumption of Innocence, the person
is considered innocent until proven guilty.
That means the judge must find evidence that
convinces him “beyond a reasonable doubt.”
This phenomenon of “Beyond a reasonable
doubt” can be understood as Probability (Judge
Decided Guilty | Person is Innocent) should be
small.
The basic concepts of Hypothesis Testing are actually
quite analogous to this situation.
HYPOTHESIS TESTING
Hypothesis testing helps in data analysis by providing a way to make inferences about a
population based on a sample of data.
It allows analysts to make decisions about whether to accept or reject a given assumption
or hypothesis about the population based on the evidence provided by the sample data.
For example, hypothesis testing can be used to determine whether a sample mean is
significantly different from a hypothesized population mean or whether a sample
proportion is significantly different from a hypothesized population proportion.
This information can be used to make decisions about whether to accept or reject a given
assumption or hypothesis about the population.
In statistical analysis, hypothesis testing is used to make inferences about a population
based on a sample of data.
HYPOTHESIS TESTING
In machine learning, hypothesis testing is used to evaluate the performance of a model and
determine the significance of its parameters.
For example, a t-test or z-test can be used to compare the means of two groups of data to
determine if there is a significant difference between them.
This information can then be used to improve the model, or select the best set of features.
Additionally, hypothesis testing can be used to evaluate the overall accuracy of a model and
make decisions about how to proceed with further development or deployment.
In fact, we can even test the statistical validity of machine learning algorithms like linear
regression and logistic regression on a given dataset using the process of hypothesis testing.
Steps to Perform Hypothesis Testing
Set the Null and Alternate Hypotheses
Set the Significance Level, Criteria for a decision
Compute the test statistic
Make a decision
P VALUE
How do you interpret a p-
value?
How much importance should
we place in the p-value?
How will you explain the
significance of p-value to a
non-data science person (a
stakeholder for example)?
Critical Value, P-Value
Let’s understand the logic of Hypothesis Testing with the graphical representation
for Normal Distribution.
The above visualization helps to understand the z-
value and its relation to the critical value. Typically, we
set the Significance level at 10%, 5%, or 1%. If our
test score lies in the Acceptance Zone, we fail to reject
the Null Hypothesis. If our test score lies in the Critical
Zone, we reject the Null Hypothesis and accept the
Alternate Hypothesis.
Z TEST
ONE SAMPLE Z TEST
EXAMPLE : Z TEST
TWO SAMPLE Z TEST
TWO SAMPLE Z TEST