Module -3

Statistical Experiments and Significance Testing

 A/B testing
 hypothesis testing
 resampling
 statistical significance & p-values
 t-tests
 multiple testing
 degrees of freedom.

1 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
A/B testing

A/B testing or split testing, in a nutshell, is a means to compare two iterations of

an email, website, or other marketing asset and assess the
performance differences between them.

A/B tests are common in web design and marketing, since results are so readily
measured. Some examples of A/B testing include:

 Testing two soil treatments to determine which produces better seed

germination
 Testing two therapies to determine which suppresses cancer more effectively
 Testing two prices to determine which yields more net profit
 Testing two web headlines to determine which produces more clicks
 Testing two web ads to determine which generates more conversions

How Is A/B Testing Conducted?

To conduct an A/B test, you must produce two variants of the same content, each
with a single variable altered. Afterward, you'll present these two versions to two
groups of people with identical sizes and compare their performance over a certain
amount of time long enough to draw precise judgments regarding your findings.
A/B testing enables marketers to compare the performance of two different
versions of marketing content. The following is the examples of A/B test types.
Imagine you run an online store that sells clothing, and your goal is to boost the
lead generation on your product pages. You choose to do A/B testing on two
variants of your product page.

You keep the initial product page layout in control group A, which places the
image of the product, synopsis, pricing, and "Add to Cart" button above the fold.
You make a minor adjustment in experimental group B by moving the product
review area under the "Add to Cart" button.

In the A/B test, 50% of the people who visit your website are randomly assigned
to Group A, and the remaining 50% are randomly assigned to Group B. You keep
track of the number of visitors from each category that add the product to their cart
and then check out.

2 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
You evaluate the data from the A/B test after a week of operation and discover that
Group B had a 10% greater conversion rate than Group A. This indicates that the
inclusion of the product review area had a favorable effect on website users'
purchasing decisions.

Based on these findings, you decide to add a product review area to every one of
your product pages to boost sales and conversion rates. This is but one illustration
of how A/B testing in marketing may be utilized to make data-driven decisions and
enhance the efficacy of your marketing initiatives.

You also need to pay attention to the test statistic or metric you use to compare
group A to group B. Perhaps the most common metric in data science is a binary
variable: click or no-click, buy or don’t buy, fraud or no fraud, and so on. Those
results would be summed up in a 2×2 table. Table 3-1 is a 2×2 table for an actual
price test.

3 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
Hypothesis testing compares two opposite ideas about a group of people or things
and uses data from a small part of that group (a sample) to decide which idea is
more likely true. We collect and study the sample data to check if the claim is
correct.

Hypothesis Testing

For example, if a company says its website gets 50 visitors each day on average,
we use hypothesis testing to look at past visitor data and see if this claim is true or
if the actual number is different.
Defining Hypotheses
 Null Hypothesis (H₀): The starting assumption. For example, "The average
visits are 50."
 Alternative Hypothesis (H₁): The opposite, saying there is a difference. For
example, "The average visits are not 50."
Key Terms of Hypothesis Testing
To understand the Hypothesis testing firstly we need to understand the key terms
which are given below:
 Significance Level (α): How sure we want to be before saying the claim is
false. Usually, we choose 0.05 (5%).
 p-value: The chance of seeing the data if the null hypothesis is true. If this is
less than α, we say the claim is probably false.
 Test Statistic: A number that helps us decide if the data supports or rejects
the claim.
 Critical Value: The cutoff point to compare with the test statistic.

4 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
  Degrees of freedom: A number that depends on the data size and helps find
the critical value.
Types of Hypothesis Testing
It involves basically two types of testing:

Types of Hypothesis testing

1. One-Tailed Test
Used when we expect a change in only one direction either up or down, but not
both. For example, if testing whether a new algorithm improves accuracy, we only
check if accuracy increases.
There are two types of one-tailed test:
 Left-Tailed (Left-Sided) Test: Checks if the value is less than expected.
Example: H0:μ≥50μ≥50 and H1: μ<50μ<50
 Right-Tailed (Right-Sided) Test: Checks if the value is greater than
expected. Example: H0 : μ≤50μ≤50 and H1:μ>50μ>50
2. Two-Tailed Test
Used when we want to see if there is a difference in either direction higher or
lower. For example, testing if a marketing strategy affects sales, whether it goes up
or down
Example: H0: μ=μ= 50 and H1: μ≠50μ =50
To go deeper into differences into both types of test: Refer to link
What are Type 1 and Type 2 errors in Hypothesis Testing?
In hypothesis testing Type I and Type II errors are two possible errors that can
happen when we are finding conclusions about a population based on a sample of

5 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
data. These errors are associated with the decisions we made regarding the null
hypothesis and the alternative hypothesis.
 Type I error: When we reject the null hypothesis although that hypothesis
was true. Type I error is denoted by alpha(αα).
 Type II errors: When we accept the null hypothesis but it is false. Type II
errors are denoted by beta(ββ).
Null Hypothesis is
Null Hypothesis is True False

Null Hypothesis is True Type II Error (False

Correct Decision
(Accept) Negative)

Alternative Hypothesis Type I Error (False

Correct Decision
is True (Reject) Positive)

How does Hypothesis Testing work?

Working of Hypothesis testing involves various steps:

Steps of Hypothesis Testing

6 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
Step 1: Define Hypotheses:
 Null hypothesis (H₀): Assumes no effect or difference.
 Alternative hypothesis (H₁): Assumes there is an effect or difference.
Example: Test if a new algorithm improves user engagement.
Note: In this we assume that our data is normally distributed.
Step 2: Choose significance level
We select a significance level (usually 0.05). This is the maximum chance we
accept of wrongly rejecting the null hypothesis (Type I error). It also sets the
confidence needed to accept results.
Step 3: Collect and Analyze data.
 Now we gather data this could come from user observations or an
experiment. Once collected we analyze the data using appropriate statistical
methods to calculate the test statistic.
 Example: We collect data on user engagement before and after
implementing the algorithm. We can also find the mean engagement scores
for each group.
Step 4: Calculate Test Statistic
The test statistic measures how much the sample data deviates from what we did
expect if the null hypothesis were true. Different tests use different statistics:
 Z-test: Used when population variance is known and sample size is large.
 T-test: Used when sample size is small or population variance unknown.
 Chi-square test: Used for categorical data to compare observed vs.
expected counts.
Step 5: Make a Decision
We compare the test statistic to a critical value from a statistical table or use the p-
value:
 Using Critical Value:
o If test statistic > critical value → reject H0.
o If test statistic ≤ critical value → fail to reject H0.
 Using P-value:
o If p-value ≤ α → reject H0.
o If p-value > α → fail to reject H0.
Example: If p-value is 0.03 and α is 0.05, we reject the null hypothesis because
0.03 < 0.05.
Step 6: Interpret the Results
Based on the decision, we conclude whether there is enough evidence to support
the alternative hypothesis or if we should keep the null hypothesis.

7 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
Resampling

Re-sampling is a statistical technique for gathering more data samples from which
inferences about the population or the process by which the initial data were
produced can be made. These methods are widely used in data analysis when it is
necessary to estimate a population parameter from the given data or when there are
few accessible data points. Resampling approaches typically use techniques like
bootstrapping, jackknifing, and permutation testing to estimate standard errors,
confidence intervals, and p-values.

There are two main types of resampling procedures: the bootstrap and
permutation tests

The Bootstrap

Bootstrap Method is a powerful statistical technique widely used in mathematics

for estimating the distribution of a statistic by resampling with replacement from
the original data.
Bootstrap Method or Bootstrapping is a statistical procedure that resamples
a single data set to create many simulated samples. This process allows for
the calculation of standard errors, confidence intervals, and hypothesis
testing,” according to a post on bootstrapping statistics from statistician Jim Frost.
Bootstrapping is a resampling technique used to estimate population statistics by
sampling from a dataset with replacement. It can be used to estimate summary
statistics such as the mean and standard deviation. It is used in applied machine
learning to estimate the quality of a machine learning model at predicting
data that is not included in the training data.
“Bootstrapping is a statistical procedure that resamples a
single data set to create many simulated samples.”

How Bootstrapping Statistics Works?

In the bootstrap method, a sample of size n is drawn from a population. We'll
call this sample S. Then, rather than using theory to determine all possible
estimates, a sampling distribution is created by resampling observations from
S with replacement m times, where each resampled set contains n observations.
With proper sampling, S will be representative of the population. Thus, by

8 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
resampling S m times with replacement, it is as if m samples were drawn from the
original population, and the derived estimates will represent the theoretical
distribution from the traditional approach.
Increasing the number of replicate samples m does not increase the information
content of the data; that is, resampling the original dataset 100,000 times is not as
useful as resampling it 1,000 times. The information content of a dataset depends
on the sample size n, which remains constant for each replicate sample. Thus, the
benefit of a larger number of replicate samples is that they provide a more accurate
estimate of the sampling distribution.

Conceptually, you can imagine the bootstrap as replicating the original sample
thousands or millions of times so that you have a hypothetical population that
embodies all the knowledge from your original sample (it’s just larger). You can
then draw samples from this hypothetical population for the purpose of estimating
a sampling distribution. See Figure 2-7.

In practice, it is not necessary to actually replicate the sample a huge number of

times. We simply replace each observation after each draw; that is, we sample with
replacement. In this way we effectively create an infinite population in which the
probability of an element being drawn remains unchanged from draw to draw. The
algorithm for a bootstrap resampling of the mean is as follows, for a sample of size
n:
1. Draw a sample value, record, replace it.
2. Repeat n times.
3. Record the mean of the n resampled values.
4. Repeat steps 1–3 R times.
5. Use the R results to:
a. Calculate their standard deviation (this estimates sample mean standard error).
b. Produce a histogram or boxplot.

9 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
c. Find a confidence interval.

R, the number of iterations of the bootstrap, is set somewhat arbitrarily. The more
iterations you do, the more accurate the estimate of the standard error, or the
confidence interval. The result from this procedure is a bootstrap set of sample
statistics or estimated model parameters, which you can then examine to see how
variable they are.
The R package boot combines these steps in one function. For example, the
following applies the bootstrap to the incomes of people taking out loans:

OR

Bootstrap Method
Bootstrap Method or Bootstrapping is a statistical technique for estimating an
entire population quantity by averaging estimates from multiple smaller data
samples. Importantly, the sample is created by extracting observations one at a
time from a larger data sample and adding them back to the selected data sample.
This allows a given observation to be included multiple times in a given smaller
sample. This sampling technique is called sampling with replacement.

10 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 Bootstrap Method

The process of creating a sample can be summarized as follows: Choose a

sample size. If the sample size is smaller than the size you selected Randomly
select observations from the dataset Add them to the sample Bootstrapping
methods can be used to estimate population abundance. This is done by repeatedly
taking small samples, computing statistics, and averaging the computed statistics.
The procedure can be summarized as follows:
 Choose the number of bootstrap samples to take
 Choose your sample size For each bootstrap sample, draw a replacement
sample of the size you selected
 Calculate the statistics for the samples Calculate the average of the
computed sample statistics

Bootstrap method is additionally a suitable for controlling and actually look

at the solidness of the outcomes. In spite of the fact that for most issues it is
difficult to know the genuine certainty span, bootstrap is asymptotically more exact
than the standard stretches got utilizing test change and presumptions of
ordinariness.
Example of samples created using Bootstrap method
Example of how bootstrap samples are created and used to estimate a
statistic of interest.
Solution:
11 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
 Let's say we have a small dataset of 5 observations:
Original Data: [3, 4, 5, 6, 7]

Create bootstrap samples by resampling with replacement:

We'll create 3 bootstrap samples of size 5 by randomly drawing observations from
the original data with replacement.

Each bootstrap sample will have the same size as the original dataset.
 Bootstrap Sample 1: [5, 6, 3, 4, 7]
 Bootstrap Sample 2: [4, 3, 6, 4, 6]
 Bootstrap Sample 3: [7, 5, 7, 3, 4]

Calculate the statistic of interest (median) for each bootstrap sample:

 Bootstrap Sample 1 median: 5
 Bootstrap Sample 2 median: 4
 Bootstrap Sample 3 median: 5

Repeat steps 1 and 2 many times (e.g., 10,000 times):

By repeating the process of creating bootstrap samples and calculating the
median, we can build an empirical sampling distribution of the median.
Use the empirical sampling distribution to calculate confidence intervals or
perform hypothesis tests:

For example, if we want to construct a 95% confidence interval for the median,
we can find the 2.5th and 97.5th percentiles of the empirical sampling distribution
of the median.

Let's say the 2.5th percentile is 4, and the 97.5th percentile is 6.

Then, the 95% confidence interval for the median would be [4, 6].

In this example, we used bootstrapping to estimate the median by resampling

from the original data multiple times and calculating the statistic of interest
(median) for each bootstrap sample. By repeating this process many times, we can
build an empirical sampling distribution of the median, which can be used to
construct confidence intervals or perform hypothesis tests without relying on
assumptions about the underlying population distribution.

12 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 Advantages of Bootstrap Method
Bootstrap method offers several key advantages that make it a valuable tool in
statistical analysis and mathematical research:
 Non-parametric Nature: The Bootstrap method does not rely on
assumptions about the underlying distribution of the data. This makes it
particularly useful when dealing with complex or unknown distributions,
allowing for more flexible and robust statistical analysis.
 Versatility: It can be applied to a wide range of statistical measures,
including means, medians, variances, and regression coefficients. This
versatility extends to various types of data, whether continuous, discrete, or
categorical.
 Accuracy in Small Samples: In cases where sample sizes are small,
traditional methods may not provide reliable estimates. The Bootstrap
method can improve the accuracy of these estimates by effectively
increasing the sample size through resampling.
 Simple Implementation: The Bootstrap method is straightforward to
implement using modern computational tools. It involves repeated
resampling and can be easily programmed, making it accessible for
researchers and analysts.
 Internal Validation: By generating multiple resampled datasets, the
Bootstrap method allows for internal validation of statistical models. This
helps in assessing the stability and reliability of the models without the need
for additional external data.
 Confidence Interval Estimation: The Bootstrap method is particularly
effective for constructing confidence intervals for various statistics. This
provides a clearer understanding of the precision and variability of the
estimates, which is crucial for decision-making and hypothesis testing.
 Handling Complex Data Structures: The Bootstrap method is capable of
dealing with complex data structures, such as time-series data or data with
hierarchical relationships. This adaptability makes it suitable for a broad
range of applications across different fields.
Limitations of Bootstrap Methods
Various limitations of Bootstrap Methods are:
 Time-Consuming: Accurate bootstrap requires thousands of simulated
samples.
 Computationally Intensive: Because bootstrap requires thousands of
samples and is time-consuming, it also requires more computing power.

13 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
  Sometimes Incompatible: Bootstrapping is not always the best solution for
your situation, especially when dealing with spatial data or time series.
 Prone to Bias: Bootstrapping does not always take into account the
variability of the distribution, which introduces errors and bias into your
calculations.

Permutation Test

In a permutation procedure, two or more samples are involved, typically the

groups in an A/B or other hypothesis test. Permute means to change the order of a
set of values. The first step in a permutation test of a hypothesis is to combine the
results from groups A and B (and, if used, C, D, …) together. This is the logical
embodiment of the null hypothesis that the treatments to which the groups were
exposed do not differ. We then test that hypothesis by randomly drawing groups
from this combined set, and seeing how much they differ from one another. The
permutation procedure is as follows:
1. Combine the results from the different groups in a single data set.
2. Shuffle the combined data, then randomly draw (without replacing) a
resample of the same size as group A.
3. From the remaining data, randomly draw (without replacing) a resample
of the same size as group B.
4. Do the same for groups C, D, and so on.
5. Whatever statistic or estimate was calculated for the original samples
(e.g., difference in group proportions), calculate it now for the
resamples, and record; this constitutes one permutation iteration.
6. Repeat the previous steps R times to yield a permutation distribution of
the test statistic.
Now go back to the observed difference between groups and compare it to the set
of permuted differences. If the observed difference lies well within the set of
permuted differences, then we have not proven anything — the observed
difference is within the range of what chance might produce. However, if the
observed difference lies outside most of the permutation distribution, then we
conclude that chance is not responsible. In technical terms, the difference is
statistically significant.

Advantages

14 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 No assumptions are made on the population distribution used in the

permutation test. They are flexible tools that work with a range of data types
and experimental layouts a consequence.
 Permutation testing is used to generate new samples from the original
dataset, which produces exact estimations of uncertainty and significance.
They are more reliable since they don't rely on assumptions about sample
size or population distribution like traditional tests do.
 Permutation testing can be applied to run a variety of statistical tests,
including t-tests, ANOVAs, and correlation analyses.
 Permutation tests are typically more successful than traditional tests in
situations when there is a limited sample size or a non-normal distribution of
the data.
Disadvantages
 When the sample size or the number of permutations is big, permutation
testing can be computationally demanding.
 Several types of data or experimental designs may not be suitable for
permutation testing, particularly if the data includes outliers or missing
values.
 It might be challenging to explain permutation testing to non-experts since it
can be less understandable than standard tests.

Statistical Significance and P-Values

Statistical significance is how statisticians measure whether an experiment (or even

a study of existing data) yields a result more extreme than what chance might
produce. If the result is beyond the realm of chance variation, it is said to be
statistically significant.
Consider in Table 3-2 the results of the web test shown earlier.

15 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
Price A converts almost 5% better than price B (0.8425% versus 0.8057% — a
difference of 0.0368 percentage points), big enough to be meaningful in a high
volume business. We have over 45,000 data points here, and it is tempting to
consider this as “big data,” not requiring tests of statistical significance (needed
mainly to account for sampling variability in small samples). However, the
conversion rates are so low (less than 1%) that the actual meaningful values — the
conversions — are only in the 100s, and the sample size needed is really
determined by these conversions. We can test whether the difference in conversions
between prices A and B is within the range of chance variation, using a resampling
procedure. By “chance variation,” we mean the random variation produced by a
probability model that embodies the null hypothesis that there is no difference
between the rates (see “The Null Hypothesis”). The following permutation
procedure asks “if the two prices share the same conversion rate, could chance
variation produce a difference as big as 5%?”

16 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
Reusing the function perm_fun defined in “Example: Web Stickiness”, we can
create a histogram of randomly permuted differences in conversion rate:

See the histogram of 1,000 resampled results in Figure 3-5: as it happens, in this
case the observed difference of 0.0368% is well within the range of chance
variation.

17 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
P-Value
The p-value in statistics quantifies the evidence against a null hypothesis. A low p-
value suggests data is inconsistent with the null, potentially favoring an alternative
hypothesis. Common significance thresholds are 0.05 or 0.01.

A p-value, or probability value, is a number describing how likely it is that your

data would have occurred by random chance (i.e., that the null hypothesis is true).

The level of statistical significance is often expressed as a p-value between 0 and 1.

The smaller the p-value, the less likely the results occurred by random chance, and
the stronger the evidence that you should reject the null hypothesis.

Remember, a p-value doesn’t tell you if the null hypothesis is true or false. It just
tells you how likely you’d see the data you observed (or more extreme data) if the
null hypothesis was true.

It’s a piece of evidence, not a definitive proof.

Example: Test Statistic and p-Value

Suppose you’re conducting a study to determine whether a new drug has an effect
on pain relief compared to a placebo.

18 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
If the new drug has no impact, your test statistic will be close to the one predicted
by the null hypothesis (no difference between the drug and placebo groups), and
the resulting p-value will be close to 1. It may not be precisely 1 because real-
world variations may exist.

Conversely, if the new drug indeed reduces pain significantly, your test statistic
will diverge further from what’s expected under the null hypothesis, and the p-
value will decrease.

The p-value will never reach zero because there’s always a slim possibility, though
highly improbable, that the observed results occurred by random chance.

P-value interpretation
The significance level (alpha) is a chosen threshold (often 0.05), while the p-value
is the probability calculated from your data.

A p-value less than or equal to your significance level (typically ≤ 0.05) is

statistically significant.

A p-value less than or equal to a predetermined significance level (often 0.05 or

0.01) indicates a statistically significant result, meaning the observed data provide
strong evidence against the null hypothesis.
This suggests the effect under study likely represents a real relationship rather than
just random chance.

For instance, if you set α = 0.05, you would reject the null hypothesis if your p-
value ≤ 0.05.

19 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
It indicates strong evidence against the null hypothesis, as there is less than a 5%
probability the null is correct (and the results are random).
Therefore, we reject the null hypothesis and accept the alternative hypothesis.

Significance of P-value
 The p-value provides a quantitative measure of the strength of the evidence
against the null hypothesis.
 Decision-Making in Hypothesis Testing
 P-value serves as a guide for interpreting the results of a statistical test. A
small p-value suggests that the observed effect or relationship is statistically
significant, but it does not necessarily mean that it is practically or clinically
meaningful.
Limitations of P-value
 The p-value is not a direct measure of the effect size, which represents the
magnitude of the observed relationship or difference between variables. A
small p-value does not necessarily mean that the effect size is large or
practically meaningful.
 Influenced by Various Factors

T-test
The t-test is part of Hypothesis testing where you start with an assumption the null
hypothesis that the two group means are the same. Then the test helps you decide if
20 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
there’s enough evidence to reject that assumption and conclude that the groups are
different.

Assumptions in T-test
 Independence: The observations within each group must be independent of
each other means that the value of one observation should not influence the
value of another observation.
 Normality: The data within each group should be approximately normally
distributed i.e., the data within each group being compared should resemble
a normal bell-shaped distribution.
 Homogeneity of Variances: The variances of the two groups being
compared should be equal. This assumption ensures that the groups have a
similar spread of values.
 Absence of Outliers: There should be no outliers in the data as outliers can
influence the results especially when sample sizes are small.
In R, the function is [Link]:

Types of T-tests
There are three types of t-tests and they are categorized as dependent and
independent t-tests.
21 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
One sample T-test
One sample t-test is used for comparison of the sample mean of the data to a
particularly given value. We can use this when the sample size is small. (under 30)
data is collected randomly and it is approximately normally distributed. It can be
calculated as:

Independent sample T-test

An Independent sample t-test commonly known as an unpaired sample t-test is
used to find out if the differences found between two groups is actually significant
or just a random occurrence. We can use this when:
 the population mean or standard deviation is unknown. (information about
the population is unknown)
 the two samples are separate/independent. For i.e. boys and girls (the two are
independent of each other)
It can be calculated using:
22 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
Paired Two-sample T-test
Paired sample t-test also known as dependent sample t-test is used to find out if the
difference in the mean of two samples is 0. The test is done on dependent samples
usually focusing on a particular group of people or things. In this each entity is
measured twice resulting in a pair of observations.
We can use this when:
 Two similar samples are given. [i.e. Scores obtained in English and Math
(both subjects)]
 The dependent variable data is continuous.
 The observations are independent of one another.
 The dependent variable is approximately normally distributed.
It can be calculated using

23 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
Multiple Testing
Multiple Testing in statistics deals with the problem of testing many hypotheses at
once. When performing multiple statistical tests, the chance of making a Type I
error (false positive) increases, meaning there's a higher probability that at least
one of the null hypotheses will be falsely rejected. There are various methods to
adjust for this "multiple comparisons problem" and control the overall error rate.

This issue is related to the problem of overfitting in data mining, or “fitting the
model to the noise.” The more variables you add, or the more models you run, the
greater the probability that something will emerge as “significant” just by chance.
In supervised learning tasks, a holdout set where models are assessed on data that
the model has not seen before mitigates this risk. In statistical and machine
learning tasks not involving a labeled holdout set, the risk of reaching conclusions
based on statistical noise persists.

In statistics, there are some procedures intended to deal with this problem in very
specific circumstances. For example, if you are comparing results across multiple
treatment groups you might ask multiple questions. So, for treatments A–C, you
might ask:
 Is A different from B?
 Is B different from C?
 Is A different from C?
Or, in a clinical trial, you might want to look at results from a therapy at multiple
stages. In each case, you are asking multiple questions, and with each question,
you are increasing the chance of being fooled by chance. Adjustment procedures
in statistics can compensate for this by setting the bar for statistical significance
more stringently than it would be set for a single hypothesis test. These adjustment
24 by Vidyashree.R (Assistant prof. POP)
CSE(DS)
procedures typically involve “dividing up the alpha” according to the number of
tests. This results in a smaller alpha (i.e., a more stringent bar for statistical
significance) for each test. One such procedure, the Bonferroni adjustment, simply
divides the alpha by the number of observations n.

However, the problem of multiple comparisons goes beyond these highly

structured cases and is related to the phenomenon of repeated data “dredging” that
gives rise to the saying about torturing the data. Put another way, given sufficiently
complex data, if you haven’t found something interesting, you simply haven’t
looked long and hard enough. More data is available now than ever before, and
the number of journal articles published nearly doubled between 2002 and 2010.
This gives rise to lots of opportunities to find something interesting in the data,
including multiplicity issues such as:
 Checking for multiple pairwise differences across groups
 Looking at multiple subgroup results (“we found no significant treatment
effect overall, but we did find an effect for unmarried women younger than
30”)
 Trying lots of statistical models
 Including lots of variables in models
 Asking a number of different questions (i.e., different possible outcomes)

25 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 Degrees of Freedom

What are Degrees of Freedom?

Degrees of Freedom is defined as the maximum number of independent values
that can vary in a sample space. The degree of freedom is generally calculated
when we subtract one from the given sample of data. Degrees of freedom are
very helpful for ensuring the validity of chi-square tests, t-tests, high f-tests, and
others.
These tests are very helpful for deducing the validity of any hypothesis. Degree
of Freedom helps us to make various crucial decisions.

Degrees of Freedom Definition

We define Degree of Freedom as choices that any given sample of data has.
Suppose you are given to choose an option from four different options and
you have to choose one option then the degree of freedom in this case is 3.

Degrees of Freedom Formula

Formulas to Calculate Degrees of Freedom with respect to the number of
samples,
Degree of freedom formula is shown in the image shown below. In the image,
N is total number of options available to us then degree of freedom is N - 1.

dF = n – 1
One-Sample T-Test Formula

dF = n1 + n2 − 2
Two-Sample T-Test Formula

Simple Linear Regression Formula dF = n – 2

26 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 dF = k – 1
Chi-Square Goodness of Fit Test Formula

dF = (r − 1)(c − 1)
Chi-Square Test for Homogeneity Formula
Degrees of Freedom are often discussed in relation to various methods of
hypothesis testing in mathematics, such as chi-square. It is important to
calculate degrees of freedom when trying to understand the importance of the
chi-square arithmetic and the validity of the null hypothesis.
Degree freedom formula is equal to the size of a sample of data minus one.

Degrees of Freedom Explaination

We can understand the meaning and concept of the degree of freedom by
going through the following example.
Suppose we take four marbles that are to be distributed between four
children A, B, C, and D in the same order respectively. Now the degrees of
freedom they have is explained as,
 For A we have all four choices of marbles available then the degrees of
freedom for A is 4-1 = 3
 For B we have all three choices of marbles available then the degrees of
freedom for B is 3-1 = 2
 For C we have all two choices of marbles available then the degrees of
freedom for C is 2-1 = 1
 For D we have all only one choice of marbles available then the degrees of
freedom for D is 1-1 = 0
Degrees of freedom of the given sample of data is given using the formula,

27 by Vidyashree.R (Assistant prof. POP)

CSE(DS)
 Degrees of Freedom Examples
Example 1: Choosing Meals
Degree of Freedom can easily be understood with the help of the following
example. Suppose you have three packets of A, B, and C of food to eat in a day.
 For Breakfast: You can have any of the three packets (A, B, and C), but you
choose to eat packet A.
 For Lunch: As, you have only two packets left (B, C) you have two choices
and you choose B.
 For Dinner: You have only one packet left i.e. C so you are forced to eat
that one only.
So, we have two levels of freedom to choose our food for breakfast in a day, so
the degree of freedom in this case is two, but for lunch, we have one freedom so
the degree of freedom in this case is one and for dinner, we have no choices so
the degree of freedom is one.
Example 2: Assigning Seats
Suppose you have four friends and four seats labeled 1, 2, 3, and 4 to assign at a
dinner table.
 For the First Friend: You can choose any of the four seats (1, 2, 3, or 4), so
you have 4 choices.
 For the Second Friend: Three seats remain, so you have 3 choices.
 For the Third Friend: Two seats remain, so you have 2 choices.
 For the Fourth Friend: Only one seat is left, so you have 1 choice.

28 by Vidyashree.R (Assistant prof. POP)

CSE(DS)