Statistical Machine Learning for Data Science (BAD702)

MODULE-3
Statistical Experiments and Significance Testing
INTRODUCTION

 In statistics, our main aim is not just to describe data, but to make inferences – that is, to
draw conclusions about a larger population from a smaller sample of data.

 To make such conclusions, researchers design statistical experiments. These experiments are
structured in such a way that they can test whether observed outcomes are due to real effects
(like a new drug working better) or whether they happened simply by random chance.

 Once an experiment is conducted, we need tools to interpret the results. This is where
significance testing (also called hypothesis testing) comes in. It is a systematic way of deciding
whether an observed effect is likely real or could be explained by randomness.

What is a Statistical Experiment?

A statistical experiment is a carefully planned investigation in which we collect data under

controlled conditions.

Key Elements of an Experiment:

1. Subjects (experimental units):

o The items or individuals under study.
o Example: patients in a medical trial, seeds in an agricultural study, or website
visitors in an online A/B test.
2. Treatments (conditions applied):
o The interventions or changes applied to subjects.
o Example: Drug A vs Drug B, Fertilizer X vs Fertilizer Y, Webpage design A vs
B.
3. Outcomes (response variables):
o The measurable results of interest.
o Example: recovery rate, crop yield, conversion rate, session time.

Example 1: In a clinical trial, patients (subjects) are given two different drugs (treatments), and
their blood pressure reduction is measured (outcome).
Example 2: In e-commerce, website visitors (subjects) are shown two versions of a webpage
(treatments A and B), and the purchase decision (outcome: buy or not buy) is recorded.

Department of AI & DS, NCE, Hassan Page 1

Statistical Machine Learning for Data Science (BAD702)

Purpose of Statistical Experiments is

 To test a hypothesis: for example, "Drug A works better than Drug B".
 To compare treatments: Is price A better than price B in terms of sales?
 To establish causality: If differences are found, can we confidently say they are caused
by the treatment, not by other external factors?
 To control variability: By randomization and design, we minimize the effect of
confounding variables (other influences).

Why Do We Need Significance Testing?

Even when we observe differences between groups, random chance can create apparent
differences. For example:

 If we flip a fair coin 10 times, we might get 7 heads and 3 tails. Does that mean the coin
is biased? Not necessarily — such results can occur by chance.
 Similarly, if Group A (Drug A) shows better recovery than Group B (Drug B), is it due to
the drug or just random variability?

Significance testing provides a structured way to answer:

 Is the observed difference statistically significant (unlikely to be due to chance)?

 Or is it statistically insignificant (likely explainable by randomness)?

The Classical Statistical Inference Pipeline

The traditional process of statistical inference consists of the following steps:

1. Formulate a hypothesis
o Example: "Price A is better than Price B."
2. Design the experiment
o Randomly assign subjects to treatment and control groups.
3. Collect the data
o Record observed outcomes (conversions, recovery, clicks, etc.).
4. Analyze the data
o Use appropriate statistical tests (t-test, chi-square, resampling, etc.).

Department of AI & DS, NCE, Hassan Page 2

Statistical Machine Learning for Data Science (BAD702)

5. Draw conclusions
o Decide whether the results are statistically significant and interpret them in real-
world context.

Example: A/B Testing in Practice

 Suppose we want to test two webpage designs:

o Group A (control): sees the current webpage.
o Group B (treatment): sees a new webpage design.
 Outcome measured: % of visitors who click “Buy Now”.
 After running the experiment:
o Group A: 200 purchases out of 23,739 visitors.
o Group B: 182 purchases out of 22,588 visitors.
 Group A looks slightly better, but is this difference real or just random noise?
→ This is where significance testing will help.

Importance of Randomization and Control

 Randomization: Randomly assigning subjects ensures that the groups are comparable
and results are not biased.
 Control group: Acts as a baseline. Without it, we cannot say for sure whether the
observed change is due to treatment or external factors.

📌 Example: If we only try a new drug on one group and compare it to past records, many other
factors (environment, diet, patient demographics) might differ. Randomized control avoids this
issue.

 Statistical experiments = Designing structured investigations.

 Significance testing = Deciding if observed effects are real or chance.

A/B TESTING
 A/B testing is one of the most widely used types of statistical experiments, especially in
business, marketing, medicine, and web analytics. The basic idea is simple: we compare
two groups (A and B) to determine which of two treatments, options, or strategies works
better.
 Group A (Control): This group receives the standard treatment, the existing version, or
no treatment at all.
 Group B (Treatment): This group receives the new option or the experimental treatment
being tested.

Department of AI & DS, NCE, Hassan Page 3

Statistical Machine Learning for Data Science (BAD702)

The goal of A/B testing is to measure whether there is a significant difference in outcomes
between the two groups, and if so, to decide which option should be chosen going forward.

 A/B testing (also called a two-sample experiment or two-group comparison) is one of

the simplest and most powerful tools in statistics for decision-making.
 The purpose of an A/B test is to determine which option performs better on a specific
outcome (called the response variable).
 The method is heavily used in scientific research, marketing, product design, and web
analytics, because it provides direct evidence for choosing between alternatives.

A/B tests are common in web design and marketing, since results are so readily measured. Some
examples of A/B testing include:
 Testing two soil treatments to determine which produces better seed germination
 Testing two therapies to determine which suppresses cancer more effectively
 Testing two prices to determine which yields more net profit
 Testing two web headlines to determine which produces more clicks (Figure 3-2)
 Testing two web ads to determine which generates more conversions

Department of AI & DS, NCE, Hassan Page 4

Statistical Machine Learning for Data Science (BAD702)

A proper A/B test has subjects that can be assigned to one treatment or another. The subject
might be a person, a plant seed, a web visitor; the key is that the subject is exposed to the
treatment. Ideally, subjects are randomized (assigned randomly) to treatments. In this way, you
know that any difference between the treatment groups is due to one of two things:
• The effect of the different treatments
• Luck of the draw in which subjects are assigned to which treatments (i.e., the random
assignment may have resulted in the naturally better-performing subjects being concentrated in
A or B)
You also need to pay attention to the test statistic or metric you use to compare group A to group
B. Perhaps the most common metric in data science is a binary variable: click or no-click, buy or
don’t buy, fraud or no fraud, and so on. Those results would be summed up in a 2×2 table. Table
3-1 is a 2×2 table for an actual price test.

If the metric is a continuous variable (purchase amount, profit, etc.) or a count (e.g., days in
hospital, pages visited), the result might be displayed differently. If one were interested not in
conversion but in revenue per page view, the results of the price test in Table 3-1 might look like
this in typical default software output:
Department of AI & DS, NCE, Hassan Page 5
Statistical Machine Learning for Data Science (BAD702)

Revenue/page view with price A: mean = 3.87, SD = 51.10

Revenue/page view with price B: mean = 4.11, SD = 62.98
“SD” refers to the standard deviation of the values within each group.

Real-World Examples of A/B Testing

 Web & Marketing:
o Testing two headlines on a webpage to see which one attracts more clicks.
o Comparing two email subject lines to see which has a higher open rate.
o Comparing two prices to see which yields higher revenue.
 Medical Trials:
o Group A receives the existing standard drug, Group B receives a new
experimental drug.
o Outcome: Which treatment leads to higher recovery rates?
 Agriculture:
o Group A seeds treated with fertilizer type X, Group B with fertilizer type Y.
o Outcome: Which fertilizer leads to higher crop yield?

Key Elements of A/B Testing

1. Treatment (variable being tested):
o This is the intervention applied to subjects.
o Example: new drug, new web design, new fertilizer.
2. Treatment Group (Group B):
o Subjects exposed to the new option.
3. Control Group (Group A):
o Subjects exposed to the standard or no treatment.
o Provides a baseline for comparison.
4. Subjects (experimental units):
o The individuals or items being tested (patients, visitors, seeds, etc.).
5. Randomization:
o Subjects must be assigned randomly to groups.
o This ensures fairness and eliminates systematic bias.
6. Test Statistic:
o The numerical measure used to compare groups.
o Examples: conversion rate, average time on page, mean crop yield.

Why is Randomization Important?

Department of AI & DS, NCE, Hassan Page 6

Statistical Machine Learning for Data Science (BAD702)

Randomization ensures that differences between groups are caused by the treatment itself and
not by other hidden factors.

Example:

 If younger patients are placed in Group A and older patients in Group B, the observed
difference may be due to age, not treatment.
 Randomization distributes such variables evenly, making the comparison fair and
unbiased.

Why have a Control Group?

 Without a control group, there is no assurance that “all other things are equal” and that
any difference is really due to the treatment (or to chance). When you have a control
group, it is subject to the same conditions (except for the treatment of interest) as the
treatment group. If you simply make a comparison to “baseline” or prior experience,
other factors, besides the treatment, might differ.

 Without a control group, you might mistakenly conclude that a treatment works when the
change is due to external conditions.
 Example:
o Imagine a company launches a new website design and sees a sales increase.
o But what if the increase happened because it was holiday season, not because of
the design?
 A control group exposed to the old design at the same time would account for seasonal
effects, allowing a fair comparison.

Blinding in A/B Testing (Especially in Medicine)

 Single-blind study: Subjects do not know whether they are receiving treatment A or B.
 Double-blind study: Neither subjects nor investigators know which treatment is given.
 Blinding avoids psychological and researcher bias.

Example: If patients know they are receiving the new drug, they may feel better simply because
of expectations (placebo effect).

Measuring Outcomes in A/B Tests

 The choice of metric is crucial.
 Common metrics include:

Department of AI & DS, NCE, Hassan Page 7

Statistical Machine Learning for Data Science (BAD702)

o Binary outcomes: click/no-click, buy/no-buy, success/failure.

o Continuous outcomes: revenue, time spent, blood pressure reduction.
 The metric should be decided before the experiment starts to avoid bias in interpreting
results.

Example: In a website test, if the primary goal is sales, then conversion rate should be the
metric. Looking at multiple metrics after the test (like clicks, time, engagement) may lead to
biased conclusions.

Example of A/B Testing in Action

Let’s say an e-commerce company wants to test two product prices.

 Group A (Price A): 23,739 visitors → 200 purchases.

 Group B (Price B): 22,588 visitors → 182 purchases.
 Conversion rates:
o Price A = 200 / 23,739 ≈ 0.8425%
o Price B = 182 / 22,588 ≈ 0.8057%
 At first glance, Price A seems better.
 But is this difference real? Or could it be random chance due to sampling?
o Significance testing (hypothesis tests, p-values, etc.) will answer this.

Strengths and Limitations of A/B Testing

Strengths:

 Easy to understand and implement.

 Provides clear evidence for decision-making.
 Applicable across fields: medicine, marketing, agriculture, social sciences.
 Flexible: works with both binary outcomes (buy/not buy) and continuous outcomes (time
spent, amount purchased).

Limitations:

 Requires careful planning (choice of metric, randomization, sample size).

 Small samples may produce misleading results due to randomness.
 Large samples may produce statistical significance even for tiny, practically
meaningless differences.
 Only compares two options at a time — if many alternatives exist, the process becomes
complex.

Department of AI & DS, NCE, Hassan Page 8

Statistical Machine Learning for Data Science (BAD702)

Why just A/B? Why Not C, D, …?

 A/B tests are popular in the marketing and ecommerce worlds, but are far from the only
type of statistical experiment. Additional treatments can be included. Subjects might have
repeated measurements taken. Pharmaceutical trials where subjects are scarce, expensive,
and acquired over time are sometimes designed with multiple opportunities to stop the
experiment and reach a conclusion.
 Traditional statistical experimental designs focus on answering a static question about the
efficacy of specified treatments. Data scientists are less interested in the question:
Is the difference between price A and price B statistically significant?
than in the question:
Which, out of multiple possible prices, is best?
For this, a relatively new type of experimental design is used: the multi-arm bandit.
 While A/B testing involves just two groups, experiments can be expanded to include
multiple treatments (A/B/C/D).
 Example: Testing 4 different versions of a webpage simultaneously.
 However, more groups increase complexity and sample size requirements.
 In modern data science, advanced techniques like the multi-armed bandit algorithm are
used to balance exploration (trying new options) and exploitation (using the best option).

A/B testing is a fundamental experimental design technique that helps us make decisions
based on data rather than guesswork. By comparing a control group (A) with a treatment group
(B), and using proper randomization and statistical testing, we can determine whether observed
differences are real or just due to chance. It is the foundation of modern data-driven decision
making in both scientific research and business applications.

HYPOTHESIS TESTING

Hypothesis tests, also called significance tests, are ubiquitous in the traditional statistical analysis
of published research. Their purpose is to help you learn whether random chance might be
responsible for an observed effect.

Department of AI & DS, NCE, Hassan Page 9

Statistical Machine Learning for Data Science (BAD702)

Why Do We Need a Hypothesis in A/B Testing?

 The answer lies in the tendency of the human mind to underestimate the scope of natural
random behavior. One manifestation of this is the failure to anticipate extreme events, or
so-called “black swans”.
 Another manifestation is the tendency to misinterpret random events as having patterns of
some significance. Statistical hypothesis testing was invented as a way to protect
researchers from being fooled by random chance.
 In a properly designed A/B test, you collect data on treatments A and B in such a way
that any observed difference between A and B must be due to either:
o Random chance in assignment of subjects
o A true difference between A and B
 A statistical hypothesis test is further analysis of an A/B test, or any randomized
experiment, to assess whether random chance is a reasonable explanation for the
observed difference between groups A and B.

The Role of a Hypothesis in Experiments

 An A/B test is always built around a specific hypothesis.
o Example: “Price B produces higher profit than Price A.”
 Without a hypothesis, one might be tempted to simply look at the raw outcomes and
choose whichever option performed better in the experiment.
 But the danger is: outcomes can be misleading due to random variation.
 Hypotheses serve as a protective framework. They allow us to compare results not just
at face value, but against what we would expect if chance alone were operating.

Why Not Just Pick the Better Outcome?

Department of AI & DS, NCE, Hassan Page 10
Statistical Machine Learning for Data Science (BAD702)

At first glance, it seems logical to just pick the treatment that performs better in the test. But
there are two major problems with this approach:

1. Randomness Creates Illusions

o In small samples, random chance can create differences that look real.
o Example: Flip a fair coin 10 times → you might get 7 heads and 3 tails. Does this
mean the coin is biased? No — random chance can produce such “imbalances.”
2. Humans Misinterpret Randomness
o The human brain is wired to see patterns even in noise.
o We are quick to believe “Treatment B is better” when the difference may simply
be random fluctuation.
o This is why a formal hypothesis test is needed — to mathematically evaluate
whether the difference is likely real or just due to chance.

Cognitive Biases in Understanding Randomness

(a) Failure to Anticipate Extreme Events (Black Swans)

 Sometimes rare, extreme events occur — events that seem impossible but are part of
natural randomness.
 Example: A startup runs an A/B test on 100 customers, and one customer buys a huge
order that skews results in favor of one version.
 Without hypothesis testing, one might incorrectly conclude that the new version is
superior.

(b) Misinterpreting Random Events as Patterns

 Humans naturally see order in randomness.

 Example: A stock price goes up 5 days in a row. We might think there’s a trend, but it
could easily be random.
 In experiments, small fluctuations between groups A and B may tempt us to declare a
“winner,” even if no real difference exists.

Hypothesis testing was invented to protect us from these traps — it forces us to measure how
compatible our results are with pure chance.

Hypothesis Testing as a Safeguard

 Statistical hypothesis testing acts as a filter between observed outcomes and final
conclusions.
 It asks: “If there was really no difference, how likely would I see results like these?”

Department of AI & DS, NCE, Hassan Page 11

Statistical Machine Learning for Data Science (BAD702)

 If such results are very unlikely under chance alone → we reject the null hypothesis and
accept the alternative.
 If the results are consistent with chance → we avoid making false claims.

In short:

 Without hypothesis testing: We risk being fooled by randomness.

 With hypothesis testing: We require evidence strong enough to rule out chance as a
reasonable explanation.

Practical Example
Imagine two webpage designs tested on 1000 visitors each:

 Version A: 52 purchases (5.2% conversion).

 Version B: 57 purchases (5.7% conversion).

At first glance, Version B looks better. But is that 0.5% difference real, or just luck of the
draw?

 If we run a hypothesis test and the p-value = 0.42, then such a difference (or larger)
would occur 42% of the time due to chance.
 Conclusion: This difference is not statistically significant — we should not switch
designs based on this result.

The Null Hypothesis

The null hypothesis (H₀) is a fundamental concept in inferential statistics used to make
decisions or draw conclusions about populations based on sample data. It represents a default or
baseline assumption that there is no effect, no difference, or no relationship between
variables being studied. In hypothesis testing, researchers start by assuming the null hypothesis is
true, and then use statistical evidence from data to test whether this assumption should be
rejected.

Hypothesis testing operates on a key principle:

Humans tend to interpret random or unusual events as meaningful patterns. Therefore, to guard
against false conclusions, statistical tests demand strong evidence before rejecting the
assumption of “no real effect.”

This means that:

Department of AI & DS, NCE, Hassan Page 12

Statistical Machine Learning for Data Science (BAD702)

 Any observed difference between groups (e.g., Group A vs. Group B) is initially
assumed to be caused by random variation or chance.
 Only when the observed difference is too extreme to be reasonably attributed to
chance do we reject the null hypothesis.

Definition:
The null hypothesis (H₀) states that there is no significant difference between two or more
groups, conditions, or populations being compared. This involves a baseline assumption that the
treatments are equivalent, and any difference between the groups is due to chance. This baseline
assumption is termed the null hypothesis.
Formally, it can be expressed as:

 H0: There is no difference or relationship between variables.

Examples:

 In comparing two means: H0:μ1=μ2

 In comparing proportions: H0:p1=p2
 In regression: H0:β=0 (no linear relationship)

Thus, the null hypothesis assumes equivalence between treatments, interventions, or groups.

Role of the Null Hypothesis in Experiments

 The null hypothesis provides the starting point or reference model for testing.
 It assumes no real effect of treatments or variables, implying that observed differences
are due purely to random sampling error.
 Researchers collect sample data and use statistical tests (e.g., t-test, ANOVA, chi-square
test) to evaluate whether the evidence is strong enough to reject this assumption.

Alternative Hypothesis
Hypothesis tests by their nature involve not just a null hypothesis but also an offsetting
alternative hypothesis. Here are some examples:
 Null = “no difference between the means of group A and group B”;
alternative =“A is different from B” (could be bigger or smaller)
 Null = “A ≤ B”; alternative = “A > B”
 Null = “B is not X% greater than A”; alternative = “B is X% greater than A”
Taken together, the null and alternative hypotheses must account for all possibilities. The nature
of the null hypothesis determines the structure of the hypothesis test.

The alternative hypothesis (H₁ or Ha) represents what the researcher aims to prove — that
there is a real difference or effect.

Department of AI & DS, NCE, Hassan Page 13

Statistical Machine Learning for Data Science (BAD702)

Hypothesis Type Description Example

Null (H₀) No difference/effect 𝜇1 = 𝜇2
Alternative (H₁) Difference/effect exists 𝜇1 ≠𝜇2

One-Way Versus Two-Way Hypothesis Tests

One-tailed or two-tailed — depends on the direction of the expected outcome.

One-tailed (One-way) Hypothesis Test

Definition:

A one-tailed test is used when the research question is directional, i.e., we only care about
whether B is better (or worse) than A, not both.

Key Characteristics:

 Considers extreme values in only one direction of the probability distribution.

 p-value is computed using only one tail of the distribution curve.
 Provides more statistical power to detect an effect in the specified direction.

Example in A/B Testing:

 Scenario: You will switch to the new design (B) only if it performs better than A.
 Hypotheses:

Interpretation:

 Only the upper tail (right side) of the sampling distribution is considered.
 If the test statistic falls into this extreme region, H₀ is rejected, and B is accepted as
better.

Two-tailed (Two-way) Hypothesis Test

Department of AI & DS, NCE, Hassan Page 14

Statistical Machine Learning for Data Science (BAD702)

Definition:

A two-tailed test is used when we are interested in any difference between A and B — whether
B is better or worse than A.

Key Characteristics:

 Considers both tails of the sampling distribution.

 p-value accounts for extreme outcomes in either direction.
 More conservative (harder to reject H₀) but avoids directional bias.

Example in A/B Testing:

 Scenario: You want to know if B is different from A, regardless of direction.

 Hypotheses:

Interpretation:

 Both the upper and lower tails are evaluated.

 If the test statistic is extreme in either direction, H₀ is rejected.

Directional vs. Non-directional Hypotheses

Choosing Between One-tailed and Two-tailed Tests

Basis One-tailed Test Two-tailed Test
Research Goal Test if B is better than A Test if B is different from A
Ignores difference in the opposite Protects against errors in both
Risk Consideration
direction directions
More powerful for detecting
Power Less powerful, more conservative
effects in one direction

Department of AI & DS, NCE, Hassan Page 15

Statistical Machine Learning for Data Science (BAD702)

Decision-based testing (A/B tests Exploratory research, scientific

Use Case
with default option) validation
p-value region Only one extreme tail Both tails of the distribution

Summary

Concept One-tailed Test Two-tailed Test

Alternative
Directional (e.g., B > A) Non-directional (e.g., B ≠ A)
Hypothesis
Tail(s) Considered One tail of distribution Both tails of distribution
When only one outcome (better When either difference is
When to Use
or worse) matters meaningful
Reject H₀ if result lies in one Reject H₀ if result lies in either
Decision Rule
extreme tail extreme tail
General scientific or academic
Common in A/B testing with a default option
research

RESAMPLING
Resampling in statistics means to repeatedly sample values from observed data, with a general
goal of assessing random variability in a statistic. It can also be used to assess and improve the
accuracy of some machine-learning models.
There are two main types of resampling procedures: the bootstrap and permutation tests. The
bootstrap is used to assess the reliability of an estimate (discussed in previous Module-2).
Permutation tests are used to test hypotheses, typically involving two or more groups.

Department of AI & DS, NCE, Hassan Page 16

Statistical Machine Learning for Data Science (BAD702)

Permutation Test
A Permutation Test (also known as a Randomization Test or Exact Test) is a non-
parametric statistical method used to test hypotheses about differences between two or more
groups. It is often applied in A/B testing, experimental design, or comparative studies where
we want to determine if observed differences between groups are statistically significant or
merely due to random chance.

 The term “permute” means to rearrange or change the order of data values.
 In a permutation test, data from all groups (e.g., A and B) are combined and shuffled to
simulate the situation under the null hypothesis (H₀) — that there is no real difference
between the groups.
 The test then measures how likely the observed difference between groups could occur
purely by random reallocation of the data.

Under the null hypothesis, we assume:

The treatments or conditions applied to different groups (A, B, C, etc.) do not have any real
effect — any observed differences are due to random variation.

Therefore:

 By mixing (combining) all observations together, we effectively create a dataset that

reflects what the world would look like if H₀ were true.
 By repeatedly reshuffling (permuting) the data and recalculating the test statistic, we
can determine how often we would see a difference as large as (or larger than) the
observed one just by chance.

Steps in the Permutation Procedure

1. Combine the results from the different groups into a single data set.
2. Shuffle the combined data and then randomly draw (without replacement) a resample of the
same size as group A (clearly it will contain some data from the other groups).
3. From the remaining data, randomly draw (without replacement) a resample of the same size as
group B.
4. Do the same for groups C, D, and so on. You have now collected one set of resamples that
mirror the sizes of the original samples.
5. Whatever statistic or estimate was calculated for the original samples (e.g., difference in group
proportions), calculate it now for the resamples, and record; this constitutes one permutation
iteration.
6. Repeat the previous steps R times to yield a permutation distribution of the test statistic.

Department of AI & DS, NCE, Hassan Page 17

Statistical Machine Learning for Data Science (BAD702)

Interpreting the results:

 Compare the observed statistic (from the actual, unshuffled data) to the permutation
distribution.

 The p-value is estimated as the proportion of permutation results that are as extreme
or more extreme than the observed value.

If this p-value is small (e.g., < 0.05), we reject the null hypothesis, concluding that the
observed difference is unlikely to be due to chance alone.

Advantages of the Permutation Test

1. Non-parametric method: It does not rely on assumptions about the data’s underlying
distribution (e.g., normality). This makes it suitable for datasets that violate parametric
test assumptions.
2. Applicable to small samples: The test can provide exact significance levels for small
datasets because all possible rearrangements can be computed.
3. High flexibility: The permutation framework can be applied to a wide range of statistics
— such as means, medians, correlations, or regression coefficients.
4. Conceptually intuitive: The logic of reshuffling data to simulate the null hypothesis is
easy to understand and directly tied to the data itself.
5. Robustness: Because it depends only on the observed data and random rearrangements,
it remains reliable even when data contain outliers or are not normally distributed.
6. No reliance on asymptotic theory: The test works accurately even for small sample
sizes without needing large-sample approximations.

Limitations of the Permutation Test

1. Computationally intensive: For large datasets or many groups, performing thousands of

permutations can be time-consuming and computationally demanding.
2. Random variation in estimated p-values: Since the test uses random sampling for
permutations, the p-value may vary slightly across runs unless a large number of
iterations are performed.
3. Exchangeability assumption: The test assumes that data points are exchangeable under
the null hypothesis — meaning that observations can be freely swapped among groups.

Department of AI & DS, NCE, Hassan Page 18

Statistical Machine Learning for Data Science (BAD702)

Violating this condition (e.g., in time-series or dependent data) can lead to incorrect
results.
4. Limited interpretability for complex models: When applied to regression or
multivariate models, the interpretation of permuted results can become more complicated.
5. Not ideal for extremely large data: While modern computing power helps, the method
may still be impractical for very large datasets with millions of observations.

Example: Web Stickiness

Objective of the Experiment
The goal is to determine which of two web page presentations — Page A and Page B — leads to
higher user engagement for a high-value service. Since sales are infrequent and take time to
complete, the company uses a proxy variable to measure performance.

A company selling a relatively high-value service wants to test which of two web presentations
does a better selling job. Due to the high value of the service being sold, sales are infrequent and
the sales cycle is lengthy; it would take too long to accumulate enough sales to know which
presentation is superior. So the company decides to measure the results with a proxy variable,
using the detailed interior page that describes the service.

A proxy variable is a measurable variable that substitutes for a true variable of interest when the
latter is:

 Hard to measure,
 Time-consuming, or
 Too costly to obtain.

In this experiment:

 True variable: Actual sales conversions

 Proxy variable: Average session time (time spent on the detailed page)

Longer session time implies higher engagement and possibly higher conversion likelihood.

One potential proxy variable for our company is the number of clicks on the detailed landing
page. A better one is how long people spend on the page. It is reasonable to think that a web
presentation (page) that holds people’s attention longer will lead to more sales. Hence, our metric
is average session time, comparing page A to page B. Google Analytics is used to measure
session time. Instead of deleting that session from the data, though, Google Analytics records it
as a zero, so the data requires additional processing to remove those sessions. The result is a total

Department of AI & DS, NCE, Hassan Page 19

Statistical Machine Learning for Data Science (BAD702)

of 36 sessions for the two different presentations, 21 for page A and 15 for page B. Using ggplot,
we can visually compare the session times using side-by-side boxplots:

R Code:
ggplot(session_times, aes(x=Page, y=Time)) + geom_boxplot()

Python Code:
ax = session_times.boxplot(by='Page', column='Time')
ax.set_xlabel('')
ax.set_ylabel('Time (in seconds)')
[Link]('')

Page B shows a higher median and overall longer session time than Page A, suggesting that Page
B may be more engaging.

Compute Mean Session Times

R Code:
mean_a <- mean(session_times[session_times['Page'] == 'Page A', 'Time'])
mean_b <- mean(session_times[session_times['Page'] == 'Page B', 'Time'])
mean_b - mean_a
# [1] 35.66667

Python Code:
mean_a = session_times[session_times.Page == 'Page A'].[Link]()
mean_b = session_times[session_times.Page == 'Page B'].[Link]()
mean_b - mean_a

Result:
Page B’s sessions are 35.67 seconds longer on average.

Department of AI & DS, NCE, Hassan Page 20

Statistical Machine Learning for Data Science (BAD702)

Is this observed difference (35.67 seconds) statistically significant — i.e., could it have
occurred by random chance?

To test this, a Permutation Test is applied.

Permutation Test Procedure

A permutation test involves repeatedly shuffling the data to simulate what differences might
arise if the null hypothesis (no difference between A and B) were true.

R Code (Permutation Function):

perm_fun <- function(x, nA, nB) {
n <- nA + nB
idx_b <- sample(1:n, nB)
idx_a <- setdiff(1:n, idx_b)
mean_diff <- mean(x[idx_b]) - mean(x[idx_a])
return(mean_diff)
}

Python Code:
def perm_fun(x, nA, nB):
n = nA + nB
idx_B = set([Link](range(n), nB))
idx_A = set(range(n)) - idx_B
return [Link][idx_B].mean() - [Link][idx_A].mean()

Explanation:

 Randomly assign session times to “Page A” and “Page B.”

 Compute the mean difference between the two random groups.
 Repeat the process 1,000 times to form a permutation distribution of differences.

Generating the Permutation Distribution

This function works by sampling (without replacement) nB indices and assigning them to the B
group; the remaining nA indices are assigned to group A. The difference between the two means
is returned. Calling this function R = 1,000 times and specifying nA = 21 and nB = 15 leads to a
distribution of differences in the session times that can be plotted as a histogram. In R this is
done as follows using the hist function:

R Code:
perm_diffs <- rep(0, 1000)
for (i in 1:1000) {
perm_diffs[i] = perm_fun(session_times[, 'Time'], 21, 15)
}
hist(perm_diffs, xlab='Session time differences (in seconds)')
abline(v=mean_b - mean_a)

Department of AI & DS, NCE, Hassan Page 21

Statistical Machine Learning for Data Science (BAD702)

Python Code:
perm_diffs = [perm_fun(session_times.Time, nA, nB) for _ in range(1000)]

fig, ax = [Link](figsize=(5, 5))

[Link](perm_diffs, bins=11, rwidth=0.9)
[Link](x = mean_b - mean_a, color='black', lw=2)
[Link](50, 190, 'Observed\ndifference', bbox={'facecolor':'white'})
ax.set_xlabel('Session time differences (in seconds)')
ax.set_ylabel('Frequency')

Figure 3-4 shows that mean difference of random permutations often exceeds the observed
difference in session times (the vertical line). For our results, this happens in 12.6% of the cases:

Compute p-Value
mean(perm_diffs > (mean_b - mean_a))
---
0.126
As the simulation uses random numbers, the percentage will vary. For example, in the Python
version, we got 12.1%:
[Link](perm_diffs > mean_b - mean_a)

Department of AI & DS, NCE, Hassan Page 22

Statistical Machine Learning for Data Science (BAD702)

---
0.121
This suggests that the observed difference in session time between page A and page B is well
within the range of chance variation and thus is not statistically significant.

Result:
p-value ≈ 0.12 (12%)

 Since p = 0.12 > 0.05, we fail to reject the null hypothesis.

 The observed difference of 35.67 seconds is not statistically significant.
 Thus, Page B’s longer session time may be due to random variation, not a true
difference.

Exhaustive and Bootstrap Permutation Tests

The permutation test is a non-parametric statistical method used to evaluate whether observed
differences between groups are due to random chance or a true effect.
There are two key variants of the standard random permutation (randomization) test:

1. Exhaustive Permutation Test

2. Bootstrap Permutation Test

These variants differ mainly in how the resampling is done and the assumptions they model.

Exhaustive Permutation Test

Definition
An Exhaustive Permutation Test evaluates all possible ways the data can be divided into
groups, instead of relying on random sampling.

Key Characteristics
 All combinations of group assignments are enumerated.
 Provides an exact test because it includes every possible permutation.
 Guarantees that, under the null hypothesis, the probability of falsely declaring
significance does not exceed the pre-defined alpha level (e.g., 0.05).
 Sometimes called an Exact Test due to its statistical precision.

Practicality
 Only feasible for small sample sizes because the number of possible permutations grows
factorially with sample size.

Department of AI & DS, NCE, Hassan Page 23

Statistical Machine Learning for Data Science (BAD702)

 For example, if 10 observations are divided into two groups of 5, the number of unique
permutations is:

But for 40 observations, this number becomes astronomically large.

Approximation
For large datasets, a random permutation test (with many random shuffles) provides results
that closely approximate those of the exhaustive version.

Bootstrap Permutation Test

Definition
In a Bootstrap Permutation Test, the sampling steps are modified so that data are drawn with
replacement instead of without replacement.

Key Idea
 The bootstrap models two sources of randomness:
1. Assignment randomness: how observations are assigned to groups (as in the
regular permutation test).
2. Sampling randomness: how the sample itself represents the underlying
population.

Process
 Randomly resample (with replacement) from the combined dataset to form new groups A
and B.
 Compute the test statistic for each resampled pair.
 Repeat many times to build a bootstrap permutation distribution.

Advantages
 Reflects both sampling variability and assignment variability.
 Especially useful when the data represent only a subset of a larger population.

Limitations
 The distinction between bootstrap and random permutation can become blurred.
 In practical data science, the difference between the two approaches is usually not
critical.

Department of AI & DS, NCE, Hassan Page 24

Statistical Machine Learning for Data Science (BAD702)

Comparison Summary

Feature Exhaustive Permutation Test Bootstrap Permutation Test

All possible reallocations (without Random sampling with
Sampling Type
replacement) replacement
Result Type Exact test (full enumeration) Approximate test (simulation-based)
Feasibility Only for small datasets Scalable to large datasets
Randomness in both assignment and
Represents Randomness in group assignment
sampling
Statistical Controls false positives exactly at Approximation—depends on
Guarantee α-level number of resamples
When dataset represents a
Typical Use When dataset is small
population sample

Permutation Tests: The Bottom Line for Data Science

From a data science perspective, permutation tests in general—whether random, exhaustive, or

bootstrap—serve as powerful heuristic tools for exploring the influence of random variation.
They are straightforward to implement, easy to interpret, and require very few assumptions about
the underlying data distribution. Unlike formula-based inferential statistics that depend heavily
on theoretical models (such as normality), permutation tests are flexible and applicable to
numeric or binary data, regardless of sample size differences or distribution shape. This makes
them highly versatile and appealing for real-world applications like A/B testing or experimental
validation. Overall, permutation methods provide a practical, assumption-free approach to
statistical inference, offering meaningful insights without the false precision often associated
with traditional formula-based techniques.

STATISTICAL SIGNIFICANCE AND P-VALUES

Statistical significance is how statisticians measure whether an experiment (or even a study of
existing data) yields a result more extreme than what chance might produce. If the result is
beyond the realm of chance variation, it is said to be statistically significant.

Statistical significance refers to the method used by statisticians to determine whether the
results of an experiment (or data study) are too extreme to be explained by random chance
alone.

 In other words, it tells us whether an observed effect — such as a difference in

conversion rates, means, or proportions — is likely to be real rather than the result of
random variation in the data.
 If the result is beyond the range of what chance alone might produce, it is said to be
statistically significant.

Department of AI & DS, NCE, Hassan Page 25

Statistical Machine Learning for Data Science (BAD702)

The p-value (probability value) quantifies this idea by measuring the probability of obtaining a
result as extreme as the observed one, assuming the null hypothesis is true.

Consider in Table 3-2 the results of the web test shown earlier.

 Total sessions for Price A: 23,739

 Total sessions for Price B: 22,588
 Total combined sample size: 46,327

Conversion Rate Calculation

Department of AI & DS, NCE, Hassan Page 26

Statistical Machine Learning for Data Science (BAD702)

Interpretation:
Price A’s conversion rate is 5% higher than Price B’s, which seems meaningful in a high-
volume business, where even small improvements can affect revenue significantly.

Why Test for Significance Even with Big Data

Even though we have over 45,000 data points, the actual conversion events (the ones and zeros
representing conversions vs. no conversions) are only in the hundreds.

Thus:

 The effective sample size is determined by the number of conversions, not total sessions.
 Random chance could still produce apparent differences of similar size.
Therefore, it is still essential to test whether the observed difference is statistically
significant.

The Null Hypothesis

Under the null hypothesis (H₀):

 There is no real difference in conversion rates between Price A and Price B.

Any observed difference is due to chance variation.
 We test this by simulating what kind of differences could arise if both prices had the
same true conversion rate.

The Resampling (Permutation) Procedure

To test the hypothesis, we use a permutation test — a resampling-based method that

randomizes outcomes to model chance variation.

Department of AI & DS, NCE, Hassan Page 27

Statistical Machine Learning for Data Science (BAD702)

Steps:

OR

1. Combine all results (1s = conversions, 0s = non-conversions) into one dataset.

o There are 382 conversions (1s) and 45,945 non-conversions (0s).
o The overall conversion rate = 382 / (382 + 45,945) = 0.008246 (0.8246%).
2. Shuffle this combined dataset (randomly mix 1s and 0s).
3. Draw:
o A resample of 23,739 observations (same size as Price A group).
o Count how many 1s (conversions) it contains.
4. From the remaining data, draw 22,588 observations (same size as Price B group).
o Count how many 1s it contains.
5. Compute the difference in conversion rates between the two resampled groups.
6. Repeat steps 2–5 many times (e.g., 1,000 iterations).
7. Examine how often the simulated differences are greater than or equal to the observed
difference (0.0368%).

This frequency gives an empirical p-value — the probability of observing a difference this large
under the null hypothesis.

We can create a histogram of randomly permuted differences in conversion rate in R:

Explanation:

 obs_pct_diff computes the observed difference in percentage.

Department of AI & DS, NCE, Hassan Page 28

Statistical Machine Learning for Data Science (BAD702)

 conversion creates a combined vector of 0s and 1s (all outcomes).

 The loop generates 1,000 permuted differences.
 A histogram visualizes the distribution of differences under random chance.
 The vertical line (abline) marks the observed difference.

Explanation:

 Creates a dataset of conversions (1) and non-conversions (0).

 Uses the permutation function perm_fun() to simulate 1,000 random reallocations.
 Draws a histogram of simulated conversion rate differences.
 Adds a vertical line to mark the actual observed difference (0.0368%).

The histogram of 1,000 resampled results in Figure 3-5: as it happens, in this case the observed
difference of 0.0368% is well within the range of chance variation.

Department of AI & DS, NCE, Hassan Page 29

Statistical Machine Learning for Data Science (BAD702)

p-Value

Definition and Concept

The p-value (probability value) is a numerical measure used to determine how likely it is to
observe the obtained results (or something more extreme), assuming that the null hypothesis
(H₀) is true. This is the frequency with which the chance model produces a result more extreme
than the observed result. We can estimate a pvalue from our permutation test by taking the
proportion of times that the permutation test produces a difference equal to or greater than the
observed difference:

In other words, it quantifies the strength of evidence against the null hypothesis.

 A small p-value (typically < 0.05) indicates that the observed result is unlikely to have
occurred by random chance, suggesting statistical significance.
 A large p-value indicates that the observed difference could easily occur by chance,
meaning there is no strong evidence against H₀.

Mathematically, the p-value represents:

Department of AI & DS, NCE, Hassan Page 30

Statistical Machine Learning for Data Science (BAD702)

Visual Interpretation (Conceptual)

Imagine a sampling distribution under the null hypothesis.

 The p-value is the area in the tails of the distribution beyond the observed statistic.
 It measures how “rare” the observed value is, given random variation.

|-------------------|-------------------|
low p-value → observed result far in tail
high p-value → observed result near center

p-Value in the Context of the Web Conversion Example

Recall the e-commerce experiment comparing Price A and Price B conversions:

 Observed difference in conversion rate:

0.8425% (Price A) – 0.8057% (Price B) = 0.0368 percentage points

The goal is to determine whether this observed difference is statistically significant or could
simply result from random chance.

Estimating the p-Value Using the Permutation Test

Since we already conducted a permutation test (randomly shuffling data 1,000 times under the
null hypothesis), we can use it to calculate the empirical p-value.

R Code
mean(perm_diffs > obs_pct_diff)
# [1] 0.308

Python Code
[Link]([diff > obs_pct_diff for diff in perm_diffs])
# 0.308

Department of AI & DS, NCE, Hassan Page 31

Statistical Machine Learning for Data Science (BAD702)

Explanation

 The expression perm_diffs > obs_pct_diff returns a logical vector:

o TRUE (1) when the random (permuted) difference ≥ observed difference
o FALSE (0) otherwise
 Taking the mean of this logical vector gives the proportion of permutations where the
random difference equals or exceeds the observed difference.

Thus,

Result

The computed p-value = 0.308 (30.8%).

Interpretation

A p-value of 0.308 means that:

If there were truly no difference in conversion rates between the two prices,
we would still observe a difference as large (or larger) than 0.0368 percentage points
about 31% of the time due to random chance.

Conclusion:
Since p = 0.308 > 0.05, the result is not statistically significant. There is no strong evidence
that Price A converts better than Price B — the difference could be due to random variation.

p-Value Using Analytical (Formula-Based) Tests

While permutation tests provide a simulation-based approach, we can also compute an

approximate p-value using classical statistical tests such as the two-sample proportion test
(based on the binomial or normal approximation).

a. Binomial / Proportion Test in R

[Link](x=c(200, 182), n=c(23739, 22588), alternative='greater')

Output:

2-sample test for equality of proportions with continuity correction

data: c(200, 182) out of c(23739, 22588)
X-squared = 0.14893, df = 1, p-value = 0.3498

Department of AI & DS, NCE, Hassan Page 32

Statistical Machine Learning for Data Science (BAD702)

alternative hypothesis: greater

95 percent confidence interval:
-0.001057439 1.000000000
sample estimates:
prop 1 prop 2
0.008424955 0.008057376

Explanation

 x = number of conversions (successes) in each group.

 n = total number of sessions (trials) in each group.
 alternative='greater' specifies a one-tailed test, testing if Price A’s conversion rate is greater
than Price B’s.

The resulting p-value = 0.3498, very close to the permutation test p-value (0.308).

b. Chi-Square Test Using Python

survivors = [Link]([[200, 23739 - 200],
[182, 22588 - 182]])

chi2, p_value, df, _ = stats.chi2_contingency(survivors)

print(f'p-value for single sided test: {p_value / 2:.4f}')

Explanation

 Creates a 2×2 contingency table of conversions and non-conversions.

 Uses the chi-square test (chi2_contingency) to check independence between price and
conversion.
 Because the test is two-sided by default, we divide the p-value by 2 for a one-sided test
(Price A > Price B).

The result gives a p-value ≈ 0.35, again similar to the other methods.

Comparison of Results

Method p-Value Conclusion

Permutation Test 0.308 Not Significant
Proportion Test (R) 0.3498 Not Significant
Chi-Square Test (Python) 0.35 Not Significant

All three methods consistently show high p-values (> 0.05), confirming that the observed
difference in conversion rates is not statistically significant.

Department of AI & DS, NCE, Hassan Page 33

Statistical Machine Learning for Data Science (BAD702)

Interpretation of p-Values in General

p-Value Range Interpretation Decision

p < 0.01 Very strong evidence against H₀ Reject H₀
0.01 ≤ p < 0.05 Moderate evidence against H₀ Reject H₀
0.05 ≤ p < 0.10 Weak evidence Often fail to reject H₀
p ≥ 0.10 No evidence against H₀ Fail to reject H₀

In this example, p = 0.308 clearly falls into the “no evidence against H₀” range.

Alpha
Definition

In statistical hypothesis testing, alpha (α) — also called the significance level — is the
predefined threshold used to decide whether a result is considered statistically significant.

It represents the probability of rejecting the null hypothesis (H₀) when it is actually true — that
is, the risk of a Type I error (false positive).

Common Alpha Levels

 α = 0.05 (5%) — most commonly used

 α = 0.01 (1%) — more conservative
 α = 0.10 (10%) — occasionally used in exploratory studies

These thresholds are arbitrary choices, not mathematical certainties. They simply represent how
much risk we are willing to accept in declaring a difference “significant” when it might have
occurred by chance.

Interpretation

When we say α = 0.05:

We are willing to accept a 5% probability of incorrectly rejecting the null hypothesis — that is,
finding a statistically significant difference even if no real difference exists.

This means that, under the null hypothesis, only 5% of results would be as extreme (or more
extreme) than the observed result.

Important Clarification

The alpha value does not guarantee correctness.

Department of AI & DS, NCE, Hassan Page 34

Statistical Machine Learning for Data Science (BAD702)

It does not mean:

“The result is correct 95% of the time.”

Instead, it defines a decision rule:

If p-value ≤ α, reject H₀.

If p-value > α, fail to reject H₀.

Underlying Probability Question

A common misunderstanding arises because people assume α (and p-values) answer the
question:

“What is the probability that this result happened by chance?”

In reality, the probability question being answered is:

“Given a chance model (the null hypothesis), what is the probability of obtaining a result this
extreme?”

Thus, we are evaluating how incompatible the data are with the null hypothesis model, not
assigning a probability that the null is true or false.

The p-Value Controversy

Over time, the p-value has been widely misused and misunderstood, leading to significant
debate in the scientific and statistical communities.

a. The Problem

Many researchers and journal editors have over-relied on p-values to determine the validity of
scientific findings.

As a result:

 Studies with “significant” p-values (< 0.05) are more likely to be published.
 Researchers often search through multiple hypotheses or tweak their analyses until
they find a p-value below 0.05 — a practice known as “p-hacking.”

Department of AI & DS, NCE, Hassan Page 35

Statistical Machine Learning for Data Science (BAD702)

This has contributed to false discoveries, irreproducible results, and a crisis in scientific
credibility, especially in psychology and biomedical research.

b. Misinterpretation of p-Values

What People Think the p-Value Means:

“The probability that the observed result happened by chance.”

They hope that a low p-value proves that the result is real and not random.

What the p-Value Actually Means:

“The probability of observing results as extreme as, or more extreme than, the actual results if
the null hypothesis is true.”

This distinction is subtle but crucial. A low p-value indicates that the data are unlikely under the
null model, but it does not prove that the null hypothesis is false, nor does it indicate the
probability that the alternative hypothesis is true.

c. Real-World Reactions

In 2015, a major psychology journal banned the use of p-values, arguing that publication
decisions based solely on them were leading to poor-quality science.

This debate led to broader recognition that p-values are often overinterpreted — they give less
“proof” than people assume.

d. ASA (American Statistical Association) Statement (2016)

In March 2016, the ASA issued an official statement clarifying the meaning and proper use of p-
values.
The ASA statement stressed six principles for researchers and journal editors:
1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability
that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a
p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance
of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or
hypothesis.

Department of AI & DS, NCE, Hassan Page 36

Statistical Machine Learning for Data Science (BAD702)

ASA Principle Explanation

1. P-values indicate how incompatible the A small p-value suggests the data disagree
data are with a specified statistical model. with the null hypothesis.
2. P-values do not measure the probability
They are conditional on the assumption that
that the studied hypothesis is true, or that data
the null hypothesis is true.
were produced by random chance alone.
3. Scientific conclusions or decisions should
Significance should not be the only criterion
not be based solely on whether the p-value
for decision-making.
crosses a threshold (e.g., 0.05).
4. Proper inference requires full Researchers must disclose all analyses, tests,
transparency and reporting. and data handling steps.
5. A p-value or “statistical significance” does
A small p-value does not imply a large or
not measure the size or importance of an
meaningful difference.
effect.
6. A p-value alone does not provide strong Other factors such as effect size, context, and
evidence about a model or hypothesis. reproducibility matter.

Practical Significance vs. Statistical Significance

Even when a result is statistically significant, it may not be practically significant — that is,
meaningful or important in real-world terms.

Example

A study with a huge dataset may find that Price A converts 0.02% more than Price B, and this
difference is statistically significant (p < 0.05). However, if this difference translates into
negligible business impact, it lacks practical value.

Key Distinction
Aspect Statistical Significance Practical Significance
Whether the result is unlikely under Whether the result is large or
Definition
random chance (based on p-value/α) meaningful in real-world terms
Depends On Sample size, variability, test method Context, effect size, impact
But difference = 0.02% ⇒ not
Example p < 0.05 ⇒ statistically significant
practically meaningful
Over-interpreting trivial results as Ignoring meaningful but statistically
Risk
“important” weak results

Explanation

 Large sample sizes can make tiny, unimportant effects statistically significant,
because even small differences can achieve very low p-values.
 Conversely, small samples might fail to reach statistical significance even if the effect is
substantial.

Department of AI & DS, NCE, Hassan Page 37

Statistical Machine Learning for Data Science (BAD702)

Hence, ruling out chance does not automatically make a result important. A truly valuable
finding must be both statistically and practically significant.

Type 1 and Type 2 Errors

In assessing statistical significance, two types of error are possible:

 A Type 1 error, in which you mistakenly conclude an effect is real, when it is really just
due to chance
 A Type 2 error, in which you mistakenly conclude that an effect is not real (i.e., due to
chance), when it actually is real
Actually, a Type 2 error is not so much an error as a judgment that the sample size is too small to
detect the effect. When a p-value falls short of statistical significance (e.g., it exceeds 5%), what
we are really saying is “effect not proven.” It could be that a larger sample would yield a smaller
p-value.
The basic function of significance tests (also called hypothesis tests) is to protect against being
fooled by random chance; thus they are typically structured to minimize Type 1 errors.

Data Science and p-Values

In traditional academic research, p-values play a central role in determining statistical
significance — that is, whether an observed result is too extreme to be due to random chance.

However, in the field of data science, the role of p-values is more pragmatic and contextual
rather than decisive. Data scientists use p-values as informative indicators, not as the ultimate
decision-making criterion.

Role of p-Values in Data Science

a. Indicator of Random Variation

A p-value helps data scientists assess whether a seemingly interesting model result — such as a
strong correlation or feature effect — might simply be due to random chance rather than a real
underlying relationship.

For example:

If a feature shows a very low p-value in a regression model, it suggests that the feature’s
relationship with the target variable is unlikely to be due to chance alone.

Department of AI & DS, NCE, Hassan Page 38

Statistical Machine Learning for Data Science (BAD702)

Thus, p-values help identify which model outputs are stable and meaningful, versus those that
could be artifacts of noise.

b. A Supporting Decision Metric

In experiments or A/B testing, data scientists use p-values to support decision-making, not to
control it.

They consider p-values along with:

 Effect size (how large the difference is),

 Confidence intervals,
 Cost-benefit trade-offs,
 Practical impact, and
 Business goals.

Hence, a low p-value is not automatically a “go” signal, and a high p-value is not always a
“stop” signal.
Instead, it is additional evidence to weigh before making a data-driven decision.

c. Intermediate Role in Modeling

In some modeling workflows, p-values are used as intermediate selection criteria for feature
inclusion or feature elimination.

For instance:

 In statistical models such as linear regression or logistic regression, features with high
p-values (indicating weak or non-significant relationships with the target variable) are
often removed to simplify the model.
 Conversely, features with low p-values may be retained, as they are more likely to have
a statistically meaningful association.

This use of p-values as a filtering mechanism helps improve model interpretability,

parsimony, and generalization.

Practical Interpretation
In data science practice:

 p-values are not controlling, meaning decisions are not solely based on them.

Department of AI & DS, NCE, Hassan Page 39

Statistical Machine Learning for Data Science (BAD702)

 They act as contextual evidence, supplementing other metrics like:

o Accuracy, precision, recall (for classification)
o R² or RMSE (for regression)
o Feature importance or SHAP values (for machine learning interpretability)
 A good decision framework integrates statistical evidence (like p-values) with domain
knowledge and business objectives.

T-TESTS

Introduction
The t-test is one of the most widely used statistical significance tests. It was developed by
William Sealy Gosset under the pseudonym “Student” in the early 1900s while working at the
Guinness Brewery. The t-test is based on the Student’s t-distribution, which provides a way to
test hypotheses when the sample size is small and the population standard deviation is
unknown.

The t-test helps determine whether the difference between sample means (or between a sample
mean and a known population mean) is statistically significant, i.e., not likely due to random
chance.

Purpose of a t-Test
The t-test is used to:

 Compare the means of one or two samples.

 Assess whether the observed difference between groups is real or due to random
variation.

Department of AI & DS, NCE, Hassan Page 40

Statistical Machine Learning for Data Science (BAD702)

 Evaluate hypotheses in experiments, such as determining if a new drug, web page, or

teaching method has an effect.

In short, it tests whether a difference in means is statistically significant.

Background: Significance Testing

All significance tests involve the following key steps:

1. Defining hypotheses:
o Null hypothesis (H₀): Assumes there is no effect or no difference.
o Alternative hypothesis (H₁ or Ha): Assumes there is an effect or difference.
2. Selecting a test statistic:
o A test statistic is calculated from the sample data to measure the magnitude of
difference or effect.
o For the t-test, the test statistic follows the t-distribution when the null hypothesis
is true.
3. Computing the p-value:
o The p-value indicates the probability of observing the sample data (or something
more extreme) if the null hypothesis were true.
o A small p-value (typically < 0.05) leads us to reject H₀.
4. Decision-making:
o Compare the calculated t-value with the critical value from the t-distribution
table.
o Or, equivalently, compare the p-value to the chosen significance level (α).

In the early 20th century, computational methods like resampling tests (permutation or bootstrap
tests) were not feasible because of limited computing power. In the 1920s and 1930s, when
statistical hypothesis testing was being developed, it was not feasible to randomly shuffle data
thousands of times to do a resampling test. Statisticians found that a good approximation to the
permutation (shuffled) distribution was the t-test, based on Gosset’s t-distribution. It is used for
the very common two-sample comparison—A/B test—in which the data is numeric. But in order
for the t-distribution to be used without regard to scale, a standardized form of the test statistic
must be used.

Therefore, Gosset developed the t-distribution as an analytical approximation to the

randomization (permutation) distribution. It allows researchers to assess significance using small
samples without extensive shuffling of data.

Formula and Concept

Department of AI & DS, NCE, Hassan Page 41

Statistical Machine Learning for Data Science (BAD702)

The t-statistic is calculated as:

For a single-sample t-test (comparing a sample mean to a known population mean):

The calculated t-value is compared against a critical t-value from the t-distribution table with
(n – 1) degrees of freedom.

Types of t-Tests
The t-test can be classified into three major types:

1. One-Sample t-Test

Used to determine whether the mean of a single sample differs significantly from a known or
hypothesized population mean.

Example: Testing if the average height of a class is different from the national average.

Department of AI & DS, NCE, Hassan Page 42

Statistical Machine Learning for Data Science (BAD702)

2. Independent Two-Sample t-Test

Used to compare the means of two independent groups.

Example: Comparing average exam scores of two different classes.

There are two versions:

 Equal variances assumed (pooled t-test)

 Unequal variances assumed (Welch’s t-test)

3. Paired-Sample t-Test

Used when the same subjects are measured before and after an intervention, or when samples
are related.

Example: Testing the effect of a training program by comparing scores before and after.

Formula:

Department of AI & DS, NCE, Hassan Page 43

Statistical Machine Learning for Data Science (BAD702)

Assumptions of the t-Test

To ensure valid results, the following assumptions must be satisfied:

1. Normality: Data should be approximately normally distributed.

2. Independence: Observations must be independent of each other.
3. Scale of measurement: Data should be continuous (interval or ratio scale).
4. Equal variances (for independent t-test): Both groups should have similar variance
(homogeneity of variance).

If these assumptions are violated, non-parametric tests (e.g., Mann–Whitney U test) can be
used instead.

All statistical software, as well as R and Python, includes commands that embody the formula. In
R, the function is [Link]:
R Example
[Link](Time ~ Page, data=session_times, alternative='less')

Output:

Welch Two Sample t-test

data: Time by Page
t = -1.0983, df = 27.693, p-value = 0.1408
alternative hypothesis: true difference in means is less than 0
95 percent confidence interval:
-Inf 19.59674
sample estimates:
mean in group Page A mean in group Page B
126.3333 162.0000

Interpretation:

 The t-value = -1.0983

 The p-value = 0.1408 > 0.05
 Hence, we fail to reject H₀, meaning there is no significant difference between the
session times of Page A and Page B.

Python Example
from scipy import stats

res = stats.ttest_ind(
session_times[session_times.Page == 'Page A'].Time,
session_times[session_times.Page == 'Page B'].Time,
equal_var=False

Department of AI & DS, NCE, Hassan Page 44

Statistical Machine Learning for Data Science (BAD702)

)
print(f'p-value for single sided test: {[Link] / 2:.4f}')

Result:

p-value for single sided test: 0.1408

This p-value matches the R result, indicating the same conclusion — no significant difference.

Comparison with Resampling Tests

 In resampling (permutation) tests, we shuffle the data many times to build a reference
distribution.
 In the t-test, we rely on theoretical assumptions and compare our statistic with the t-
distribution.
 Today, with computing power available, resampling and permutation tests provide a
more flexible alternative, but the t-test remains a fast and practical approximation.

Decision Rules
Criterion Decision
p-value < α (0.05) Reject H₀ (Significant difference)
p-value ≥ α (0.05) Fail to reject H₀ (No significant difference)

Applications
 Comparing the effectiveness of two treatments (medical trials)
 A/B testing in marketing (click-through rates, session times)
 Comparing test scores, reaction times, or machine learning model performances
 Testing differences in means under experimental designs

MULTIPLE TESTING
In statistics, multiple testing (or multiple comparisons) refers to the practice of performing
many hypothesis tests on the same dataset. While testing multiple hypotheses can reveal
interesting insights, it also increases the risk of false positives — that is, incorrectly concluding
that an effect exists when it actually does not.

This issue is summarized by the well-known saying:

“Torture the data long enough, and it will confess.”

Department of AI & DS, NCE, Hassan Page 45

Statistical Machine Learning for Data Science (BAD702)

This means that if you analyze your data through enough perspectives and perform enough
statistical tests, you will almost inevitably find some result that appears statistically significant,
even if it occurred purely by chance.

The Problem of Alpha Inflation (Type I Error Inflation)

Basic Concept

In hypothesis testing, we typically choose a significance level (α) — commonly 0.05, meaning
we accept a 5% chance of wrongly rejecting a true null hypothesis (Type I error).

However, when many tests are conducted simultaneously, the chance of making at least one
Type I error increases dramatically.

Example

Suppose you have:

 20 predictor variables
 1 outcome variable, and all are randomly generated (i.e., no real relationship exists).

If you perform 20 independent t-tests, each at α = 0.05:

 Probability that one test is not significant = 0.95

 Probability that all 20 tests are not significant = (0.95)20=0.36

Department of AI & DS, NCE, Hassan Page 46

Statistical Machine Learning for Data Science (BAD702)

Therefore, the probability that at least one test appears significant (by chance) is:

1−0.36=0.64

This means there is a 64% chance of finding at least one “significant” result purely due to
random noise.

Term: Alpha Inflation

This inflation of the overall false positive rate when performing multiple tests is called alpha
inflation.
It leads to an increased probability of Type I errors, where we mistakenly find “effects” that do
not actually exist.

Relation to Overfitting and Data Mining

Overfitting Analogy

Multiple testing is conceptually similar to overfitting in data mining or machine learning:

 In overfitting, a model learns the noise instead of the underlying pattern.

 In multiple testing, by running many tests, you increase the chance that random noise
appears as a statistically significant effect.

Example

If you try 100 different models or variable combinations, some model will likely appear to fit the
data very well — but its performance may be due to random coincidence, not genuine
predictive power.

Mitigation in Machine Learning

In supervised learning, the use of a holdout (test) set or cross-validation helps mitigate
overfitting.
These techniques ensure the model’s performance is evaluated on data it has not seen before,
reducing the likelihood of false discoveries due to multiple experimentation.

However, in statistical analysis without labeled data (e.g., exploratory hypothesis testing),
such protections do not exist — and multiple testing remains a serious issue.

Department of AI & DS, NCE, Hassan Page 47

Statistical Machine Learning for Data Science (BAD702)

Real-World Scenarios of Multiple Testing

Multiple testing arises naturally in many contexts:

1. Comparing multiple treatment groups

o Example: Treatments A, B, and C
o Questions asked:
 Is A different from B?
 Is B different from C?
 Is A different from C?
→ Each comparison increases the probability of a false positive.

2. Clinical trials with multiple time points

o Analyzing results at multiple stages of therapy (week 2, week 4, week 6)
increases the number of comparisons.
3. Exploratory data analysis
o Checking multiple subgroups (e.g., “effect is significant only for women under
30”)
o Trying many model specifications
o Testing many predictor variables against one outcome
o Performing numerous pairwise tests among several variables

Each additional test increases the chance of obtaining at least one spurious result.

Statistical Adjustment Procedures

To correct for alpha inflation, statisticians developed adjustment procedures that make the
significance threshold more stringent when multiple tests are performed.

Bonferroni Adjustment

One of the simplest and most widely used corrections.

Department of AI & DS, NCE, Hassan Page 48

Statistical Machine Learning for Data Science (BAD702)

Example:
If you conduct 10 tests at α = 0.05, the adjusted threshold for each test becomes

This ensures that the overall probability of making at least one Type I error remains
approximately 0.05.

Limitation:
Bonferroni is conservative — it reduces false positives but can increase false negatives (missed
true effects).

Tukey’s Honest Significant Difference (HSD)

Used when comparing multiple group means (post-hoc test after ANOVA).

 It calculates the maximum difference among all group means.

 Compares it against a critical value based on the t-distribution.
 Adjusts for the number of pairwise comparisons.

Conceptually, it’s similar to reshuffling all data values and recalculating maximum mean
differences repeatedly to find how extreme the observed differences are.

Other Adjustment Methods

 Holm–Bonferroni: Sequentially adjusts α values to be less conservative.

 Benjamini–Hochberg: Controls the False Discovery Rate (FDR) instead of the overall
Type I error rate.

Multiplicity issues

 Checking for multiple pairwise differences across groups

 Looking at multiple subgroup results (“we found no significant treatment effect
 overall, but we did find an effect for unmarried women younger than 30”)
 Trying lots of statistical models
 Including lots of variables in models
 Asking a number of different questions (i.e., different possible outcomes)

Department of AI & DS, NCE, Hassan Page 49

Statistical Machine Learning for Data Science (BAD702)

False Discovery Rate (FDR)

Definition

The False Discovery Rate (FDR) is the expected proportion of false positives among all
significant results.

Originally developed for large-scale hypothesis testing, such as genomic research, where
thousands of tests are conducted simultaneously.

Example in Genomics

In gene sequencing projects:

 Researchers test thousands of genes to find those linked to a disease.

 Even with α = 0.05, 5% of genes could appear “significant” by chance.
 FDR controls the rate of these false findings to ensure meaningful results.

Benjamini–Hochberg (BH) Procedure:

1. Rank p-values from smallest to largest: p1,p2,...,pmp_1, p_2, ..., p_mp1,p2,...,pm

2. For each test iii, compute:

where q = desired FDR level (e.g., 0.05)

3. Reject all hypotheses up to the largest iii satisfying the condition.

FDR in Machine Learning (Classification Context)

In data mining or classification:

 FDR represents the misclassification rate within the positive class (class 1).
 It answers: “Given that we labeled a record as positive (1), what is the probability that
this prediction is false?”

Department of AI & DS, NCE, Hassan Page 50

Statistical Machine Learning for Data Science (BAD702)

Example:

 If 100 samples are predicted as positive and 10 are actually false,

 This is especially relevant when positives are rare and negatives are abundant.

Consequences of Ignoring Multiple Testing

When researchers ignore multiple testing, it leads to:

 False claims of significance

 Poor reproducibility of studies
 Overestimation of effect sizes

Example: Replication Crisis

A study by Bayer (2011) attempted to replicate 67 scientific studies and succeeded in fully
replicating only 14 of them — roughly 21%. Nearly two-thirds of results could not be
replicated, largely due to false discoveries and multiple testing effects.

Guidelines for Data Scientists

For Predictive Modeling:

 Use cross-validation and holdout sets to validate model performance.

 These techniques help detect and avoid illusory patterns that arise from chance.

For Exploratory or Statistical Analysis:

 Be aware that repeated querying and manipulation of data increases false discovery
risk.
 Use adjustment methods (Bonferroni, FDR control) when conducting many tests.
 Employ resampling and simulation heuristics (e.g., permutation tests) to estimate the
expected number of false positives under random conditions.

Department of AI & DS, NCE, Hassan Page 51

Statistical Machine Learning for Data Science (BAD702)

DEGREES OF FREEDOM

Introduction

The term degrees of freedom (df) is a fundamental concept in statistics that appears in many
areas such as hypothesis testing, regression, and probability distributions.
It represents the number of independent values in a sample that can vary freely while calculating
a statistical estimate.

In simple terms, degrees of freedom tell us how many pieces of information are available to
estimate another piece of information.

Definition

Degrees of Freedom (df) refer to the number of independent observations in a dataset that
can vary when estimating a statistical parameter.

Mathematically,

It shows how many independent comparisons or quantities can be used in statistical calculations.

Concept Explanation

Suppose you have a sample of 10 numbers. If you know the mean of those numbers, then only
9 numbers are free to vary — because once you know 9 of them and the mean, the 10th number
is automatically determined.

Hence:

df=n−1=10−1=9

This single restriction (knowing the mean) reduces one degree of freedom.

Department of AI & DS, NCE, Hassan Page 52

Statistical Machine Learning for Data Science (BAD702)

Importance in Statistics

Degrees of freedom are important because they affect:

 The shape of various probability distributions (like t, chi-square, F).

 The accuracy of estimations such as variance, standard deviation, and regression
coefficients.
 The critical values used in hypothesis testing.

Degrees of Freedom in Variance and Standard Deviation

When calculating variance or standard deviation for a sample, we divide by (n − 1) instead of n.

Reason:

 Using n would underestimate the population variance (biased estimate).

 Dividing by (n − 1) corrects this bias — known as Bessel’s correction.

Formula:

This ensures that the sample variance is an unbiased estimator of the population variance.

Role in Probability Distributions

Many probability distributions are defined in terms of degrees of freedom:

(a) t-Distribution

 Used when estimating population mean from a small sample.

 Shape depends on df = n – 1.
 As df increases, the t-distribution approaches the normal distribution.

Department of AI & DS, NCE, Hassan Page 53

Statistical Machine Learning for Data Science (BAD702)

(b) F-Distribution

 Used in comparing two variances or in ANOVA.

 It has two degrees of freedom:
o Numerator df: n1−1
o Denominator df: n2−1

(c) Chi-Square Distribution

 Used in tests of independence and goodness of fit.

 The shape depends on the number of degrees of freedom (equal to the number of
independent variables).

Practical Relevance in Data Science

In data science, formal statistical tests (like t-test or F-test) are less frequently used, especially
with large datasets.

 When sample size (n) is large, the difference between using n and n − 1 becomes
negligible because:

 Hence, the bias disappears for large samples.

However, in small datasets, using n − 1 is essential for accurate and unbiased results.

Degrees of Freedom in Regression Analysis

In regression models, degrees of freedom represent the number of observations available to

estimate the model parameters.

Formula:
df = n - k

where:

 n = total number of observations

 k = number of estimated parameters (including intercept)

Department of AI & DS, NCE, Hassan Page 54

Statistical Machine Learning for Data Science (BAD702)

Example:
If you have 100 data points and estimate 4 regression coefficients,
then df=100−4=96.

Degrees of Freedom with Categorical Variables (Dummy Variables)

When a categorical variable with k categories is included in a regression model, it creates k−1
dummy (indicator) variables.

Reason:
Including all k dummies causes perfect multicollinearity, also known as the dummy variable
trap.

Example – “Day of Week” variable:

 7 categories → Monday to Sunday

 Only 6 degrees of freedom If you include indicators for Monday through Saturday, the
model can automatically infer Sunday (the omitted category).

Including all 7 dummies would cause a linear dependency:

Thus, one category must be omitted to avoid multicollinearity.

Example Problem
Example:
You have a sample of 5 test scores: [78, 82, 85, 90, 95]. Calculate degrees of freedom for
variance.
Solution:
n=5
df = n – 1 = 5 – 1 = 4
When you calculate variance:

You divide by 4, not 5, because one value is dependent once the mean is fixed.

Department of AI & DS, NCE, Hassan Page 55

Statistical Machine Learning for Data Science (BAD702)

ANOVA

ANOVA is used when comparing 3 or more group means to determine if differences are
statistically significant. ANOVA (Analysis of Variance) tests whether multiple groups have the
same mean or not.
Table 3-3 shows the stickiness of four web pages, defined as the number of seconds a visitor
spent on the page. The four pages are switched out so that each web visitor receives one at
random. There are a total of five visitors for each page, and in Table 3-3, each column is an
independent set of data. The first viewer for page 1 has no connection to the first viewer for page
2.
Note that in a web test like this, we cannot fully implement the classic randomized sampling
design in which each visitor is selected at random from some huge population. We must take the
visitors as they come. Visitors may systematically differ depending on time of day, time of week,
season of the year, conditions of their internet, what device they are using, and so on. These
factors should be considered as potential bias when the experiment results are reviewed.

Department of AI & DS, NCE, Hassan Page 56

Statistical Machine Learning for Data Science (BAD702)

With four means, there are six possible comparisons between groups:
• Page 1 compared to page 2
• Page 1 compared to page 3
• Page 1 compared to page 4
• Page 2 compared to page 3
• Page 2 compared to page 4
• Page 3 compared to page 4

The more such pairwise comparisons we make, the greater the potential for being fooled by
random chance. Instead of worrying about all the different comparisons between individual
pages we could possibly make, we can do a single overall test that addresses the question,
“Could all the pages have the same underlying stickiness, and the differences among them be due
to the random way in which a common set of session times got allocated among the four pages?”

The procedure used to test this is ANOVA. The basis for it can be seen in the following
resampling procedure (specified here for the A/B/C/D test of web page stickiness):
1. Combine all the data together in a single box.
2. Shuffle and draw out four resamples of five values each.
3. Record the mean of each of the four groups.
4. Record the variance among the four group means.
5. Repeat steps 2–4 many (say, 1,000) times.
The proportion of the time did the resampled variance exceed the observed variance. This is the
p-value.
Fortunately, the aovp function in the lmPerm package computes a permutation test for this case:

Department of AI & DS, NCE, Hassan Page 57

Statistical Machine Learning for Data Science (BAD702)

F-Statistic
The F-statistic is based on the ratio of the variance across group means (i.e., the treatment effect)
to the variance due to residual error. The higher this ratio, the more statistically significant the
result.

Department of AI & DS, NCE, Hassan Page 58

Statistical Machine Learning for Data Science (BAD702)

Df is “degrees of freedom,” Sum Sq is “sum of squares,” Mean Sq is “mean squares” (short for
mean-squared deviations), and F value is the F-statistic. For the grand average, sum of squares is
the departure of the grand average from 0, squared, times 20 (the number of observations). The
degrees of freedom for the grand average is 1, by definition.
For the treatment means, the degrees of freedom is 3 (once three values are set, and then the
grand average is set, the other treatment mean cannot vary). Sum of squares for the treatment
means is the sum of squared departures between the treatment means and the grand average.

Mean squares (MS) is the sum of squares divided by the degrees of freedom. The F-statistic is
MS(treatment)/MS(error). The F value thus depends only on this ratio and can be compared to a
standard F-distribution to determine whether the differences among treatment means are greater
than would be expected in random chance variation.

Chi-Square Test
The chi-square test was originally developed by Karl Pearson in 1900. The term chi comes from
the Greek letter Χ used by Pearson in the article.

Department of AI & DS, NCE, Hassan Page 59

Statistical Machine Learning for Data Science (BAD702)

Chi-Square Test: A Resampling Approach Suppose you are testing three different headlines—A,
B, and C—and you run them each on 1,000 visitors, with the results shown in Table 3-4.

Department of AI & DS, NCE, Hassan Page 60

Statistical Machine Learning for Data Science (BAD702)

Department of AI & DS, NCE, Hassan Page 61

Statistical Machine Learning for Data Science (BAD702)

Department of AI & DS, NCE, Hassan Page 62

Statistical Machine Learning for Data Science (BAD702)

Department of AI & DS, NCE, Hassan Page 63