Hypothesis Testing: Comprehensive Study Material

1. Introduction to Hypothesis Testing

Definition and Purpose

Hypothesis testing is a statistical method for making inferences about population parameters based on
sample data. It provides a systematic framework for evaluating claims or hypotheses about populations
when direct measurement of the entire population is impractical or impossible.

Core Methodology

The process involves formulating a hypothesis about a population parameter, collecting sample data, and
using statistical tests to determine whether the observed sample evidence is sufficiently strong to reject the
initial hypothesis.

Fundamental Principle

Sample statistics are compared to hypothesized population parameters. Small discrepancies between sample
and hypothesized values suggest the hypothesis may be correct, while large discrepancies provide evidence
against it.

Key Distinction from Proof

Hypothesis testing does not prove hypotheses with absolute certainty. Rather, it quantifies the strength of
evidence against a null hypothesis, allowing researchers to make probabilistic statements about population
parameters.

Applications

Quality control: Verifying manufacturing processes meet specifications

Medical research: Evaluating treatment efficacy

Marketing analytics: Assessing campaign effectiveness

Educational assessment: Comparing teaching methodologies

2. Null and Alternative Hypotheses

Every hypothesis test involves two competing statements about the population parameter.

1
2.1 Null Hypothesis (𝐻0 )
Definition: The null hypothesis represents the default position or status quo, asserting no effect, no
difference, or a specific parameter value.

Symbol: 𝐻0

Characteristics:

• Contains an equality operator (=, ≤, or ≥)

• Assumed true unless evidence indicates otherwise
• Represents the conservative position

Examples:

• 𝐻0 ∶ 𝜇 = 100 (population mean equals 100)

• 𝐻0 ∶ 𝜇1 = 𝜇2 (two population means are equal)
• 𝐻0 ∶ 𝑝 = 0.5 (population proportion equals 0.5)

Rationale for Starting Position

In scientific methodology, we adopt a skeptical stance, maintaining the null hypothesis until sufficient
evidence warrants its rejection. This parallels the legal principle of presumption of innocence.

Example: A pharmaceutical company claims efficacy for a new medication. 𝐻0 states the medication has
no effect beyond placebo, requiring the company to provide evidence to the contrary.

2.2 Alternative Hypothesis (𝐻1 or 𝐻𝑎 )

Definition: The alternative hypothesis contradicts the null hypothesis, suggesting a significant effect,
difference, or deviation from the hypothesized value.

Symbol: 𝐻1 or 𝐻𝑎

Characteristics:

• Contains an inequality operator (≠, <, or >)

• Accepted only when sufficient evidence contradicts 𝐻0
• Represents the research hypothesis

Types of Alternative Hypotheses:

Two-tailed test: 𝐻1 ∶ 𝜇 ≠ 𝜇0

Tests for deviation in either direction from the hypothesized value

Example: Testing whether coin fairness differs from 0.5 (could be biased toward heads or tails)

One-tailed test (right): 𝐻1 ∶ 𝜇 > 𝜇0

Tests for increase only

Example: Evaluating whether a new training program increases productivity

One-tailed test (left): 𝐻1 ∶ 𝜇 < 𝜇0

2
Tests for decrease only

Example: Testing whether a new process reduces manufacturing defects

Complementary Nature

The null and alternative hypotheses are mutually exclusive and exhaustive. Evidence supporting one
necessarily contradicts the other.

3. Significance Level and Decision Making

3.1 Significance Level (𝛼)
Definition: The significance level represents the probability of Type I error - rejecting a true null hypothesis.
It establishes the threshold for determining statistical significance.

Symbol: 𝛼 (alpha)

Common Values:

• 𝛼 = 0.05: Standard in most research (5% Type I error rate)

• 𝛼 = 0.01: Stringent criterion (1% Type I error rate)
• 𝛼 = 0.10: Exploratory research (10% Type I error rate)

Interpretation of 𝛼 = 0.05:

The test maintains a 5% probability of false positive results, corresponding to 95% confidence in decision-
making.

Selection Criteria:

The choice of 𝛼 depends on context and error consequences:

High-stakes applications (medical trials, safety assessments): 𝛼 = 0.01

Standard research contexts: 𝛼 = 0.05

Exploratory or preliminary studies: 𝛼 = 0.10

3.2 Decision Rules

Statistical decisions involve comparing test results to predetermined thresholds.

Method 1: Test Statistic Comparison

Calculate test statistic from sample data

Compare absolute value to critical value from statistical tables

If |test statistic| > critical value: Reject 𝐻0 (statistically significant)

If |test statistic| ≤ critical value: Fail to reject 𝐻0 (not statistically significant)

Method 2: P-value Method (Standard Approach)

Calculate p-value from test statistic

3
Compare p-value to significance level 𝛼

If p-value < 𝛼: Reject 𝐻0 (statistically significant)

If p-value ≥ 𝛼: Fail to reject 𝐻0 (not statistically significant)

Terminology Standards:

“Reject 𝐻0 ”: Evidence indicates significant effect/difference

“Fail to reject 𝐻0 ”: Insufficient evidence to conclude significant effect/difference

Note: We never “accept 𝐻0 ” as absence of evidence does not constitute evidence of absence

3.3 Critical Region

Definition: The critical region (rejection region) comprises the range of test statistic values that lead to
null hypothesis rejection.

Structure:

For two-tailed test (𝛼 = 0.05):

Central 95% constitutes acceptance region

Extreme 2.5% in each tail constitutes critical region

For one-tailed test (𝛼 = 0.05):

95% constitutes acceptance region

Extreme 5% in one tail constitutes critical region

Decision Rule:

Test statistic falling in critical region → Reject 𝐻0

Test statistic falling in acceptance region → Fail to reject 𝐻0

4. P-values
Definition

The p-value quantifies the probability of obtaining sample results at least as extreme as those observed,
assuming the null hypothesis is true.

Interpretation Framework:

Small p-value (< 𝛼): Observed data would be improbable under 𝐻0

Strong evidence against 𝐻0 → Reject 𝐻0

Large p-value (≥ 𝛼): Observed data reasonably consistent with 𝐻0

Weak evidence against 𝐻0 → Fail to reject 𝐻0

Interpretation Examples:

4
p-value = 0.001 (0.1%)

Only 1 in 1000 samples would produce results this extreme if 𝐻0 were true. Very strong evidence against
𝐻0 .

p-value = 0.04 (4%)

4% probability of observing this result under 𝐻0 . Marginally significant at 𝛼 = 0.05.

p-value = 0.15 (15%)

15% probability of observing this result under 𝐻0 . Insufficient evidence to reject 𝐻0 .

p-value = 0.50 (50%)

Results completely consistent with 𝐻0 . No evidence against null hypothesis.

Common Misconception

The p-value does NOT represent the probability that 𝐻0 is true. Rather, it quantifies the probability of the
observed data (or more extreme) given that 𝐻0 is true.

Application Example

Testing coin fairness: 100 flips yield 65 heads

p-value = 0.002

Interpretation: If the coin were fair, only 0.2% of 100-flip experiments would yield 65+ heads. This provides
strong evidence of bias (reject 𝐻0 : p = 0.5).

5. Test Statistics Overview

5.1 Concept and Formula
Definition: A test statistic is a standardized value calculated from sample data that measures the deviation
between the sample statistic and the hypothesized parameter.

General Form:

Sample Statistic − Hypothesized Parameter

Test Statistic =
Standard Error

The magnitude of the test statistic (absolute value) indicates the strength of evidence against 𝐻0 .

5.2 Types of Test Statistics

t-statistic:

Application: Population standard deviation (𝜎) unknown

Utilizes sample standard deviation (s) as estimator

Formula: 𝑡 = 𝑥−𝜇
̄√
𝑠/ 𝑛

5
Distribution: t-distribution with (n-1) degrees of freedom

z-statistic:

Application: Population standard deviation (𝜎) known, or large sample (n > 30)

Formula: 𝑧 = 𝑥−𝜇
̄√
𝜎/ 𝑛

Distribution: Standard normal distribution

z-statistic for proportions:

Application: Testing population proportions

Formula: 𝑧 = 𝑝−𝑝
̂ 0
√𝑝0 (1−𝑝0 )/𝑛

Distribution: Standard normal distribution (by Central Limit Theorem)

5.3 Degrees of Freedom

Definition: Degrees of freedom represent the number of independent values in a dataset that are free to
vary when estimating a parameter.

Conceptual Explanation

Given n observations and their mean, only (n-1) values are independent; the final value is determined by
the constraint that the mean must equal the calculated value.

Formulas:

One sample: df = n - 1

Two samples: df varies by method (commonly 𝑛1 + 𝑛2 − 2)

Significance

Degrees of freedom determine the shape of the t-distribution. Lower df values produce distributions with
heavier tails, reflecting greater uncertainty in parameter estimates.

6. One-Sample t-Test
Purpose: Test whether a sample mean differs significantly from a known or hypothesized population mean
when population standard deviation is unknown.

Assumptions:

• Population 𝜎 unknown (typical case)

• Sample size small (n < 30) or any size
• Data approximately normally distributed

Hypotheses:

• 𝐻0 ∶ 𝜇 = 𝜇0
• 𝐻1 ∶ 𝜇 ≠ 𝜇0

6
Test Statistic:

𝑥̄ − 𝜇
𝑡= √
𝑠/ 𝑛

Where:

• 𝑥̄ = sample mean
• 𝜇 = hypothesized population mean
• s = sample standard deviation
• n = sample size

Degrees of Freedom: df = n - 1

Implementation:
[Link](202315)
sample_data = [Link](loc=105, scale=15, size=20)
population_mean = 100

t_statistic, p_value = stats.ttest_1samp(sample_data, population_mean)

stats.ttest_1samp(sample_data, population_mean) - Performs one-sample t-test comparing sample

mean to hypothesized value

Decision Process:
alpha = 0.05
if p_value < alpha:
print("Reject H�: Significant difference detected")
else:
print("Fail to reject H�: Insufficient evidence of difference")

Example Output:

T-statistic: 1.0979
P-value: 0.2860
Fail to reject the null hypothesis

Interpretation:

T-statistic = 1.0979: Sample mean deviates 1.0979 standard errors above hypothesized mean

P-value = 0.2860: 28.6% probability of observing this deviation under 𝐻0

Decision: p-value (0.2860) > 𝛼 (0.05), therefore fail to reject 𝐻0

Conclusion: Insufficient statistical evidence to conclude sample mean differs from 100

Application Example

Quality control scenario: Manufacturer claims average component weight is 100g. Random sample of 20
components yields mean = 105g, SD = 15g.

7
Test: Does actual mean differ from claimed 100g?

Result: p = 0.286 → Not significant. No evidence contradicting manufacturer’s claim.

7. Two-Sample t-Test
Purpose: Compare means from two independent samples to determine if they differ significantly.

Assumptions:

• Two independent groups

• Population 𝜎 unknown for both groups
• Interest in comparing group means

Hypotheses:

• 𝐻0 ∶ 𝜇1 = 𝜇2
• 𝐻1 ∶ 𝜇1 ≠ 𝜇2

Test Statistic:

𝑥1̄ − 𝑥2̄
𝑡=
𝑆𝐸

Where SE (standard error) calculation depends on variance equality assumption

Implementation:
sample_data1 = [Link](loc=105, scale=15, size=100)
sample_data2 = [Link](loc=100, scale=18, size=120)

t_statistic, p_value = stats.ttest_ind(sample_data1, sample_data2, equal_var=False)

stats.ttest_ind(sample1, sample2, equal_var) - Performs independent two-sample t-test

Parameters:

equal_var=False: Applies Welch’s t-test (robust to unequal variances)

equal_var=True: Applies Student’s t-test (assumes equal variances)

Example Output:

T-statistic: 1.9211
P-value: 0.0560
Fail to reject the null hypothesis

Interpretation:

T-statistic = 1.9211: Group 1 mean exceeds Group 2 mean by 1.92 standard errors

P-value = 0.0560: 5.6% probability of observing this difference under equal means

Decision: p-value (0.056) > 𝛼 (0.05), marginally exceeds significance threshold

8
Conclusion: Insufficient statistical evidence to conclude means differ

Alternative Example:
response_times_a = [Link](loc=20, scale=5, size=100)
response_times_b = [Link](loc=25, scale=6, size=100)
t_statistic, p_value = stats.ttest_ind(response_times_a, response_times_b)

Output:

t-statistic: -6.4442
p-value: 8.66e-10
Reject the null hypothesis

Interpretation: Extremely small p-value indicates strong evidence of difference. Group B demonstrates
significantly longer response times than Group A.

8. One-Sample z-Test
Purpose: Test whether sample mean differs from known population mean when population standard
deviation is known.

Assumptions:

• Population 𝜎 known (rare in practice)

• Large sample size (n > 30), or
• Data normally distributed

Hypotheses:

• 𝐻0 ∶ 𝜇 = 𝜇0
• 𝐻1 ∶ 𝜇 ≠ 𝜇0

Test Statistic:

𝑥̄ − 𝜇
𝑧= √
𝜎/ 𝑛

Where:

• 𝑥̄ = sample mean
• 𝜇 = hypothesized population mean
• 𝜎 = population standard deviation (known)
• n = sample size

Implementation:
sample_data = [Link](loc=105, scale=15, size=100)
population_mean = 100
population_std = 15 # Known �

9
z_statistic = ([Link](sample_data) - population_mean) / (population_std / [Link](len(sample_data
p_value = 2 * (1 - [Link]([Link](z_statistic)))

[Link](array) - Calculates sample mean

[Link](n) - Computes square root

[Link](z) - Returns cumulative probability up to z-value

[Link](value) - Returns absolute value

P-value Calculation:

2 * (1 - [Link]([Link](z))) - Computes two-tailed p-value

Factor of 2 accounts for both tails in two-tailed test

Example Output:

Z-statistic: 3.9314
P-value: 0.0001
Reject the null hypothesis

Interpretation:

Z-statistic = 3.93: Sample mean deviates 3.93 standard deviations above population mean

P-value = 0.0001: Only 0.01% probability under 𝐻0

Decision: Strong evidence to reject 𝐻0

Conclusion: Sample mean significantly differs from 100

Distinction from t-test:

z-test: Known 𝜎 → standard normal distribution

t-test: Unknown 𝜎 → t-distribution with df = n-1

For large samples, distributions converge

Practical Application: Standardized assessments (SAT, IQ) where population 𝜎 is published

9. Two-Sample z-Test
Purpose: Compare means from two independent samples when population standard deviations are known.

Assumptions:

• Two independent groups

• Population 𝜎 known for both (rare), or
• Large samples (𝑛1 , 𝑛2 > 30)

Hypotheses:

• 𝐻0 ∶ 𝜇1 = 𝜇2

10
 • 𝐻1 ∶ 𝜇1 ≠ 𝜇2

Test Statistic:

𝑥1̄ − 𝑥2̄
𝑧= 2 𝜎22
√ 𝑛𝜎1 + 𝑛2
1

Implementation:
sample_data1 = [Link](loc=105, scale=15, size=100)
sample_data2 = [Link](loc=100, scale=18, size=120)
sample_std1 = 15 # Known ��
sample_std2 = 18 # Known ��

mean_difference = [Link](sample_data1) - [Link](sample_data2)

std_error = [Link]((sample_std1**2 / len(sample_data1)) + (sample_std2**2 / len(sample_data2)))
z_statistic = mean_difference / std_error
p_value = 2 * (1 - [Link]([Link](z_statistic)))

len(array) - Returns sample size

** operator - Exponentiation (e.g., 𝜎2 )

Example Output:

Z-statistic: 1.9385
P-value: 0.0526
Fail to reject the null hypothesis

Interpretation:

Z-statistic = 1.94: Difference equals 1.94 standard errors

P-value = 0.0526: Marginally above 𝛼 = 0.05 threshold

Decision: Fail to reject 𝐻0

Conclusion: Insufficient evidence to conclude means differ

10. One-Sample Proportion Test

Purpose: Test whether sample proportion differs significantly from hypothesized population proportion.

Assumptions:

• Categorical data (binary outcomes)

• Large sample (np� > 5 and n(1-p�) > 5)
• Random sampling

Hypotheses:

• 𝐻0 ∶ 𝑝 = 𝑝0

11
 • 𝐻1 ∶ 𝑝 ≠ 𝑝0

Test Statistic:

𝑝 ̂ − 𝑝0
𝑧=
√ 𝑝0 (1−𝑝
𝑛
0)

Where:

• 𝑝̂ = sample proportion
• 𝑝0 = hypothesized population proportion
• n = sample size

Implementation:
from [Link] import proportions_ztest

sample_size = 100
success_count = 60
population_proportion = 0.5

sample_proportion = success_count / sample_size

z_statistic, p_value = proportions_ztest(success_count, sample_size, population_proportion)

proportions_ztest(count, nobs, value) - Performs one-sample proportion z-test

Parameters:

• count: Number of successes

• nobs: Sample size
• value: Hypothesized proportion under 𝐻0

Example Output:

Sample Proportion: 0.60

Z-statistic: 2.0412
P-value: 0.0412
Reject the null hypothesis

Interpretation:

Sample proportion = 0.60 (60%)

Z-statistic = 2.04: Sample proportion exceeds hypothesized value by 2.04 standard errors

P-value = 0.0412: 4.12% probability under 𝐻0

Decision: p-value (0.0412) < 𝛼 (0.05), reject 𝐻0

Conclusion: Sufficient evidence that true proportion differs from 0.5

Application Example

12
Quality control: Testing fairness of random number generator

100 trials yield 60 successes (expected 50 under fairness)

Result: p = 0.0412 → Significant deviation from expected 50-50 distribution

11. Two-Sample Proportion Test

Purpose: Compare proportions from two independent samples.

Assumptions:

• Two independent samples

• Categorical outcomes
• Large samples (sufficient successes and failures in both groups)

Hypotheses:

• 𝐻0 ∶ 𝑝1 = 𝑝2
• 𝐻1 ∶ 𝑝1 ≠ 𝑝2

Test Statistic:

𝑝1̂ − 𝑝2̂
𝑧=
̄ − 𝑝)̄ ( 𝑛1 +
√𝑝(1 1
𝑛2 )
1

Where 𝑝̄ represents pooled sample proportion

Implementation:
success_count1 = 60
sample_size1 = 100
success_count2 = 45
sample_size2 = 100

successes = [Link]([success_count1, success_count2])

sample_sizes = [Link]([sample_size1, sample_size2])
z_statistic, p_value = proportions_ztest(successes, sample_sizes)

[Link]([values]) - Creates numpy array

Example Output:

Z-statistic: 2.1240
P-value: 0.0337
Reject the null hypothesis

Interpretation:

Z-statistic = 2.12: Difference equals 2.12 standard errors

P-value = 0.0337: 3.37% probability under equal proportions

13
Decision: p-value (0.0337) < 𝛼 (0.05), reject 𝐻0

Conclusion: Group 1 (60%) demonstrates significantly higher success rate than Group 2 (45%)

Application Example

A/B testing in web design:

Design A: 60/100 visitors convert (60%)

Design B: 45/100 visitors convert (45%)

Result: p = 0.0337 → Design A significantly outperforms Design B

12. Practical Applications

Complete Hypothesis Testing Workflow
Step 1: Define Research Question

Formulate specific, testable question about population parameter

Step 2: Establish Hypotheses

𝐻0 : Null hypothesis (no effect/difference)

𝐻1 : Alternative hypothesis (effect/difference exists)

Step 3: Select Significance Level

𝛼 = 0.05 (standard), adjust based on context

Step 4: Choose Appropriate Test

Consider: sample size, known/unknown 𝜎, data type, number of groups

Step 5: Collect Data and Calculate Test Statistic

Step 6: Determine P-value

Step 7: Make Statistical Decision

If p-value < 𝛼: Reject 𝐻0

If p-value ≥ 𝛼: Fail to reject 𝐻0

Step 8: Interpret Results in Context

Translate statistical decision into practical conclusions

Application Examples
Quality Control

Problem: Factory claims 2% defect rate. Sample of 500 units yields 15 defects (3%).

Test: One-sample proportion test

14
 • 𝐻0 ∶ 𝑝 = 0.02
• Sample: 𝑝̂ = 15/500 = 0.03
• Objective: Determine if observed 3% significantly exceeds claimed 2%

Medical Research

Problem: Evaluate efficacy of new antihypertensive medication

Test: Two-sample t-test

• 𝐻0 ∶ 𝜇𝑛𝑒𝑤 = 𝜇𝑠𝑡𝑎𝑛𝑑𝑎𝑟𝑑
• Measure blood pressure reduction in treatment and control groups
• Objective: Assess statistical significance of difference

Marketing Analysis

Problem: Assess impact of email marketing campaign on sales

Test: Two-sample t-test (or paired t-test if before/after design)

• 𝐻0 ∶ 𝜇𝑏𝑒𝑓𝑜𝑟𝑒 = 𝜇𝑎𝑓𝑡𝑒𝑟
• Compare average sales pre- and post-campaign
• Objective: Quantify campaign effectiveness

Educational Assessment

Problem: Compare online versus traditional instruction outcomes

Test: Two-sample t-test

• 𝐻0 ∶ 𝜇𝑜𝑛𝑙𝑖𝑛𝑒 = 𝜇𝑡𝑟𝑎𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙
• Compare mean test scores between delivery methods
• Objective: Evaluate pedagogical effectiveness

Test Selection Framework

One sample vs. two samples:

One sample: Comparison to known/hypothesized value

Two samples: Comparison between groups

Mean vs. proportion:

Mean: Continuous quantitative data

Proportion: Categorical binary data

t-test vs. z-test:

t-test: 𝜎 unknown (most common)

z-test: 𝜎 known or very large sample

15
13. Key Functions Reference
NumPy Functions:

[Link](value) - Sets random seed for reproducibility

[Link](loc, scale, size) - Generates normal random variates

[Link](array) - Computes sample mean

[Link](value) - Calculates square root

[Link](value) - Returns absolute value

[Link]([values]) - Creates numpy array

len(array) - Returns number of elements

Statistical Tests:

stats.ttest_1samp(sample, popmean) - One-sample t-test

stats.ttest_ind(sample1, sample2, equal_var) - Independent two-sample t-test

[Link](z) - Standard normal cumulative distribution function

proportions_ztest(count, nobs, value) - One-sample proportion z-test

proportions_ztest(counts, nobs) - Two-sample proportion z-test

Parameters:

equal_var=False - Welch’s t-test (unequal variances)

equal_var=True - Student’s t-test (equal variances assumed)

Return Values:

(statistic, p_value) - Standard return format for test functions

Common Patterns:

alpha = 0.05 - Standard significance level

if p_value < alpha: - Decision rule for hypothesis rejection

2 * (1 - [Link]([Link](z))) - Two-tailed p-value calculation

** - Exponentiation operator

16