STAT 301
A/B testing
Hypothesis testing 中比较来自于两个分布的mean或 A/A testing：在进行正式实验前先对A/B两组用户采用相同的实验条件，
proportion, 比如引入了新技术、新流水线等，对产品 看两组之间的差异,如果差异显著则舍弃，重新选组
的均值的影响（变大，变小，还是不变） • For A/A testing， we know H0 is true.
Population
• Means Early stopping in A/B testing：ending the experiment earlier than originally
designed. Stopping a reject H0 as soon as the p-value go below /t-
• Proportion statistics go beyond the specified siginificance level, which can
drasticaly increase the type I error rate.
A/B testing：【week13-d1】
定义：比较Control A和variation B, to determine which one
is the best.
Sequential tests: 按数据出现的顺序，sample size 是变化的如果实验early
e.g. new products, new methods, new website/design, app, stopping。有两种分类
digits, etc. Group(分组）: the analyst pre-specifies when to inspect the
For example, to increase the donation by a new website, data and performs each analysis as a fixed sample one
using A/B testing to decide the new website （some Full: the analyst performs an analysis after every new
observation
changes in pictures，in buttons， in colours，etc.）is
better than control one
Some terms： Peeking problem : check the intermediate results for statistical significance
Response variable：与testing的目的有关，用于测度变化后 between Control A和variation B, and make decisions.
的效应，比如新website是否提高了浏览量 Peeking will increase p-value
Covariable(类似于回归中的input variables/independent Peeking is ok, if the sample size is fixed in advanced
variables): 识别那些能影响response variable的变量，
比如新website的style、color、功能按钮等变量的变 Type-I error: H0是真却拒绝，对应的概率为a
化 Type-II error: H0是假而接受，对应的概率为B
Statistical methods统计方法：可以参考two-samples Power of test: H0是假而拒绝H0的概率，即1-B, 越大越好，表示犯第二类
错误的概率越小
hypothesis testing
Controlling type I error rate methods:
Bonferroni adjustmentP-Values(significant level, critical values),
!注意： Pocock methodless conservative conservative 控制第一类错误率in
Design procedure：post questions/aimsdesign sequential tesing with early stops
experimentalidentify methodsanalysis O’Brien-Fleming method less conservative 控制第一类错误率 in
sequential tesing with early stops
datadecisitions
Controlling type I error rate methods:【week13-
d2】
Bonferroni correction (P15/32)to adjust P-
Values(significant level, critical values),
For group or full,
Pocock method (P17/32) less conservative 控
制第一类错误率in sequential tesing with early
stops
O’Brien-Fleming method less conservative 控
制第一类错误率 in sequential tesing with early
stops
Pocock method
Advantage: easy to use ,and p-value is the same in the interim
analysis
Disadvantage:
• Fixed number of the interim analysis, analyses can
not be added after starting trails
• Just report overall p-value not the p-value when it is
stopped. E.g. report p-value=0.05,
Two-samples hypothesis
• A two-sample hypothesis test is a statistical method that compares two random samples from
different populations to determine if there is a statistically significant difference between
them. The test is often used in research to compare the effectiveness of two treatments or
processes
• The two-sample t-test (also known as the independent samples t-test) is a method used to test
whether the unknown population means of two groups are equal or not.
– a two-sample t-test is used to analyze the results from A/B tests.
– The test statistic(t-score) is
Degrees of freedom
• The two-sample z-test is a method used to test proportions are equal or not.
• two-sides test(双侧检验）
•
left-tailed (左侧检验） right-tailed（右侧检验）
– The test statistic(Z-score) is
– Pc is pooled proportion
Exercises:
[Link]
/10%3A_Hypothesis_Testing_with_Two_Samples
Some notes:
• Bootstrapping create CI, approximate the sampling distribution of the LS estimators in LR,
compute SE
• Statistically significant means that p-value of the ovserved statistic is below the significance
level of the test
– If p-value 小于significant level, 那么 reject H0,
– If p-value 大于significant level, 那么fail to reject H0,
考点：H0原假设
答案：C
考点：H1原假设
答案：A
考点：Two-samples hypothesis test
答案：D
考点：significance level，
Statistically significant
答案：C
考点：rules of test H0, H1【参考
week3-d2】(P11/32)
答案：B