Introduction to A/B
testing
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Ryan Grossman
Data Scientist, EDO
Overview
Introduction to A/B testing
How to design an experiment
Understand the logic behind A/B testing
Analyze the results of a test
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
A/B test: an experiment where you...
Test two or more variants against each other
to evaluate which one performs "best",
in the context of a randomized experiment
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Control and treatment groups
Testing two or more ideas against each other:
Control: The current state of your product
Treatment(s): The variant(s) that you want
to test
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
A/B Test - improving our app paywall
Question: Which paywall has a higher conversion
rate?
Current Paywall: "I hope you enjoyed your
free-trial, please consider subscribing" (control)
Proposed Paywall: “Your free-trial has ended,
don’t miss out, subscribe today!” (treatment)
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
A/B testing process
Randomly subset the users and show one set
the control and one the treatment
Monitor the conversion rates of each group to
see which is better
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
The importance of randomness
Random assignment helps to...
isolate the impact of the change made
reduce the potential impact of confounding variables
Using an assignment criteria may introduce confounders
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
A/B testing exibility
A/B testing can be use to...
improve sales within a mobile application
increase user interactions with a website
identify the impact of a medical treatment
optimize an assembly lines ef ciency
and many more amazing things!
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Good problems for A/B testing
Users are impacted individually
Testing changes that can directly impact their behavior
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Bad problems for A/B testing
Cases with network effects among users
Challenging to segment the users into groups
Dif cult to untangle the impact of the test
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Let's practice!
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Initial A/B test
design
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Ryan Grossman
Data Scientist, EDO
Increasing our app's revenue with A/B testing
Speci c Goals:
Test change to our consumable purchase paywall to...
Increase revenue by increasing the purchase rate
General Concepts:
A/B testing techniques transfer across a variety of contexts
Keep in mind how you would apply these techniques
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Paywall views & Demographics data
demographics_data = pd.read_csv('user_demographics.csv')
demographics_data.head(n=2)
uid reg_date device gender country age
0 52774929 2018-03-07 and F FRA 27
1 84341593 2017-09-22 iOS F TUR 22
paywall_views = pd.read_csv('paywall_views.csv')
paywall_views.head(n=2)
uid date purchase sku price
0 52774929 2018-03-11 04:11:01 0 NaN NaN
1 52774929 2018-03-13 21:28:54 0 NaN NaN
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Chapter 3 goals
Introduce the foundations of A/B testing
Walk through the code need to apply these concepts
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Response variable
The quantity used to measure the impact of your change
Should either be a KPI or directly related to a KPI
The easier to measure the better
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Factors & variants
Factors: The type of variable you are changing
The paywall color
Variants: Particular changes you are testing
A red versus blue paywall
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Experimental unit of our test
The smallest unit you are measuring the change over
Individual users make a convenient experimental unit
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Calculating experimental units
# Join our paywall views to the user demographics
purchase_data = demographics_data.merge(
paywall_views, how='left', on=['uid'])
# Find the total purchases for each user
total_purchases = purchase_data.groupby(
by=['uid'], as_index=False).[Link]()
# Find the mean number of purchases per user
total_purchases.[Link]()
3.15
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Calculating experimental units
# Find the minimum number of purchases made by a user
# over the period
total_purchases.[Link]()
0.0
# Find the maximum number of purchases made by a user
# over the period
total_purchases.[Link]()
17.0
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Experimental unit of our test
User-days: User interactions on a given day
More convenient than users by itself
Not required to track user's actions across time
Can treat simpler actions as responses to the test
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Calculating user-days
# Group our data by users and days, then find the total purchases
total_purchases = purchase_data.groupby(
by=['uid', 'date'], as_index=False)).[Link]()
# Calcualte summary statistics across user-days
total_purchases.[Link]()
total_purchases.[Link]()
total_purchases.[Link]()
0.0346
0.0
3.0
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Randomness of experimental units
Best to randomize by individuals regardless of our experimental unit
Otherwise users can have inconsistent experience
This can impact the tests results
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Designing your A/B test
Good to understand the qualities of your metrics and experimental units
Important to build intuition about your users and data overall
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Let's practice!
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Preparing to run an
A/B test
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Ryan Grossman
Data Scientist, EDO
A/B testing example - paywall variants
Paywall Text: Test & Control
Current Paywall: "I hope you are enjoying the relaxing bene ts of our app. Consider making a
purchase."
Proposed Paywall Don’t miss out! Try one of our new products!
Questions
Will updating the paywall text impact our revenue?
How do our three different consumable prices impact this?
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Considerations in test design
1. Can our test be run well in practice?
2. Will we be able to derive meaningful results
from it?
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Test sensitivity
First question: What size of impact is meaningful to detect
1%...?
20%...?
Smaller changes = more dif cult to detect
can be hidden by randomness
Sensitivity: The minimum level of change we want to be able to detect in our test
Evaluate different sensitivity values
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Revenue per user
# Join our demographics and purchase data
purchase_data = demographics_data.merge(
paywall_views,how='left', on=['uid'])
# Find the total revenue per user over the period
total_revenue = purchase_data.groupby(by=['uid'], as_index=False).[Link]()
total_revenue.price = [Link](
[Link](total_revenue.price), 0, total_revenue.price)
# Calculate the average revenue per user
avg_revenue = total_revenue.[Link]()
print(avg_revenue)
16.161
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Evaluating different sensitivities
avg_revenue * 1.01 # 1% lift in revenue per user
16.322839545454478
# Most reasonable option
avg_revenue * 1.1 # 10% lift in revenue per user
17.77
avg_revenue * 1.2 # 20% lift in revenue per user
19.393
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Data variability
Important to understand the variability in your
data
Does the amount spent vary a lot among users?
If it does not then it will be easier to detect a
change
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Standard deviation
[Link]() : Calculate the standard deviation of a pandas DataFrame
# Calculate the standard deviation of revenue per user
revenue_variation = total_revenue.[Link]()
print(revenue_variation)
17.520
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Variability of revenue per user
# Calculate the standard deviation of revenue per user
revenue_variation = total_revenue.[Link]()
17.520
Good to contextualize standard deviation (sd) by calculating: mean / standard deviation?
revenue_variation / avg_revenue
1.084
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Variability of purchases per user
# Find the average number of purchases per user
avg_purchases = total_purchases.[Link]()
3.15
# Find the variance in the number of purchases per user
purchase_variation = total_purchases.[Link]()
2.68
purchase_variation / avg_purchases
0.850
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Choosing experimental unit & response variable
Primary Goal: Increase revenue
Better Metric: Paywall view to purchase conversion rate
more granular than overall revenue
directly related to the our test
Experimental Unit: Paywall views
simplest to work with
assuming these interactions are independent
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Finding our baseline conversion rate
Baseline conversion rate: Conversion rate before we run the test
# Aggregate our data sets
purchase_data = demographics_data.merge(
paywall_views, how='inner', on=['uid']
)
# conversion rate = total purchases / total paywall views
conversion_rate = (sum(purchase_data.purchase) /
purchase_data.[Link]())
print(conversion_rate)
0.347
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Let's practice!
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Calculating sample
size
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N
Ryan Grossman
Data Scientist, EDO
Calculating the sample size of our test
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Null hypothesis
Hypothesis that control & treatment have the same impact on the response
Updated paywall does not improve conversion rate
Any observed difference is due to randomness
Rejecting the Null Hypothesis
Determine their is a difference between the treatment and control
Statistically signi cant result
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Types of error & con dence level
Con dence Level: Probability of not making
Type 1 Error
Higher this value, larger test sample needed
Common values: 0.90 & 0.95
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Statistical power
Statistical Power: Probability of nding a statistically signi cant result when the Null Hypothesis is
false
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Connecting the Different Components
Estimate our needed sample size from:
needed level of sensitivity
our desired test power & con dence level
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Power formula
Sample size increases = Power increases
Con dence level increases = Power decreases
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Sample size function
# Calculate the test power (some details omitted)
def get_power(n, p1, p2, cl):
alpha = 1 - cl
qu = [Link](1 - alpha/2)
diff = abs(p2 - p1)
bp = (p1 + p2) / 2
...
power = power_part_one + power_part_two
return(power)
# Calculate the sample size needed for the specified
# power and confidence level
def get_sample_size(power, p1, p2, cl, max_n = 1000000):
n = 1
while n <= max_n:
tmp_power = get_power(n, p1, p2, cl)
if tmp_power >= power:
return n
else:
n = n + 1
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Calculating our needed sample size
Baseline Conversion Rate: 0.03468 (calculated previously)
Con dence Level: 0.95 (chosen by us)
Desired Power: 0.80 (chosen by us)
Sensitivity: 0.1 (chosen by us)
sample_size_per_group = get_sample_size(
0.8 # Desired Power
conversion_rate,
conversion_rate * 1.1 # Lifted conversion rate,
0.95 # Confidence level)
print(sample_size_per_group)
45788
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Generality of this function
Function shown speci c to conversion rate calculations
Different response variables have different but analogous formulas
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Decreasing the needed sample size
Choose a unit of observation with lower variability
Excluding users irrelevant to the process/change
Think through how different factors relate to the sample size
CUSTOMER ANALYTICS AND A/B TESTING IN PYTHON
Let's practice!
C U S TO M E R A N A LY T I C S A N D A / B T E S T I N G I N P Y T H O N