DIGITAL PRODUCT MANAGEMENT (DPCCBJ23-5)
– A/B TESTING (OUTPUT ANALYSIS)
Status messages - Actions
➢Most software won't declare an outcome unless an
experiment has received data for at least two weeks
➢Waiting for data
➢No experiment sessions
➢Not enough experiment sessions
➢ Consider adjusting your variant weights
➢Keep your experiment running
➢Run your experiment until at least one variant has a 95
percent Probability to beat Original (PBO)
➢What happens if there’s no difference between variations?
➢Is that a problem?
➢Is the interpretation same as in development of a new
pharmacological drug?
Status messages – Actions –
Continued
➢When it is safe to end the experiment now since the
outcome is unlikely to change and it would be better to
try a new experiment?
➢No leader found
➢The original is the leader
➢At least one variant is better than the original
➢Deploy the one with the highest probability to be best
➢If the experiment is still running - wait longer to find the absolute best
➢When It's safe to end the experiment now since the
outcome is unlikely to change?
➢One or more leaders found - Deploy the variant with the
highest PBB
➢A variant is the leader - Deploy or implement the leading
variant
Interpretation of Conversion rate
– Use of Credible Interval
➢Conversion rate provides X% (+/- Y)
➢In A/B testing - Credible intervals
➢Based on the range of likely values for a variant – A probability
statement
➢Used to managing the risk associated with implementing a new
variant
➢We are 95% confident that the conversion rate is X% +/-
Y%
➢If the 95% credible interval for a probability distribution is
[0:35; 0:40]
➢We are 95% confident that the true value of the parameter
➢Is contained in the interval [0:35; 0:40]
➢If the two conversion ranges overlap
➢Continue testing to get a valid result
Bayesian Approach in A/B Testing –
Pageviews per session using R – NOT
in Syllabus
➢library(rstanram) → May need to install
➢Fits generalized linear models (GLMs) using Bayesian methods
➢model_A = stan_glm(pageviews ~ 1, family =
poisson(link = "log"), data = [Link](x = rep(1,
sessions)))
➢Prior → Normal (0,10) for Intercept (~ 1)
➢posterior_predict(model_A, draws = n)
➢95% Credible interval
➢quantile(difference, c(0.025, 0.975))
➢Online R compiler →
[Link]
A/B Testing Recap Questions
1. What is the need for an A/B Testing platform?
2. What is the difference among treatment, variant, and
combination?
3. What is the difference between primary and secondary
metrics?
4. What is the difference between an event and a session?
5. When should multi-arm bandit be used? What is its
advantage over the custom distribution of traffic?
6. When should less than 100% traffic be allocated for the
experiment? How do we calculate the traffic percentage
that is directed to control?
A/B Testing Recap Questions –
Continued
7. What is the problem with the flicker effect? How can it be
controlled?
8. Should one user always see the same treatment during the
experiment? If so, how can this be ensured?
9. What is the difference between confidence interval and
credible interval?
10. On which calculation is A/B testing decision made – (a)
conversion rate, (b) PBO (c) PBB? What are the differences
between them?
11. What is the problem if the frequentist approach is adopted?
12. How do you decide to “continue the present experiment” or
“start a new experiment”? Explain numerically and graphically.
Common A/B Testing Mistakes
1. Testing without a reason
2. Testing more than one element
3. Spending all your time in A/B testing
4. Blindly following A/B testing best practices
5. Testing elements at different times
6. Not eliminating confounding variables
7. Testing elements that won’t bring a lift
8. Giving up after one test
9. Some common issues for wrong outcomes
➢ Peeking/Novelty effect/Carryover effect