DIGITAL PRODUCT MANAGEMENT (DPCCBJ23-5)
– A/B TESTING AND VISUAL WEBSITE OPTIMIZER
(VWO) – PART 1
What are we trying to test?
➢Will changing the colour of the “Download” button in
a music app increase the song downloads?
➢ Will the proposed new package design of an FMCG
product sold from its online store increase its sale?
➢ Will moving from list to tile format in the home page
of an e-commerce site increase sale?
➢ Does changing the human models webrooming
apparels increase conversion if the models appears to
be from the countries the visitors are located?
➢ Will a banner announcing the upcoming bumper
Diwali sale actually increase the conversion or will it be
perceived as a “hard sell” by the customers?
Tests using A/B Testing Software –
Examples
➢A/B test
➢A/B/n test
➢Multivariate test
➢Testing two headlines and three hero images simultaneously
➢Split URL test or Redirect test
➢Allows test separate web pages against each other
➢Two different landing pages/ complete redesign of a page
➢Personalization
➢Example → Promoting seasonal clothing based on visitor’s
geography or offering free shipping to the best customers
➢Banner
Some Important Terms
➢Variant
➢Change to a single element, or changes to multiple elements, or a
totally different page in an experiment
➢Treatment
➢The original and all variants
➢Section
➢A group of variants
➢A/B test - one section (with one or more variants)
➢MVT - multiple sections simultaneously
➢Session
➢Time a user is active on experiment site or app
➢New session – If user is inactive for X unit time or more
➢Part of the original session - leaving site and returning within X unit
time
The Objective
➢Pick a treatment that has the largest conversion rate
in the long-run
➢Goal (e.g. sign-up) or Transaction/e-commerce conversion
➢Only way to know this with certainty is to
➢Observe every single website visitor for the treatment’s entire
lifetime
➢OR → Take a random sample of users to estimate
➢Which treatment has the largest conversion rate in the long-
run
➢Why DO we NEED a platform to test
➢Difference with physical product versions
➢Introduction to
Flicker Effect – Common
Reasons
➢The Code Snippet is in the wrong place → Many other
scripts are loaded before testing the tool script
➢The Code Snippet was installed through a third-party
tag management tool like GTM (instead of directly on
the page and you don’t control the load order)
➢The Code Snippet is too large
➢Other third-party programs are interfering
➢Image size or if the image is loaded from an external
CDN (Content Delivery Network)
Flicker Effect – Common
Reasons – Continued
➢Running scripts asynchronously
➢Your testing tool code was added to the <body> on some
pages and the <header> on others
➢An element of your test triggers an action that disables or
changes your testing tool code
➢Overall website load speed is slow
➢Many new sites are based on frameworks like Angular or
React
➢This makes jQuery an unnecessary burden
➢Page-hiding snippet is used to prevent the flicker effect
Additional Concepts
➢ Server-side A/B Testing – Feature Testing
➢ Different from UI/ UX Testing
➢ Full Stack Testing (includes App)
➢ Canary deployment (Feature roll-out)
➢Difference with Personalization
➢ Bandit Approach in Testing
➢ In traditional A/B testing methodologies
➢Traffic is evenly split between two variations (both get 50%)
➢Multi-armed bandits dynamically allocate
➢More traffic to variants that are performing well – Without performing a
new experiment
➢While allocating less and less traffic to underperforming variants
The Theory Part
Frequentist Approach in A/B
Testing – Issues
➢The REAL question → What is the probability that
variation B is better than variation A
➢𝑃 𝜆𝐵 > 𝜆𝐴
➢Frequentist technique will choose
➢Null Hypothesis → 𝐻0 : 𝜆𝐵 = 𝜆𝐴
➢Will use → Maximal likelihood estimate
➢Will compute → 𝑝 = P t ≥ 𝑡𝑒 𝐻0
➢Many A/B testing tools use Null Hypothesis
Significance Testing (NHST)
➢Checks if the observed difference is random
➢Make a recommendation when the chance of a false positive is
small
Issues with focusing on false
positive
➢False positive (Type I error) - A difference between
treatments when in fact there was no difference
➢In A/B Test → DO NOT focus on false positives
➢False positive rate in A/B Test is likely higher
➢There is always some difference between
treatments
➢It is just a matter of the magnitude of the difference and
which treatment is better
➢Instead of focusing on false positives
➢Need to → Make a recommendation if it is confident that
➢The recommendation is suboptimal by only a negligible amount
Issues with Statistical significance
and p-value
➢Statistical significance by itself
➢Should NOT be considered → As a stopping rule in A/B Test
➢p-values are NOT the same as
➢The probabilities needed in A/B Test
➢A p-value
➢INFORMS a probability → The observed difference occurred due to
random chance if the null hypothesis is true
➢DOES NOT INFORM → The probability that B is better than A
➢The probability we WRONGLY use → In selecting B over A
➢Significance in statistical hypothesis testing
➢ABLE to say → The probability of concluding a difference between
treatments (when no true difference) is at most 5%
➢NOT able to say → The variant has a 95% probability of beating the
original
A/B test with an original and a
single variant
➢Bayesian considers two hypotheses:
➢Check through H1: The original is better than the variant
➢Check through H2: The variant is better than the original
➢P(H | data) → Probability of a hypothesis being true
given the data we’ve observed
➢Use Bayes' theorem to determine P(H1 | data) and
P(H2 | data)
➢Must decide on what’s called a prior
➢Bayes' theorem
➢Allows us to take data from a random sample of users
➢Make estimates on something that is unobservable
Bayesian Approach in A/B Testing –
Procedure Followed
➢ Visitors in a Treatment = 𝛼 + 𝛽 → Conversion = 𝛼
➢ Assume BETA Distribution
➢Higher 𝛼 = distribution has more weight towards higher
probabilities → Higher Conversion
➢Higher 𝛽 = distribution has more weight towards lower probabilities
→ Lower Conversion
➢Define PRIOR → Uniform prior
➢ Consider PRIOR as BETA (1,1) → Here 𝛼 = 1, 𝛽 = 1
➢ Update to get POSTERIOR
➢Assume Conversion happened in 𝕄 cases out of 𝕋 cases
➢Then POSTERIOR is BETA (1 + 𝕄, 2 + 𝕋 − 1 − 𝕄)
= BETA (1 + 𝕄, 1 + 𝕋 − 𝕄)
➢ Compare Treatments → Get samples from POSTERIOR
➢Fraction of cases where → Conversion Treatment B > Treatment A
Why Bayesian Approach in
place of Frequentist Approach
➢Interpretation is easier than p-value
➢Does NOT require to determine a fixed sample size
before starting the experiment
➢Not a binary decision like Rejecting or Not rejecting
➢Continuously updated and dynamic
➢Online R compiler →
[Link]
➢ 2-sample z test →
[Link]
spx
Bayesian Approach in A/B Testing –
Reproduce Results using R
➢ library(MASS) → rbeta (𝒏, 𝜶, 𝜷)
Create Account in VWO