Core Concept: Design of Experiments (DOE)
The core concept of these lessons is Design of Experiments (DOE),
which is the process of designing a method to
collect a subset of data quickly and efficiently
when a full dataset is unavailable
impossible or take too long to obtain
The goal is to ensure the collected data is sufficient to answer specific
analytical questions, such as which marketing strategy is most effective or
which factors lead to a startup's success.
--------------------------------------------------------------------------------
1. Foundations of DOE: Comparison, Control, and Blocking
Before applying specific methods, the sources establish foundational
theories for valid experimentation.
Comparison and Control: To determine the effect of a specific
factor (e.g., the color of a car), you must compare it against
alternatives while controlling for other factors. For instance, if
comparing red and blue car prices, you must ensure the datasets
have a similar mix of car ages and types to avoid confounding
variables.
Blocking: A blocking factor is an element that creates variation
(e.g., car type: sports car vs. family car). By accounting for these
factors—analyzing sports cars and family cars separately—you
reduce variability in your estimates and gain more precise results.
Feature Control Blocking
What is it? Ensuring test groups Grouping data by a
are similar across known source of
important variables variation
Making both groups Grouping by a known
similar before testing noise factor
Remove bias before Reduce noise within
Goal
the experiment the experiment
During design —
When does it During analysis — how
before data is
happen? you structure the data
collected
"Is something else
"Are my groups fair to
creating unwanted
The question it compare?"
variance?"
answers "Are both groups fair
"Is training time hiding
to compare?"
inside my data?"
Does a Protein Supplement Improve Workout Performance?
One group is full of Some people train in
The Problem it beginners, the other the morning, some at
solves full of athletes — night — energy levels
unfair comparison differ naturally
Ensure both groups
Analyze morning
have a similar mix of
What you do trainers and night
fitness levels, age, and
trainers separately
diet
Without it, supplement
Without it, morning
looks better just
trainers naturally
Why it matters because the athletes
outperform and inflate
happened to be in that
supplement results
group
If ignored Results are biased Results are noisy
--------------------------------------------------------------------------------
2. A/B Testing
A/B testing is a simple statistical method used to choose the best option
between two alternatives.
Theory and Math: This method typically uses binomial data (e.g.,
clicked vs. not clicked). A hypothesis test is then applied to
determine if the difference in performance (such as a 10% click rate
vs. 5%) is statistically significant.
Application: Choosing between two banner ad designs to see
which generates more clicks.
Requirements:
1. Data must be collected quickly enough to be useful.
2. The data must come from a representative sample.
3. The sample size must be small compared to the total
population to ensure there are enough people left to benefit
from the results.
Disadvantage: It can be wasteful because it may continue to
show an inferior alternative to many users just to complete the fixed
test period.
The A/B Testing Disadvantage Explained
Imagine you're testing two banner ads:
Ad A (Red Ad B (Blue
Button) Button)
Click
3% 12%
rate
Status Clearly losing Clearly winning
The Problem
In a proper A/B test, you commit to a fixed test period — say 30 days.
Even if by Day 5 it is already obvious that Ad B is winning, you must
keep showing Ad A to users until Day 30 to complete the test.
That means for 25 more days, thousands of users are seeing an ad that
you already know is worse.
Why is that wasteful?
What's Being Wasted
Thousands of people saw the inferior ad
Users
unnecessarily
You lost potential clicks that Ad B would have
Clicks
generated
Revenu Every day Ad A runs instead of Ad B = lost
e money
One Line Summary
You already know the answer, but the test forces you to keep running
the losing option anyway.
--------------------------------------------------------------------------------
3. Factorial Designs
When there are multiple factors to test simultaneously, factorial designs
are used to determine how much each factor contributes to the outcome.
Full Factorial Design
Theory: This involves testing every possible combination of
factors.
Math: If you have 2 fonts, 2 wordings, and 2 backgrounds, you have
combinations. If you have 7 factors with 3 choices each, you face
combinations.
Analysis: ANOVA (Analysis of Variance) is used to determine the
importance of each factor.
Disadvantage: It becomes impractical when the number of
combinations is too high.
Fractional Factorial Design
Theory: You test only a balanced subset of combinations.
Properties of a Balanced Design: In a good fractional design,
each choice appears the same number of times, and each
pair of choices appears together the same number of times.
Advantage: It allows you to test a wide range of values with
significantly fewer trials.
What are the two defining properties of a "balanced design"? According to
the sources, a balanced fractional design has these two properties:
o Each choice appears the same number of times.
o Each pair of choices appears together the same number of
times.
Why Does Balance Matter?
Without Balance With Balance
Some combinations tested more than others Every factor gets equal representation
Results are biased toward over-tested
Fair comparison across all factors
combinations
Can't trust which factor actually caused the Can confidently estimate each factor's
outcome effect
--------------------------------------------------------------------------------
4. Multi-Armed Bandits
This method addresses the limitations of A/B testing by balancing the
need for information with the desire for immediate value.
Core Theory: Exploration vs. Exploitation:
o Exploration: Focusing on gathering more information to
determine which alternative is truly best.
o Exploitation: Focusing on immediate value by showing the
alternative that has performed best so far. Maximize reward
based on what you already know
Mechanism:
o Start with no information and equal probabilities for all
alternatives.
o Run tests and update probabilities (often using Bayesian
updates) of each alternative being the "best".
o Assign future tests based on these updated probabilities,
meaning the better-performing options are shown more
often while still testing others.
Advantage: It minimizes lost value by not wasting as many trials
on bad alternatives and helps you learn faster "on the fly".
Disadvantage: There is no simple rule for the best parameters
(e.g., how often to recalculate or how to assign tests); it requires
careful tuning.
Multi-armed bandit models increasingly favor the best-performing
option (exploitation) as confidence in its effectiveness grows, while still
allowing for exploration of other options to a lesser extent.
Here’s how it works:
● Say we have k alternatives
● We start with no information
● We have an equal probability of selecting each alternative
● After performing tests, we get more information
● We repeat
● Choose an alternative to test based on probability of each being best
● After the test, update the probabilities of each one being the best
We continue repeating this process of collecting data and updating
probabilities until we’re reasonably sure which option is the best.
--------------------------------------------------------------------------------
Why "No Simple Rule" is a Problem
Think of Multi-Armed Bandit like a slot machine 🎰
You have 3 slot machines (Ad A, Ad B, Ad C) and you want to find the best
one while still making money.
The Decisions You Have to Make
Decision The Question Why it's hard
How many tests Do you pull each machine Too few = unreliable data.
before 10 times? 100 times? Too many = wasted time on
recalculating? 1000 times? bad options
After Ad A wins 3 times,
How do you
how much do you No standard formula —
update
increase its chances of everyone does it differently
probabilities?
being shown?
Too aggressive = you stop
How do you Do you show the winning exploring too early. Too
assign new tests? ad 70% of the time? 90%? cautious = you keep
showing bad ads
A/B Testing vs Multi-Armed Bandit
A/B Testing Multi-Armed Bandit
Rules Clear and fixed No standard rules
Parameter Simple — just set test Multiple parameters to tune
A/B Testing Multi-Armed Bandit
s period
Consistenc Everyone runs it the Every practitioner does it
y same way differently
Risk Wasteful but predictable Flexible but unpredictable
One Line Summary
A/B Testing has a rulebook everyone follows. Multi-Armed Bandit hands
you blank paper and says "figure it out."
Summary of Relation to Core Concept
Each method relates back to the core concept of efficient data
collection for decision-making:
A/B Testing is the simplest form of efficient selection between two
items.
Factorial Designs handle complex scenarios with many
dimensions, using fractional sets to maintain efficiency.
Multi-Armed Bandits maximize the value of the data collection
process itself by dynamically shifting between exploration and
exploitation