SDE-INTERNAL
Module 5 Required Assignment
Case: Advertising Experiments at RestaurantGrades
<Your Name>
Suggested time: 120 minutes
Note: This is a required assignment and counts towards programme completion.
Task: You have gone through the case study to understand the advertising
experiments at RestaurantGrades. In this assignment, you are provided
with a data set consisting of results from a hypothetical experiment. You
are expected to make recommendations based on the data. Studying this
case and completing this assignment will help you understand how to
analyse, design and interpret experiments. Read the questions carefully
and analyse the data before you make a recommendation about which, if
either, advertising package is effective.
1. Why should organisations run experiments? What are the
core concepts in designing and interpreting experimental
data?
To establish causal relations. Experiments are one of the few methods that allow
organisations to determine cause and effect, not just correlation.
Core concepts in designing and interpreting experimental data
- Treatment and Control Groups
- Randomisation
- Independent & dependent variables
- Statistical signicance
- Confounding factors & validity
- Effect size and practical significance
SMU – Data Science & Analytics for Strategic Decisions 1
SDE-INTERNAL
2. How should firms use experimental results to make
managerial decisions? Mention at least two points.
Firms should use experimental results to make data-driven managerial decisions,
relying on empirical evidence rather than assumptions to determine which
strategies or initiatives genuinely improve performance. Experiments help identify
cause-and-effect relationships, guiding managers on whether to adopt, scale, or
discard an intervention.
However, decisions should consider not just statistical significance but also
practical significance, evaluating whether the observed effect is large enough to
justify the associated costs and align with business objectives.
In short, experiments enable organisations to make informed, evidence-based
choices that balance analytical rigor with strategic relevance.
3. Is the current advertising package good or should
RestaurantGrades switch to the new advertising package?
Justify your answer with at least two reasons.
RestaurantGrades should switch to the new advertising package (Treatment 2)
rather than continue with the current one (Treatment 1), as it demonstrates
stronger and more meaningful improvements across key performance metrics.
Treatment 2 significantly outperformed both the control group and the current
package in generating pageviews, calls, and reservations. Compared with the
control, Treatment 2 produced an average of +63 more pageviews and +7.7 more
calls per restaurant, both with p < 0.001, indicating strong statistical significance.
Although Treatment 1 also increased traffic, its impact on conversions
(reservations) was negligible and statistically insignificant (p ≈ 0.51). In contrast,
Treatment 2 not only raised engagement but also improved the reservation-to-call
rate, suggesting better conversion efficiency.
The two-proportion tests further reveal that Treatment 2 achieved higher
reservation and call rates per pageview, indicating more effective ad performance
and a greater likelihood of revenue generation. Treatment 1, by contrast, showed
lower or even negative differences versus the control group for conversion metrics.
From a managerial perspective, this means the current advertising package does
SMU – Data Science & Analytics for Strategic Decisions 2
SDE-INTERNAL
not justify continued investment, while the new package offers tangible, scalable
benefits.
4. What additional information would you need if there was
opportunity to collect more data? List at least two points.
1. Customer and restaurant characteristics: Collecting data on factors such as
restaurant size, cuisine type, location, pricing level, and prior popularity would
help determine whether the advertising effects vary across different
segments. This allows for targeted recommendations, e.g., identifying if the
new package works better for premium restaurants or those in urban areas.
Controlling for these variables would also improve the internal validity of the
experiment by reducing confounding effects.
2. Financial and behavioural outcomes : Including revenue, profit margins, and
repeat customer behaviour would help evaluate the true business impact and
ROI of each advertising package. Current metrics (pageviews, calls,
reservations) measure engagement but don’t reveal whether the increased
traffic translates into higher sales or long-term customer value.
5. Did RestaurantGrades have to run the experiment, or could they
have just used the data for businesses that advertise versus
businesses that do not advertise? Justify your answer with two
reasons.
RestaurantGrades had to run the experiment rather than simply comparing
businesses that advertise with those that do not, because only an experiment can
establish a causal relationship and provide unbiased, reliable insights.
1. To establish causation rather than correlation : If RestaurantGrades merely
compared advertisers and non-advertisers, any observed difference in
performance (e.g., pageviews, calls, or reservations) could be due to pre-
existing differences, such as restaurant popularity, budget size, or location,
rather than the advertising itself. Running a randomised controlled
experiment ensures that businesses are comparable across all
characteristics except for the advertising treatment, allowing
RestaurantGrades to confidently attribute performance differences to the
advertising package.
2. To avoid selection bias and confounding factors : Businesses that choose to
SMU – Data Science & Analytics for Strategic Decisions 3
SDE-INTERNAL
advertise may already be more motivated, better managed, or have higher
marketing budgets, which could independently influence their outcomes.
Without random assignment, these confounding factors would bias the
results and lead to incorrect conclusions about advertising effectiveness. The
experiment eliminates this bias by randomly assigning restaurants to
treatment and control groups, ensuring that differences in outcomes reflect
the true impact of the advertising package.
SMU – Data Science & Analytics for Strategic Decisions 4