Chapter 10
Data – Understanding the
impact on the Product change
Bill Clinton (sort of)
„Data, you fool!”
Words to remember
„Measure, analyze,
conclude and act!”
Data summary
• Being data driven is the highest value for best Product
Managers
• When presented, make sure it’s easy to understand the
data
• Start your day by looking at your key data – the KPIs
• Look for different anomalies and observe trends
• Nothing not set in data, is just an opinion
Definition
„A Key Performance Indicator is a
measurable value that demonstrates
how effectively a company is
achieving key business objectives.”
Source: [Link]
Leading metric
„Leading indicators are sometimes
described as inputs. They define what
actions are necessary to achieve your goals
with measurable outcomes. They “lead” to
successfully meeting overall business
objectives, which is why they are called
‚leading’.”
Source: [Link]
Leading metrics examples
Example leading indicators for an enterprise business
software company with an annual subscription fee might be:
• Percent of customers that sign up for two-year
agreements
• Number of customers that login daily
• Number of customers that made at least a single purchase
Lagging metric
„Lagging indicators on the other hand
measure current production and performance.
While a leading indicator is dynamic but
difficult to measure, a lagging indicator is
easy to measure but hard to change. They are
opposites, and as such a lagging indicator is
sometimes compared to an output metric.”
Source: [Link]
Lagging metrics examples
Lagging indicators measure output that’s already occurred to
gain insight on future success. They do this by measuring
things like:
• Profit
• Expenses
• Retention
• Revenue
KPI summary
• You will be monitoring different number of KPIs, make sure
to keep focus on the rigth ones
• Observe all of them in case of anomalies
• Use KPI impact when prioritizing backlog (value indicator)
• Make sure you are observing the correct figures
• Act as a guardian of the true company goals
Words to remember
„Stand on the shoulders
of (data) giants”
Source: [Link]
How to get the right data?
• Proximity is your best ally
• Competitors data is usually a good source
• People willing to share
• Try to look for related science papers
(google scholar)
• Last and least: online blogs and articles
Words to remember
„KPIs are simply
numerical
representation of user
actions”
User segmentation summary
• Best way to know your users is to ask them who they are
during profile creation
• You can also use tracking tools
• Understanding the groups who use your product allows
you to develop better tailored updates
• You can prioritize your backlog based on the tasks that
benefit most valueable group of users
Cohort definition
„Cohort is a group of users that
joined your product within a
certain span of time, usually
with a single day or a week.”
Cohort segmentation summary
• Similar to A/B testing, but less reliable
• Helps you determine the feature impact on new and
existing users
• Helps you compare different iterations of features
dedicated for new users or behaviour based on time of
joining
Presenting the data summary
• Make sure you present the key data, not all relevant data
• Make the data easy to understand
• Be ready to provide additonal details upon request
• If you are not sure ask yourself: „Would my mum
understand this?
The perfect data reporting slide
• 1-5 numbers per slide
• The key figures get the largest font and bolding, the less
important one stand out less
• Not more than two charts on a slide
• Introduce addtional charts on a diagram slide by slide
• Put axis legends on your diagrams
• Make your data visually pleasing
• Add small explanations for abbreviations or KPIs
Booking rates for day example day – version 1
HOTEL 1 HOTEL 2 HOTEL 3 HOTEL 4
Room 1 Room 1 Room 1 Room 1
Room 2 Room 2 Room 2 Room 2
Room 3 Room 3 Room 3 Room 3
Room 4 Room 4 Room 4 Room 4
Room 5 Room 5 Room 5 Room 5
Booking rates for day example day – version 2
HOTEL 1 HOTEL 2 HOTEL 3 HOTEL 4
60% 60% 60% 60%
Booking rates for day example day – version 3
60%
The perfect slide
60%
Average booking rate at our hotels
A good slide slide
Average hotel online rating Average restaurants rating
4.6/5 4.8/5
60%
Average booking rate
Rooms ready for guests Open positions
95% 23
Iterative chart – step 1
income (in millions of dollars)
70
60
50
40
income
30
20
10
0
1 3 5 7 9 11
month
Iterative chart – step 2
Income and revenue
70
60
income and revenue (mil $)
50
40
30
20
10
0
1 2 3 4 5 6 7 8 9 10 11 12
month
income (in millions of dollars) revenue (in millions of dollars)
Iterative chart – step 3
Income and revenue
120
100
80
60
40
20
0
1 2 3 4 5 6 7 8 9 10 11 12
month
income (in millions of dollars) revenue (in millions of dollars) booking rate [%]
Words to remember
„Never reject an idea
just because it is
difficult to pull off.”
Data vs hunch summary
• Use common sense
• Make sure you correctly manage risk
• If not data available, try to generate it (with user survey)
• Find the least impactful way to test a risky idea (MVP,
limited release, hackaton, etc)
• Invest in risky initiatives only if product is stable
• Disclose your ideas with stakeholders early on
AB
Words to remember
„A/B testing is perhaps
the best tool to verify
any product
hypothesis”
A/B testing summary
AB
• Typically done for even, 2-4 groups of users
• 2-4 tested variants of a increment
• Enables you to choose the best option
• Provides estimates for the long-term impact of the
increment on the product
• Consolidates different opinions of team and stakeholders
• Make sure to prepare and test A/B test dashboard ahead of
time
A/B testing summary
AB
• Avoid holidays when planning the A/B test period
• Try to run one test at a time or devide the test between
diffent audiences
• Run it from 1 to 4 weeks, but as short as possible
• Make sure you can start and stop the test at any time
• Don’t be afraid to make a decision on ending the test
quicker
A/B testing summary
AB
• Monitor the test daily, early morning
• Gossip on the early results to your team
• Analyze the results on your own first
• Delegate the in-depth report, look at different user groups
and cohorts
• Don’t toss away a failed test likely
• Communicate the key KPI impact to stakeholders once the
test is done
Fallout report
Observing users’ screen directly
Narrated in-office product session
Heatmap