Measuring effectiveness

Merry Baskin
Source: WARC Best Practice, July/August 2010
Downloaded from WARC

The pressure to measure effectiveness has risen with the increasing influence of client procurement
departments and the global recession. We need to demonstrate not just attitude or behaviour changes,
but also the financial returns. To know whether or not you have been successful is to know what you
set out to achieve in the first place. It is not enough to know whether or not you succeeded in meeting
your objectives; you also need to understand how or why you did (or didn't). Successes do not happen
overnight, but evolve from a virtuous circle of hypothesis, testing, learning and feedback. Knowing how
to predict and measure effectiveness will encourage clients to invest in the long-term.

Merry Baskin advises on how to demonstrate the effectiveness of marketing budgets through ROI,
behavioural and intermediate measurements

The burden of proof for marketing funds to have been invested efficiently and effectively has always been
Important. That pressure has intensified with the increasing influence of client procurement departments and the
frugality imposed by the global recession.

In the 'Era of Accountability', we need to demonstrate not just the attitudinal or behaviour changes effected by
our campaigns, the perceptions of the brand we have altered, but also the financial return of those campaigns.

But only very rarely is there a direct link between the ad campaign and its impact on sales. However, there are
other roles for performance measurement: to monitor progress against objectives; to check the strategy is
working as intended; to identify ways in which communications can become more effective in future; to set and
decide future budgets and media spends; and to quantify the impact we have had (Return on Marketing
Investment (ROI or ROMI). Even if your campaign bombs, a return on learning (from your mistakes) is most
valuable.

MEASUREMENT TYPES
There are three different kinds of measurement:
Intermediate measures: awareness, message communication, brand image and attitude shifts are all usually
covered by some sort of consumer tracking study.

Behavioural measures:

response rates, enquiries, or compliance with proposed changes (such as getting tax returns in on time or
different usage occasions).

ROI measures and effects:

Sales correlations, monetised payback, efficiency, human and societal impact.

Some people dispute the value of intermediate measures versus hard commercial ones. Some claim that
intermediate measures are the most important, given they can't control the consumer experience beyond the
advertising. However, the reality is that the most effective campaigns tend to show a wide range of effects to
build really compelling cases, so the more measures that you can demonstrate to have shifted, the better.

A BRAVE NEW WORLD

As new media and marketing channels proliferate, so do the options for evaluation. It is important to be able to
distinguish the individual contribution of each component in the mix, not least to justify its continuing presence in
the budget.

There are also exciting developments in how we understand communication works, such as neuroscience,
which may rewrite the rules. In particular, we have been forced to recognise the crucial role played by our
emotions in how we not only make decisions, but also how we respond to advertising and other brand stimuli. It
has been powerfully demonstrated that emotions are central to our decision-making and our rational
consciousness is considerably less dominant.

These findings have huge implications for the research industry, those traditional providers of pre-testing and
tracking surveys, some of whom are still wrestling with the shift from face-to-face and telephone interviewing to
online. For decades, they have been promulgating research techniques (let alone fieldwork methods) that have
barely moved on from the Awareness, Interest, Desire, Action (AIDA) model of how communication works. Highly
rational, high attention processing of information, brand recall and recognition and persuasion scores, not
forgetting the dreaded norms, are all trademark measures of the wrong sort of measurement for today's
marketers and their brands. Nowadays, measuring emotional response should be central, not peripheral, to
consumer communications research. It has been proved that the way people feel about an ad is a better
predictor of its potential effectiveness than rational, information-processing measures.

Let's not forget the typical multinational brand manager who measures their ad's effectiveness by putting a TV
animatic through some sort of quantitative pretest among 100 representative members of the UK population
(who may or may not have been the actual target audience). Back comes a key one number score generated by
a black box technique that determines whether or not the ad is going to get made. Once over that hurdle, and on
air, a benchmark data point is established, and any changes in brand and ad awareness, ad recall, brand image
and purchase decision are tracked, and if positive, linked to the timing of the campaign. I suspect that there are
still many brand managers operating under such a system and I suspect their days are numbered. It's time to
wake up to the New World of Performance Measurement.

EVALUATING THE OPTIONS

One key thing is to remember that, to know whether or not you have been successful is to know what you set
out to achieve in the first place, so start by revisiting your campaign objectives and see how they relate to the
overarching business objectives. They should be clear and well defined, giving a hint as to the quantitative
results and timescale they are intended to perform under.

The second thing is that it is not enough to know whether or not you succeeded in meeting your objectives; you
also need to understand how or why you did (or didn't). Only then does true learning start. So, you need to
identify your communications strategy, and figure out how that contributed to the outcome. If possible, you will
also want to isolate the various effects of your different channels and messages (rather than the fact you were
running a BOGOF at the same time).

The third thing to understand is the context in which your campaign was operating – because it is never in a
vacuum. Other things will have been happening, which you may or may not have anticipated, and these may
have also influenced the outcome.

You need to develop a conceptual model of how your communication is going to work on the consumer. Some
research companies build this into their research development model and are often surprised at how hard some
clients and agencies seem to find it to state how they expect their campaign to work against the target to deliver
the intended results. But all parties agree how helpful it is to focus their thinking and unite the teams involved
against a common goal.

It is known as the By, By, By technique (Figure 1) and it works by asking yourself: 'what are we going to
achieve?' and 'how are we going to achieve it?' at every stage. It was originated by Marilyn Baxter. Each layer,
from Business Objectives to Business Strategy, down to Advertising Objectives and Creative Strategy, answers
that 'how?' in the form of 'we'll do that by doing such and such' and 'we'll achieve that by doing something else'…
and so on.
Figure 1: The By, By, By Model – Police Recruitment

You can add as many layers as you like to this ladder, by setting out in more and more detail how your different
strategies are intended to work – sometimes right down to the executional detail of the creative or media
strategy. The example in Figure 1 comes from M&C Saatchi's Police Recruitment case study, winner of APG
Gold and IPA Silver awards.

Once the theoretical model of your strategy has been agreed, you can use it to identify the effects you expect to
see if it works. So, in this example, you would expect to achieve an increase in the number of policemen and
women hired into the force (note – not see more applications in your postbag). At level two, you would expect
the quality of applications to improve. At levels three and four, you would like to see the campaign generating
these responses from the audience, and so on.

From here, you can figure out exactly how you are going to gather this data. You can use it to develop the
methodology and questionnaire design for any consumer research. You would want to know the number of new
police recruits for starters, compared with previous years. Also, an analysis of the educational calibre of the
applicants, or possibly anecdotal feedback from the recruitment interviews, compared with previous years. Then
some qualitative research into the creative concepts to gauge responses from the audience. Followed by image
tracking on perceptions of how hard it is to be a member of the police force, and how much respect you accord
them (having established a benchmark first), analysed by those who had ad recall. The important thing is not to
overwhelm with too many measurements, figure out your KPIs, and include some business numbers as well as
behavioural and attitudinal measures. Keep an eye on the end game.

CONCLUSIONS
Communications ideas are powerful but expensive things. They can work in both the short-term and the long-
term, and it is the latter that tends to transform the brands and the businesses behind them.

These valuable successes do not happen overnight, however; they evolve from a virtuous circle of hypothesis,
testing, learning and feedback. Knowing how to predict and measure effectiveness will encourage clients to
invest for the long-term and value the contribution that communication makes.
It has never been easy to prove communication effectiveness, and it has just got a lot harder and more
complicated. On the plus side, it has become a lot more accurate. Reading around the subject, developing
hypotheses, starting early, being well prepared and investing appropriately, thinking both short and long-term,
are all 'best behaviours'.

FURTHER READING ON [Link]

Home Office/Police: How Thinking Negatively Ended the Negative Thinking, Richard Storey, IPA Silver
and Best Insight award, 2002

Using an Emotional Model to Measure Ad Effectiveness, Orlando Wood, Admap Jan 2010

On the road to a New Effectiveness Model: Measuring Emotional Responses to Television

How Effective is Creativity? Emotive Content in TV Advertising Does Not Increase Attention, R Heath,
A Nairn, P Bottomley, JAR Dec 2009

Payback Calculations – How to Make Sure You Get Your Sums Right, Les Binet, IPA

15 ways NOT to evaluate your communications, Les Binet, Admap, February 2006

OTHER READING
Marketing in The Era of Accountability, Les Binet and Peter Field, IPA/Warc 2007

How public service advertising works, Judie Lannon (ed), COI, IPA, Warc 2008

Best Practice Guide to Measuring Marketing Payback, [Link]

ABOUT THE AUTHOR

Merry Baskin founded planning consultancy Baskin Shark in 2000. A former chair of the Account Planning
Group and JWT planning director, she teaches planning craft skills all over the globe. She is co-editor of A
Master Class in Brand Planning: The Timeless Works of Stephen King.

merry@[Link]
© Copyright Ascential Events (Europe) Limited 2010
Ascential Events (Europe) Limited
Americas: 229 West 43rd Street, 7th Floor, New York, NY 10036, United States - Tel: +1 212 201 2800
APAC: 5 Shenton Way, UIC Building #10-01, Singapore 068808 - Tel: +65 3157 6200
EMEA: 33 Kingsway, London, WC2B 6UF, United Kingdom - Tel: +44 (0)20 7467 8100

[Link]

All rights reserved including database rights. This electronic file is for the personal use of authorised users based at the subscribing
company's office location. It may not be reproduced, posted on intranets, extranets or the internet, e-mailed, archived or shared electronically
either within the purchaser's organisation or externally without express written permission from Warc.