Chapter 9:
Concept Testing
Reference Book:
Product Design and Development
Chapter 9
Karl T. Ulrich, Steven D. Eppinger and Maria C. Yang
7th Edition
1
Concept Development Process
Mission Development
Statement Identify Establish Generate Select Test Set Plan Plan
Customer Target Product Product Product Final Downstream
Needs Specifications Concepts Concept(s) Concept(s) Specifications Development
Perform Economic Analysis
Benchmark Competitive Products
Build and Test Models and Prototypes
2
Concept Testing
• In this chapter, the focus is on testing done during the
concept development phase.
• In a concept test, the development team solicits a
response to a description of the product concept from
potential customers in the target market.
• This type of testing may be used to select which of two or
more concepts should be pursued, to gather information
from potential customers on how to improve a concept,
and to estimate the sales potential of the product.
• Concept development process shows concept testing
relative to other concept development activities.
3
Concept Testing
• Concept testing is closely related to concept selection in
that both activities aim to further narrow the set of
concepts under consideration.
• However, concept testing is distinct in that it is based on
data gathered directly from potential customers and
relies to a lesser degree on judgments made by the
development team.
• The reason that concept testing generally follows
concept selection is that a team cannot feasibly test
more than a few concepts directly with potential
customers.
4
Concept Testing
• A team may choose not to do any concept testing at all
if the time required to test the concept is large relative to
the product life cycles in the product category, or if the
cost of testing is large relative to the cost of actually
launching the product.
• For example, in the Internet software business, some
observers and practitioners argue that just launching a
product and iteratively refining it with subsequent
product generations is a better strategy than carefully
testing a concept before developing it fully.
5
Nature of concept testing
• Further narrow the set of concepts under
consideration,
– based data gathered from potential customers in
the target markets, rather than the judgments
made by the development team
• Specific Objectives
– Select one from multiple concepts,
– Gather information on how to improve a concept,
and
– Estimate the sales potential of the product
6
Input and output
• Input to the potential customer
– Prototype
• Output from the potential customer
– Likelihood for the potential customer to buy
the product
– Estimate of how many units of the product
the company is likely to sell
7
Steps of Concept Testing Process
1. Define the purpose of the concept testing
2. Choose a survey population and sample
size
3. Choose a survey format
4. Communicate the concept
5. Measure customer response
6. Interpret the results
7. Reflect on the results and the process
8
Step 1: Define the Purpose
• As a first step in concept testing, the team explicitly articulate
in writing the questions that the team wishes to answer with
the test.
• Concept testing is essentially an experimental activity, and as
with any experiment, knowing the purpose of the experiment
is essential to designing an effective experimental method.
• The primary questions addressed in concept testing are
typically:
✓ Which of the alternative concepts should be pursued?
✓ How can the concept be improved to better meet customer
needs?
✓ Approximately how much units are likely to be sold?
✓ Should the development be continued?
9
Step 2: Choose a Survey Population
• An assumption underlying the concept test is that the
population of potential customers surveyed reflects that of
the target market for the product.
• If the survey population is either more or less enthusiastic
about the product than will be the eventual target audience
for the product, then inferences based on the concept test
will be biased.
• As a result, the team should choose a survey population
that mirrors the target market in as many ways as possible.
• Often a product addresses multiple market segments. In
such cases, an accurate concept test requires that potential
customers from each target segment be surveyed.
10
Step 2: Choose a Survey Population
❑ Surveying every possible segment may be prohibitively
expensive in cost or time, and in such cases, the team may
choose to survey potential customers from only the largest
segment.
❑ However, when only one segment is sampled, inferences
about the response of the entire market are likely to be
biased.
• For a scooter, there were two primary consumer
segments: college students and urban commuters. The
team may decide to form a survey population from both
segments.
• The team may identify several smaller secondary
segments, including transportation for factory and airport
employees.
11
Sample Size
❑ The sample size of the survey should be large enough that
the team’s confidence in the results is high enough to guide
decision making.
❑ Sample sizes for concept testing are sometimes as small as
10 (qualitative assessment) or as large as 1,000
(quantitative assessment).
❑ Although there are no simple formulas for determining
sample size, some of the factors driving sample size are
mentioned in the next slide.
❑ Depending on the desired data to be collected from the
concept-testing process, the team may actually structure
multiple surveys with different objectives.
❑ Each of these surveys may involve a different sample
population and a different sample size. 12
Sample Size
13
Summary of Survey Population
1. Sample size varies from a few to thousands
2. Factors affecting the sample size
1. The stage of product development
2. Cost to conduct survey
3. Nature and intent of the survey
4. Budget (amount) of the development project
5. How possible to collect the intended information.
3. Possible to structure multiple surveys with
different objectives at different stages.
14
Step 3: Choosing a Survey Format
❑ The following formats are commonly used in concept
testing.
❖Face to face interaction
❖Telephone
❖Postal mail
❖Electronic mail
❖Internet (a test site on the internet)
15
Survey Formats
Face-to-face interaction:
➢ In this format, an interviewer interacts directly with the respondent.
➢ Face-to-face interactions can take the form of intercepts (i.e.,
stopping people at a mall, in a park, or on a city street), interviews
prearranged by telephone, interviews with potential customers at a
trade-show booth, or focus groups (i.e., prearranged group
discussions with 6–12 people).
Telephone:
➢ Telephone interviews may be prearranged and targeted at very
specific individuals (e.g., pediatric dentists) or may be “cold calls”
of consumers from a target population.
16
Survey Formats
Postal mail:
➢ In mail surveys, concept-testing materials are sent and
respondents are asked to return a completed form.
➢ Postal surveys are somewhat slower than other methods and
suffer from relatively poor response rates.
➢ Some kind of incentive—often cash or a gift—is sometimes offered
to increase response.
Electronic mail:
➢ Electronic mail surveys are very similar to postal mail surveys,
except that (as of this writing) respondents seem slightly more
likely to reply than via postal mail.
➢ Electronic mail surveys be used only when respondents are likely
to perceive a benefit to their participation, or when the team has
already established some kind of positive relationship with the
target population.
17
Survey Formats
Internet:
➢ Using the Internet, a team may create a virtual concept-
testing site in which survey participants can observe
concepts and provide responses.
➢ An electronic mail message is usually used to recruit
respondents to visit the test site.
18
Pros and Cons of Survey Formats
❑ Each of these formats presents risks of sample bias.
❑ For example, the use of electronic formats may bias the
sample toward those who are technologically
sophisticated.
❑ For some products, this sophistication is part of the
profile of the target market (e.g., the target market for
Internet software products is likely to be comfortable
with electronic survey formats).
❑ Conversely, an Internet survey might be a particularly
bad format for testing a concept targeted at people
without personal computers.
19
Suggestions about Survey Formats
➢ Exploratory testing, typical in the early phases of concept
development, benefits from open-ended interactive formats.
➢ It is recommended that the team use face-to-face formats when
presenting multiple concept alternatives or when soliciting ideas for
improving a concept.
➢ In these settings, the product developers themselves benefit from
performing the interviews because they can directly observe
reactions to the product in rich detail.
➢ As the purpose of the concept test becomes more focused, more
structured formats such as mail and telephone become more
appropriate.
➢ If the questions are very focused, the team can hire a market
research firm to implement the concept test.
20
Step 4: Communicate the Concept
❑ The choice of survey format is closely linked to the way in which the
concept will be communicated.
❑ Concepts can be communicated in any of the following ways, listed in
order of increasing richness of the description.
➢ Verbal description
➢ Sketch
➢ Photos and renderings
➢ Storyboard (a series of images shown a temporal sequence of
actions involving the products)
➢ Video (allowing more dynamic than the story board)
➢ Simulation
➢ Interactive multimedia (video and simulation)
➢ Physical appearance model (looks-like)
➢ Working prototypes (works-like)
21
Means of Communication
Verbal description:
✓ A verbal description is generally a short paragraph or a collection of
bullet points summarizing the product concept.
✓ This description may be read by the respondent or may be read aloud
by the person administering the survey.
Sketch:
✓ Sketches are usually line drawings showing the product in perspective,
perhaps with annotations of key features.
Photos and renderings:
✓ Photographs can be used to communicate the concept when
appearance models exist for the product concept.
✓ Renderings are nearly photo-realistic illustrations of the concept.
✓ Renderings can be created with pens and markers or using computer-
aided design tools.
22
Sketch and Photos
Sketch Photos and
Renderings
23
Means of Communication
Storyboard:
✓ A storyboard is a series of images that communicates a temporal
sequence of actions involving the product.
✓ For example, one of the potential benefits of the scooter is that it can be
easily stored and transported.
Video:
✓ Video images allow even more dynamism than the storyboard.
✓ With video, the form of the product itself can be clearly communicated,
as can the way in which the product is used. The scooter team used a
video in its purchase-intent survey.
Simulation:
✓ Simulation is generally implemented as software that mimics the
function or interactive features of the product.
✓ Simulation would probably not be the ideal way to communicate the key
features of a scooter, but in some other cases simulation can be
effective.
24
Means of Communication
Interactive multimedia:
✓ Interactive multimedia combines the visual richness of video with the
interactivity of simulation.
✓ Using multimedia, you can display video and still images of the product.
✓ The respondent can view verbal and graphical information and can listen
to audio information.
✓ Interaction allows the respondent to choose from among several
sources of available information on the product, and in some cases to
experience the controls and displays of a simulated product.
Physical appearance models:
✓ Physical appearance models, also known as “looks-like” models, vividly
display the form and appearance of a product.
✓ They are often made of wood or polymer foams and are painted to look
like real products.
✓ In some cases, limited functionality is included in the model.
25
Means of Communication
Working prototypes:
✓ When available, working prototypes, or works-like models, can be
useful in concept testing.
✓ However, the use of working prototypes is also risky.
✓ The primary risk is that the respondents will equate the prototype
with the finished product.
✓ In some cases, prototypes perform better than the ultimate
product (e.g., because the prototype uses better, more expensive
components such as motors or batteries).
✓ In most cases, the prototype performs worse than the ultimate
product and is almost always less visually attractive than the
ultimate product.
26
Survey Format and Means of Communication
❑ The choice of survey format is tightly linked to the
means of communicating the product concept.
❑ For example, the team obviously cannot demonstrate
the scooter with a working model using a telephone
survey.
❑ Exhibit 9-9 identifies which means of communicating
concepts are appropriate for each survey format
27
Exhibit 9-9
Appropriateness of different survey formats for different ways of
communicating the product concept.
28
Issues in Communicating the Concept
✓ When communicating the product concept, the team must decide how
aggressively to promote the product and its benefits.
✓ The scooter could be described as an “electric-powered personal
mobility device” or as an “exciting new electric scooter that provides
freedom from gridlock.”
✓ The description of the concept should closely mirror the information
that the user is likely to consider when making a purchase decision.
✓ There is a debate about whether the purchase price of the product
should be included as part of the concept description.
✓ Price is a very powerful lever on customer response, and, therefore,
pricing information can dramatically influence the results of a concept
test.
✓ It is recommended that price be omitted from the concept description
unless the price of the product is expected to be unusually high or
low.
29
Issues in Communicating the Concept
✓ Instead of showing a single concept, the team may choose to ask a
respondent to select from several alternatives.
✓ This approach is attractive when the team is trying to decide
among several concepts under consideration.
✓ A variant on this approach is to present the concept for the new
product along with descriptions and pictures of the most
successful existing products.
✓ This approach has the advantage of allowing respondents to
directly assess attributes of the product concept in comparison to
those of competitors.
✓ Assuming the products would be equally distributed and
promoted, this approach also allows the team to estimate potential
market share.
✓ Using a forced-choice survey technique is likely to be most
effective in cases for which there is a narrowly defined product
category with relatively few existing products.
30
Step 5: Measure Customer Response
✓ Most concept test surveys first communicate the product concept
and then measure customer response.
✓ When a concept test is performed early in the concept
development phase, customer response is usually measured by
asking the respondent to choose from two or more alternative
concepts.
✓ Additional questions focus on why respondents react the way they
do and on how the product concepts could be improved.
✓ Concept tests also generally attempt to measure purchase intent.
✓ The most commonly used purchase-intent scale has five response
categories:
➢Definitely would buy ➢Probably would buy
➢Might or might not buy ➢Probably would not buy
➢Definitely would not buy
31
32
Step 6: Interpret the Results
✓ If the team is simply interested in comparing two or more
concepts, interpretation of the results is straightforward.
✓ If one concept dominates the others and the team is confident
that the respondents understood the key differences among
the concepts, then the team can simply choose the preferred
concept.
✓ If the results are not conclusive, the team may decide to
choose a concept based on cost or other considerations or
may decide to offer multiple versions of the product.
✓ Note that care must be applied in making this judgment for
cases in which manufacturing costs are dramatically different
among the concepts under comparison and in which no price
information is communicated to the respondents.
✓ In such cases, respondents may be biased to select the most
costly alternative.
33
Estimating Demand for Durables
• The team is also interested in estimating the demand for
a product in the period following launch, usually one
year.
• Now, a model for estimating the sales potential of
durables will be discussed.
• By durables products that last several years are meant,
and for which there is, therefore, a negligible repeat-
purchase rate.
• These products are in contrast to consumer packaged
goods, like razor blades, toothpaste, or frozen food, for
which forecasting models must consider rates of trial
and subsequent repeat purchase.
34
Estimating Demand for Durables
Q=NxAxP
Where, P = Cdefinitely x Fdefinitely + Cprobably x Fprobably
• Q = the quantity of the expected sales
• N = the number of potential customers expected to purchase
• A = the fraction of these potential customers aware of the product when the product is
available
• P = the probability that the product is purchased if the customer is aware of it and it is
available.
• Fdefinitely = the fraction of survey respondents indicating that they would definitely purchase
• Fprobably = the fraction of survey respondents indicating that they would probably purchase
• Cdefinitely = calibration constant that those in Fdefinitely will actually buy (0.1 to 0.5)
• Cprobably = calibration constant that those in Fprobably will actually buy (0-0.25)
Generally the value of Cdefinitely is considered 0.4
and the value of Cprobably is considered 0.2
35
EXAMPLE
36
Estimating Demand for Durables
➢ The results of forecasts based on concept testing should be
interpreted with caution.
➢ Some firms, mostly after repeated experience with similar
products, have achieved impressive levels of accuracy in
their forecasts.
➢ While forecasts do tend to be correlated with actual sales,
most individual forecasts exhibit substantial errors.
➢ Some of the factors that can cause actual purchase patterns
to differ from the purchase intentions expressed in surveys
include:
✓ Importance of word-of-mouth
✓ Fidelity of the concept description
✓ Pricing
✓ Level of promotion
37
Estimating Demand for Durables
❑ Importance of word-of-mouth:
✓ When the benefits of a product are not immediately
obvious, the enthusiasm of existing users may be an
important factor in generating demand.
✓ This factor is not generally captured in concept testing.
❑ Fidelity of the concept description:
✓ If the actual product differs substantially from the
description of the product in the concept test, then actual
sales are likely to differ from the forecast.
38
Estimating Demand for Durables
❑ Pricing:
✓ If the price of the product deviates substantially from the
price indicated in the survey, or from the expectations of
survey respondents, then forecasts are likely to be
inaccurate.
❑ Level of promotion:
✓ Spending on advertising and other forms of promotion can
increase demand for most products.
✓ The influence of promotion is accounted for only weakly in
the forecasting model via the awareness/availability term
and via the materials used to present the concept(s).
39
Step 7: Reflect on the Result and the Process
✓ The primary benefit of the concept test is in getting
feedback from real potential customers.
✓ The qualitative insights gathered through open-ended
discussions with respondents about the proposed
concepts may be the most important result of concept
testing, especially early in the development process.
✓ The team should reflect on this evidence as well as on
the numerical outcome of its forecast.
40
Step 7: Reflect on the Result and the Process….
➢ The team benefits from thinking about the impact of the three
key variables in the forecasting model: (1) the overall size of
the market, (2) the availability and awareness of the product,
and (3) the fraction of customers who are likely to purchase.
➢ Considering alternative markets for the product can
sometimes increase the first factor.
➢ The second factor can be increased through distribution
arrangements and promotion plans.
➢ The third factor can be increased through changes to the
product design (and possibly advertising) that improve the
attractiveness of the product.
➢ In considering these factors, a sensitivity analysis can yield
useful insights and aids in decision making.
41
Step 7: Reflect on the Result and the Process….
❑ For example, what would be the impact on sales if the team
were able to secure a partnership with a retailer and therefore
increase A by 20%?
❑ In reflecting on the results of the concept test, the team
should ask two key diagnostic questions.
✓ First, was the concept communicated in a way that is likely
to elicit customer response that reflects true intent?
✓ Second, is the resulting forecast consistent with observed
sales rates of similar products?
❑ The team can benefit from its experience by documenting the
results of its concept testing and by attempting to reconcile
these results with subsequent observations of product
success.
42
THANK YOU
43