The Delphi Method

Techniques and Applications

Edited by

Harold A. Linstone
Portland State University

Murray Turoff
New Jersey Institute of Technology

With a Foreword by
Olaf Helmer
University of Southern California

©2002 Murray Turoff and Harold A. Linstone

UA-740
I. Introduction

UA-740
I. Introduction
HAROLD A. LINSTONE and MURRAY TUROFF

General Remarks

It is common, in a book of this kind, to begin with a detailed and explicit

definition of the subject- the Delphi technique. However, if we were to attempt this,
the reader would no doubt encounter at least one contribution to this collection which
would violate our definition. There is in addition a philosophical perspective that
when something has attained a point at which it is explicitly definable, then progress
has stopped; such is the view we hold with respect to Delphi.
In 1969 the number of Delphi studies that had been done could be counted in
three digits; today, in 1974, the figure may have already reached four digits. The
technique and its application are in a period of evolution, both with respect to how it
is applied and to what it is applied. It is the objective of this book to expose the
richness of what may be viewed as an evolving field of human endeavor. The reader
will encounter in these pages many different perspectives on the Delphi method and
an exceedingly diverse range of applications.
For a technique that can be considered to be in its infancy, it would be
presumptuous of us to present Delphi in the cloak of a neatly wrapped package,
sitting on the shelf and ready to use, Rather, we have adopted the approach, through
our selection of contributions, of exhibiting a number of different objects having the
Delphi label and inviting you to sculpt from these examples your own view and
assessment of the technique. For, if anything is "true" about Delphi today, it is that in
its design and use Delphi is more of an art than a science.
However, as editors, we would be remiss if there were not some common thread
underlying the articles brought together in this volume. As long as we restrict
ourselves to a very general view, it is not difficult to present an acceptable definition
of the Delphi technique which can be taken as underlying the contributions to this
book:
Delphi may be characterized as a method for structuring a group communication
process so that the process is effective in allowing a group of individuals, as a whole, to
deal with a complex problem.

To accomplish this "structured communication" there is provided: some

feedback of individual contributions of information and knowledge; some assessment
of the group judgment or' view; some opportunity for individuals to revise views; and
some degree of anonymity for the individual responses, As the reader will discover,
there are many different views on what are the "proper," "appropriate," "best," and/or
"useful" procedures for accomplishing the various specific aspects of Delphi. We
hope that the reader will find this book a rich menu of procedures from which he may
select his own repast if he should seek to employ the Delphi technique.
When viewed as communication processes, there are few areas of human
endeavor which are not candidates for application of Delphi. While many people label
3

UA-740
4 Harold A. Linstone and Murray Turoff
Delphi a forecasting procedure because of its significant use in that area, there is a
surprising variety of other application areas. Among those already developed we find:

• Gathering current and historical data not accurately known or available

• Examining the significance of historical events
• Evaluating possible budget allocations
• Exploring urban and regional planning options
• Planning university campus and curriculum development
• Putting together the structure of a model
• Delineating the pros and cons associated with potential policy options
• Developing causal relat ionships in complex economic or social
phenomena
• Distinguishing and clarifying real and perceived human motivations
• Exposing priorities of personal values, social goals

It is not, however, the explicit nature of the application which determines

the appro priateness of utilizing Delphi; rather, it is the particular circumstances
surrounding the necessarily associated group communication process: "Who is
it that should communicate about the problem, what alternative mechanisms are
available for that communication, and what can we expect to obtain with these
alternatives?" When these questions are addressed, one can then decide if
Delphi is the desirable choice. Usually, one or more of the following properties
of the application leads to the need for employing Delphi:

• The problem does not lend itself to precise analytical techniques but cart
benefit from subjective judgments on a collective basis
• The individuals needed to contribute to the examination of a broad or
complex problem have no history of adequate communication and may
represent diverse backgrounds with respect to experience or expertise
• More individuals are needed than can effectively interact in a face-to-face
exchange
• Time and cost make frequent group meetings infeasible
• The efficiency of face-to-face meetings can be increased by a supplemental
group communication process
• Disagreements among individuals are so severe or politically unpalatable
that the communication process must be refereed and/or anonymity assured
• The heterogeneity of the participants must be preserved to assure validity of
the results, i.e., avoidance of domination by quantity or by strength of
personality ("bandwagon effect")

Hence, for the application papers in this book tire emphasis is not on the results
of a particular application but, rather, on discussion of why Delphi was used and how
it was implemented. From this the reader may be able to transpose the considerations

UA-740
Introduction 5
to his own area of endeavor and to evaluate the applicability of Delphi to his own
problems.
Those who seek to utilize Delphi usually recognize a need to structure a group
communication process in order to obtain a useful result for their objective.
Underlying this is a deeper question: "Is it possible, via structured communications, to
create any sort of collective human intelligence1 capability?" This is an issue
associated with the utility of Delphi that. has not as vet received the attention it
deserves and the reader will only find it addressed here indirectly. It will, therefore,
be a subjective evaluation on his part to determine if the material in this book
represents a small, but initial, first step in the long-term development of collective
human intelligence processes.

Characteristics of the Delphi

The Delphi process today exists in two distinct forms. The most common is the
paper-and-pencil version which is commonly referred to as a "Delphi Exercise." In
this situation a small monitor team designs a questionnaire which is sent to a larger
respondent group, After the questionnaire is returned the monitor team summarizes
the results and, based upon the results, develops a new questionnaire for the
respondent group. The respondent group is usually given at least one opportunity to
reevaluate its original answers based upon examination of the group response. To a
degree, this form of Delphi is a combination of a polling procedure and a conference
procedure which attempts to shift a significant portion of the effort needed for
individuals to communicate from the larger respondent group to the smaller mo nitor
teat). We shall denote this form conventional Delphi.
A newer form, sometimes called a "Delphi Conference," replaces the monitor
teat) to a large degree by a computer which has been programmed to carry out the
compilation of the group results. This latter approach has the advantage of eliminating
the delay caused in summarizing each round of Delphi, thereby turning the process
into a real-time communications system. However, it does require that the
characteristics of the communication be well defined before Delphi is undertaken,
whereas in a paper-and-pencil Delphi exercise the monitor team can adjust these
characteristics as a function of the group responses. This latter form shall be labeled
real-lucre Delphi.
Usually Delphi, whether it be conventional or real-tune, undergoes four distinct
phases. The first phase is characterized by exploration of the subject under discussion,
wherein each individual contributes additional information he feels is pertinent to the
issue. The second phase involves the process of reaching an understanding of how the
group views the issue (i.e., where the members agree or disagree and what they mean
by relative terms such as importance, desirability, or feasibility). if there is significant
disagreement, then that disagreement is explored in the third phase to bring out the

1
We refer to "intelligence" in this context as including attitudes and feelings which are part of
the process of human motivation and action.

UA-740
6 Harold A. Linstone and Murray Turoff
underlying reasons for the differences and possibly to evaluate them. The last phase, a
final evaluation, occurs when all previously gathered information has been initially
analyzed and the evaluations have been fed back for consideration.
On the surface, Delphi seems like a very simple concept that can easily be
employed. Because of this, many individuals have jumped at trying the procedure
without carefully considering the problems involved in carrying out such an exercise.
There are perhaps as many individuals who have had disappointing experiences with
a Delphi as there are users who have had successes. Some of the common reasons for
the failure of a Delphi are:

• Imposing monitor view's and preconceptions of a problem upon the

respon dent group by overspecifying the structure of the Delphi and not
allowing for the contribution of other perspectives related to the
problem
• Assuming that Delphi can be a surrogate for all other human
communica tions in a given situation
• Poor techniques of summarizing and presenting the group response and
ensuring common interpretations of the evaluation scales utilized in the
exercise
• Ignoring and not exploring disagreements, so that discouraged
dissenters drop out and an artificial consensus is generated
• Underestimating the demanding nature of a Delphi and the fact that tire
respondents should he recognized as consultants and properly
compensated for their time if the Delphi is not an integral part of their
job function

In addition to the latter problems associated with the Delphi technique

another class of criticisms directed at Delphi is often raised in the literature.
These are the "virtual" problems that do not in themselves affect the utility of
the technique. 2 Typic al of these is the question of how to choose a "good"
respondent group. This is, in fact, a problem for the formation of any group
activity -committees, panels, study groups, etc. One has this problem no matter
what communication mode is used; therefore, wh ile it is a real arid significant
problem, it is not a problem unique to Delphi. However, the nature of certain
applications does, in fact, dictate special consideration of this problem and it is
discussed in a number of articles. Another virtual problem frequently - arises
when a particular Delphi design for a particular application is taken as
representative of all Delphis, whereupon it is then observed that this design
does not work for some other application. The problem here is that of making
too explic it and restrictive a definition for Delphi. A third virtual problem is the
honesty of the monitor team, and it is of the same concern as the honesty of any study
or analysis group. In fact, there is probably more likelihood in most instances of

2
See, for example, Cordon Welty, "A Critique of the Delphi Technique," Proceedings of the American
Statistical Association, 1971. Social Statistics Section.

UA-740
Introduction 7
exposure of misrepresentation in a Delphi summary than in a typical ,group study
report. Finally, misunderstandings may arise from differences in language and logic if
participants come from diverse cultural backgrounds. Since we consider these virtual
issues to be s omewhat irrelevant to Delphi per sc, we have made no attempt to give
them special attention within this book. Other problems will be discussed in Chapter
VIII.
It is quite clear that in any one application it is impossible to eliminate all
problems associated with Delphi. There is, for example. a natural conflict in (he goal
of allowing a wide latitude in the contribution of information and the goal of keeping
the communication process efficient. It. is the task of the Delphi designer to minimize
(here problems as much as possible and to balance the various communication "goals"
within the context of the objective of the particular Delphi and the nature of the
participants. Arriving at a balanced design for the communication structure is still
very Much an art, even though there is considerable experience on how to ask and
summarize various types of questions.
It can be expected that the use of Delphi will continue to grow. AS a result of
this, one can observe that a body of knowledge is developing on how to structure the
human communication process for particular classes of problems. The abuse, as well
as the use, of the technique is contributing to the development of this design
methodology.
Table 1 compares the properties of normal group communication modes and the
Delphi conventional and real-time modes. The major differences lie in such areas as
the ability of participants to interact with the group at their own convenience (i.e.,
random as opposed to coincident), the capacity to handle large groups, and the
capability to structure the communication. With respect to time consideration there is
a certain degree of similarity between a committee and a conventional Delphi process,
since delays between committee meetings and Delphi rounds are unavoidable. Also,
the real-time Delphi is conceptually somewhat analogous to a randomly occurring
conference call with a written record automatically produced. It is interesting to
observe that within the context of the normal operation of these communication
modes in the typical organization--government, academic, or industrial the Delphi
process appears to provide the individual with the greatest degree of individuality or
freedom from restrictions on his expressions. The items highlighted in tire table will
be discussed in more detail in many of the articles in this book,
While the written word allows fur- emotional content, the Delphi process does
tend to minimize the feelings Mid information normally communicated in such
manners as the tone of a voice, the gesture of a hand, or the look of an eve. In many
instances these are a vital and highly informative part of a communication process.
Our categorization of group communication processes is not meant to imply that the
choice for a particular objective is limited, necessarily, to one communication mode.
As the readers will see from some of the contributions to this book, there are instances
where it is desirable to use a mix of these approaches.

UA-740
 8

TABLE 1
Group Communication Techniques
Conference Committee Formal Conference Conventional Real-Time
Telephone Call Meeting or Seminar Delphi Delphi
Effective Group Small Small to Medium Small to Large Small to Large Small to Large
Size

Occurrence of Coincident with Coincident with Coincident with Random Random

Interaction by group group group
Individual

Length of Short Medium to Long Long Short to Medium Short

Interaction

Number of Multiple, as Multiple, necessary Single Multiple, necessary Multiple, as

Interactions required by time delays time delays required by
group between between individual

Normal Mode Equality to Equality to Presentation Equality to monitor Equality to monitor

Range chairman control chairman control (directed) control control or group
(flexible) (flexible) (structured) control and no
monitor
(structured)
Harold A. Linstone and Murray Turoff

UA-740
 TABLE 1 (continued)
Group Communication Techniques
Conference Committee Formal Conference Conventional Real-Time
Telephone Call Meeting or Seminar Delphi Delphi
Introduction

Principal Costs Communications • Travel • Travel • Monitor Time • Communications

• Individual’s Time • Individual’s Time • Clerical • Computer Usage
• Fees • Secretarial

Time -urgent Forced delays Forced delays Time -urgent

considerations considerations

Other
Characteristics • Equal flow of information to and from • Efficient flow of • Equal flow of information to and from
all information from all
• Can maximize psychological effects few to many • Can minimize psychological effects
• Can minimize time demanded of
respondents or conferees
9

UA-740
10 Harold A. Linstone and Murray Turoff
The Evolution of Delphi

The Delphi concept may be viewed as one of the spinoffs of defense research.
"Project Delphi" was the name given to an Air Force-sponsored Rand Corporation
study, starting in the early 1950's, concerning the use of expert opinion.3 The
objective of the original study was to "obtain the most reliable consensus of opinion
of a group of experts ... by a series of intensive questionnaires interspersed with
controlled opinion feedback."
It may be a surprise to some that the subject of this first study was the
application of "expert opinion to the selection, from the point of view of a Soviet
strategic planner, of an optimal U. S. industrial target system and to the estimation of
the number of A-bombs required to reduce the munitions output by a prescribed
amount." It is interesting to note that the alternative method of handling this problem
at that time would have involved a very extensive and costly data-collection process
and the programming and execution of computer models of a size almost prohibitive
on the computers available in the early fifties. Even if this alternative approach had
been taken, a great many subjective estimates on Soviet intelligence and policies
would still have dominated the results of the model. Therefore, the original
justifications for this first Delphi study are still valid for many Delphi applications
today, when accurate information is unavailable or expensive to obtain, or evaluation
models require subjective inputs to the point where they become the dominating
parameters. A good example of this is in the "health care" evaluation area, which
currently has a number of Delphi practitioners.
However, because of the topic of this first notable Delphi study, it took a later
effort to bring Delphi to the attention of individuals outside the defense community.
This was the "Report on a Long-Range Forecasting Study," by T. J. Gordon and Olaf
Helmer, published as a Rand paper in 1964.4 Its aim was to assess "the direction of
long-range trends, with special emphasis on science and technology, and their
probable effects on our society and our world." "Long-range" was defined as the span
of ten to fifty years. The study was done to explore both the methodological aspects
of the technique and to obtain substantive results. The authors found themselves in "a
near-vacuum as far as tested techniques of long-range forecasting are concerned." The
study covered six topics: scientific breakthroughs; population control; automation;
space progress; war prevention; weapon systems. Individual respondents were asked
to suggest future possible developments, and then the group was to estimate the year
by which there would be a 50 percent chance of the development occurring. Many of
the techniques utilized in that Delphi are still common to the pure forecasting Delphis
being done today. That study, together with an excellent related philosophical paper
providing a Lockean-type justification for the Delphi technique,5 formed the

3
N. Dalkey and O. Helmer, "An Experimental Application of the Delphi Method to the Use of Experts,"
Management Saence 9, No. 3 (April 1963), p, 458.
4
Rand Paper P -2982. Most of the study was later incorporated into Helmer's Social Technology, Basic
Books, New York, 1966.
5
O. Helmer and N. Rescher, "On the Epistemology of the Inexact Sciences," Project Rand Report R-353,
February 1960.

UA-740
Introduction 11
foundation in the early and mid-sixties for a number of individuals to begin
experimentation with Delphi in non-defense areas.
At the same time that Delphi was beginning to appear in the open literature,
further interest was generated in the defense area: aerospace corporations and the
armed services. The rapid pace of aerospace and electronics technologies and the
large expenditures devoted to research and development leading to new systems in
these areas placed a great burden on industry and defense planners. Forecasts were
vital to the preparation of plans 'as well as the allocation of R&D (research and
development) resources, and trend extrapola tions were clearly inadequate. As a result,
the Delphi technique has become a fundamental tool for those in the area of
technological forecasting and is used today in many technologically oriented
corporations. Even in the area of "classical" management science and operations
research there is a growing recognition of the need to incorporate subjective
information (e.g., risk analysis) directly into evaluation models dealing with the more
complex problems facing society: environment, health, transportation, etc. Because of
this, Delphi is now finding application in these fields as well.
From America, Delphi has spread in the past nine years to Western Europe,
Eastern Europe, and the Far East. With characteristic vigor the largest Delphi
undertaken to date is a Japanese study. Starting in a nonprofit organization, Delphi
has found its way into government, industry, and finally academe. This explosive rate
of growth in utilization in recent years seems, on the surface, incompatible with the
limited amount of controlled experimentation or academic investigation that has taken
place. It is, however, responding to a demand for improved communications among
larger and/or geographically dispersed groups which cannot be satisfied by other
available techniques. As illustrated by the articles in this book, aside from some of the
Rand studies by Dalkey, most "evaluations" of the technique have been secondary
efforts associated with some application which was the primary interest. It is hoped
that in coming years experimental psychologists and others in related academic areas
will take a more active interest in exploring the Delphi technique.
While many of the early articles on Delphi are quite significant and liberally
mentioned in references throughout this book, we have chosen to concentrate on work
that has taken place in the past five years and which represents a cross section of
diverse applications.
Although the majority of the Delphi efforts are still in the pure forecasting area,
that application provides only a small part of the contents of this volume. Chapters II
and III of this book consist of articles which provide an overview of the Delphi
technique, its utility, the underlying philosophy, and broad classes of application.
Chapter IV takes up recent studies in the evaluation of the technique. Precision
and accuracy are considered in this context. Between the reviews, articles, and
associated references, the reader should obtain a good perspective on the state of the
art with respect to experimentation.
Chapters V and VI describe some of the specialized techniques that have
evolved for asking questions and evaluating responses. Foremost among them is
cross-impact analysis (Chapter V). This concept reflects recognition of the
complexity of the systems dealt with in most Delphi activities, systems where

UA-740
12 Harold A. Linstone and Murray Turoff
"everything interacts with everything." In essence, these sections explore the
quantitative techniques available for deeper analysis of the subjective judgments
gathered through the Delphi mechanism.
The effect computers can have on Delphi and speculations on the future of the
technique itself are discussed in Chapter VII. The book concludes with a summary of
pitfalls which can serve the practitioner as a continuing checklist (Chapter VIII).
We have striven to avoid making this volume a palimpsest of previously
published papers: all but four of the articles have been especially prepared for this
work, The four reprinted articles were selected from the journal Technological
Forecasting and Social Change, a rich lode of material on Delphi. The extensive
bibliography in the Appendix provides a guide to those who wish to probe the subject
further. It is thus our hope that this volume will serve the reader as a useful reference
work on Delphi for a number of years.

UA-740