The Delphi method, impeachment and terrorism: Accuracies of short-range forecasts for volatile world events

Two experiments examined the effects of iterated polling and feedback on prediction accuracy for volatile world events using the Delphi paradigm. In both experiments, participants predicted rapidly changing geopolitical events. The first experiment occurred during the litigation surrounding the Clinton/Lewinsky affair in 1998. Results indicated that feedback improved individual and group accuracy for predictions of whether an event would occur. These types of predictions derived from group consensus were approximately 90% accurate, which was significantly higher than the average individual participant's accuracy. Neither polling nor feedback had much effect on time course predictions. The second experiment occurred during the American military action against the Taliban regime in Afghanistan in 2001. In Experiment 2, participants were polled three times to determine if increasing the number of iterated polls would improve Delphi accuracy. The results showed that accuracy improved from the first to the second poll but not from the second to the third. The groups predicted whether these scenarios would occur with approximately 75% accuracy, which was significantly higher than the accuracy of any individual participant. Once again, the Delphi procedure did not enhance time course predictions in Experiment 2.     

ExpertLens: A system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge

The complexity of policy decision-making raises the need to elicit opinions from large and heterogeneous groups of stakeholders with broad and diverse sets of expertise. Existing options for elicitation include small face-to-face panels of experts by using the Nominal Group Technique (NGT), large Delphi panels whose members do not interact with each other face-to-face, and crowdsourcing, which involves an open call for input issued to a large community of people. In an attempt to close the gap between the practical needs of policy makers and the methodological challenges associated with eliciting opinions of large, diverse, and distributed groups, we have developed a new online elicitation system and methodology called ExpertLens. By optimizing the direct interactions of NGT with the larger number of Delphi participants and the wisdom of “selected crowds,” our approach is designed to save on the costs associated with traditional expert panels, while increasing accuracy in elicitation by reducing the potential for group process losses that can occur in large, diverse, and non-collocated panels whose members interact via asynchronous online discussion boards. The ExpertLens approach is iterative, does not require participants to develop consensus, and determines what the group “thinks” by statistically analyzing data collected in all rounds of the elicitation. This paper describes the ExpertLens system and methodology, briefly discusses recent ExpertLens trials, provides conceptual arguments for why it is an appropriate model for eliciting expert opinions, illustrates its main components and analytics by using an infrastructure investment example, and discusses a research agenda for testing the underlying tenets of the ExpertLens approach.     

Design and evaluation of ranking-type Delphi studies using best-worst-scaling

The ranking-type Delphi method has received widespread use in a large number of research areas. In order to rank a set of items, ranking-type Delphi studies use different ranking approaches which may be biased. This is intensified by the fact that the number of issues a participant can reasonably rank is limited. This paper describes an approach for the design and evaluation of ranking-type Delphi studies using best-worst-scaling (BWS). The statistical BWS design is based on a balanced incomplete block design to construct the comparison sets. The statistical evaluation encompasses the measurement of stability and consensus with the use of the coefficient of variation. Overall, our approach (1) enables to design a ranking with up to 22 items in a manner that participants can reasonably rank, (2) avoids and overcomes shortcomings of common ranking approaches, and (3) offers a statistical procedure for an unambiguous evaluation.     

Improving the Delphi process: Lessons from social psychological research

The Delphi technique was largely developed to avoid the problems of freely interacting groups such as dominant individuals and pressure to conform to the majority view. Our review of the social psychological literature reveals some obstacles to Delphi achieving its full potential relative to other cheaper and easier methods of aggregating judgment. We identify residual normative and informational pressures towards consensus that potentially reduce process gain that might otherwise be achieved. For instance, panelist confidence may act as a signal of status rather than be a valid cue to expertise, whereas consensus appears to have a strong influence on the final outcome that can reduce its accuracy when there are valid minority opinions. We argue that process gain in Delphi must occur through those further from the "truth" changing their opinion more than those closer to the truth, with the general direction of opinion change being towards the truth. For such virtuous opinion change to occur we suggest the need to both facilitate opinion change and provide good cues as to where the truth lies. Research on Judge Advisor Systems shows that people usually do not change their opinion as much as they should, giving too much weight to their own opinion and too little to the views of others—this bias can be reduced by increasing involvement and motivation. In addition, we propose that the best way to provide good cues as to the direction of the truth is to elicit rich reasoning from panelists about the judgment or choice in question, then use this as feedback. We suggest practical ways of focusing and deepening panelists’ consideration and evaluation of such reasoning—such that all proffered opinions are well-evaluated. Additionally, we propose a model of opinion change in Delphi for use as a paradigm for future process-orientated research.     

The Delphi method as early warning: Linking global societal trends to future radicalization and terrorism in The Netherlands

A Delphi study was performed to answer the question: Which global societal trends relate to future radicalization and subsequent terrorism in the Netherlands? An inventory of two hundred global societal trends and a literature study of radicalization together served as briefing of the participants in the Delphi study. The first two rounds were conducted in anonymous writing. The third round was a face to face focus group meeting. In the first and second round participants scored the trends, adjusted their scores relative to others and provided arguments, and in the focus group meeting, themes were discussed in which the participants remained to have strong opposite views. This approach emphasizes divergence in opinion (‘dissensus’), as opposed to convergence (‘consensus’). Consensus seeking to us seems to be the dominant application of Delphi, but we emphasize the value of seeking dissensus. We hypothesize that in those cases where the opposite views that were discussed kept their ground, this Delphi study may have found early warnings of future radicalization.The essence of this paper is that it is possible, with limited effort, to get a handle on the complex and poorly defined subject of global societal trends influencing future radicalization. Application of the Dissensus Delphi method provided a selection of early warnings that may be looked into with future research.     

Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.     

The use of Delphi methodology in agrifood policy development: Some lessons learned

Developing policy in the agrifood area is an inexact process, usually relying upon effective integration of opinions from multiple experts from different disciplines, organisational types, and regions/countries. Delphi would appear to have the potential to overcome some of the typical limitations related to soliciting expert opinion and identifying consensus on future activities or options, particularly where relevant experts are dispersed geographically, and international consensus is required, as is the case in this domain. Three case studies, focused on the application of Delphi to emerging policy needs in international or European agrifood policy, are presented here to exemplify the utility of the technique. A number of practical recommendations are drawn from these case studies that may be applicable to other major policy making arenas. Among these recommendations are; that an exploratory workshop to refine round one Delphi questions is essential; that the implementation of “cascade” methodology (utilizing the personal contacts of researchers or members of existing policy networks) appears to increase response rates in subsequent Delphi rounds; and that the policy issue under discussion should be particularly relevant to stakeholders in order to increase participation rates. Further research might usefully focus on developing ways to incorporate measurements of uncertainty associated with stakeholder judgement into quantitative responses, and on establishing how best to utilise such information in feedback in subsequent Delphi rounds. Ensuring how best to inform policy uptake of the outputs of Delphi merits further research in particular.     

The decision Delphi

If Delphi is used as a prognostic tool in the social sciences, it runs the danger of producing a self-fulfilling prophecy. In a Delphi application conducted at the Austrian Academy of Sciences about the present situation and future developments of the scientific-technical information and documentation system in Austria, an attempt was made to apply this principle intentionally.A group of actual decision makers in this field were included in a Delphi panel and were therefore confronted with problems and possible future developments in a field which is considerably influenced by their own actions. The Delphi method was therefore used to structure the decision-making process and to help to “create” the future in reality rather than just predicting it. In this way, an attempt was made to transfer an important field of socioeconomic development from a pattern of accidental development through uncoordinated decisions toward a pattern of broad discussion among all involved social groups and thus toward goal-oriented and well-considered management.It seems that this study represents a further variant of the well-known Delphi technique, a variant which has fundamental differences from the classical and the policy Delphi. For a Delphi application of this type the expression “Decision Delphi” is suggested.     

Shang inquiry as an alternative to Delphi: Some experimental findings

A new group judgment method known as Shang is presented as an alternative to Delphi. Shang is structured to avoid Delphi problems associated with convergence pressures and response commitment while incorporating Delphi advantages. The new method is evaluated through an experiment comparing it with a control method and two Delphi techniques, one using point estimates and one using distributional estimates. Findings indicate that Shang is generally superior to Delphi in producing accurate numerical estimates from the judgments of a group of experts.     

Desirability bias in foresight: Consequences for decision quality based on Delphi results

In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.     

Delphi questionnaires versus individual and group interviews: A comparison case

The article reports on a methodological investigation for different types of the communication device represented by the Delphi method. Traditionally mailed or computer-mediated questionnaires are used. The Delphi method was invented to overcome undesirable effects of group interaction while retaining the positive effects of interactive group judgments. This article supplies evidence for the opposite: the superiority of face- to-face techniques (individual and group interviews) as compared to mailed questionnaires in a three-round Policy Delphi using half-structured questionnaires. Moreover, the case study proves that they might produce different results. In a field- or quasi-experimental design the three techniques were compared for their effects. The performance of 100 panel members on a large number of quality standards in the Delphi method is described. A list of pros and cons of the three techniques in a (Policy) Delphi serves as a summary.     

Personal attitudes in the assessment of the future of science and technology: A factor analysis approach

In 1996, the second German Delphi study (“Delphi '98”) was started. The Delphi '98 is a two-round Delphi expert survey being conducted by the Fraunhofer Institute for Systems and Innovation Research (ISI) on behalf of the German Federal Ministry of Education, Research, Science, and Technology (BMBF). The study was published in February 1998, and is now getting into its implementation phase. Its inherent focus is on the development of science and technology in 12 thematic fields in the next 30 years. To arrive at a better understanding of the influence of personal attitudes towards general developments in natural environment and society, the respondents were asked in the first round of the Delphi survey for their personal opinion towards several megatrends concerning the natural environment, economic, sociological, and political developments. Over 2,300 answers led to a very solid database, which gives insights into the general attitudes of the German R&D experts. On some topics, there is a high consensus, whereas in others, opposite opinions coexist. These results may serve as the database for a factor analysis leading to the identification of five different expert types. This paper examines the crucial question of whether different patterns in assessing the future development in science and technology by expert types can be observed. In general, it turned out that differences in personal attitudes towards megatrends do not influence the estimation of developments in science and technology. However, differences exist in specific topics and the distribution of the five experts types among the respondents differs significantly in the 12 fields.     

Consensus among Economics Teachers from Transition Economies

The authors analyze the economic opinions of teachers and economists from the former Soviet Union who participated in economic education programs sponsored by the U.S. Department of Education under the auspices of the National Council on Economic Education from 1995–2001. They sought to determine the level of consensus on economic topics among the participants, and then analyzed their concordance with U.S. economists. They conclude that there is a significant level of consensus among the participants from the former Soviet Union, but this consensus is often at odds with the views of American economists.     

GOS tree (Goal-Objective-Strategy tree) approach to strategic planning using a fuzzy-Delphi process: An application to the Indian Shrimp Industry

Setting long-term goals, specifying short-term objectives, and formulating strategies to achieve the objectives are the key elements in a strategic planning process. In this paper, we have used hierarchical relationships among these key elements to propose a Goal-Objective-Strategy (GOS) tree. We have proposed an algorithm to validate the structural relationship between two hierarchically adjacent elements of the GOS tree by getting their feasibility score rated in a five-point linguistic scale with the help of a Delphi questionnaire survey. The linguistic scores were then converted into fuzzy scores and consensus of Delphi experts' opinions on feasibility of various statements were derived mathematically using fuzzy aggregation algorithm. We have applied the GOS-tree-based approach as a tool for strategic planning for the Indian Shrimp Industry that has displayed a cyclic performance in the past and has shown signs of rejuvenation in recent times.     

Enhancing a Delphi study on family-focused prevention

This article discusses how additional inquiries can enhance Delphi findings. We argue that inquiries aimed at supporting and refining Delphi findings is both theoretically and practically meaningful. We illustrate our argument on the basis of a framework for family-focused prevention that was developed through a Delphi study. The results of individual and group interviews conducted as a follow-up to the implementation of the framework provided us with effective ways to support and refine it. We draw the conclusion that adequate follow-up inquiries can enhance Delphi findings, from theoretical and application perspectives.     

Stability and agreement criteria for the termination of Delphi studies

This paper addresses the question of the methodology of testing for stability between successive Delphi rounds and the necessity of doing so prior to the analysis of the level or type of consensus. Following a discussion of a hierarchical set of a criteria for determining the termination of such studies, the paper reviews some of the more common treatments of the subject in the literature. It is finally proposed that a χ² test be used to test the stability of responses between rounds. The method is demonstrated by a numerical example.     

Individual stability in Delphi studies

This article carries forward the work discussed in a recent article in this journal by Dajani, Sincoff, and Talley (Technol. Forecast. Soc. Change 13, 83, 1979) by investigating individual stability as opposed to group stability for successive Delphi rounds. It is demonstrated that individual stability implies group stability but that the converse does not hold. A χ² test is proposed for testing the stability of individual responses between consecutive Delphi rounds. Also, an estimation procedure is presented for measuring the extent of individual stability.     

An empirical study of a new method for forming group judgments: Qualitative Controlled Feedback

The Qualitative Controlled Feedback (QCF) method was developed by Press [1] to assist policy makers in forming judgments and making decisions that reflect the careful interactive reasoning and arguments of all of the members of a group or population. Since the QCF method involves controlled feedback, it tends to minimize the effects of face-to-face group interaction pressures. Since the feedback is “qualitative,” however, the procedure tends not to artificially induce a consensus on the group. This paper summarizes a feasibility study of the procedure. A sample of 111 faculty and staff members of the University of British Columbia participated in the testing of the method. The participants were asked to make a judgment on the importance of building an aquatic center on campus. A second (control) group of 89 faculty and staff members was surveyed on the same issue, but using the conventional survey method, that is, no feedback. It was observed that Qualitative Controlled Feedback created a good interaction (in the sense of exchanging arguments and reasons) among group members. Changes in judgments occurred as subjects went from one stage to another after having qualitative feedback of information. It was also found that the judgments given by the subjects in the qualitative controlled feedback group were distributed quite differently from those given by the control group. The method suggests a significantly new way of collecting and interpreting group judgments.     

The future of economics: A Delphi Study

In an attempt to forecast the shape of economics in the year 2001, a Delphi study was undertaken, involving over 200 professional economists from 18 different countries. The end result was a list of 28 breakthroughs thought likely to occur in the next 20 years. The paper describes each breakthrough in detail, the process by which consensus and dissent were communicated, and the method used to select the panel of experts surveyed. Although the overall outcome of the study can be interpreted from several perspectives, one inescapable impression that emerges is that the liberal tradition in economics is latent but not dead, or equivalently, that the present conservative drift in the discipline is likely to be shortlived.