Desirability bias in foresight: Consequences for decision quality based on Delphi results

In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.     

RT Delphi: An efficient, “round-less” almost real time Delphi method

The authors have recently developed a new approach to performing a Delphi study that does not involve the use of sequential “rounds” and as a result, greatly improves the efficiency of the process and shortens the time to perform such studies. This paper describes this process, RT Delphi, and illustrates its use in a decision-making application drawn from the Millennium Project of the American Council for the United Nations University. The illustrative application involves setting priorities among strategies for dealing with anticipated terrorist activities that might be initiated by a single deranged individual.     

Current validity of the Delphi method in social sciences

The Delphi method is a popular technique for forecasting and an aid in decision-making based on the opinions of experts, which has been in existence for over half a century. This work evaluates its methodology and reviews its validity in the present day, especially in the area of Social Sciences. Three recent applications in this field are also explained, professional in nature, which have some characteristics that are not frequent with respect to other Delphi studies published. The main aim of two of these studies was to provide input for economic or statistical quantitative models, using the judgement of expert groups, while the third study aimed to analyse a complex social reality by means of a Policy Delphi in order to obtain reliable information before taking a policy decision. These applications highlight how this technique may be adapted to different social realities and requirements, making a positive contribution to social progress, provided it is applied with the necessary methodological rigour and with a good knowledge of the social medium in which it is being applied. Finally, there is an explanation of a number of lessons learned from the theory and aforementioned experiences, which may contribute to the successful outcome of a Delphi exercise.     

A Delphi technology forecasting approach using a semi-Markov concept

This paper suggests a technology forecasting approach based on a semi-Markov model, which appropriately describes the probabilistic nature of a sequential technology development process. This approach focuses primarily on the utilization of the information that has been skipped in conventional Delphi survey data. That is, through a simple statistic, the interrelationships among sequential technology developments can be extracted in a formal structure of a semi-Markov model from the original Delphi panel's estimates. A simulation technique is developed to forecast the development process by utilizing the information on such interrelationships. This technique provides a flexible and useful tool for R&D planners or project managers, especially in postanalysis of Delphi forecasting. To make good use of the approach, a computer-based interactive Delphi data analysis system (IDEAS) is implemented in IBM PC.     

Judgment change during Delphi-like procedures: The role of majority influence, expertise, and confidence

This study investigates individual opinion change and judgmental accuracy in Delphi-like groups. Results reveal that the accuracy of judgmental probability forecasts increases over Delphi rounds (in terms of proportion correct and appropriateness of confidence) when statistical summaries or written rationales are provided from other members of an individual's nominal group, but does not increase in a control iteration condition (without feedback). Additionally, subjects who gave more appropriate probability forecasts on the first round exhibited least opinion change, although measures of confidence were unrelated to opinion change. Results also show that majority opinion exerts strong opinion pull on minority opinion even when the majority favours an incorrect answer (irrespective of the nature of feedback provided). The implications of these results for the utility and conduct of the Delphi technique are discussed, in particular, with respect to selecting panellists and choosing an appropriate feedback format.     

The Delphi method as a ritual: Inquiring the Delphic Oracle

Is Delphi just a namesake? Or does it make sense to connect the Delphi method to the ancient Delphic Oracle? To bring some elements of response to these questions, this article will present the Delphic Oracle in the first part and its actions with its prophecy seekers and the Oracle's staff, including the Pythia. This first part will show the importance of conforming to the procedure in the revelation of the Oracle and the latter's role in policy-making. The second part of the article will compare and contrast the oracular practice in Delphi with the Delphi method, and more generally with expert-based futures methods, from the characteristics of the knowledge revealed, on the one hand, and the role of the actors in this inquiring process on the other.     

Enhancing rigour in the Delphi technique research

The cornerstone of good research is establishing integrity. However, identifying and gauging methodological rigour for the Delphi technique remains elusive. This is due to a number of reasons such as the ongoing epistemological debate, along with continual modifications. Consequently, the scant studies exploring rigour are mainly experimental, component specific and outdated. This paper discusses the literature on establishing rigour in Delphi studies, the methodological trinity of reliability, validity and trustworthiness. In addition it presents a discussion of the principal forms of establishing rigour, such as the application of rigour using both qualitative and quantitative measurements and corroborating results with relevant evidence in the field for each individual Delphi. Addressing such issues will help enhance the development and utilisation of rigour in the future.     

Scenario construction via Delphi and cross-impact analysis

Since its origins, decision makers have broadly used the Delphi method as a collaborative technique for generating important events and scenarios about what may happen in the future. This is a complex process because of the different interrelations and the potential synergetic effects among the relevant events related to a decision. This fact, along with the uncertainty about the occurrence or non-occurrence of the events, makes the scenario generation task a challenging issue in Delphi processes. In the 1960's, Cross-Impact Analysis (CIA) appeared as a methodological tool for dealing with this complexity. CIA can be used for creating a working model out from a set of significant events. CIA has been combined with other methodological approaches in order to increase its functionality and improve its final outcome. In this paper, the authors propose a new step-by-step model for scenario-analysis based on a merger of Turoff's alternative approach to CIA and the technique called Interpretive Structural Modeling (ISM). The authors' proposal adds tools for detecting critical events and for producing a graphical representation to the previous scenario-generation methods based on CIA. Moreover, it allows working with large sets of events without using large computational infrastructures. The authors present sufficient information and data so that anyone who wishes to may duplicate the implementation of the process. Additionally they make explicit a set of requirements for carrying out a Delphi process for a group to develop a set of significant events, collectively make the estimations of cross impacts, and to support a continuous planning process within an organization. They use two examples to discuss operational issues and practical implications of the model.     

Design and evaluation of ranking-type Delphi studies using best-worst-scaling

The ranking-type Delphi method has received widespread use in a large number of research areas. In order to rank a set of items, ranking-type Delphi studies use different ranking approaches which may be biased. This is intensified by the fact that the number of issues a participant can reasonably rank is limited. This paper describes an approach for the design and evaluation of ranking-type Delphi studies using best-worst-scaling (BWS). The statistical BWS design is based on a balanced incomplete block design to construct the comparison sets. The statistical evaluation encompasses the measurement of stability and consensus with the use of the coefficient of variation. Overall, our approach (1) enables to design a ranking with up to 22 items in a manner that participants can reasonably rank, (2) avoids and overcomes shortcomings of common ranking approaches, and (3) offers a statistical procedure for an unambiguous evaluation.     

Future prospects of alternative agro-based bioenergy use in Finland—Constructing scenarios with quantitative and qualitative Delphi data

In this paper, the potential of bioenergy production in agriculture (preferred and probable futures) is scrutinised using the Delphi method. We present a case of northern possibilities to utilise renewable energy sources within agriculture in a form of alternative bioenergy scenarios. Altogether 20 experts participated in the Delphi process which outlines the future of bioenergy production in Finland. The first round of the Delphi study was carried out by semi-structured interviews and the second feedback round by means of a mail questionnaire. Background information of key variables was presented to the panellists who responded with their views on developments between 2004 and 2025. Alternative scenarios were then constructed from these dimensions with cluster analysis in line with the Disaggregative Policy Delphi (DPD) approach. Quantitative statements were complemented with the experts' argumentation. Five scenarios were constructed: 1) Renewable prosperity, 2) Incremental change, 3) Vision of sun, wind and wood, 4) Let's burn it all, and 5) Flood of waterpower. After the Delphi rounds, a dialogue seminar for the agri-technology experts and policy-makers was organised. These results bring to the table an agri-food technology expert community view of the future directions for Finnish agro-bioenergy use.     

The Delphi method, impeachment and terrorism: Accuracies of short-range forecasts for volatile world events

Two experiments examined the effects of iterated polling and feedback on prediction accuracy for volatile world events using the Delphi paradigm. In both experiments, participants predicted rapidly changing geopolitical events. The first experiment occurred during the litigation surrounding the Clinton/Lewinsky affair in 1998. Results indicated that feedback improved individual and group accuracy for predictions of whether an event would occur. These types of predictions derived from group consensus were approximately 90% accurate, which was significantly higher than the average individual participant's accuracy. Neither polling nor feedback had much effect on time course predictions. The second experiment occurred during the American military action against the Taliban regime in Afghanistan in 2001. In Experiment 2, participants were polled three times to determine if increasing the number of iterated polls would improve Delphi accuracy. The results showed that accuracy improved from the first to the second poll but not from the second to the third. The groups predicted whether these scenarios would occur with approximately 75% accuracy, which was significantly higher than the accuracy of any individual participant. Once again, the Delphi procedure did not enhance time course predictions in Experiment 2.     

Results of a Delphi survey in drawing up the input-output tables for Catalonia

This work presents the results of an ambitious and innovative use of the Delphi technique in supplying a quantitative economic model, specifically in the preparation of the Input-Output Tables for Catalonia (TIOC 2001). Using this technique for obtaining subjective information from experts comes forward as a valid and reliable option for improving the quality of the data with which the Input-Output tables are constructed, also providing further economic and social advantages. The application presented thus opens up an interesting field of development for this technique and also gives several contributions for appraising the execution of a Delphi exercise of this type.     

Enhancing a Delphi study on family-focused prevention

This article discusses how additional inquiries can enhance Delphi findings. We argue that inquiries aimed at supporting and refining Delphi findings is both theoretically and practically meaningful. We illustrate our argument on the basis of a framework for family-focused prevention that was developed through a Delphi study. The results of individual and group interviews conducted as a follow-up to the implementation of the framework provided us with effective ways to support and refine it. We draw the conclusion that adequate follow-up inquiries can enhance Delphi findings, from theoretical and application perspectives.     

The use of Delphi methodology in agrifood policy development: Some lessons learned

Developing policy in the agrifood area is an inexact process, usually relying upon effective integration of opinions from multiple experts from different disciplines, organisational types, and regions/countries. Delphi would appear to have the potential to overcome some of the typical limitations related to soliciting expert opinion and identifying consensus on future activities or options, particularly where relevant experts are dispersed geographically, and international consensus is required, as is the case in this domain. Three case studies, focused on the application of Delphi to emerging policy needs in international or European agrifood policy, are presented here to exemplify the utility of the technique. A number of practical recommendations are drawn from these case studies that may be applicable to other major policy making arenas. Among these recommendations are; that an exploratory workshop to refine round one Delphi questions is essential; that the implementation of “cascade” methodology (utilizing the personal contacts of researchers or members of existing policy networks) appears to increase response rates in subsequent Delphi rounds; and that the policy issue under discussion should be particularly relevant to stakeholders in order to increase participation rates. Further research might usefully focus on developing ways to incorporate measurements of uncertainty associated with stakeholder judgement into quantitative responses, and on establishing how best to utilise such information in feedback in subsequent Delphi rounds. Ensuring how best to inform policy uptake of the outputs of Delphi merits further research in particular.     

Opinion convergence in location: A spatial version of the Delphi method

This paper proposes a modified version of the Delphi method, starting from the assumption that in many situations the convergence of opinions can be considered in a spatial context. The Spatial Delphi we suggest is based, like the classical Delphi, on the judgments of experts, and it is useful in the consultations for decision and/or forecast purposes, provided that they concern matters of spatial location. The basis for the questionnaire is a map, on which each expert provides, as answer(s), one or more opinion-points, i.e., locations that, according to their opinion, are best for a specific purpose. We propose a method for narrowing the area of convergence along a multi-step procedure so that the final result of the consultation is a small portion of the initial territory with respect to the initial area considered. Besides a simulation study, we report also a prototype application with a panel of twelve experts.     

Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.     

The Delphi technique: Past, present, and future prospects — Introduction to the special issue

The Delphi technique has been around for over half a century, so now seems a proper time to consider its past, present and future. This introduction characterises the papers in this special Delphi issue, which include both conceptual and empirical works. It summarises the main lessons that have been learned from these for the conduct of the technique, and provides a call for more and better empirical studies in the future.     

The decision Delphi

If Delphi is used as a prognostic tool in the social sciences, it runs the danger of producing a self-fulfilling prophecy. In a Delphi application conducted at the Austrian Academy of Sciences about the present situation and future developments of the scientific-technical information and documentation system in Austria, an attempt was made to apply this principle intentionally.A group of actual decision makers in this field were included in a Delphi panel and were therefore confronted with problems and possible future developments in a field which is considerably influenced by their own actions. The Delphi method was therefore used to structure the decision-making process and to help to “create” the future in reality rather than just predicting it. In this way, an attempt was made to transfer an important field of socioeconomic development from a pattern of accidental development through uncoordinated decisions toward a pattern of broad discussion among all involved social groups and thus toward goal-oriented and well-considered management.It seems that this study represents a further variant of the well-known Delphi technique, a variant which has fundamental differences from the classical and the policy Delphi. For a Delphi application of this type the expression “Decision Delphi” is suggested.     

Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education

The focus of this study is the Delphi method. First, a short history of the Delphi method is given. Then, different types of the Delphi method are described, and the validity and reliability of the Delphi method are discussed. Finally, this study reports on the selection processes and assessments faced when a policy Delphi was conducted into qualification issues in Senior Secondary Vocational Education in the Netherlands.     

Delphi analysis of national specificities in selected innovative areas in Germany and France

The increasing complexity of the relations between technologies and economics combined with more social pressure, global competition, technological change, as well as national budget restrictions, imply new challenges for public policies. Thus, to be able to forecast the development of knowledge and technological change in some well-known trajectories could be one of the major stakes for science technology and industrial policies. It is then not surprising that recent years have brought a significant revival of public foresight activities in many countries. The purpose of the paper is precisely to propose a new foresight method in order to obtain a taxonomy of the future technologies, and consequently to provide a better understanding of industrial dynamics. We present a statistical analysis of a Delphi investigation, based on scientific and technological knowledge complementarities, in order to obtain coherent clusters, which may be looked upon as a theoretical tool for political decisions. Our methodology is then applied to French and the German sectors of life sciences, elementary particles, energy, environment and natural resources.