Enhancing a Delphi study on family-focused prevention

This article discusses how additional inquiries can enhance Delphi findings. We argue that inquiries aimed at supporting and refining Delphi findings is both theoretically and practically meaningful. We illustrate our argument on the basis of a framework for family-focused prevention that was developed through a Delphi study. The results of individual and group interviews conducted as a follow-up to the implementation of the framework provided us with effective ways to support and refine it. We draw the conclusion that adequate follow-up inquiries can enhance Delphi findings, from theoretical and application perspectives.     

Desirability bias in foresight: Consequences for decision quality based on Delphi results

In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.     

Current validity of the Delphi method in social sciences

The Delphi method is a popular technique for forecasting and an aid in decision-making based on the opinions of experts, which has been in existence for over half a century. This work evaluates its methodology and reviews its validity in the present day, especially in the area of Social Sciences. Three recent applications in this field are also explained, professional in nature, which have some characteristics that are not frequent with respect to other Delphi studies published. The main aim of two of these studies was to provide input for economic or statistical quantitative models, using the judgement of expert groups, while the third study aimed to analyse a complex social reality by means of a Policy Delphi in order to obtain reliable information before taking a policy decision. These applications highlight how this technique may be adapted to different social realities and requirements, making a positive contribution to social progress, provided it is applied with the necessary methodological rigour and with a good knowledge of the social medium in which it is being applied. Finally, there is an explanation of a number of lessons learned from the theory and aforementioned experiences, which may contribute to the successful outcome of a Delphi exercise.     

Validating an innovative real-time Delphi approach - A methodological comparison between real-time and conventional Delphi studies

A novel and innovative real-time Delphi technique is introduced in order to address previously identified weaknesses of the conventional Delphi method, such as complicated facilitator tasks, lack of real-time presentation of results, and difficulties in tracking progress over time. We demonstrate how the real-time (computer-based) method increases the efficiency of the process, accommodates expert availability, and reduces drop-out-rates. Modifications in the Delphi procedure (e.g. change of iteration principle) not only increase efficiency but also change the nature and process of the survey technique itself. By identifying and analysing three individual effects (initial condition effect, feedback effect, and iteration effect) we examine whether the modifications in the survey process cause deviations to the survey results. Empirical data obtained from both conventional as well as real-time Delphi studies is analysed based on multiple statistical analyses. The research findings indicate that significant differences between the two Delphi survey formats do not exist and final survey results are not affected by changes in the survey procedure.     

Design and evaluation of ranking-type Delphi studies using best-worst-scaling

The ranking-type Delphi method has received widespread use in a large number of research areas. In order to rank a set of items, ranking-type Delphi studies use different ranking approaches which may be biased. This is intensified by the fact that the number of issues a participant can reasonably rank is limited. This paper describes an approach for the design and evaluation of ranking-type Delphi studies using best-worst-scaling (BWS). The statistical BWS design is based on a balanced incomplete block design to construct the comparison sets. The statistical evaluation encompasses the measurement of stability and consensus with the use of the coefficient of variation. Overall, our approach (1) enables to design a ranking with up to 22 items in a manner that participants can reasonably rank, (2) avoids and overcomes shortcomings of common ranking approaches, and (3) offers a statistical procedure for an unambiguous evaluation.     

Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education

The focus of this study is the Delphi method. First, a short history of the Delphi method is given. Then, different types of the Delphi method are described, and the validity and reliability of the Delphi method are discussed. Finally, this study reports on the selection processes and assessments faced when a policy Delphi was conducted into qualification issues in Senior Secondary Vocational Education in the Netherlands.     

Enhancing rigour in the Delphi technique research

The cornerstone of good research is establishing integrity. However, identifying and gauging methodological rigour for the Delphi technique remains elusive. This is due to a number of reasons such as the ongoing epistemological debate, along with continual modifications. Consequently, the scant studies exploring rigour are mainly experimental, component specific and outdated. This paper discusses the literature on establishing rigour in Delphi studies, the methodological trinity of reliability, validity and trustworthiness. In addition it presents a discussion of the principal forms of establishing rigour, such as the application of rigour using both qualitative and quantitative measurements and corroborating results with relevant evidence in the field for each individual Delphi. Addressing such issues will help enhance the development and utilisation of rigour in the future.     

Identification of future fields of standardisation: An explorative application of the Delphi methodology

This paper investigates the application of the Delphi methodology for the identification of future fields of standardisation complemented by a methodological extension by using various science and technology indicators. By the term standardisation, we broadly mean the process of developing and implementing technical standards within a standardisation body. Underlining the explorative nature of this paper, we describe the process of identifying future fields of standardisation.To provide a systematic forecasting view on complex science and technology fields, a combination of quantitative indicator-based analyses and qualitative in-depth Delphi surveys is choosen. Firstly, statistical analyses of suitable indicators are used to identify dynamic developments in such fields. Secondly, to identify detailed challenges for future standardisation, qualitative Delphi surveys are conducted. To collect and evaluate relevant issues the respective expert communities were included. They were identified by using information derived from the science and technology databases used.The paper concludes with the assessment of the chosen approach and give practical insights for its feasibility based on a review of the existing literature on the Delphi methodology. In addition, an outlook for further improvements and other possible fields of application is given.     

The Delphi technique: Past, present, and future prospects — Introduction to the special issue

The Delphi technique has been around for over half a century, so now seems a proper time to consider its past, present and future. This introduction characterises the papers in this special Delphi issue, which include both conceptual and empirical works. It summarises the main lessons that have been learned from these for the conduct of the technique, and provides a call for more and better empirical studies in the future.     

Exploring the application of the Delphi method as a forecasting tool in Information Systems and Technologies research

The Delphi method has been used widely in clarifying decision-making based on the opinions of experts in all research areas. This study focuses on Information Systems and Information Technology (IS/IT) research area. Its aim is to explore the use of the Delphi method in topics related to IS/IT as a forecasting tool. A sizeable amount of research has been examined about IS/IT using the Delphi method. This work proposes a guideline for applying the Delphi method in IS/IT studies based on a literature review of high impact IS/IT papers that apply Delphi with a view to answering forecasting questions. This guideline defines the most relevant aspects which should be considered when applying this method. Previously, the authors conducted a literature review of 2502 articles from different research topics to obtain evidence about the diffusion of the Delphi method in IS/IT compared to other areas.     

The importance of a common understanding in emergency management

An emergency situation brings together individuals belonging to many different organizations, representing different organizational cultures including different usage of language. In this paper we propose that a common understanding is vital in managing emergency situations. Firstly, a simple emergency situation is presented where one of the causes for increased damage was simply due to a misunderstanding of language. Secondly, we propose a theoretical framework of how different organization cultures with their different uses of language can be integrated on the terminological level where the communication takes place. Thirdly, the presented example is reviewed. We show how situations such as the one in our example can be analyzed by means of the theoretical framework. Finally, we present the findings from a Delphi study conducted in Finland, which support the central thesis, i.e. the importance of a common understanding in emergency situations, and we propose using the Delphi method to find the critical domains in the flow of information between the different actors involved in disaster management.     

The Delphi method as a ritual: Inquiring the Delphic Oracle

Is Delphi just a namesake? Or does it make sense to connect the Delphi method to the ancient Delphic Oracle? To bring some elements of response to these questions, this article will present the Delphic Oracle in the first part and its actions with its prophecy seekers and the Oracle's staff, including the Pythia. This first part will show the importance of conforming to the procedure in the revelation of the Oracle and the latter's role in policy-making. The second part of the article will compare and contrast the oracular practice in Delphi with the Delphi method, and more generally with expert-based futures methods, from the characteristics of the knowledge revealed, on the one hand, and the role of the actors in this inquiring process on the other.     

The use of Delphi methodology in agrifood policy development: Some lessons learned

Developing policy in the agrifood area is an inexact process, usually relying upon effective integration of opinions from multiple experts from different disciplines, organisational types, and regions/countries. Delphi would appear to have the potential to overcome some of the typical limitations related to soliciting expert opinion and identifying consensus on future activities or options, particularly where relevant experts are dispersed geographically, and international consensus is required, as is the case in this domain. Three case studies, focused on the application of Delphi to emerging policy needs in international or European agrifood policy, are presented here to exemplify the utility of the technique. A number of practical recommendations are drawn from these case studies that may be applicable to other major policy making arenas. Among these recommendations are; that an exploratory workshop to refine round one Delphi questions is essential; that the implementation of “cascade” methodology (utilizing the personal contacts of researchers or members of existing policy networks) appears to increase response rates in subsequent Delphi rounds; and that the policy issue under discussion should be particularly relevant to stakeholders in order to increase participation rates. Further research might usefully focus on developing ways to incorporate measurements of uncertainty associated with stakeholder judgement into quantitative responses, and on establishing how best to utilise such information in feedback in subsequent Delphi rounds. Ensuring how best to inform policy uptake of the outputs of Delphi merits further research in particular.     

RT Delphi: An efficient, “round-less” almost real time Delphi method

The authors have recently developed a new approach to performing a Delphi study that does not involve the use of sequential “rounds” and as a result, greatly improves the efficiency of the process and shortens the time to perform such studies. This paper describes this process, RT Delphi, and illustrates its use in a decision-making application drawn from the Millennium Project of the American Council for the United Nations University. The illustrative application involves setting priorities among strategies for dealing with anticipated terrorist activities that might be initiated by a single deranged individual.     

The Delphi method, impeachment and terrorism: Accuracies of short-range forecasts for volatile world events

Two experiments examined the effects of iterated polling and feedback on prediction accuracy for volatile world events using the Delphi paradigm. In both experiments, participants predicted rapidly changing geopolitical events. The first experiment occurred during the litigation surrounding the Clinton/Lewinsky affair in 1998. Results indicated that feedback improved individual and group accuracy for predictions of whether an event would occur. These types of predictions derived from group consensus were approximately 90% accurate, which was significantly higher than the average individual participant's accuracy. Neither polling nor feedback had much effect on time course predictions. The second experiment occurred during the American military action against the Taliban regime in Afghanistan in 2001. In Experiment 2, participants were polled three times to determine if increasing the number of iterated polls would improve Delphi accuracy. The results showed that accuracy improved from the first to the second poll but not from the second to the third. The groups predicted whether these scenarios would occur with approximately 75% accuracy, which was significantly higher than the accuracy of any individual participant. Once again, the Delphi procedure did not enhance time course predictions in Experiment 2.     

Personal attitudes in the assessment of the future of science and technology: A factor analysis approach

In 1996, the second German Delphi study (“Delphi '98”) was started. The Delphi '98 is a two-round Delphi expert survey being conducted by the Fraunhofer Institute for Systems and Innovation Research (ISI) on behalf of the German Federal Ministry of Education, Research, Science, and Technology (BMBF). The study was published in February 1998, and is now getting into its implementation phase. Its inherent focus is on the development of science and technology in 12 thematic fields in the next 30 years. To arrive at a better understanding of the influence of personal attitudes towards general developments in natural environment and society, the respondents were asked in the first round of the Delphi survey for their personal opinion towards several megatrends concerning the natural environment, economic, sociological, and political developments. Over 2,300 answers led to a very solid database, which gives insights into the general attitudes of the German R&D experts. On some topics, there is a high consensus, whereas in others, opposite opinions coexist. These results may serve as the database for a factor analysis leading to the identification of five different expert types. This paper examines the crucial question of whether different patterns in assessing the future development in science and technology by expert types can be observed. In general, it turned out that differences in personal attitudes towards megatrends do not influence the estimation of developments in science and technology. However, differences exist in specific topics and the distribution of the five experts types among the respondents differs significantly in the 12 fields.     

The UK technology foresight programme: An assessment of expert estimates

Since the early 1990s, ‘Technology Foresight’ exercises with special emphasis on the use of Delphi surveys have played an important role in science and technology (S + T) policy across Europe in an effort to focus resource allocation. Yet, none of the estimates made in the European Delphi surveys have been formally assessed in retrospect, while this process has been incorporated into the Japanese surveys since 1996. Taking the UK Technology Foresight Programme, this research sets out to assess the estimates of three of the fifteen panel Delphi surveys. Whilst on average 2/3 of Delphi statements were predicted to be realised by 2004, it will be shown that only a fraction of these statements had been realised by 2006. Based on the evidence collected from the published panel reports, the ‘Hindsight on Foresight’ survey conducted by OST in 1995 and interviews with panel members, it will be argued that the overwhelming majority of estimates were overly optimistic. While optimism and strategic gaming of experts is the most convincing explanation for these results, process factors were also explored, including the quality of expert panels used, the Delphi statements and the respondents of the Delphi questionnaire. It is argued that at least the issue of short-range optimism and strategic gaming of experts should be addressed in future Delphi exercises, as decision makers relying on expert advice cannot deal with this issue alone.     

The Delphi method as early warning: Linking global societal trends to future radicalization and terrorism in The Netherlands

A Delphi study was performed to answer the question: Which global societal trends relate to future radicalization and subsequent terrorism in the Netherlands? An inventory of two hundred global societal trends and a literature study of radicalization together served as briefing of the participants in the Delphi study. The first two rounds were conducted in anonymous writing. The third round was a face to face focus group meeting. In the first and second round participants scored the trends, adjusted their scores relative to others and provided arguments, and in the focus group meeting, themes were discussed in which the participants remained to have strong opposite views. This approach emphasizes divergence in opinion (‘dissensus’), as opposed to convergence (‘consensus’). Consensus seeking to us seems to be the dominant application of Delphi, but we emphasize the value of seeking dissensus. We hypothesize that in those cases where the opposite views that were discussed kept their ground, this Delphi study may have found early warnings of future radicalization.The essence of this paper is that it is possible, with limited effort, to get a handle on the complex and poorly defined subject of global societal trends influencing future radicalization. Application of the Dissensus Delphi method provided a selection of early warnings that may be looked into with future research.     

Future prospects of alternative agro-based bioenergy use in Finland—Constructing scenarios with quantitative and qualitative Delphi data

In this paper, the potential of bioenergy production in agriculture (preferred and probable futures) is scrutinised using the Delphi method. We present a case of northern possibilities to utilise renewable energy sources within agriculture in a form of alternative bioenergy scenarios. Altogether 20 experts participated in the Delphi process which outlines the future of bioenergy production in Finland. The first round of the Delphi study was carried out by semi-structured interviews and the second feedback round by means of a mail questionnaire. Background information of key variables was presented to the panellists who responded with their views on developments between 2004 and 2025. Alternative scenarios were then constructed from these dimensions with cluster analysis in line with the Disaggregative Policy Delphi (DPD) approach. Quantitative statements were complemented with the experts' argumentation. Five scenarios were constructed: 1) Renewable prosperity, 2) Incremental change, 3) Vision of sun, wind and wood, 4) Let's burn it all, and 5) Flood of waterpower. After the Delphi rounds, a dialogue seminar for the agri-technology experts and policy-makers was organised. These results bring to the table an agri-food technology expert community view of the future directions for Finnish agro-bioenergy use.     

Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.