Methods for analyzing data from Delphi panels: Some evidence from a forecasting study

Delphi and other methods of using expert opinion to generate forecasts can be a useful tool for planning, impact assessment, and policy analysis. Unfortunately, little is known about the accuracy of forecasts produced using these methods, so their utility is limited at present. Based on the logic of the Delphi method, I suggest that: 1) forecast accuracy should increase across rounds of a Delphi iteration, 2) there is a positive correlation between a panelist's uncertainty about a forecast and his or her shift in forecast from round to round, 3) forecasts weighted by self-reported confidence will be more accurate than unweighted forecasts, and 4) the use of robust estimates of location as summaries of expert opinion yield better forecasts than nonrobust measures. A Delphi experiment provides little support to any of these hypotheses. This finding suggests that traditional assumptions about the proper methods for analyzing a Delphi study may be inappropriate.     

Delphi questionnaires versus individual and group interviews: A comparison case

The article reports on a methodological investigation for different types of the communication device represented by the Delphi method. Traditionally mailed or computer-mediated questionnaires are used. The Delphi method was invented to overcome undesirable effects of group interaction while retaining the positive effects of interactive group judgments. This article supplies evidence for the opposite: the superiority of face- to-face techniques (individual and group interviews) as compared to mailed questionnaires in a three-round Policy Delphi using half-structured questionnaires. Moreover, the case study proves that they might produce different results. In a field- or quasi-experimental design the three techniques were compared for their effects. The performance of 100 panel members on a large number of quality standards in the Delphi method is described. A list of pros and cons of the three techniques in a (Policy) Delphi serves as a summary.     

The UK technology foresight programme: An assessment of expert estimates

Since the early 1990s, ‘Technology Foresight’ exercises with special emphasis on the use of Delphi surveys have played an important role in science and technology (S + T) policy across Europe in an effort to focus resource allocation. Yet, none of the estimates made in the European Delphi surveys have been formally assessed in retrospect, while this process has been incorporated into the Japanese surveys since 1996. Taking the UK Technology Foresight Programme, this research sets out to assess the estimates of three of the fifteen panel Delphi surveys. Whilst on average 2/3 of Delphi statements were predicted to be realised by 2004, it will be shown that only a fraction of these statements had been realised by 2006. Based on the evidence collected from the published panel reports, the ‘Hindsight on Foresight’ survey conducted by OST in 1995 and interviews with panel members, it will be argued that the overwhelming majority of estimates were overly optimistic. While optimism and strategic gaming of experts is the most convincing explanation for these results, process factors were also explored, including the quality of expert panels used, the Delphi statements and the respondents of the Delphi questionnaire. It is argued that at least the issue of short-range optimism and strategic gaming of experts should be addressed in future Delphi exercises, as decision makers relying on expert advice cannot deal with this issue alone.     

The delphi technique: an exposition and application

The article provides a n overview of the Delphi technique and its related adaptations, as well as practical hints on procedures gained from its use to forecast changes in the market analysis industry. The Lielphi technique has been used many times as a method of forecasting the future ofestab1ish:ed industries, but it has never been used to consider the future of a nascent industry. A case is presented which suggests that companies i n nascent industries have a greater need for long-range planning than those at other stages of a n industry's development, and a greater need for an industry, rather than a company, perspective. Problems such as time pressure on executives and the need for confidentiality are more acute when using judgmental forecasting techniques in new industries. The review of the Delphi process highlights some of the major considerations, s,uch as panel selection, questionnaire construction and administration, response analysis and number of rounds, as well as how Delphi can be modified. Since new industries rarely have established trade organizations to curry out such Delphi studies, the role could be played by institutes of management education.     

The delphi technique: an exposition and application

The article provides a n overview of the Delphi technique and its related adaptations, as well as practical hints on procedures gained from its use to forecast changes in the market analysis industry. The Lielphi technique has been used many times as a method of forecasting the future ofestab1ish:ed industries, but it has never been used to consider the future of a nascent industry. A case is presented which suggests that companies i n nascent industries have a greater need for long-range planning than those at other stages of a n industry's development, and a greater need for an industry, rather than a company, perspective. Problems such as time pressure on executives and the need for confidentiality are more acute when using judgmental forecasting techniques in new industries. The review of the Delphi process highlights some of the major considerations, s,uch as panel selection, questionnaire construction and administration, response analysis and number of rounds, as well as how Delphi can be modified. Since new industries rarely have established trade organizations to curry out such Delphi studies, the role could be played by institutes of management education.     

The decision Delphi

If Delphi is used as a prognostic tool in the social sciences, it runs the danger of producing a self-fulfilling prophecy. In a Delphi application conducted at the Austrian Academy of Sciences about the present situation and future developments of the scientific-technical information and documentation system in Austria, an attempt was made to apply this principle intentionally.A group of actual decision makers in this field were included in a Delphi panel and were therefore confronted with problems and possible future developments in a field which is considerably influenced by their own actions. The Delphi method was therefore used to structure the decision-making process and to help to “create” the future in reality rather than just predicting it. In this way, an attempt was made to transfer an important field of socioeconomic development from a pattern of accidental development through uncoordinated decisions toward a pattern of broad discussion among all involved social groups and thus toward goal-oriented and well-considered management.It seems that this study represents a further variant of the well-known Delphi technique, a variant which has fundamental differences from the classical and the policy Delphi. For a Delphi application of this type the expression “Decision Delphi” is suggested.     

Disaggregative policy Delphi: Using cluster analysis as a tool for systematic scenario formation

A critical phase of scenario making is the choosing of scenarios. In the worst case, a futures researcher creates scenarios according to his/her subjective views and cannot see the real quality of the study material. Oversimplification is a typical example of this kind of bias. In this study, an attempt towards a more data sensitive method was made using Finnish transport policy as an example. A disaggregative Delphi method as opposed to traditional consensual Delphi was applied. The article summarises eight Delphi pitfalls and gives an example how to avoid them. A two-rounded disaggregative Delphi was conducted, the panelists being representatives of interest groups in the traffic sector. Panelists were shown the past development of three correlating key variables in Finland in 1970-1996: GDP, road traffic volume and the carbon dioxide emissions from road traffic. The panelists were invited to give estimates of their organisation to the probable and the preferable futures of the key variables for 1997-2025. They were also asked to give qualitative and quantitative arguments of why and the policy instruments of how their image of the future would occur. The first round data were collected by a fairly open questionnaire and the second round data by a fairly structured interview. The responses of the quantitative three key variables were grouped in a disaggregative way by cluster analysis. The clusters were complemented with respective qualitative arguments in order to form wider scenarios. This offers a relevance to decision-making not afforded by a nonsystematic approach. Of course, there are some problems of cluster analysis used in this way: The interviews revealed that quantitatively similar future images produced by the panelists occasionally had different kind of qualitative background theory. Also, cluster analysis cannot ultimately decide the number of scenarios, being a choice of the researcher. Cluster analysis makes the choice well argued, however.     

Exploring the application of the Delphi method as a forecasting tool in Information Systems and Technologies research

The Delphi method has been used widely in clarifying decision-making based on the opinions of experts in all research areas. This study focuses on Information Systems and Information Technology (IS/IT) research area. Its aim is to explore the use of the Delphi method in topics related to IS/IT as a forecasting tool. A sizeable amount of research has been examined about IS/IT using the Delphi method. This work proposes a guideline for applying the Delphi method in IS/IT studies based on a literature review of high impact IS/IT papers that apply Delphi with a view to answering forecasting questions. This guideline defines the most relevant aspects which should be considered when applying this method. Previously, the authors conducted a literature review of 2502 articles from different research topics to obtain evidence about the diffusion of the Delphi method in IS/IT compared to other areas.     

Scenario construction via Delphi and cross-impact analysis

Since its origins, decision makers have broadly used the Delphi method as a collaborative technique for generating important events and scenarios about what may happen in the future. This is a complex process because of the different interrelations and the potential synergetic effects among the relevant events related to a decision. This fact, along with the uncertainty about the occurrence or non-occurrence of the events, makes the scenario generation task a challenging issue in Delphi processes. In the 1960's, Cross-Impact Analysis (CIA) appeared as a methodological tool for dealing with this complexity. CIA can be used for creating a working model out from a set of significant events. CIA has been combined with other methodological approaches in order to increase its functionality and improve its final outcome. In this paper, the authors propose a new step-by-step model for scenario-analysis based on a merger of Turoff's alternative approach to CIA and the technique called Interpretive Structural Modeling (ISM). The authors' proposal adds tools for detecting critical events and for producing a graphical representation to the previous scenario-generation methods based on CIA. Moreover, it allows working with large sets of events without using large computational infrastructures. The authors present sufficient information and data so that anyone who wishes to may duplicate the implementation of the process. Additionally they make explicit a set of requirements for carrying out a Delphi process for a group to develop a set of significant events, collectively make the estimations of cross impacts, and to support a continuous planning process within an organization. They use two examples to discuss operational issues and practical implications of the model.     

An evaluation of Delphi

The literature concerning quantitative applications of the Delphi method is reviewed. No evidence was found to support the view that Delphi is more accurate than other judgment methods or that consensus in a Delphi is achieved by dissemination of information to all participants. Existing data suggest that consensus is achieved mainly by group pressure to conformity, mediated by the statistical group response that is fed back to all participants.     

Public sector use of the Delphi technique

The Delphi Technique is gaining extensive use as a multipurpose tool by government planners and policy makers. This paper begins by briefly discussing the historical development of Delphi and some of its advantages and disadvantages. A considerable number of Delphis conducted in the public sector for the purposes of forecasting, communication, budgeting, and goal setting are then reviewed. Additionally, three prior reviews of Delphi use in the field of education are discussed. In reviewing these studies every attempt was made to detail the research intentions, special features and modifications, cautions, and benefits of each study. These characteristics are provided in tabular form in the summary section as a ready reference for the potential Delphi user. It is recommended that one should take full advantage of what has been learned from earlier research before undertaking a new Delphi.     

Theory and applications of the Delphi technique: A bibliography (1975–1994)

Delphi is a popular, long-range, qualitative forecasting technique that has been extensively applied to a wide variety of problems in different domains. Since the method was conceived in the early 1950s at the Rand Corporation, different variations of Delphi have evolved in an effort to meet the unique forecasting needs of different decision makers. This bibliographic study surveys the literature on the methodology and applications of Delphi over a period of two decades (1975–1994). A total of 463 papers were identified out of which 254 papers treat Delphi as a primary subject while the remaining 209 papers treat Delphi as a secondary subject. The study concludes with a brief commentary on the Delphi technique that may be useful for researchers and practitioners in qualitative forecasting.     

Delphi: An investigation from a Bayesian viewpoint

This paper presents an analysis of Delphi from a Bayesian point of view and brings to attention several hitherto neglected, but highly relevant findings of research in areas other than Delphi proper. The aim has been to seek avenues of improvement in the Delphi technique. This has led to development of a methodology, which is based on a concept of second order probabilities as a measure of one's fuzzy thinking. The justification of this new methodology lies in explicit recognition and implementation of an optimum, determined by the trade-off between advantages and complexities of hierarchial inference. It is essential that conventional Delphi applications, at the very least, be accompanied by a Turoff-type cross impact analysis. It is indicated that the tremendous potential of Bayesianized Delphi in appropriate situations has remained utapped.     

Foresight with Delphi Surveys in Japan

Foresight activities are conducted in many countries, but the Japanese were the most active in this field for many years. Most of their activities are based on the Delphi methodology. They published their first Delphi study in 1971 and repeated studies every five years. There is more knowledge available on more than 30 years of foresight with the Delphi approach in Japan. This provides an excellent opportunity to examine the established foresight system with the use of some examples. The forecast of the fax machine as an example of success, and the earthquake warning as a failure are illustrated in more detail, and the question is raised why—in spite of this mixed picture—foresight with the Delphi method is regarded as useful in Japan.     

Foresight with Delphi Surveys in Japan

Foresight activities are conducted in many countries, but the Japanese were the most active in this field for many years. Most of their activities are based on the Delphi methodology. They published their first Delphi study in 1971 and repeated studies every five years. There is more knowledge available on more than 30 years of foresight with the Delphi approach in Japan. This provides an excellent opportunity to examine the established foresight system with the use of some examples. The forecast of the fax machine as an example of success, and the earthquake warning as a failure are illustrated in more detail, and the question is raised why—in spite of this mixed picture—foresight with the Delphi method is regarded as useful in Japan.     

Opinion convergence in location: A spatial version of the Delphi method

This paper proposes a modified version of the Delphi method, starting from the assumption that in many situations the convergence of opinions can be considered in a spatial context. The Spatial Delphi we suggest is based, like the classical Delphi, on the judgments of experts, and it is useful in the consultations for decision and/or forecast purposes, provided that they concern matters of spatial location. The basis for the questionnaire is a map, on which each expert provides, as answer(s), one or more opinion-points, i.e., locations that, according to their opinion, are best for a specific purpose. We propose a method for narrowing the area of convergence along a multi-step procedure so that the final result of the consultation is a small portion of the initial territory with respect to the initial area considered. Besides a simulation study, we report also a prototype application with a panel of twelve experts.     

The Delphi method as early warning: Linking global societal trends to future radicalization and terrorism in The Netherlands

A Delphi study was performed to answer the question: Which global societal trends relate to future radicalization and subsequent terrorism in the Netherlands? An inventory of two hundred global societal trends and a literature study of radicalization together served as briefing of the participants in the Delphi study. The first two rounds were conducted in anonymous writing. The third round was a face to face focus group meeting. In the first and second round participants scored the trends, adjusted their scores relative to others and provided arguments, and in the focus group meeting, themes were discussed in which the participants remained to have strong opposite views. This approach emphasizes divergence in opinion (‘dissensus’), as opposed to convergence (‘consensus’). Consensus seeking to us seems to be the dominant application of Delphi, but we emphasize the value of seeking dissensus. We hypothesize that in those cases where the opposite views that were discussed kept their ground, this Delphi study may have found early warnings of future radicalization.The essence of this paper is that it is possible, with limited effort, to get a handle on the complex and poorly defined subject of global societal trends influencing future radicalization. Application of the Dissensus Delphi method provided a selection of early warnings that may be looked into with future research.     

An analysis of prior delphi applications and some observations on its future applicability

Since the inception of the Delphi method, it has experienced increased application. Despite its spreading utilization, very little is known about its past applications and its actual value. This article reports on the results of an international research project concerning the Delphi method. The research project focused primarily upon an analysis of the purposes for which the Delphi method has been used and its degree of successfulness, the extent of its use in various substantive fields, the future of the Delphi method, its relationship with other methodologies and the types of organizations financing Delphi studies.     

A fuzzy multiple criteria comparison of technology forecasting methods for predicting the new materials development

New materials have been recognized as key drivers for corporate profitability and growth in today's fast changing environments. To predict correctly the development of the new materials becomes a critical issue. However, little has been done in discussing the selection of technology forecasting methods for the new materials development. Accordingly, this study adopted the fuzzy AHP method to obtain professional's opinions on this issue. The efforts result in seven evaluation criteria with one, the “data validity” having the highest weight, followed by “method adaptability” and “technology predictability”. Delphi method and case study method are the two most applicable technology forecasting methods for predicting the new materials development.     

Design and evaluation of ranking-type Delphi studies using best-worst-scaling

The ranking-type Delphi method has received widespread use in a large number of research areas. In order to rank a set of items, ranking-type Delphi studies use different ranking approaches which may be biased. This is intensified by the fact that the number of issues a participant can reasonably rank is limited. This paper describes an approach for the design and evaluation of ranking-type Delphi studies using best-worst-scaling (BWS). The statistical BWS design is based on a balanced incomplete block design to construct the comparison sets. The statistical evaluation encompasses the measurement of stability and consensus with the use of the coefficient of variation. Overall, our approach (1) enables to design a ranking with up to 22 items in a manner that participants can reasonably rank, (2) avoids and overcomes shortcomings of common ranking approaches, and (3) offers a statistical procedure for an unambiguous evaluation.