Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.     

A two-stage technology foresight model with system dynamics simulation and its application in the Chinese ICT industry

Technology foresight is a process that identifies the critical technologies a country or industry needs to develop in order to shape a desired future. In this paper, a two-stage technology foresight approach is proposed. During the first stage, critical technologies are identified and evaluated by nationwide experts through Delphi surveys. In the second stage, a system dynamics simulation model is used to estimate how critical parameter values are likely to impact the attainment of foresight goals. A detailed illustration of this two-stage process is provided by a technology foresight case study in the Chinese information and communication technologies (ICT) industry. Because it was identified by the first-stage Delphi method as a critical technology, the diffusion of 4th generation wireless telecommunication technology (4G) was simulated, and experimentation was conducted. Suggestions are provided regarding how the Chinese information industry would need to expand the IP network infrastructure in order to support technology foresight goals under different scenarios.     

The UK technology foresight programme: An assessment of expert estimates

Since the early 1990s, ‘Technology Foresight’ exercises with special emphasis on the use of Delphi surveys have played an important role in science and technology (S + T) policy across Europe in an effort to focus resource allocation. Yet, none of the estimates made in the European Delphi surveys have been formally assessed in retrospect, while this process has been incorporated into the Japanese surveys since 1996. Taking the UK Technology Foresight Programme, this research sets out to assess the estimates of three of the fifteen panel Delphi surveys. Whilst on average 2/3 of Delphi statements were predicted to be realised by 2004, it will be shown that only a fraction of these statements had been realised by 2006. Based on the evidence collected from the published panel reports, the ‘Hindsight on Foresight’ survey conducted by OST in 1995 and interviews with panel members, it will be argued that the overwhelming majority of estimates were overly optimistic. While optimism and strategic gaming of experts is the most convincing explanation for these results, process factors were also explored, including the quality of expert panels used, the Delphi statements and the respondents of the Delphi questionnaire. It is argued that at least the issue of short-range optimism and strategic gaming of experts should be addressed in future Delphi exercises, as decision makers relying on expert advice cannot deal with this issue alone.     

Disaggregative policy Delphi: Using cluster analysis as a tool for systematic scenario formation

A critical phase of scenario making is the choosing of scenarios. In the worst case, a futures researcher creates scenarios according to his/her subjective views and cannot see the real quality of the study material. Oversimplification is a typical example of this kind of bias. In this study, an attempt towards a more data sensitive method was made using Finnish transport policy as an example. A disaggregative Delphi method as opposed to traditional consensual Delphi was applied. The article summarises eight Delphi pitfalls and gives an example how to avoid them. A two-rounded disaggregative Delphi was conducted, the panelists being representatives of interest groups in the traffic sector. Panelists were shown the past development of three correlating key variables in Finland in 1970-1996: GDP, road traffic volume and the carbon dioxide emissions from road traffic. The panelists were invited to give estimates of their organisation to the probable and the preferable futures of the key variables for 1997-2025. They were also asked to give qualitative and quantitative arguments of why and the policy instruments of how their image of the future would occur. The first round data were collected by a fairly open questionnaire and the second round data by a fairly structured interview. The responses of the quantitative three key variables were grouped in a disaggregative way by cluster analysis. The clusters were complemented with respective qualitative arguments in order to form wider scenarios. This offers a relevance to decision-making not afforded by a nonsystematic approach. Of course, there are some problems of cluster analysis used in this way: The interviews revealed that quantitatively similar future images produced by the panelists occasionally had different kind of qualitative background theory. Also, cluster analysis cannot ultimately decide the number of scenarios, being a choice of the researcher. Cluster analysis makes the choice well argued, however.     

Industry- and company-level factors influencing the development of the forest energy business — insights from a Delphi Study

The global business environment of today requires industries to be increasingly agile in order to create added value. There is a particularly urgent need to innovate and redefine business models in the mature pulp and paper industry, which is continually announcing mill closures and reporting persistent profitability problems. This paper focuses on the emerging forest energy business, which appears to offer many novel opportunities for both the forest and the energy industry. We conducted a qualitative dissensus-based online Delphi study and carried out themed expert interviews in order to identify the main industry- and company-level factors that are most likely to influence the bioenergy sector, its value-creation potential and forest and energy companies' future roles in it. The Delphi technique proved to be a valuable research tool with which we were able to obtain comprehensive information on a subject that lacks historical and financial data, and yet requires input from many quarters. The results suggest that the complementary resources held by forest and energy companies make collaboration in the bioenergy business favorable. Moreover, the procurement and logistics of forest raw material appear to be key success factors in terms of yielding the most synergetic gains. Uncertainty about policy interventions nevertheless causes concern given their relatively rapid effect on the prospects of the bioenergy business.     

Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy

This paper further enhances the analytical power of Delphi methodology by identifying the advantages, disadvantages and challenges presented by increasing diversity among panel groups. Using Delphi survey data on the future of nuclear energy in France, we analyze the origins of the variety of judgments within and between two panels: one of experts and one of laypeople. We investigate the determinants of the stability of those opinions both in one given round and over several rounds of opinion-formation. We reach an apparently paradoxical conclusion: that non-expert judgment is less stable, but not necessarily less accurate, than that of experts, judgments on the part of experts sometimes being clouded by self-interest. Apart from highlighting some factors underlying the controversy over nuclear power, our paper calls for greater participatory democracy in Delphi panels, but also demonstrates the limits of such an extension.     

The unholy marriage? Integrating qualitative and quantitative information in Delphi processes

This study analyses the strengths, weaknesses and pitfalls encountered when combining qualitative and quantitative information in a Delphi process and when reporting the results as scenarios or images of the future. The paper draws material from seven Disaggregative Policy Delphi processes conducted in Finland in 1999-2008, in which the authors were researchers or advisors. The cases are analysed in terms of the level of integration and the ways to overcome the difficulties. A learning ‘community of practice’ was created by these afterthoughts and by organising an international conference workshop on the issue. Qualitative and quantitative material was holistically integrated in one case. In the other cases, solutions led to domination of one material type over another but even then the other material did give relevant points to scenario formation. Finally, we give recommendations for tackling the pitfalls: 1) balancing between qualitative and quantitative, 2) balancing between formal structure and questions raised in the process, 3) framing questions to discover alternative future states, 4) paying attention to panellists' style, 5) dealing with lack of data for comprehensive cluster analysis, 6) considering scenario consistency, 7) understanding manager's responsibility and, 8) understanding the epistemological aspects of Delphi data. We conclude that integrating qualitative and quantitative material by using mixed methods to form coherent scenarios is at the same time desirable, possible and difficult - making the ‘unholy marriage’ a worthy adventure.     

Results of a Delphi survey in drawing up the input-output tables for Catalonia

This work presents the results of an ambitious and innovative use of the Delphi technique in supplying a quantitative economic model, specifically in the preparation of the Input-Output Tables for Catalonia (TIOC 2001). Using this technique for obtaining subjective information from experts comes forward as a valid and reliable option for improving the quality of the data with which the Input-Output tables are constructed, also providing further economic and social advantages. The application presented thus opens up an interesting field of development for this technique and also gives several contributions for appraising the execution of a Delphi exercise of this type.     

Desirability bias in foresight: Consequences for decision quality based on Delphi results

In foresight activities uncertainty is high and decision makers frequently have to rely on human judgment. Human judgment, however, is subject to numerous cognitive biases. In this paper, we study the effects of the desirability bias in foresight. We analyze data from six Delphi studies and observe that participants systematically estimate the probability of occurrence for desirable (undesirable) future projections higher (lower) than the probability for projections with neutral desirability. We also demonstrate that in the course of a multi-round Delphi process, this bias decreases but is not necessarily eliminated. Arguably, the quality of decisions based on Delphi results may be adversely affected if experts share a pronounced and common desirability for a future projection. Researchers and decision makers have to be aware of the existence and potential consequences of such a desirability bias in Delphi studies when interpreting their results and taking decisions. We propose a post-hoc procedure to identify and quantify the extent to which the desirability bias affects Delphi results. The results of this post-hoc procedure complement traditional Delphi results; they provide researchers and decision makers with information on when and to which extent results of Delphi-based foresight may be biased.     

Personal attitudes in the assessment of the future of science and technology: A factor analysis approach

In 1996, the second German Delphi study (“Delphi '98”) was started. The Delphi '98 is a two-round Delphi expert survey being conducted by the Fraunhofer Institute for Systems and Innovation Research (ISI) on behalf of the German Federal Ministry of Education, Research, Science, and Technology (BMBF). The study was published in February 1998, and is now getting into its implementation phase. Its inherent focus is on the development of science and technology in 12 thematic fields in the next 30 years. To arrive at a better understanding of the influence of personal attitudes towards general developments in natural environment and society, the respondents were asked in the first round of the Delphi survey for their personal opinion towards several megatrends concerning the natural environment, economic, sociological, and political developments. Over 2,300 answers led to a very solid database, which gives insights into the general attitudes of the German R&D experts. On some topics, there is a high consensus, whereas in others, opposite opinions coexist. These results may serve as the database for a factor analysis leading to the identification of five different expert types. This paper examines the crucial question of whether different patterns in assessing the future development in science and technology by expert types can be observed. In general, it turned out that differences in personal attitudes towards megatrends do not influence the estimation of developments in science and technology. However, differences exist in specific topics and the distribution of the five experts types among the respondents differs significantly in the 12 fields.     

Political risk analysis for Canada and Mexico

A new methodology for political risk analysis was developed and applied to Canada and Mexico. The ideas of an interdisciplinary team of corporate and country experts were obtained in mid-1982 via a two-round Delphi inquiry on socioeconomic change and a survey on business contingencies in Canada and Mexico. The information provided by the panelists was used to describe possible future conditions in Canada and Mexico in the context of alternative U.S. and international conditions.     

Assessing market penetration combining scenario analysis, Delphi, and the technological substitution model: The case of the OLED TV market

The technological substitution model, proposed in 1971, was designed to analyze the penetration process of new-generation technologies replacing old ones. It was expanded in 1979 to consider more than two generations and the substitution process among various entities in the market. However, this expanded model is often limited by a relative lack of data on the latest-generation technology.On the other hand, scenario analysis, as another breed to forecasting tool, provides rich and complex portraits of possible future development of technologies. However, it fails to provide quantifiable forecasts. Therefore, previous researchers combined these two methods to analyze the development of new technologies.Nevertheless, to make forecasting more holistic, the current opinions of seasoned experts should also be taken into account. However, because of the often-diverged expert opinions, a method for consensus building, such as the Delphi method, is also necessary. Therefore, we combined the scenario analysis with the Delphi method and the technological substitution model to analyze the development of a new technology, namely, the latest-generation television (TV), or the organic light-emitting diode (OLED) TV. We elaborate three possible scenarios, and forecast the market share of OLED TV and four other TV technologies in the global market over the next 10 years: cathode ray tube, rear projection, plasma display panel, and liquid crystal display.     

The use of Delphi methodology in agrifood policy development: Some lessons learned

Developing policy in the agrifood area is an inexact process, usually relying upon effective integration of opinions from multiple experts from different disciplines, organisational types, and regions/countries. Delphi would appear to have the potential to overcome some of the typical limitations related to soliciting expert opinion and identifying consensus on future activities or options, particularly where relevant experts are dispersed geographically, and international consensus is required, as is the case in this domain. Three case studies, focused on the application of Delphi to emerging policy needs in international or European agrifood policy, are presented here to exemplify the utility of the technique. A number of practical recommendations are drawn from these case studies that may be applicable to other major policy making arenas. Among these recommendations are; that an exploratory workshop to refine round one Delphi questions is essential; that the implementation of “cascade” methodology (utilizing the personal contacts of researchers or members of existing policy networks) appears to increase response rates in subsequent Delphi rounds; and that the policy issue under discussion should be particularly relevant to stakeholders in order to increase participation rates. Further research might usefully focus on developing ways to incorporate measurements of uncertainty associated with stakeholder judgement into quantitative responses, and on establishing how best to utilise such information in feedback in subsequent Delphi rounds. Ensuring how best to inform policy uptake of the outputs of Delphi merits further research in particular.     

A study on the bioenergy crop production function of land use in China

Based on the analysis of the bioenergy crop production function of land use,combined with the status quo of Chinese land use,the cultivation of energy plants and the bioenergy crop production function of land use had been analyzes and discusses in this paper.Results show that there were a lot of unused lands and marginal lands which can be planted bioenergy crops to perform the bioenergy crop production function of land use with great potentials; and currently there were no food production problems.Therefore,it was very important for China to emphasize bioenergy crops planting in order to fully use land resources in our country,moderate the energy crisis and increase peasants＇ income.     

Delphi as a tool for identifying future appropriate technologies in India

The concept of appropriate technologies with particular reference to developing countries like India is discussed. It is suggested that appropriate technologies should: (1) include the entire range of technologies from very primitive to very modern, (2) be available when required, and (3) have the maximum impact on the largest fraction of the poorest sections of the population, especially through increasing employment. The problem of identifying appropriate technologies that would meet these criteria has also been discussed. In this article we have attempted to use the Delphi technique of technology forecasting for identifying future appropriate technologies in the areas of food, health, and energy using respondents in India. Subsequently, we have analyzed our results in terms of the criteria previously assigned to appropriate technologies and have thereby evaluated our experiment.It is concluded that the Delphi is an adequate first step in identification of such technologies; however, it must be followed by more rigorous techniques (possibly quantitative) for assigning further priorities within the technologies identified by the Delphi. This will help in arriving at suitable policy decisions and allocation of funds to a few of the most important areas.Finally, two alternate scenarios have been presented (using the Delphi results as well as our own perceptions and observations of the Indian society), depending on whether appropriate technologies and policies are adopted.     

A case study of long-term Delphi accuracy

We evaluated the long-term accuracy of predictions from a Delphic poll of the future of the mental health profession [1]. Six hundred participants provided predictions about the likely occurrence and probable time courses for 18 scenarios that could possibly have occurred over the 30-year interval between 1981 to present. Each of the panelists was polled twice with distributional feedback from the first poll provided on the second poll questionnaires. The data from the second polling was used to make predictions regarding the future of various issues that may have affected the mental health profession over the forecast interval. It has now been 30 years since the original poll was conducted; the purpose of this study was to evaluate the accuracy of the original group predictions. Results indicated that the Delphi panelists correctly predicted the occurrence of 14/18 scenarios. For those scenarios that did occur, the time course predictions were accurate within approximately 1-5 years.     

Systemic analysis of UK foresight results: Joint application of integrated management model and roadmapping

This paper proposes a new systems approach to foresight studies. The paper will first indicate the complex and conflicting nature of long-term decision-making process. Then, the need for systems approach will be highlighted by the analysis of 1995 UK Delphi survey results and the scenarios of 2000 UK foresight scenarios. The paper proposes two methodologies, namely Integrated Management Model (IMM) and Roadmapping, in order to overcome challenges introduced by the multidimensional characteristics and complex nature of foresight studies. Based on systemic approach, IMM offers a useful way of developing long-term normative policies and strategies and their transformations into actions by considering necessary changes in organizational structures and behaviors. In addition, roadmapping is used to capture, manipulate and manage information to decrease complexity in the foresight by constructing roadmaps. In the paper, IMM and roadmapping are employed first to analyze UK foresight results and then to develop a new methodology to formulate Delphi events and scenarios for the successful implementation of foresight. This paper also promotes the integrated use of foresight techniques such as scenarios and Delphi rather than one for another.     

Future issues of science and technology

The Millennium Project involved 237 scientists, futurists, and policymakers around the world in a two-round Delphi on the future issues of science and technology (S&T) over the next 25 years. This is the first of a 3-year study. The study began by asking science attaches to Washington, DC, what questions were worth asking to their leading scientists and issues that were important to explore on an international basis. A series of questions were identified and rated. Actions to address the questions were also suggested and rated. The results are presented in this paper with some regional differences discussed. The next year of the study interviewed S&T policymakers for the management and policy implications of these findings, and the third year developed scenarios based on these implications.     

The decision Delphi

If Delphi is used as a prognostic tool in the social sciences, it runs the danger of producing a self-fulfilling prophecy. In a Delphi application conducted at the Austrian Academy of Sciences about the present situation and future developments of the scientific-technical information and documentation system in Austria, an attempt was made to apply this principle intentionally.A group of actual decision makers in this field were included in a Delphi panel and were therefore confronted with problems and possible future developments in a field which is considerably influenced by their own actions. The Delphi method was therefore used to structure the decision-making process and to help to “create” the future in reality rather than just predicting it. In this way, an attempt was made to transfer an important field of socioeconomic development from a pattern of accidental development through uncoordinated decisions toward a pattern of broad discussion among all involved social groups and thus toward goal-oriented and well-considered management.It seems that this study represents a further variant of the well-known Delphi technique, a variant which has fundamental differences from the classical and the policy Delphi. For a Delphi application of this type the expression “Decision Delphi” is suggested.     

Shang inquiry as an alternative to Delphi: Some experimental findings

A new group judgment method known as Shang is presented as an alternative to Delphi. Shang is structured to avoid Delphi problems associated with convergence pressures and response commitment while incorporating Delphi advantages. The new method is evaluated through an experiment comparing it with a control method and two Delphi techniques, one using point estimates and one using distributional estimates. Findings indicate that Shang is generally superior to Delphi in producing accurate numerical estimates from the judgments of a group of experts.