Stability and agreement criteria for the termination of Delphi studies

This paper addresses the question of the methodology of testing for stability between successive Delphi rounds and the necessity of doing so prior to the analysis of the level or type of consensus. Following a discussion of a hierarchical set of a criteria for determining the termination of such studies, the paper reviews some of the more common treatments of the subject in the literature. It is finally proposed that a χ² test be used to test the stability of responses between rounds. The method is demonstrated by a numerical example.     

The Delphi method as early warning: Linking global societal trends to future radicalization and terrorism in The Netherlands

A Delphi study was performed to answer the question: Which global societal trends relate to future radicalization and subsequent terrorism in the Netherlands? An inventory of two hundred global societal trends and a literature study of radicalization together served as briefing of the participants in the Delphi study. The first two rounds were conducted in anonymous writing. The third round was a face to face focus group meeting. In the first and second round participants scored the trends, adjusted their scores relative to others and provided arguments, and in the focus group meeting, themes were discussed in which the participants remained to have strong opposite views. This approach emphasizes divergence in opinion (‘dissensus’), as opposed to convergence (‘consensus’). Consensus seeking to us seems to be the dominant application of Delphi, but we emphasize the value of seeking dissensus. We hypothesize that in those cases where the opposite views that were discussed kept their ground, this Delphi study may have found early warnings of future radicalization.The essence of this paper is that it is possible, with limited effort, to get a handle on the complex and poorly defined subject of global societal trends influencing future radicalization. Application of the Dissensus Delphi method provided a selection of early warnings that may be looked into with future research.     

Participants' response to the Delphi method: An attitudinal perspective

The processes involved in an individual's response to the Delphi method are conditioned by a number of psychological affects. This paper examines aspects of two of these processes: the basis for changing opinions, and the factors related to continued participation in subsequent Delphi rounds. The attitudinal perspective is shown to be promising in explaining an individual panelist's response in the Delphi method. Cognitive dissonance apparently has a significant role to play in bolstering the assimilation of nonconforming respondents. This is, however, complicated by the respondents' perception of the credibility of the feedback received. Careful choice of panels may reduce dropouts, presumably by selecting those with a high degree of ego involvement. Nonetheless, elements of cognitive dissonance apparently lead certain panelists to abandon the Delphi process. It is evident that numerous parallels exist between psychological explanations of attitude change and the behavior of Delphi panelists.     

Reclaim-proof allocation of indivisible objects

We study desirability axioms imposed on allocations in indivisible object allocation problems. The existing axioms in the literature are various conditions of robustness to blocking coalitions with respect to agentsʼ ex ante (individual rationality and group rationality) and ex post (Pareto efficiency) endowments. We introduce a stringent axiom that encompasses and strengthens the existing ones. An allocation is reclaim-proof if it is robust to blocking coalitions with respect to any conceivable interim endowments of agents. This is an appealing property in dynamic settings, where the assignments prescribed by an allocation to be implemented need to be made in multiple rounds rather than all in one shot. We show that an allocation is reclaim-proof if and only if it is induced by a YRMH–IGYT mechanism (introduced by Abdulkadiroğlu and Sönmez, 1999) and if and only if it is a competitive allocation.     

Delphi questionnaires versus individual and group interviews: A comparison case

The article reports on a methodological investigation for different types of the communication device represented by the Delphi method. Traditionally mailed or computer-mediated questionnaires are used. The Delphi method was invented to overcome undesirable effects of group interaction while retaining the positive effects of interactive group judgments. This article supplies evidence for the opposite: the superiority of face- to-face techniques (individual and group interviews) as compared to mailed questionnaires in a three-round Policy Delphi using half-structured questionnaires. Moreover, the case study proves that they might produce different results. In a field- or quasi-experimental design the three techniques were compared for their effects. The performance of 100 panel members on a large number of quality standards in the Delphi method is described. A list of pros and cons of the three techniques in a (Policy) Delphi serves as a summary.     

Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.     

Valuing ecosystem goods and services: a new approach using a surrogate market and the combination of a multiple criteria analysis and a Delphi panel to assign weights to the attributes

A new approach to valuing ecosystem goods and services (EGS) is described which incorporates components of the economic theory of value, the theory of valuation (USappraisal), a multi-model multiple criteria analysis (MCA) of ecosystem attributes, and a Delphi panel of experts to assign weights to the attributes. The total value of ecosystem goods and services in the various tenure categories in the Wet Tropics World Heritage Area (WTWHA) in Australia was found to be in the range AUD$188 to $211 million year⁻¹, or AUD$210 to 236 ha⁻¹ year⁻¹ across tenures, as at 30 June 2002. Application of the weightings assigned by the Delphi panelists and assessment of the ecological integrity of the various tenure categories resulted in values being derived for individual ecosystem services in the World Heritage Area. Biodiversity and refugia were the two attributes ranked most highly at AUD$18.6 to $20.9 million year⁻¹ and AUD$16.6 to $18.2 million year⁻¹, respectively.     

Is diversity in Delphi panelist groups useful? Evidence from a French forecasting exercise on the future of nuclear energy

This paper further enhances the analytical power of Delphi methodology by identifying the advantages, disadvantages and challenges presented by increasing diversity among panel groups. Using Delphi survey data on the future of nuclear energy in France, we analyze the origins of the variety of judgments within and between two panels: one of experts and one of laypeople. We investigate the determinants of the stability of those opinions both in one given round and over several rounds of opinion-formation. We reach an apparently paradoxical conclusion: that non-expert judgment is less stable, but not necessarily less accurate, than that of experts, judgments on the part of experts sometimes being clouded by self-interest. Apart from highlighting some factors underlying the controversy over nuclear power, our paper calls for greater participatory democracy in Delphi panels, but also demonstrates the limits of such an extension.     

Enhancing a Delphi study on family-focused prevention

This article discusses how additional inquiries can enhance Delphi findings. We argue that inquiries aimed at supporting and refining Delphi findings is both theoretically and practically meaningful. We illustrate our argument on the basis of a framework for family-focused prevention that was developed through a Delphi study. The results of individual and group interviews conducted as a follow-up to the implementation of the framework provided us with effective ways to support and refine it. We draw the conclusion that adequate follow-up inquiries can enhance Delphi findings, from theoretical and application perspectives.     

Judgment change during Delphi-like procedures: The role of majority influence, expertise, and confidence

This study investigates individual opinion change and judgmental accuracy in Delphi-like groups. Results reveal that the accuracy of judgmental probability forecasts increases over Delphi rounds (in terms of proportion correct and appropriateness of confidence) when statistical summaries or written rationales are provided from other members of an individual's nominal group, but does not increase in a control iteration condition (without feedback). Additionally, subjects who gave more appropriate probability forecasts on the first round exhibited least opinion change, although measures of confidence were unrelated to opinion change. Results also show that majority opinion exerts strong opinion pull on minority opinion even when the majority favours an incorrect answer (irrespective of the nature of feedback provided). The implications of these results for the utility and conduct of the Delphi technique are discussed, in particular, with respect to selecting panellists and choosing an appropriate feedback format.     

Global health resource utilization associated with pacemaker complications

Aim: To estimate health resource utilization (HRU) associated with the management of pacemaker complications in various healthcare systems.

Methods: Electrophysiologists (EPs) from four geographical regions (Western Europe, Australia, Japan, and North America) were invited to participate. Survey questions focused on HRU in the management of three chronic pacemaker complications (i.e. pacemaker infections requiring extraction, lead fractures/insulation breaches requiring replacement, and upper extremity deep venous thrombosis [DVT]). Panelists completed a maximum of two web-based surveys (iterative rounds). Mean, median values, and interquartile ranges were calculated and used to establish consensus.

Results: Overall, 32 and 29 panelists participated in the first and second rounds of the Delphi panel, respectively. Consensus was reached on treatment and HRU associated with a typical pacemaker implantation and complications. HRU was similar across regions, except for Japan, where panelists reported the longest duration of hospital stay in all scenarios. Infections were the most resource-intensive complications and were characterized by intravenous antibiotics days of 9.6?13.5 days and 21.3?29.2 days for pocket and lead infections respectively; laboratory and diagnostic tests, and system extraction and replacement procedures. DVT, on the other hand, was the least resource intensive complication.

Limitations: The results of the panel represent the views of the respondents who participated and may not be generalizable outside of this panel. The surveys were limited in scope and, therefore, did not include questions on management of acute complications (e.g. hematoma, pneumothorax).

Conclusions: The Delphi technique provided a reliable and efficient approach to estimating resource utilization associated with chronic pacemaker complications. Estimates from the Delphi panel can be used to generate costs of pacemaker complications in various regions.     

The delphi technique: an exposition and application

The article provides a n overview of the Delphi technique and its related adaptations, as well as practical hints on procedures gained from its use to forecast changes in the market analysis industry. The Lielphi technique has been used many times as a method of forecasting the future ofestab1ish:ed industries, but it has never been used to consider the future of a nascent industry. A case is presented which suggests that companies i n nascent industries have a greater need for long-range planning than those at other stages of a n industry's development, and a greater need for an industry, rather than a company, perspective. Problems such as time pressure on executives and the need for confidentiality are more acute when using judgmental forecasting techniques in new industries. The review of the Delphi process highlights some of the major considerations, s,uch as panel selection, questionnaire construction and administration, response analysis and number of rounds, as well as how Delphi can be modified. Since new industries rarely have established trade organizations to curry out such Delphi studies, the role could be played by institutes of management education.     

The use of Delphi methodology in agrifood policy development: Some lessons learned

Developing policy in the agrifood area is an inexact process, usually relying upon effective integration of opinions from multiple experts from different disciplines, organisational types, and regions/countries. Delphi would appear to have the potential to overcome some of the typical limitations related to soliciting expert opinion and identifying consensus on future activities or options, particularly where relevant experts are dispersed geographically, and international consensus is required, as is the case in this domain. Three case studies, focused on the application of Delphi to emerging policy needs in international or European agrifood policy, are presented here to exemplify the utility of the technique. A number of practical recommendations are drawn from these case studies that may be applicable to other major policy making arenas. Among these recommendations are; that an exploratory workshop to refine round one Delphi questions is essential; that the implementation of “cascade” methodology (utilizing the personal contacts of researchers or members of existing policy networks) appears to increase response rates in subsequent Delphi rounds; and that the policy issue under discussion should be particularly relevant to stakeholders in order to increase participation rates. Further research might usefully focus on developing ways to incorporate measurements of uncertainty associated with stakeholder judgement into quantitative responses, and on establishing how best to utilise such information in feedback in subsequent Delphi rounds. Ensuring how best to inform policy uptake of the outputs of Delphi merits further research in particular.     

Does the Delphi process lead to increased accuracy in group-based judgmental forecasts or does it simply induce consensus amongst judgmental forecasters?

We investigate the relative impact of internal Delphi process factors - including panelists' degree of confidence, expertise, majority/minority positioning - and an external factor, richness of feedback - on opinion change and subsequent accuracy of judgmental forecasts. We found that panelists who had low confidence in their judgmental forecast and/or who were in a minority were more likely to change their opinion than those who were more confident and/or in a majority. The addition of rationales, or reasons, to the numeric feedback had little impact upon panelists' final forecasts, despite the quality of panelists' rationales being significantly positively correlated with accurate forecasts and thus of potential use to aid forecast improvement over Delphi rounds. Rather, the effect of rationales was similar to that of confidence: to pull panelists towards the majority opinion regardless of its correctness. We conclude that majority opinion is the strongest influence on panelists' opinion change in both the ‘standard’ Delphi, and Delphi-with-reasons. We make some suggestions for improved variants of the Delphi-with-reasons technique that should help reduce majority influence and thereby permit reasoned arguments to exert their proper pull on opinion change, resulting in forecast accuracy improvements over Delphi rounds.     

Dissertation abstract: Three essays on the determinants of behavior in the commons-experimental evidence from fishing communities in Colombia

This dissertation presents the results of a series of common pool experiments conducted in three regions of rural Colombia with individuals who face a social dilemma in their everyday lives that is similar to what was presented in the experiment. The research objectives are to develop an empirical characterization of how individual behavior deviates from purely self-interested Nash behavior and to further our understanding of the effects of alternative institutions to promote more conservative choices in common pool experiments.Groups of five subjects participated in a 20-period common pool resource game framed as a harvest decision from a fishery. Every group first played 10 rounds of a baseline limited access common pool resource game and then 10 additional rounds under one of five institutions: face-to-face communication, one of two external regulations, and communication combined with one of the two regulations. The two external regulations consisted of an individual harvest quota that was set at the efficient outcome, but differ with respect to the level of enforcement. A total of 420 individuals participated in the experiments, with individual earnings averaging slightly more than a day’s wages. The results are presented in three essays.The first essay, What Motivates Common Pool Resource Users?, develops and tests several models of pure Nash strategies of individuals who extract from a common pool resource when they are motivated by combinations of self-interest, altruism, reciprocity, inequity aversion or conformity. The results suggest that a model which balances self-interest with a strong preference for conformity best describes average strategies. The data are inconsistent with a model of pure self-interest, as well as models that combine self-interest with individual preferences for altruism, reciprocity and inequity aversion.The second essay, Communication and Regulation to Conserve Common Pool Resources, tests for interaction effects between formal regulations imposed on a community to conserve a local natural resource and non-binding verbal agreements to do the same. The results indicate that formal regulations and informal communication are mutually reinforcing in some instances, but this result is not robust across regions or regulations. Therefore, the hypothesis of a complementary relationship of formal and informal control of local natural resources cannot be supported in general; instead the effects are likely to be community-specific. There is some evidence to suggest that these effects are correlated with the relative importance of formal regulations versus informal community efforts in the community.The third essay, Within and Between Group Variation in Individual Strategies in Common Pools, analyzes the relative effects of groups and individuals within groups in explaining variation in individual harvest decisions for particular institutions, and uses a hierarchical linear model to examine how these sources of variation may vary across institutions. Communication serves to effectively coordinate individual strategies within groups, but these coordinated strategies vary considerably among groups. In contrast, externally-imposed regulatory schemes (as well as unregulated limited access) produce significant variation in the individual strategies within groups, but these strategies are roughly replicated across groups so that there is little between-group variation.     

The delphi technique: an exposition and application

The article provides a n overview of the Delphi technique and its related adaptations, as well as practical hints on procedures gained from its use to forecast changes in the market analysis industry. The Lielphi technique has been used many times as a method of forecasting the future ofestab1ish:ed industries, but it has never been used to consider the future of a nascent industry. A case is presented which suggests that companies i n nascent industries have a greater need for long-range planning than those at other stages of a n industry's development, and a greater need for an industry, rather than a company, perspective. Problems such as time pressure on executives and the need for confidentiality are more acute when using judgmental forecasting techniques in new industries. The review of the Delphi process highlights some of the major considerations, s,uch as panel selection, questionnaire construction and administration, response analysis and number of rounds, as well as how Delphi can be modified. Since new industries rarely have established trade organizations to curry out such Delphi studies, the role could be played by institutes of management education.     

Technological forecasting and planning exercises in engineering curricula

To sensitize graduate students in engineering to the multidisciplinary nature of decision making in contemporary situations, the elements of technological forecasting have been used as the basis for a graduate course. The principal objective is to use the technological forecasting approach in a structured problem resolution sequence, one of which is discussed here. Faced with a hypothetical problem of diversification in a medium-sized, high-technology organization, the student group employed objective trees, cross-impact matrices and structured interviews (Delphi rounds) in order to define quantitatively the nature and scope of opportunities open within the framework of economic, societal, political, and technological environments pertaining to their situation. Delphi-style debates, using a Consensor, were used to resolve group or individual conflicts. The exercise is intended to expedite adaptation of graduate engineers to the environment (industrial, social, etc.) within which they will be expected to perform as useful decision makers.     

Disaggregative policy Delphi: Using cluster analysis as a tool for systematic scenario formation

A critical phase of scenario making is the choosing of scenarios. In the worst case, a futures researcher creates scenarios according to his/her subjective views and cannot see the real quality of the study material. Oversimplification is a typical example of this kind of bias. In this study, an attempt towards a more data sensitive method was made using Finnish transport policy as an example. A disaggregative Delphi method as opposed to traditional consensual Delphi was applied. The article summarises eight Delphi pitfalls and gives an example how to avoid them. A two-rounded disaggregative Delphi was conducted, the panelists being representatives of interest groups in the traffic sector. Panelists were shown the past development of three correlating key variables in Finland in 1970-1996: GDP, road traffic volume and the carbon dioxide emissions from road traffic. The panelists were invited to give estimates of their organisation to the probable and the preferable futures of the key variables for 1997-2025. They were also asked to give qualitative and quantitative arguments of why and the policy instruments of how their image of the future would occur. The first round data were collected by a fairly open questionnaire and the second round data by a fairly structured interview. The responses of the quantitative three key variables were grouped in a disaggregative way by cluster analysis. The clusters were complemented with respective qualitative arguments in order to form wider scenarios. This offers a relevance to decision-making not afforded by a nonsystematic approach. Of course, there are some problems of cluster analysis used in this way: The interviews revealed that quantitatively similar future images produced by the panelists occasionally had different kind of qualitative background theory. Also, cluster analysis cannot ultimately decide the number of scenarios, being a choice of the researcher. Cluster analysis makes the choice well argued, however.     

Zero-sum condition: A necessary and sufficient condition for a transitive voting system

A simple and quick way to ascertain whether or not any given majority voting system can always produce a transitive social preference orderings without imposing any restriction on the distribution of diverse individual preference orderings is to examine whether all individual voting (preference) vectors satisfy the Addition Rule or not. This conclusion was obtained by first reformulating the voting mechanism into that of a linear mapping from T^m defined by q = Σp_i. It was found that the subset P of T that can accommodate all possible individual preference ordering profiles and such that every sum vector q = Σ p_i of its member vectors p_i is contained in T can be expressed as P = {p: p∈ T, s(p) = 0}. It was also pointed out that this is equivalent to the requirement that all individual preference (voting) functions must satisfy the Addition Rule. Finally, Borda's Rule and Saposnik's Contributive Rule were shown to be examples of transitive voting rules which satisfy these necessary and sufficient conditions.