Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case

This article focuses on the stability of rankings of academics by research productivity in the context of short-term decision-making. In particular, the growing use of national research assessment exercises (NRAE) has increased interest in identifying the contributions of individual researchers to an assessment unit’s output and ranking. The article concentrates on the assessment of individuals using plausible journal ranking schemes. We find that despite statistical evidence of a high degree of stability across journal ranking schemes as indicated by rank correlation coefficients, the particular ranking scheme used is of great importance to individual researchers. This applies with particular force to academics working within a NRAE environment based on individual assessment such as New Zealand’s PBRF.     

Quantifying the development of agreement among experts in Delphi studies

Delphi studies are often conducted with the aim of achieving consensus or agreement among experts. However, many Delphi studies fail to offer a concise interpretation of the meaning of consensus or agreement. Whereas several statistical operationalizations of agreement exist, hardly any of these indices is used in Delphi studies. In this study, computer simulations were used to study different indices of agreement within different Delphi scenarios. A distinction was made between the indices of consensus (Demoivre index), agreement indices (e.g., Cohen's kappa and generalizations thereof), and association indices (e.g., Cronbach's alpha, intraclass correlation coefficient). Delphi scenarios were created by varying the number of objects, the number of experts, the distribution of object ratings, and the degree to which agreement increased between subsequent rounds. Each scenario consisted of three rounds and was replicated 1000 times. The simulation study showed that in the same data, different indices suggest different levels of agreement, and also, different levels of change of agreement between rounds. In applied Delphi studies, researchers should be more transparent regarding their choice of agreement index and report the value of the chosen index within every round as to provide insight into how the suggested agreement level has developed across rounds.     

经济学实验结果外部效度排序探索研究

经济学实验在政策研究方面具有独特优点,但针对同一问题的不同实验设计,其结论的外部效度并不相同,明确现有实验研究结果的外部效度差异可以为解决实际问题提供更精确的政策制定参考,并改进相应实验设计。目前针对具体问题的经济学实验结果外部效度量化排序方法尚属空白,本文基于Harrison和List(2004)以及Levitt和List(2007)的思想提出一个对经济学实验结果外部效度进行排序的两标准判断法,两个标准分别强调实验设计类型及实验处理中的个体行为因素,并通过对嵌套社会困境环境下的公共品自愿贡献实验结果进行外部效度排序说明这一方法的操作。     

Comparison of statistical analysis methods for object case best–worst scaling

Abstract

Aims: Different methods have been used to analyze “object case” best–worst scaling (BWS). This study aims to compare the most common statistical analysis methods for object case BWS (i.e. the count analysis, multinomial logit, mixed logit, latent class analysis, and hierarchical Bayes estimation) and to analyze their potential advantages and limitations based on an applied example.

Methods: Data were analyzed using the five analysis methods. Ranking results were compared among the methods, and methods that take respondent heterogeneity into account were presented specifically. A BWS object case survey with 22 factors was used as a case study, tested among 136 policy-makers and HTA experts from the Netherlands, Germany, France, and the UK to assess the most important barriers to HTA usage.

Results: Overall, the five statistical methods yielded similar rankings, particularly in the extreme ends. Latent class analysis identified five clusters and the mixed logit model revealed significant preference heterogeneity for all, with the exception of three factors.

Limitations: The variety of software used to analyze BWS data may affect the results. Moreover, this study focuses solely on the comparison of different analysis methods for the BWS object case.

Conclusions: The most common statistical methods provide similar rankings of the factors. Therefore, for main preference elicitation, count analysis may be considered as a valid and simple first-choice approach. However, the latent class and mixed logit models reveal additional information: identifying latent segments and/or recognizing respondent heterogeneity.     

Single Object Auctions With Externalities: A Tractable Model

We consider the single object auction model with allocative externalities in a private valuation and quasi‐linear setting. We model externalities by assuming that every agent has a private valuation (for the object) and a strict ranking of other agents. The utility for an agent when another agent receives the object is the product of his own valuation and a real number that depends on the rank of this agent in his ranking. When the only private information is the valuation of the agents, we characterise the implementable allocation rules and use these to derive the optimal auction. The optimal auction collects payments from agents who do not receive the object.     

Regulatory foresight: Methodologies and selected applications

This paper on regulatory foresight addresses approaches which allow future fields for regulatory action to be identified. We follow a rather wide perception of regulation and include standards and standardisation as elements of the regulatory framework. The paper presents three methodologies appropriate for performing regulatory foresight. First, an approach is presented which makes use of science and technology indicators and enables the identification of possible fields which may cause challenges for the regulatory framework and the regulatory bodies. Second, survey approaches are displayed which enable regulatory bodies to identify future needs for regulations. Finally, the usability of the Delphi methodology is discussed and results of a Delphi survey in the telecommunication area are presented. The paper concludes with a comparative analysis of the three methodological approaches regarding their effectiveness to conduct regulatory foresight.     

A dissensus based online Delphi approach: An explorative research tool

This paper presents an adapted Delphi methodology that is, contrary to the classical Delphi design is not aiming to minimize expert estimation variance, but to maximize the range of expert opinions inputted sequentially into an online system. After discussing the traditional Delphi approach and its dissensus based derivatives, the author opens the case for a dissensus Delphi based explorative research tool with special consideration of the Delphi aim, the expert sample and the Delphi design. The proposed online Delphi process is then presented conceptually. Next, the proposed tool is demonstrated based on a prototype, exploring the barrier factors to the adoption of mobile data services. A discussion on the theoretical design and practical R&D experience of the dissensus based online Delphi approach concludes the paper.     

Systemic analysis of UK foresight results: Joint application of integrated management model and roadmapping

This paper proposes a new systems approach to foresight studies. The paper will first indicate the complex and conflicting nature of long-term decision-making process. Then, the need for systems approach will be highlighted by the analysis of 1995 UK Delphi survey results and the scenarios of 2000 UK foresight scenarios. The paper proposes two methodologies, namely Integrated Management Model (IMM) and Roadmapping, in order to overcome challenges introduced by the multidimensional characteristics and complex nature of foresight studies. Based on systemic approach, IMM offers a useful way of developing long-term normative policies and strategies and their transformations into actions by considering necessary changes in organizational structures and behaviors. In addition, roadmapping is used to capture, manipulate and manage information to decrease complexity in the foresight by constructing roadmaps. In the paper, IMM and roadmapping are employed first to analyze UK foresight results and then to develop a new methodology to formulate Delphi events and scenarios for the successful implementation of foresight. This paper also promotes the integrated use of foresight techniques such as scenarios and Delphi rather than one for another.     

Preferred reporting items for studies mapping onto preference-based outcome measures: The MAPS statement

Abstract

‘Mapping’ onto generic preference-based outcome measures is increasingly being used as a means of generating health utilities for use within health economic evaluations. Despite publication of technical guides for the conduct of mapping research, guidance for the reporting of mapping studies is currently lacking. The MAPS (MApping onto Preference-based measures reporting Standards) statement is a new checklist, which aims to promote complete and transparent reporting of mapping studies. The primary audiences for the MAPS statement are researchers reporting mapping studies, the funders of the research, and peer reviewers and editors involved in assessing mapping studies for publication.

A de novo list of 29 candidate reporting items and accompanying explanations was created by a working group comprised of six health economists and one Delphi methodologist. Following a two-round, modified Delphi survey with representatives from academia, consultancy, health technology assessment agencies and the biomedical journal editorial community, a final set of 23 items deemed essential for transparent reporting, and accompanying explanations, was developed. The items are contained in a user friendly 23 item checklist. They are presented numerically and categorised within six sections, namely: (i) title and abstract; (ii) introduction; (iii) methods; (iv) results; (v) discussion; and (vi) other. The MAPS statement is best applied in conjunction with the accompanying MAPS explanation and elaboration document.

It is anticipated that the MAPS statement will improve the clarity, transparency and completeness of reporting of mapping studies. To facilitate dissemination and uptake, the MAPS statement is being co-published by seven health economics and quality of life journals, and broader endorsement is encouraged. The MAPS working group plans to assess the need for an update of the reporting checklist in five years’ time.     

A rank order and efficiency evaluation of the EU regions in a social framework

In this paper, methods of multi-criteria efficiency evaluation are implemented for ranking the socio-economic systems of the EU regions. The socio-economic rank order problem is a multi-criteria non-convex optimization problem that was solved by the implementation of a new efficiency evaluation, AOWI (Absolute Optimal Weights Index). The rank order of the 268 NUTS-2 regions from the 27 European Union countries in the social framework obtained on the basis of 16 socio-economical indicators illustrates social divergence within the European Union and in the new European Union regions.     

Variations of Delphi

This paper describes a number of variations to refine the classical Delphi technique, which have been tested in two recent Delphis. A calculated median and interquartile range are introduced. The use of parallel questionnaires to explore many questions simultaneously is described. The use of probabilities in ratio form is suggested, as are general data transforms for median and interquartile range calculation. The idea of ranking the timing of events is described, and when calculations of the different timescales people use are employed, it is shown that part of the convergence in succeeding rounds arises from changes in people's timescales. Correlation between events is used to identify events that interact and hence, Cross Impact Analysis is required.     

Testing for poverty dominance: an application to Canada

Abstract The paper proposes and applies statistical tests for poverty dominance that check for whether poverty comparisons can be made robustly over ranges of poverty lines and classes of poverty indices. This helps provide both normative and statistical confidence in establishing poverty rankings across distributions. The tests, which can take into account the complex sampling procedures that are typically used by statistical agencies to generate household‐level surveys, are implemented using the Canadian Survey of Labour and Income Dynamics (SLID) for 1996, 1999, and 2002. Although the yearly cumulative distribution functions cross at the lower tails of the distributions, the more recent years tend to dominate earlier years for a relatively wide range of poverty lines. Failing to take into account SLID’s sampling variability (as is sometimes done) can inflate significantly one’s confidence in ranking poverty. Taking into account SLID’s complex sampling design (as has not been done before) can also decrease substantially the range of poverty lines over which a poverty ranking can be inferred.     

A contingent grouping approach for stated preferences

This paper introduces the theoretical and statistical foundations of contingent grouping, a variant of attribute-based stated preference methods, and checks for its implementation difficulty. Respondents are asked to simply group a number of alternatives as better or worse than a baseline or the status quo situation. A Monte Carlo approach shows that it performs statistically better than contingent choice according to the MSE criterion, and worse than contingent ranking, while a case study suggests that contingent grouping is less difficult to answer than contingent ranking. Furthermore, contingent choice shows a greater consistency in preferences with contingent grouping than with contingent ranking.     

RANKING ECONOMICS DEPARTMENTS IN TERMS OF RESIDUAL PRODUCTIVITY: NEW ZEALAND ECONOMICS DEPARTMENTS, 2000–2006*

This paper utilises a human‐capital approach for ranking the research productivity of academic departments. Our approach provides rankings in terms of residual research output after controlling for the key characteristics of each department's academic staff. More specifically, we estimate residual research output rankings for all of New Zealand's economics departments based on their publication performance over the 2000 to 2006 period. We do so after taking into account the following characteristics of each department's academic staff: gender, experience, seniority, academic credentials and academic rank. The paper demonstrates that the rankings generated by the residual research approach and those generated by traditional approaches to research rankings may be significantly different for some departments. These differences are important in determining the likely efficiency impact of research assessment exercises.     

Public sector use of the Delphi technique

The Delphi Technique is gaining extensive use as a multipurpose tool by government planners and policy makers. This paper begins by briefly discussing the historical development of Delphi and some of its advantages and disadvantages. A considerable number of Delphis conducted in the public sector for the purposes of forecasting, communication, budgeting, and goal setting are then reviewed. Additionally, three prior reviews of Delphi use in the field of education are discussed. In reviewing these studies every attempt was made to detail the research intentions, special features and modifications, cautions, and benefits of each study. These characteristics are provided in tabular form in the summary section as a ready reference for the potential Delphi user. It is recommended that one should take full advantage of what has been learned from earlier research before undertaking a new Delphi.     

Current validity of the Delphi method in social sciences

The Delphi method is a popular technique for forecasting and an aid in decision-making based on the opinions of experts, which has been in existence for over half a century. This work evaluates its methodology and reviews its validity in the present day, especially in the area of Social Sciences. Three recent applications in this field are also explained, professional in nature, which have some characteristics that are not frequent with respect to other Delphi studies published. The main aim of two of these studies was to provide input for economic or statistical quantitative models, using the judgement of expert groups, while the third study aimed to analyse a complex social reality by means of a Policy Delphi in order to obtain reliable information before taking a policy decision. These applications highlight how this technique may be adapted to different social realities and requirements, making a positive contribution to social progress, provided it is applied with the necessary methodological rigour and with a good knowledge of the social medium in which it is being applied. Finally, there is an explanation of a number of lessons learned from the theory and aforementioned experiences, which may contribute to the successful outcome of a Delphi exercise.     

A review of selected recent advances in technological forecasting

During the past decade, there have been some significant developments in technological forecasting methodology. This paper describes developments in environmental scanning, models, scenarios, Delphi, extrapolation, probabilistic forecasts, technology measurement and some chaos-like behavior in technological data. Some of these developments are refinements of earlier methodology, such as using computerized data mining (DM) for environmental scanning, which extends the power of earlier methods. Other methodology developments, such as the use of cellular automata and object-oriented simulation, represent new approaches to basic forecasting methods. Probabilistic forecasts were developed only within the past decade, but now appear ready for practical use. Other developments include the wide use of some methods, such as the massive national Delphi studies carried out in Japan, Korea, Germany and India. Other new developments include empirical tests of various trend extrapolation methods, to assist the forecaster in selecting the appropriate trend model for a specific case. Each of these developments is discussed in detail.     

A Delphi technology forecasting approach using a semi-Markov concept

This paper suggests a technology forecasting approach based on a semi-Markov model, which appropriately describes the probabilistic nature of a sequential technology development process. This approach focuses primarily on the utilization of the information that has been skipped in conventional Delphi survey data. That is, through a simple statistic, the interrelationships among sequential technology developments can be extracted in a formal structure of a semi-Markov model from the original Delphi panel's estimates. A simulation technique is developed to forecast the development process by utilizing the information on such interrelationships. This technique provides a flexible and useful tool for R&D planners or project managers, especially in postanalysis of Delphi forecasting. To make good use of the approach, a computer-based interactive Delphi data analysis system (IDEAS) is implemented in IBM PC.     

基于因子法的上市公司投资价值分析

分析上市公司投资价值有许多方法,而如何综合评价上市公司的投资价值则是一个值得研究的重要问题。选取2011年9月30日沪深两市23家煤炭上市公司14项财务指标,采用因子分析法,使用SPSS统计软件得出综合价值排名,并与市净率进行比较,得出高估或者低估的股票。     

Foresight with Delphi Surveys in Japan

Foresight activities are conducted in many countries, but the Japanese were the most active in this field for many years. Most of their activities are based on the Delphi methodology. They published their first Delphi study in 1971 and repeated studies every five years. There is more knowledge available on more than 30 years of foresight with the Delphi approach in Japan. This provides an excellent opportunity to examine the established foresight system with the use of some examples. The forecast of the fax machine as an example of success, and the earthquake warning as a failure are illustrated in more detail, and the question is raised why—in spite of this mixed picture—foresight with the Delphi method is regarded as useful in Japan.