Classical meets modern in the IDEA protocol for structured expert judgement

Expert judgement is pervasive in all forms of risk analysis, yet the development of tools to deal with such judgements in a repeatable and transparent fashion is relatively recent. This work outlines new findings related to an approach to expert elicitation termed the IDEA protocol. IDEA combines psychologically robust interactions among experts with mathematical aggregation of individual estimates. In particular, this research explores whether communication among experts adversely effects the reliability of group estimates. Using data from estimates of the outcomes of geopolitical events, we find that loss of independence is relatively modest and it is compensated by improvements in group accuracy.     

The role of individual personality type in subjective risk elicitation outcomes

Managing uncertainty is an unavoidable challenge in a variety of decision contexts. On many occasions, objective data are not available, necessitating the use of experts. However, research into procedures to elicit and aggregate information from experts is difficult to validate or test empirically. A controlled economic experiment was used to assess subjective probability elicitation accuracy resulting from three alternative elicitation procedures and two aggregation alternatives. The empirical results provide evidence of the impacts of elicitation techniques, distribution type, and personal characteristics such as risk preferences and personality type on subjective risk assessment accuracy. Our conclusion is that experimental approaches hold promise as a technique to assess the forecast accuracy of aggregated subjective probabilities in a variety of contexts.     

Reproducibility investigation of elicitation techniques in risk assessment for hydraulic turbines

Abstract

Certain elicitation techniques exert some control on expert opinions by leading them to a consensus or to a specific choice. In the absence of such guidelines, experts rely on their own knowledge to formulate opinions. This can result in large dispersions and affects the decision maker’s judgment. In this situation, we wonder what the relevant elicitation techniques are and how we can help experts to express their knowledge. From literature review, it is hard to decide if elicitation techniques are equivalent or not, which justifies the reproducibility analysis that we carry out in this paper. In this study, multiple experts have been involved in order to predict the defect size in hydraulic turbines, according to four proposed elicitation techniques. The comparison between these techniques was performed based on a suggested algorithm using the area metric concept. Our Findings show that elicitation techniques with ‘support’ tend to limit variations between experts and might be suitable only when prior knowledge on the expected elicited variable is available. Otherwise, we can end up with a distorted opinion of the elicited variable and an erroneous risk assessment.     

Expert relevance and the use of context-driven heuristic processes in risk perception

The effectiveness of a medical treatment should not predict its risk (highly effective treatments can be either safe or risky), however, people’s use of heuristic shortcuts may lead them to judge a link between effectiveness and risk, typically a negative correlation. A particular concern is that experts might use such a strategy and that this is unlikely to provide an accurate judgement. This large-scale field-based experiment compares expert-relevant and non-expert-relevant contexts, for both expert and public judgements of risk and effectiveness in the context of blood transfusion medicine. Postal questionnaires were distributed to anaesthetists (experts, N?=?123) and a general public (non-expert) comparison group (N?=?1153); half of the participants were cued with accompanying general information about genetically-modified (GM) biotechnology and half received specific information about blood product technologies. The blood-focussed information served to emphasise the medical relevance of the questionnaire to the expert group. Regression analyses showed that generally perceived effectiveness predicted perceived risk for both experts and non-experts, which suggests heuristic processing. However, although experts appeared to engage in heuristic processing for risk perceptions in certain circumstances, this processing is strongly affected by context. Experts who received the medically relevant context rated perceptions of effectiveness independently of perceptions of risk, unlike those who received the GM context. This indicates a reduced reliance on a low-effort heuristic for experts given an expertise-relevant context. The results are considered in light of dual-process (rational-associative) accounts of reasoning.     

Expert judgement elicitation for risk assessments of critical infrastructures

Governmental bodies and companies are confronted with the problem of achieving rational consensus in the face of substantial uncertainties. The subject area of this special issue (risk and vulnerability assessments and management of critical infrastructures) might be a good example as are risk management of chemical installations and accident consequence management for nuclear power plants. Decisions with regard to infrastructures functioning and possible malfunctioning must be taken on the basis of predictions of technical and organizational system behaviour. These predictions use mathematical models containing scores of uncertain parameters. Decision makers want to take, and want to be perceived to take, these decisions in a rational manner. The question is, how can this be accomplished in the face of large uncertainties? One available source is experts in the many fields of interest within infrastructures. This paper describes the use of structured expert judgement in a formal manner. The paper refers to the Procedures Guide published by the European Union as EUR 18820. This Procedures Guide addresses two methods for using expert judgements developed at Delft University of Technology. The paired comparisons method is particularly suitable to identify the relative importance of attributes in the risk management arena, while the Classical Model, apt to arrive at subjective probability assessments, is particularly suitable to derive uncertainty distributions over model parameters. Examples will be referred to for further illustration of applications relevant in the field of risk assessment and risk management.     

The psychology of forecasting

This article reviews research on the elicitation of consistent, coherent and valid probability forecasts. From this a general model of the judgemental forecasting process is developed showing the interdependence of consistency, coherence and validity. The authors describe a forecasting aid which elicits consistent and coherent forecasts and argue that research should now focus on the validity of judgemental forecasting by attempting to identify the characteristics of those individuals who can produce valid judgements.     

Expert opinion and probabilistic volcanic risk assessment

We present data from an international survey of scientists working at volcanic observatories concerning eruption likelihoods. The scientists were asked a range of questions using different types of phrasing. The data suggest that the phrasing of questions affects the ways in which probabilities are estimated. In total, 71% of respondents (N = 70) exhibited some form of inconsistency in their answers between and/or within different question formats. The data also allow for an analysis of the use of scaling in probabilistic assessment, and the use of quantitative versus verbal risk measurements. However, some respondents were uncomfortable with providing any numerical probability estimate, perhaps suggesting that they considered the uncertainty too high for meaningful judgements to be made.     

Who Will Pay for Long‐Term Care in the UK? Projections Linking Macro‐ and Micro‐Simulation Models

The long‐term care funding system continues to attract much debate in the UK. We produce projections of state and private long‐term care expenditure and analyse the distributional impact of state‐financed care, through innovative linking of macro‐ and micro‐simulation models. Variant assumptions about life expectancy, dependency and care costs are examined and the impact of universal state‐financed (‘free’) personal care, based on need but not ability to pay, is investigated. We find that future long‐term care expenditure is subject to considerable uncertainty and is particularly sensitive to assumed future trends in real input costs. On a central set of assumptions, free personal care would, by 2051, increase public spending on long‐term care from 1.1 per cent of GDP to 1.3 per cent, or more if it generated an increase in demand. Among the care‐home population aged 85 or over, the immediate beneficiaries of free personal care would be those with relatively high incomes.     

The siren call of probability: Dangers associated with using probability for consideration of the future

Many tools for thinking about the future employ probability. For example, Delphi studies often ask expert participants to assign probabilities to particular future outcomes. Similarly, while some scenario planners reject probability, others insist that assigning probabilities to scenarios is required to make them meaningful. Formal modelling and forecasting methods often also employ probability in one way or another. The paper questions this widespread use of probability as a device for considering the future, firstly showing that objective probability, based on empirically-observed frequencies, has some well-known drawbacks when used for this purpose. However, what is less-widely acknowledged is that this is also true of the subjective probability used in, for example, Delphi. Subjective probability is less distinct from objective probability than proponents of its use might imply, meaning it therefore suffers from similar problems. The paper draws on the foundations of probability theory as set out by Kolmogorov, as-well-as the work of Keynes, Shackle, Aumann, Tversky and Kahneman, and others, to reassert the essential distinction between risk and uncertainty, and to warn about the dangers of inappropriate use of probability for considering the future. The paper sets out some criteria for appropriate use.     

Assessing inherent risk during audit planning: The development of a knowledge based model

Within the academic and professional auditing communities there has been growing concern with accurately assessing the various risks associated with the performance of an audit. One approach to developing sophisticated risk assessment models is to study how experienced expert auditors use industry and firm specific factors in making audit judgements. This study presents a model of inherent risk assessment based on literature reviews and a field study that involved structured and unstructured interviews and observations of experts in audit planning meetings. Analysis of the data gathered led to the specification of a conceptual model of inherent risk assessment which has been implemented as a computer program (a computational model). Auditors were asked to assess the behavior and performance of the computational model as a first step in evaluating the expert model.     

Accuracy of Long-Range Actuarial Projections of Health Care Costs

The Office of the Actuary, mandated to provide projections of future medical spending for use by the U.S. Medicare and Medicaid programs, publishes forecasts that have been widely used by private firms and government budget officials as a baseline for expected long-run premium trends and to estimate liabilities for retiree health benefits. Although these projections have been made publicly available since 1986, they have not yet been subject to systematic evaluation by an external reviewer. This article develops a method for assessment of both short- and long-run accuracy and applies it to the 17 sets of projections made public over the last 25 years. The more recent set of projections (1998–2010) incorporating lagged macroeconomic effects appear to be more accurate than the older (1986–1995) projections that relied more heavily on demographic cost of illness trends. The average annualized error of the forecasts is approximately 0.5–1% per year, whether assessed over a span of one, two, or 10 years. Projecting “excess” growth in health spending (the rise in the share of wages or GDP) tends to be more accurate than forecasting nominal or real spending per capita.     

Expert judgement and lethal toxicity of inhaled chemicals

The application of formal expert judgement for assessing quantitative data of the toxicity of large amounts of inhaled chemical substances after a potential major hazards incident is described. In this particular application, the expert assessments were used to derive probit relations for the acute lethal effects of five hazardous substances: ammonia, acrylonitrile, hydrogen fluoride, sulphur trioxide and azinphos-methyl. The purpose of the study was to develop a protocol for selecting experts and the elicitation and analysis of expert assessments, to apply this protocol to generate probit relations for representative substances, and to evaluate the overall performance of the protocol. The use of formal expert judgement includes quantitative estimates of variables, calibration of experts and representation of uncertainty on the variables. This enables analysts to optimize combined experts' assessments for establishing quantitative results for further use in probit relations. Twenty-seven experts, distributed over the five substances, gave medians and 90% central confidence bands of the lethal dose-response relations under several conditions. The calculated dose-response relations are shown and compared to existing probit relations with respect to its impact on risk assessment outputs. The calculated relations can be applied in quantitative risk assessments, for example, under the Dutch law on major hazards installations. The scientific and practical efforts and costs of using the expert judgement technique are described.     

Die Auswirkungen der steigenden Lebenserwartung auf die Prävalenz der Pflegebedürftigkeit in Deutschland

As a consequence of population ageing the number of persons in need of care will rise considerably in the decades to come. While the rise per se is uncontroversial a still open question is whether the increase in longevity will raise or lower the age-specific prevalence of long term care. As there are both empirical studies finding an increase or a decrease of prevalence as life expectancy increases, most existing projections of the number of persons in need of care are based on the assumption of constant age-specific prevalence rates. While this question cannot be answered ultimately from the current point of view, this paper gives some guidance on the range of the future increase in cases. To this end we first analyze the interplay of changes in mortality, incidence and prevalence within a flow figure model. This approach allows for a better understanding of the conditions which must be met for an increase or a decline of the prevalence of care in the future. Based on German data we then adapt a scenario approach to analyze the model's quantitative implications for the prevalence of care and the number of nursing cases in the future. Our results indicate that the number of nursing cases will range between 2,6?m and 7?m persons in the year 2060. Nonetheless, based on the existing empirical findings for Germany and other industrialized countries, a doubling on 4?m persons in need of care seems to be the most realistic scenario.     

(Almost) Model‐Free Recovery

Under mild assumptions, we recover the model‐free conditional minimum variance projection of the pricing kernel on various tradeable realized moments of market returns. Recovered conditional moments predict future realizations and give insight into the cyclicality of equity premia, variance risk premia, and the highest attainable Sharpe ratios under the minimum variance probability. The pricing kernel projections are often U‐shaped and give rise to optimal conditional portfolio strategies with plausible market timing properties, moderate countercyclical exposures to higher realized moments, and favorable out‐of‐sample Sharpe ratios.     

Economic prediction and human action

Empirical prediction entails syllogistic reasoning, the major premise being a “proposition of regularity”. This latter can itself be tested empirically, as the hypothesis “Past regularity implies future regularity”. In the social sciences (in contrast to the physical sciences), the deterministic version of this hypothesis is demonstrably untrue. Though the “statistical” version of the hypothesis stands, the likelihood of its truth has never been expressed in probability terms. Thus empirical prediction in economics confronts not only uncertainty, which can be expressed in statistical terms, but an added element of “ignorance” which (at least as yet) cannot be so expressed.     

The impact of confidence interval information on probability judgements

Previous efforts to demonstrate the value of confidence interval financial statements for credit evaluation and investment decisions have met with limited success. The present study examined the impact of nonfinancial confidence interval reports on predecisional behavior. A Bayesian revision task was used to investigate the impact of report type, information system uncertainty and decision problem uncertainty on the accuracy of probability judgements. All three variables influenced task performance. Moreover, subjects given confidence interval reports outperformed those who were given single-figure reports when significant information system and decision problem uncertainties were present. The results obtained here suggest several facets of the decision setting which may influence the value of confidence interval financial statements.     

The impact of political pressure on novice decision makers: are auditors qualified to make going concern judgements?

Over the past decade, the public accountancy profession has been the subject of broad criticism for its failure to adequately prepare third parties for the impending failure of client organizations. Many potential causes of auditors’ failure to issue opinions reflecting client organization’s ability to remain a going concern have been put forth, including primarily poor judgement and/or client pressures to issue a clean opinion. In this paper, we argue that the uncertainty in the decision-making process may be compounded by a relatively low level of knowledge, making the auditor’s end decision more susceptible to socio-political pressure. The paper is divided into five sections. The introductory section of the paper focuses on the widespread criticism of the accountancy profession for its inability to issue going concern opinions for questionable clients who fail shortly after the audit report is issued. In the second section of the paper, we provide an overview of extant studies that have examined how auditors accumulate and process data used to make going concern judgements in a politically sensitive environment. In the third section, we contrast this environment and the approach used by auditors in making going concern judgements with the environment and approach used by insolvency practitioners in dealing with companies in financial distress. A multi-year study in building and using a computerized model of insolvency decision processes is used as the basis for understanding the approach used by insolvency practitioners. Even though the nature of the problems faced in going concern judgements and insolvency practice are very similar, striking differences in the environment, evidence and decision processes used by the two groups of accounting experts are noted. The fourth section reports the results of a field study conducted with insolvency novices and specialists in an effort to elicit the differences in their respective decision processes. The interaction with insolvency specialists from 10 different discussion sessions reveals that senior insolvency practitioners perceive that the decision strategies of auditors is very similar to junior (novice) insolvency practitioners and that the going concern decision, as currently made, fails to include the specialized knowledge of expert decision makers. The fifth and concluding sections discuss implications of these results on auditors’ susceptibility to socio-political pressures and explores the difficulties in addressing the problems identified.     

Public priorities and expectations of climate change impacts in the United Kingdom

Irrespective of the success of climate mitigation efforts, societies worldwide face the challenge of adapting to a changing climate. In this paper, we examine UK residents’ expectations of future threats and opportunities associated with climate change impacts, along with willingness to prioritise different climate change impacts for investment. Using a national survey (n = 2007), we report on three main findings. First, UK residents tend to expect threats related to flooding and wet weather to be more likely and concerning than heat extremes or opportunities. Second, UK residents’ expectations of climate change impacts do not align with expert assessments, especially showing lower estimates of heat-related threats as compared to experts. Third, willingness to allocate resources to potential climate change impacts tends to be more strongly associated with anticipated concern should they occur than climate change belief or the expected likelihood of them occurring. We discuss the implications of our findings for policies and communications about climate change adaptation in the UK and elsewhere.     

French demographic perspectives

This article shows how demographic projections are made and what the evolutionary perspectives are for France up to 2050. Assuming a zero migration balance, it is shown that the death rate should not change noticeably in the years ahead and that changes in the future population of France will be affected essentially by fertility rates, which have noticeably decreased since 1965. By using projections prepared by INSEE and based on four fertility assumptions, the author shows that the population of French origin could vary by as much as 100% between now and the middle of the next century, and that the age structure will change more or less rapidly. An ageing of the population will occur leading to major modifications in the relationships between ‘active’ people and ‘non-active’ people from the year 2000. This article presents a comprehensive view of possible futures for the French population and, by way of comparison, the demographic evolutionary paths of other European countries.