Classical meets modern in the IDEA protocol for structured expert judgement

Expert judgement is pervasive in all forms of risk analysis, yet the development of tools to deal with such judgements in a repeatable and transparent fashion is relatively recent. This work outlines new findings related to an approach to expert elicitation termed the IDEA protocol. IDEA combines psychologically robust interactions among experts with mathematical aggregation of individual estimates. In particular, this research explores whether communication among experts adversely effects the reliability of group estimates. Using data from estimates of the outcomes of geopolitical events, we find that loss of independence is relatively modest and it is compensated by improvements in group accuracy.     

Reproducibility investigation of elicitation techniques in risk assessment for hydraulic turbines

Abstract

Certain elicitation techniques exert some control on expert opinions by leading them to a consensus or to a specific choice. In the absence of such guidelines, experts rely on their own knowledge to formulate opinions. This can result in large dispersions and affects the decision maker’s judgment. In this situation, we wonder what the relevant elicitation techniques are and how we can help experts to express their knowledge. From literature review, it is hard to decide if elicitation techniques are equivalent or not, which justifies the reproducibility analysis that we carry out in this paper. In this study, multiple experts have been involved in order to predict the defect size in hydraulic turbines, according to four proposed elicitation techniques. The comparison between these techniques was performed based on a suggested algorithm using the area metric concept. Our Findings show that elicitation techniques with ‘support’ tend to limit variations between experts and might be suitable only when prior knowledge on the expected elicited variable is available. Otherwise, we can end up with a distorted opinion of the elicited variable and an erroneous risk assessment.     

Book Review

This research examines two modes of assessment of environmental health risks and the transformation of these risks into public health issues while relying on the specific case of well‐water toxicity and mega dose of electromagnetic radiation found in one prosperous town in the center of Israel – Ramat ha‐Sharon. Based on official and scientific documentation and interviews conducted at three time periods with randomly selected town residents from contaminated neighborhoods (N = 169), this study shows the discrepancy between the ‘objective’ experts’ standards for assessing environmental health risks and the public’s subjective perception and evaluation of the impact of these risks on their health and well‐being. Even though, by experts’ standards, the well‐water toxicity remained constant over the three interview sessions, Ramat ha‐Sharon town residents’ subjective levels of concern and perception of risk fluctuated as a function of news media and municipality announcements and residents’ perceived ability to minimize the risk. This study also shows the complex and multidisciplinary nature of environmental health risk assessments and the need to relocate them into the broader socioeconomic and political context in which they are embedded.     

Expert relevance and the use of context-driven heuristic processes in risk perception

The effectiveness of a medical treatment should not predict its risk (highly effective treatments can be either safe or risky), however, people’s use of heuristic shortcuts may lead them to judge a link between effectiveness and risk, typically a negative correlation. A particular concern is that experts might use such a strategy and that this is unlikely to provide an accurate judgement. This large-scale field-based experiment compares expert-relevant and non-expert-relevant contexts, for both expert and public judgements of risk and effectiveness in the context of blood transfusion medicine. Postal questionnaires were distributed to anaesthetists (experts, N?=?123) and a general public (non-expert) comparison group (N?=?1153); half of the participants were cued with accompanying general information about genetically-modified (GM) biotechnology and half received specific information about blood product technologies. The blood-focussed information served to emphasise the medical relevance of the questionnaire to the expert group. Regression analyses showed that generally perceived effectiveness predicted perceived risk for both experts and non-experts, which suggests heuristic processing. However, although experts appeared to engage in heuristic processing for risk perceptions in certain circumstances, this processing is strongly affected by context. Experts who received the medically relevant context rated perceptions of effectiveness independently of perceptions of risk, unlike those who received the GM context. This indicates a reduced reliance on a low-effort heuristic for experts given an expertise-relevant context. The results are considered in light of dual-process (rational-associative) accounts of reasoning.     

Development paths of risk management: approaches,methods and fields of application

In the most recent stage of development of modern society, the growing complexity and dynamism of the contexts firms operate in has led to a relentless increase in the level of risk in all areas of firms’ management and activities. For this reason, the discipline and practice of risk management (RM) has taken hold more and more in very different sectors and contexts: from nuclear to supply chain to healthcare. The evolutionary path of RM has followed distinct paths of development that have adopted specific perspectives and, backed by different cultural matrices, led to unique approaches and methods. The aim of this paper is to provide a contribution to the identification and characterization of new paths of development based on a thorough analysis of the academic and managerial literature. This taxonomy can serve as an integrated reference to the RM discipline allowing those who deal with RM to identify and characterize the most suitable paths of development for a given context, within a comprehensive framework. This study can also promote a sort of ‘cross fertilization’ between experts in specific areas of RM as has been the case of clinical RM which draws from the experience gained in the field of complex engineering systems.     

Embodied uncertainty: living with complexity and natural hazards

In this paper, we examine the concept of embodied uncertainty by exploring multiple dimensions of uncertainty in the context of risks associated with extreme natural hazards. We highlight a need for greater recognition, particularly by disaster management and response agencies, of uncertainty as a subjective experience for those living at risk. Embodied uncertainty is distinguished from objective uncertainty by the nature of its internalisation at the individual level, where it is subjective, felt and directly experienced. This approach provides a conceptual pathway that sharpens knowledge of the processes that shape how individuals and communities interpret and contextualise risk. The ways in which individual characteristics, social identities and lived experiences shape interpretations of risk are explored by considering embodied uncertainty in four contexts: social identities and trauma, the co-production of knowledge, institutional structures and policy and long-term lived experiences. We conclude by outlining the opportunities that this approach presents, and provide recommendations for further research on how the concept of embodied uncertainty can aid decision-making and the management of risks in the context of extreme natural hazards.     

Expert judgements of sea-level rise at the local scale

Whilst local projections of sea-level rise (SLR) are necessary to facilitate targeted climate change adaptation and communication strategies, downscaling from global climate models can be problematic. Here, we use expert probability judgement to elicit a suite of local projections, and associated uncertainties, for future SLR on the Severn Estuary in the south-west of the UK. Eleven experts from a range of policy and academic backgrounds took part in a structured probability elicitation exercise for the years 2050, 2100 and 2200. In addition to the quantitative elicitation, the experts’ reasoning during the task was qualitatively analysed. Quantitative analyses show that although there is consensus that sea levels will rise on the Estuary in future, there is wide variation between judgements and much uncertainty regarding the magnitude of future rise. For example, median estimates of SLR (compared to the 2011 level) range from 9.6 to 40 cm in the year 2050; 20 to 100 cm in 2100; and 35 to 300 cm in 2200. Fifty per cent confidence intervals and ninety per cent confidence intervals vary even more. Qualitative analyses indicate that experts’ judgements may have been influenced by their choice of methods and information sources, the ways in which they thought about the future, and heuristics. The study shows the merits of integrating qualitative and quantitative methods to explore the reasoning behind uncertainty judgements. We conclude that where expert probability judgements are to be used to characterise uncertainty such reasoning should be made explicit.     

Managing risk in an unstable world

With emerging markets like China and politically unstable countries like Saudi Arabia figuring more than ever into companies' investment calculations, business leaders are turning to political risk analysis to measure the impact of politics on potential markets, minimize risks, and make the most of global opportunities. But political risk is more subjective than its economic counterpart. It is influenced by the passage of laws, the foibles of government leaders, and the rise of popular movements. So corporate leaders must grapple not just with broad, easily observable trends but also with nuances of society and even quirks of personality. And those hard-to-quantify factors must constantly be pieced together into an ongoing narrative within historical and regional contexts. As goods, services, information, ideas, and people cross borders today with unprecedented velocity, corporations debating operational or infrastructural investments abroad increasingly need objective, rigorous assessments. One tool for measuring and presenting stability data, for example, incorporates 20 composite indicators of risk in emerging markets and scores risk variables according to both their structural and their temporal components. The indicators are then organized into four equally weighted subcategories whose ratings are aggregated into a single stability score. Countries are ranked on a scale of zero (a failed state) to100 (a fully institutionalized, stable democracy). Companies can buy political risk analyses from consultants or, as some large energy and financial services organizations have done, develop them in-house. Either way, a complete and accurate picture of any country's risk requires analysts with strong reportorial skills; timely, accurate data on a variety of social and political trends; and a framework for evaluating the impact of individual risks on stability.     

Knowing stories that matter: learning for effective safety decision-making

Ongoing safe operation of hazardous industries such as hydrocarbon production and transportation, air traffic control and nuclear power generation depends on effective decision-making by those in key positions. Safety studies often focus on the extent to which actions of operational personnel in particular are dictated by procedures or rules and hence reinforce the need for compliance to ensure the best outcomes. This article directs attention to a different area – the judgements made by experts in the cases that are not covered by rules and, in particular, the key role of stories and storytelling. This ethnographic research draws on literature related to high-reliability theory, organisational learning and naturalistic decision-making to examine how experts working in diverse critical contexts use stories to share and make sense of their experiences. It argues that such stories are vital to effective decision-making as a result of both the general and specific lessons that they embody. Our analysis shows that experts use stories as parables to nurture their ability to imagine possible outcomes and maintain a safety imagination. Stories are also embedded in work practices to support decision-making in the moment. Finally, stories are strongly linked to organisational learning for experts as a group and in mentoring less-experienced colleagues. We argue that the increased focus on incident reporting systems in hazardous industries, which is driven at least in part by a consideration of organisational learning, is failing in this regard because such systems do not facilitate story-based learning. We appeal to organisations to support story-based learning with as much vigour as formal systems for professional development and reporting.     

The attitude of Libyan auditors to inherent control risk assessment

Previous empirical evidence contradicts the practices of audit risk assessment recommended by the professional auditing standards. Key differences represent the presumption that auditors can readily distinguish between different categories of risk, how they assess such risks in practice, their application to the different account levels and their capacity to differentiate between what may constitute potentially high or low risks. The paper addresses these differences in the context of the Libyan auditors’ perceptions and practices. The nature and development of the audit profession in Libya makes this a good comparator for the US and UK contexts of audit behaviour. Equally importantly, the religious, political and socio-cultural context combined with Libya's role as an emergent economy is representative of many other economies in the world, thus providing a good comparator for assessing the validity and applicability of the established auditing management principles and procedures. One hundred and sixty four practicing auditors in Libya were investigated, initially by questionnaire and subsequently interview of a smaller sample of 20 auditors. The evidence reinforces prior empirical evidence that inherent and control risks are assessed interdependently, auditors are aware of the risk differentials depending on the level of account and are cognizant of factors normally associated with potentially high and low risk levels.     

Investor reactions to key audit matters: Financial and non-financial contexts

We investigate how a disclosed risk item and key audit matter (KAM) relatedness combine to affect investors' riskiness assessment in financial and non-financial contexts. When management disclose a high-risk item, we find that investors react the same way across contexts with KAM relatedness having no effect. When management disclose a low-risk item, investors react differently in each context. When a KAM is related to the disclosed financial (non-financial) low-risk item, investors assess investment riskiness higher (lower) than when a KAM is unrelated to the low-risk item. Our findings indicate the varying communicative value of KAMs across financial and non-financial contexts.     

What role for social scientists in risk expertise?

When it comes to risks – health and environmental risks, like those linked to the use of nanotechnologies, pesticides, etc. – three main groups of actors are easily identified, brought together through boundary organisations such as environmental and sanitary risk agencies: the natural and technical scientists, who provide their expertise to assess risks (especially toxicologists, epidemiologists and microbiologists); the policy makers, who take decisions regarding risk management and risk regulation; the lay public, who are more and more involved in participatory frameworks. Sometimes three other groups of actors are added: the ‘economists’ who can for instance conduct cost–benefit assessments or multi-criteria analyses (especially ecological economists, public economists, political economists and social economists); the ‘philosophers’/‘ethicists’ who can use ethics to highlight moral choices and responsibilities in face of risks; and the ‘jurists’/‘legal experts’ who can justify authorisation or interdiction according to law. Inversely, there is a group of actors which is not clearly identified, that of social scientists, even though a considerable quantity of social science knowledge on risk has been produced. Why is there such a discrepancy? This article, based on a critical review of the literature, aims to make sense of the fuzziness surrounding the involvement of social scientists when it comes to risk expertise. The article shows that one reason for this puzzling situation is to be found in the gap between what social scientists often want to do when they are called in as risk experts and what natural scientists and public policy makers actually expect from them.     

The constitutive element of probabilistic agency in risk: a semantic analysis of risk,danger, chance,and hazard

Defining central concepts with accuracy is crucial to any scientific discipline. A recent debate over risk definitions in this journal illustrates the far reaching consequences of divergent definitions. Aven and Renn define risk as a social construct while Rosa defines risk as an ontological fact. Both claim that their definition reflects the common usage of the word risk. Through a semantic analysis this paper points to a constitutive element of what is termed probabilistic agency in the risk concept. In this respect, risk is distinct from danger, and because Rosa’s main argument is based on the apparent synonymy between risk and danger, the premises for his risk ontology are not valid. The paper furthermore argues that Aven and Renn’s attempt to bridge between epistemology and ontology is based on a distinction between a conceptual level of risk and its practical application which is impossible to uphold if a risk definition is to be in accordance with the ordinary usage of the word. The paper concludes by arguing that risks are only real within a subjective ontology.     

Nationality and differences in auditor risk assessment: A research note with experimental evidence

This study examines whether auditors from different countries come to different conclusions when they perform analytical procedures to assess the risk of misstatement in accounts. During a laboratory experiment, auditors who worked for the same firm in the United Kingdom, France, and the United States performed analytical procedures on identical case materials. Although auditors from all three countries came to similar conclusions about the overall risk of misstatement, they attributed risk differently across the individual financial statement accounts they evaluated.     

Views of Risk in Sweden: Global Fatalism and Local Control — An Empirical Investigation of Ulrich Beck's Theory of New Risks

Ulrich Beck's theory of risk society has been criticised because there is lack of empirical evidence. By comparing people with different life contexts and experiences, the aim of this study was to investigate how these people view risk, and if ‘new’ risks are perceived differently by different groups in society. Five focus‐group interviews were conducted in Sweden, in 2004/05, with people in rural and urban areas, people with a foreign background and experts. The groups consisted of four people each and lasted for two hours. The results show that ‘new’ risks are not something people worry about; ‘risk’ is associated with personal experiences and life context. This indicates a traditional or at least modern way of viewing risk, and contradicts the idea of a reflexive view of risk. However, a division between the urban versus the rural‐migrant groups appears: the expert‐urban groups show a more global—fatalistic strategy to handle of risk, while the rural—migrant group shows a more traditional approach to risk, where control and the local context are in focus.     

Scientist and public responses to BSE‐related risk: A comparative study

Social studies of risk repeatedly have found substantial divergences in the way that experts/scientists and members of the general public appraise risks associated with health and environmental hazards. However, empirical evidence for these differences remains controversial. A recent review of literature suggests in particular that divergences between experts and lay people may potentially result from confounding socio‐demographic factors. The purpose of the present article is to investigate and to compare how medical scientists and members of the general population reacted to BSE (“mad cow disease”) in France. A sample of 401 scientists belonging to the French Institute of Health and Medical Research (INSERM) and a representative sample of 902 French citizens were asked about their cognitive, affective and behavioural responses to BSE. Results show that non‐scientists tended to express much more concern about BSE‐related risk than scientists, even when socio‐demographic variables were controlled. However, (1) no significant differences were found between these two groups concerning subjective epidemiological predictions, and (2) scientists manifested greater avoidance of beef or beef by‐products avoidance following the BSE epidemic than lay respondents did. The implications of these paradoxical results are discussed.     

Assessing inherent risk during audit planning: The development of a knowledge based model

Within the academic and professional auditing communities there has been growing concern with accurately assessing the various risks associated with the performance of an audit. One approach to developing sophisticated risk assessment models is to study how experienced expert auditors use industry and firm specific factors in making audit judgements. This study presents a model of inherent risk assessment based on literature reviews and a field study that involved structured and unstructured interviews and observations of experts in audit planning meetings. Analysis of the data gathered led to the specification of a conceptual model of inherent risk assessment which has been implemented as a computer program (a computational model). Auditors were asked to assess the behavior and performance of the computational model as a first step in evaluating the expert model.     

Risk models-at-risk

The experience from the global financial crisis has raised serious concerns about the accuracy of standard risk measures as tools for the quantification of extreme downward risks. A key reason for this is that risk measures are subject to a model risk due, e.g. to specification and estimation uncertainty. While regulators have proposed that financial institutions assess the model risk, there is no accepted approach for computing such a risk. We propose a remedy for this by a general framework for the computation of risk measures robust to model risk by empirically adjusting the imperfect risk forecasts by outcomes from backtesting frameworks, considering the desirable quality of VaR models such as the frequency, independence and magnitude of violations. We also provide a fair comparison between the main risk models using the same metric that corresponds to model risk required corrections.     

Detecting local environmental change: the role of experience in shaping risk judgments about global warming

Anthropogenic climate change is increasing the frequency and severity of extreme weather events (e.g. flooding, heat waves, and wildfires). As a result, it is often reasoned that as more individuals experience unusual weather patterns that are consistent with changing climate conditions, the more their concern about global warming will increase, and the more motivated they will become to respond and address the problem effectively. Social science research evaluating the relationships between personal experiences with and risk perceptions of climate change, however, show mixed results. Here, we analyze a representative statewide survey of Floridians and compare their risk perceptions of five-year trends in climate change with local weather station data from the five years preceding the survey. The results show that Floridians are unable to detect five-year increases in temperature, but some can detect changes in precipitation. Despite an inability to detect the correct direction of change, respondents were significantly more likely than not to correctly identify the season in which most change occurred. Nevertheless, compared to local experience, risk perceptions of climate change were more strongly predicted by subjective experiences of environmental change, personal beliefs about climate change, and political ideology. Results from the study suggest that long-term changes in climate patterns and extreme weather events need to be interpreted by weather and climate experts within the context of climate change; individuals cannot be expected to detect or comprehend such complex linkages directly.     

Expert judgement elicitation for risk assessments of critical infrastructures

Governmental bodies and companies are confronted with the problem of achieving rational consensus in the face of substantial uncertainties. The subject area of this special issue (risk and vulnerability assessments and management of critical infrastructures) might be a good example as are risk management of chemical installations and accident consequence management for nuclear power plants. Decisions with regard to infrastructures functioning and possible malfunctioning must be taken on the basis of predictions of technical and organizational system behaviour. These predictions use mathematical models containing scores of uncertain parameters. Decision makers want to take, and want to be perceived to take, these decisions in a rational manner. The question is, how can this be accomplished in the face of large uncertainties? One available source is experts in the many fields of interest within infrastructures. This paper describes the use of structured expert judgement in a formal manner. The paper refers to the Procedures Guide published by the European Union as EUR 18820. This Procedures Guide addresses two methods for using expert judgements developed at Delft University of Technology. The paired comparisons method is particularly suitable to identify the relative importance of attributes in the risk management arena, while the Classical Model, apt to arrive at subjective probability assessments, is particularly suitable to derive uncertainty distributions over model parameters. Examples will be referred to for further illustration of applications relevant in the field of risk assessment and risk management.