Are case studies more than sophisticated storytelling?: Methodological problems of qualitative empirical research mainly based on semi-structured interviews

Qualitative empirical research and case studies have, like any other scientific method, their strengths and weaknesses. But how valid are the findings stemming from such research, particularly when they are base on data gained from interviews? This paper primarily deals with the methodological problems throughout the whole research process. Areas of concerns are: (1) before collecting data and during the whole research process, (2) the process of collecting data, (3) internal validity and making sense of data, (4) external validity of the data and findings, (5) relation between the findings and social sciences as well as social practice. The analysis will reveal a mixed picture concerning the scientific value of qualitative case studies. There is some good news. There is more methodological freedom and room for creativity in qualitative research than it seems at first glance. On the other hand there is a need for a greater methodological awareness particularly concerning possible downsides of subjectivity, the generalisation of the findings, conscious and unconscious biases, influences of dominant ideologies and mainstream thinking. Above all, there is a great need for rational critique. There is great need for a more critical attitude towards (basic) assumptions—particularly, to challenge, question and criticise literally everything systematically and permanently and to put the findings sufficiently in relation to the wider historical context, i.e. epochal ideologies and societal structures. Social science research has to contribute much more to the discussion of values, interests, and ideologies that shape current and future social practices.     

Validity — the quest for reality in quantitative and qualitative research

It has been argued that fundamentally different methodological approaches have made for ‘two sociologies’. This view has obscured the fact that the problem of validity has to be tackled independently of any specific methodological premises because of the textuality of sociological data. This does not necessarily imply, however, a single, unified strategy for validity testing. In this paper, some basic theoretical presuppositions underlying the approach to validity testing in quantitative research will be contrasted with the strategies offered by Max Weber's methodological writings on the ideal type. It is argued that the use of ideal typical constructs in qualitative research (exemplified by patient's illness careers) allows systematic validity testing despite the important differences in the conceptualization of social reality which is used in quantitative research, thus serving the purpose of any empirical sociological research, that is, to gain valid insight into societies' concrete reality.     

Alternative quality standards in qualitative research?

Qualitative researchers often use other principles for judging the quality of their study than quantitative researchers. This inhibits a straightforward assessment of the quality and comparability of different types of studies, as well as decision-making about their usefulness for further research and practice. In this article, we question the use of alternative criteria, and argue for one coherent and inclusive framework of quality criteria for both qualitative and quantitative studies. We developed such a framework, based on a comparison and operationalization of different criteria, and the elaboration of procedures to realize them. Its usability is demonstrated by the application in a qualitative case study research and a mixed-methods study. This framework enhances advancing and judging the basic quality of any type of scientific research, promoting the assessment of the quality and hence usability of studies for further research or decision-making about practice.     

Triangulation: Validity and empirically-based hypothesis construction

The essay focuses on the opportunities and strengths of a multi-method approach, widely called methodological triangulation, in which different investigative methods are applied to one research object. In practice, this can be realized with the coupling of quantitative structural data concerning the life course and the interpretation and evaluation of life course data collected with qualitative methods. This approach is examined in order to shed light on the problem that research findings often show different phenomena and not the different aspects of one phenomenon. The discussion of the relationships of the findings to one another (congruent, complementary or divergent) shows that in this context a multi-method approach can nevertheless be used to increase validity and to test hypotheses. Further, its particular strengths are the empirically induced modification of existing models and theories, as well as the development of new explanations.     

Respondent Related Correlates of Response Behaviour in Audience Research

Audience research by means of surveys has a long tradition, certainly withinarts and humanities oriented research. Yet, due to selective sampling and unitnonresponse it frequently lacks the methodological rigour to make scientificallyvalid statements based on sample estimates. This is one of the first attempts toexplore unit nonresponse in audience research. More specifically, it focuses onthe explanation of nonresponse by the socio-demographic and more topic relatedcharacteristics of a theatre audience. Using a two-step procedure for the on-sitecollection of data, the characteristics of respondents are compared with those ofnonrespondents. In step 1 the composition of the theatre audience is comparedto a proxy of a theatre population benchmark based on a weighted sample of theFlemish population (APS-2000). The validity of this best available method isdiscussed. Step 2 compares respondents with nonrespondents on a micro-level:ignoring unit nonresponse in step 1, we use logistic regression to map selectionin step 2. The chance of cooperating with the survey has been found to increasewith educational attainment and vary according to occupational category. Moreover,involvement with survey topic is confirmed as a strong predictor of survey participation.Gender, age and experience with theatre remain insignificant in predicting responsebehaviour. These findings are compared with the socio-demographic correlates ofresponse behaviour in general populations. Implications for statistically controllingfor nonresponse bias in audience research are discussed. Suggestions for furtherresearch are presented.     

A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies

The practice of mixed-methods research has increased considerably over the last 10 years. While these studies have been criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been addressed. The purpose of this paper is to identify criteria to critically appraise the quality of mixed-method studies in the health literature. Criteria for critically appraising quantitative and qualitative studies were generated from a review of the literature. These criteria were organized according to a cross-paradigm framework. We recommend that these criteria be applied to a sample of mixed-method studies which are judged to be exemplary. With the consultation of critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers, further efforts are required to revise and prioritize the criteria according to importance.     

Integrating quality into quantity: survey research in the era of mixed methods

As widely recognized during the golden age of survey research thanks to the work of the Columbia school, the use of mixed strategies allows survey research to overcome its limitations by incorporating the advantages of qualitative approaches rather than seeking alternative methods. The need to re-think survey research before embarking on this course impelled the author to undertake a critical analysis of one of the survey’s most important assumptions, proposing a shift from standardization of stimulus to standardization of meanings in order to anchor the requirement of answer comparability on a more solid basis. This proposal for rapprochement with qualitative research is followed by a more detailed section in which the author distinguishes four different types of mixed survey strategies, combining two criteria (time order and function of qualitative procedures). The most significant parts of the constructed typology are then brought together in a model called the multilevel integrated survey approach. This methodological model is concretely illustrated in an empirical study of homophobic prejudice among teenagers. The example shows how in research practice analytical mixed strategies can be creatively combined in the same survey research design, contributing to improvements in data quality and the relevance of research findings.     

The Pervasiveness And Trajectory Of Methodological Choices: A 20‐Year Review Of Human Resource Management Research

This article examines the methodological choices of researchers studying the HR practices–outcome relationship via a content analysis of 281 studies published across the past 20 years. The prevalence and trajectory of change over time are reported for a wide range of methodological choices relevant to internal, external, construct, and statistical conclusion validity. While the results indicate a high incidence of potentially problematic cross‐sectional, single‐informant, and single‐level designs, they also reveal significant improvements over time across many validity relevant methodological choices. This broad‐based improvement in the methodological underpinnings of HR research suggests that researchers and practitioners can view the findings reported in the HR literature with increasing confidence. Directions for future research are provided. © 2016 Wiley Periodicals, Inc.     

A common base for quality control criteria in quantitative and qualitative research

In the social sciences, several scientific paradigms are mutually isolated owing to their use of specific sets of methodological criteria and quality control procedures. In this article, the central hypothesis, to be tested by conceptual analysis and logical reasoning, is that recommended procedures for quality control in quantitative as well as qualitative research can be derived from a common base of regulative ideas. By ‘qualitative’, we mean the complex of ethnographic, anthropological, symbolic interactionist, ethnoscience and related approaches. A second goal is to demonstrate the use of regulative ideas as a parsimonious and fruitful base for a comparative analysis of methodological canons. Although our focus is on the comparison of quantitative and qualitative (or: naturalistic) research, we also pay attention to policy research as opposed to fundamental research.     

A note on campbell's distinction between internal and external validity

The concepts of internal and external validity, developed by Norman Campbell, are widely used to structure methodological thinking about social research. This article points to ambiguities in the interpretation of those terms, both as regards the relationships they refer to as well as the sort of object that is held to be capable of internal and external validity. In addition, it is suggested that the distinction between these types of validity is fundamentally misleading because it reflects a failure to distinguish relations between events and relations between variables. It also rests on the false assumption that we can separate the discovery of causal relationships from the question of whether these apply to other cases than the ones studied. In the final section, an alternative conceptualisation of validity is sketched, one that avoids the problems identified.     

The Experience of Participatory Research: Perceptions of Oncology Employees Participating in a Workplace Study

Participatory research, a concept developed in the Third World, has been increasingly applied to community and health research in developed countries. However, little is known about attitudes to the participatory process in the context of workplace research, especially that carried out in health care settings. In this qualitative study, employees participating in a quality of work-life (QWL) project at a Canadian cancer centre were asked about their perceptions of the participatory research process. Using a phenomenological approach, the author interviewed 12 employees. The following themes emerged from the analysis of interview data: (1) The role of management and senior management was viewed as being important but employees were uncomfortable with the presence of management at meetings; (2) The desired composition of the committee was more complex than ensuring representation from workers and there may have been a natural process by which this composition was attained; (3) Participatory research without action was unacceptable; and (4) Full participation in all aspects of the project was difficult to achieve. These findings have important implications because they challenge some existing notions in the literature about participatory research. Recommendations regarding trust issues, membership recruitment, and the role of members in the participatory process are outlined.     

Challenges for spatially oriented entrepreneurship research

During the past two decades, interdisciplinary oriented entrepreneurship research focused increasingly on spatial aspects of entrepreneurial activities and support policies. This paper takes stock of central themes in entrepreneurship research at and across different geographic scales, the preferred sources of data and information as well as methodological approaches. It sets out to discuss the shifting interest of research over time and to sketch out theoretical and methodological challenges for further research. This paper is based on a review of 18 international journals in small business and entrepreneurship research, economic geography, regional economics and neighbouring sciences for the period 1990–2007. Altogether, 348 relevant articles were identified, read and classified. The analysis reveals that the entrepreneur's socio-spatial contexts in which they operate on a daily basis are still absent from much of the entrepreneurship debate. We suggest intensifying research efforts on the linkage between entrepreneurial activities and localities in order to reach a better understanding of the everydayness of entrepreneurship.     

Interviewing and surveying over the phone: a reflexive account of a research on parenting

This paper is on the role that the phone plays in a mixed-methods empirical research. In the current sociological literature, the phone is a methodological object associated with quantitative fieldwork, while little is said on the phone as a tool for qualitative research. We reflect here on a case study on the use of parental leaves by fathers in Spain, which included the phone as the only communication tool between researchers and subjects. We collect out own experience as interviewers in the qualitative phase of the research project, and interviewed surveyors working in the field. We analyze the discursive data following grounded theory principles and compare the use of both techniques in the data gathering process, including sampling and design. Thus, we consider interviewing and surveying as mediated communication situations. Our findings show that the value of the phone as a tool for research is a matter of adequacy to a given methodology, rather than an issue of validity, reliability, or credibility.     

EMPLOYEE PARTICIPATION IN POLLUTION REDUCTION: A SOCIO-TECHNICAL PERSPECTIVE

Is industrial pollution reduction solely a matter of applying technical solutions to technical problems, or do organizational strategies and human factors also play an important part? In 1991 the US Environmental Protection Agency began collecting data on organizational strategies in pollution control as part of its Toxic Release Inventory database. An exploratory analysis of the 1991–1992 Toxic Release Inventory data revealed evidence that internally based, socially driven organizational strategies such as employee participation can have a dramatic impact on pollution reduction. The results are reported here of the second phase of this research: the approach is broadened to define general ‘internal’ versus ‘external’ strategies, and ‘social’ versus ‘technical’ approaches. There is strong evidence that the combination of internal and external strategies is superior to either alone; there is even stronger evidence that a socio-technical approach is better than either alone, and still stronger support for a combination of all of these criteria.     

“Going to market”: Changing patterns in the organisation and character of process research

This paper explores a meta level question in process research: the implications of the move to a more market based mode of research now evident for the production of process knowledge. It first of all examines the implications for the changing organisation of process research. For example, there has been the emergence of a group of contract process researchers. These researchers may also form alliances with researchers from other disciplines within multidisciplinary teams. Second, it examines implications for the changing character of process knowledge in relation to criteria of internal and external validity. Different effects are evident at different stages at the research cycle, but it is argued that there is no inherent threat to concepts of validity. These trends are exemplified by a project on which the authors are currently engaged. The article concludes with a discussion about the possible future for process research.     

Qualitative methodology and its pitfalls in educational research

There is considerable controversy in educational research with respect to the use of qualitative and quantitative data and as to what constitutes scientific research. The objective of this study is to explore the degree to which in-service teachers understand the difference between qualitative/quantitative data and methods, validity/authenticity, generalization and how these can be used to solve problems faced by the teachers. The study is based on 84 participants who had registered for a 10-week course on Methodology of Investigation in Education, as part of their Master’s degree program. The course is based on 11 readings drawing on a history and philosophy of science perspective (positivism, constructivism, Popper, Kuhn, Lakatos). Course activities included written reports, class room discussions based on participants’ presentations, and written exams. Based on the results obtained it is concluded: (a) Most participants understood that the problem to be investigated precedes the method and determines the methodology to be used; (b) As all observations are theory-laden, it is preferable that interpretations based on both qualitative and quantitative data be allowed to compete in order to provide validity to our research findings; (c) The difference between validity and authenticity was controversial and most participants considered the need for interpreting data and hence favored authenticity; (d) Discussions led to the idea of ‘degrees of validity’ as both validity in the quantitative sense and authenticity in the qualitative sense, ultimately depend on critical appraisals of the community; (e) Generalizability of results obtained from qualitative studies was a controversial topic and most participants agreed that it is not desirable to generalize; (f) Discussions suggested an alternative: In both qualitative and quantitative research generalizability is possible, provided we are willing to grant that our conceptions/theories are not entirely grounded in empirical evidence but rather on the degree to which the community can uphold such a consensus; (g) Most teachers considered the use of participant observation in qualitative research as non-controversial. Class discussions led to the understanding that emphasizing observations may lead us to the Aristotelian ideal of empirical science; (h) Formulation of hypotheses, manipulation of variables, and the quest for causal variables was considered by many teachers to be equivalent to the scientific method. Discussions facilitated the understanding that this led to idealization and thus helped to reduce the complexity of a problem.     

Linking quality & quantity

A fault of many quantitative studies is the inadequate fit between the real-world processes and the formal aspects of the quantitative research designs as seen in their accounting schemes, measurements, and analysis plans. The examples reviewed in this article show how to ameliorate this problem by following the hermeneutically informed strategy of formalization of qualitative texts. Qualitative studies often possess external validity but lack internal validity. Quantitative studies often possess internal validity but lack external validity. By building quantitative studies on formalizations of the qualitative, the resulting research is more likely to possess both external and internal validity. This strategy of formalization will help to avoid two faults in present-day social science — sterility and discontinuity — and will promote the achievement of a dynamic continunity in social research.     

Systematic social observation of police decision-making: the process, logistics, and challenges in a Canadian context

Research on police decision-making and the use of discretion predominantly relies on official records or qualitative interview and participant observation data. Pioneered by Albert J. Reiss, Jr. in the 1960s, systematic social observation is a field research method simultaneously gathering quantitative and qualitative data in a natural setting. Data collection procedures are stipulated in advance reducing bias and allowing for scientific inference, replication, and measurable process outcomes as the observations recorded are independent of the observer. Although three large scale studies have been conducted in the United States to understand police behaviour, this ‘gold standard’ is used infrequently due to methodological challenges in an applied setting. The current research is the first sole observer and non-U.S. study conducted with a Canadian regional police service. Discussion of methodological innovation, challenges, and process are based on the experience of collecting data on 406 police–citizen encounters involving 568 citizens over 637 observational hours. To assist future researchers, the method and data collection using modified instruments as well as the challenges of resources, police cooperation, bias, reactivity, field role, and dynamics of conducting research as a female in an insular, male-dominated setting are explored.