Mixed methods research synthesis: definition, framework, and potential

Literature on the combination of qualitative and quantitative research components at the primary empirical study level has recently accumulated exponentially. However, this combination is only rarely discussed and applied at the research synthesis level. The purpose of this paper is to explore the possible contribution of mixed methods research to the integration of qualitative and quantitative research at the synthesis level. In order to contribute to the methodology and utilization of mixed methods at the synthesis level, we present a framework to perform mixed methods research syntheses (MMRS). The presented classification framework can help to inform researchers intending to carry out MMRS, and to provide ideas for conceptualizing and developing those syntheses. We illustrate the use of this framework by applying it to the planning of MMRS on effectiveness studies concerning interventions for challenging behavior in persons with intellectual disabilities, presenting two hypothetical examples. Finally, we discuss possible strengths of MMRS and note some remaining challenges concerning the implementation of these syntheses.     

QUALITATIVE AND MIXED‐METHODS RESEARCH IN ECONOMICS: SURPRISING GROWTH,PROMISING FUTURE

Qualitative research in economics has traditionally been unimportant compared to quantitative work. Yet there has been a small explosion in use of quantitative approaches in the past 10–15 years, including ‘mixed‐methods’ projects which use qualitative and quantitative methods in combination. This paper surveys the growing use of qualitative methods in economics and closely related fields, aiming to provide economists with a useful roadmap through major sets of qualitative methods and how and why they are used. We review the growing body of economic research using qualitative approaches, emphasizing the gains from using qualitative‐ or mixed‐methods over traditional ‘closed‐ended’ approaches. It is argued that, although qualitative methods are often portrayed as less reliable, less accurate, less powerful and/or less credible than quantitative methods, in fact, the two sets of methods have their own strengths, and how much can be learned from one type of method or the other depends on specific issues that arise in studying the topic of interest. The central message of the paper is that well‐done qualitative work can provide scientifically valuable and intellectually helpful ways of adding to the stock of economic knowledge, especially when applied to research questions for which they are well suited.     

The use of mixed methods research in the field of behavioural sciences

Mixed methods research involves the combined use of quantitative and qualitative methods in the same research study, and it is becoming increasingly important in several scientific areas. The aim of this paper is to review and compare through a mixed methods multiple-case study the application of this methodology in three reputable behavioural science journals: the Journal of Organizational Behavior, Addictive Behaviors and Psicothema. A quantitative analysis was carried out to review all the papers published in these journals during the period 2003–2008 and classify them into two blocks: theoretical and empirical, with the latter being further subdivided into three subtypes (quantitative, qualitative and mixed). A qualitative analysis determined the main characteristics of the mixed methods studies identified, in order to describe in more detail the ways in which the two methods are combined based on their purpose, priority, implementation and research design. From the journals selected, a total of 1,958 articles were analysed, the majority of which corresponded to empirical studies, with only a small number referring to research that used mixed methods. Nonetheless, mixed methods research does appear in all the behavioural science journals studied within the period selected, showing a range of designs, where the sequential equal weight mixed methods research design seems to stand out.     

Empirical research consolidation: a generic overview and a classification scheme for methods

I define research consolidation as comparing and combining (amalgamating or, at least, juxtaposing) results or other output from multiple previously conducted research activities, relevant to a goal or topic of interest. The concept, not limited to empirical research or any specific method, is similar to and subsumes what is often referred to as research synthesis or research integration; I explain the concept and my rationale for the new term. After introducing the broader concept, I focus on consolidation of empirical research. As background for this, I offer a brief introduction to empirical research. Then I provide a generic overview of empirical research consolidation, abstracted from several of its specific methods, such as quantitative meta-analysis and qualitative meta-synthesis; this sort of overview—method agnostic and capturing the commonality—is not readily available in the literature. At the core of the paper, I propose a scheme to classify methods for empirical research consolidation followed by a review and classification of selected methods, illustrating the scheme. I also discuss and clarify related terminology. The classification scheme differentiates between two major attributes of methods—consolidation technique and type of content to be consolidated—that may reflect the positivist or constructivist research paradigm. Consolidation techniques could be aggregative, interpretive, or a combination thereof. Content consolidated, from empirical research reports, could be empirical or not and could be from quantitative research, qualitative research, other kinds of empirical research, or a combination thereof.     

Accreditation and standards-driven program evaluation: implications for program quality assurance and stakeholder professional development

This study evaluates a language education program and assesses the influence of this program evaluation on program performance and stakeholder professional development. The research design makes use of mixed paradigms and methods. Positivism is followed to verify the extent to which the program meets predetermined quality standards through quantitative evaluation research, questionnaires and document and record analysis. In contrast, the qualitative paradigm is followed to uncover program evaluation impact on program components and stakeholders through qualitative evaluation and interviews. The study concludes the program maintains satisfactory overall performance with some components performing better than others. Likewise, evaluation of program evaluation improves program elements and faculty and program administration professional skills. The study therefore recommends program evaluation as an effective program improvement strategy. Program evaluation is also an effective professional development strategy that provides program administration, staff and faculty members with onsite opportunities to develop their professional skills. Moreover, evaluation of program evaluation should be an integral part of program evaluation to help program stakeholders not only do but also use evaluation alongside decreasing resistance to imposed review and reform.     

Representations of Australia in South Korean online news: a qualitative and quantitative approach utilizing Leximancer and Korean keywords in context

This article outlines original empirical research using a quantitative computer-assisted big data approach to survey and evaluate the representation of Australia in popular South Korean online news media sources. This is an exploratory content analysis of news reporting (in Korean and English) on Australia in key Korean digital news media providers and distributors over a six-month-period during which certain key events occurred that drew Korean news media attention to Australia. The research aims to address the fundamental question: what forms and patterns of representation of Australia are present in South Korean online news media? It is designed in large part to understand how Australia is present as a theme in Korean online news, and how this theme is contextualized by associated topics, such as trade, security, or tourism. The quantitative analysis is augmented using a qualitative method: a series of key informant interviews conducted with editors and journalists responsible for the production and prioritization of news. Further, it is interpreted by locating the research in the context of the Australia-South Korea relationship. We summarize the complexities that arise when analyzing large amounts of textual content, especially from two very different languages, discuss the opportunities that arise from mixed methods—by combining big data (breadth) with contextual analysis (depth) and offer suggestions for aspiring researchers.     

Taking the “Q” Out of Research: Teaching Research Methodology Courses Without the Divide Between Quantitative and Qualitative Paradigms

The purpose of this paper is to provide evidence that the debate between quantitative and qualitative is divisive and, hence, counterproductive for advancing the social and behavioral science field. We advocate that all graduate students learn to utilize and to appreciate both quantitative and qualitative research methodologies. As such, students will develop into pragmatist researchers who are able to utilize both quantitative and qualitative techniques when conducting research. We contend that the best way to accomplish this is by eliminating quantitative research methodology and qualitative research methodology courses from curricula and replacing these with research methodology courses at different levels that simultaneously teach both quantitative and qualitative techniques within a mixed methodological framework.An earlier version of this article received the 2003 Southwest Educational Research Association (SERA) Outstanding Paper Award.     

Mixed Methods Studies in Environmental Management Research: Prevalence,Purposes and Designs

Mixed methods research, that is, the combined use of quantitative and qualitative methods in the same study, is a popular approach in several fields. However, its use in the field of environmental management and sustainable development has not been examined. The authors review the application of mixed methods research in the journal Business Strategy and the Environment in the last ten years. A total of 340 articles have been read and reviewed, and 26 mixed methods studies have been identified. The main purposes and mixed methods designs are examined. Suggestions on why and how to use mixed methods research are offered, and recommendations are provided to guide future mixed methods studies to advance our understanding of environmental management topics. Copyright © 2014 John Wiley & Sons, Ltd and ERP Environment     

A common base for quality control criteria in quantitative and qualitative research

In the social sciences, several scientific paradigms are mutually isolated owing to their use of specific sets of methodological criteria and quality control procedures. In this article, the central hypothesis, to be tested by conceptual analysis and logical reasoning, is that recommended procedures for quality control in quantitative as well as qualitative research can be derived from a common base of regulative ideas. By ‘qualitative’, we mean the complex of ethnographic, anthropological, symbolic interactionist, ethnoscience and related approaches. A second goal is to demonstrate the use of regulative ideas as a parsimonious and fruitful base for a comparative analysis of methodological canons. Although our focus is on the comparison of quantitative and qualitative (or: naturalistic) research, we also pay attention to policy research as opposed to fundamental research.     

Towards a flexible online mixed method design with a feedback loop

The notion of mixed methods design relates to the research studies that combine qualitative and quantitative approaches. However, most of these studies are tailored to specific research problem in a particular study and are typically limited to a fixed sequence of qualitative and quantitative approaches (e.g. qualitative interviews followed by a survey or vice-versa). This limitation historically arises from time, cost and logistic restrictions. As an alternative, we develop an general extension of fixed mixed method design by introducing a flexible feedback-loop so that several phases can be combined in a flexible order. In practice, such designs are now increasingly feasible within an information-communication technology environment, where online respondents are readily available for immediate participation. An online experiment combining interactive series of web surveys and in-depth web interviews was performed to compare this approach with standard two-phase designs involving mixed methods. The costs, timing, quality of finiding and experiences of researchers were systematically evaluated. In summary, the proposed approach proved to be very beneficial.     

Generalization practices in qualitative research: a mixed methods case study

The purpose of this mixed methods case study was to examine the generalization practices in qualitative research published in a reputable qualitative journal. In order to accomplish this, all qualitative research articles published in Qualitative Report since its inception in 1990 (n =  273) were examined. A quantitative analysis of the all 125 empirical qualitative research articles revealed that a significant proportion (i.e., 29.6%) of studies involved generalizations beyond the underlying sample that were made inappropriately by the author(s). A qualitative analysis identified the types of over-generalizations that occurred, which included making general recommendations for future practice and providing general policy implications based only on a few cases. Thus, a significant proportion of articles published in Qualitative Report lack what we call interpretive consistency.     

The distinction between qualitative and quantitative research methods is problematic

The distinction between qualitative and quantitative research is abstract, very general and its value is usually taken for granted. In contrast, this article attempts to show that the distinction between qualitative and quantitative research is unclear, poor and therefore of limited value and that its popularity risks leading to unfortunate consequences. Various arguments are presented for this conclusion. For example, it is argued that the heterogeneity of different stand-points on important issues among qualitative researchers (for example with respect to the use of quantification and causal analysis) makes the distinction as such unstable. Moreover, the presence of substantial overlap between many features of qualitative and quantitative research often makes it difficult to separate qualitative and quantitative research. It is also shown that three obvious ways of making the distinction between qualitative and quantitative research are unsatisfactory. Use of the distinction may restrict creativity in the development of new research methods and create confusion and unnecessary work. In general, it may be preferable not to conceptualize research approaches at such abstract levels as done in the context of qualitative or quantitative approaches. Instead, it is suggested that it is more fruitful to discuss the pros and cons of specific research methods, preferably in the context of specific research problems.     

Mixed‐frequency models with moving‐average components

Temporal aggregation in general introduces a moving‐average (MA) component in the aggregated model. A similar feature emerges when not all but only a few variables are aggregated, which generates a mixed‐frequency (MF) model. The MA component is generally neglected, likely to preserve the possibility of ordinary least squares estimation, but the consequences have never been properly studied in the MF context. In this paper we show, analytically, in Monte Carlo simulations and in a forecasting application on US macroeconomic variables, the relevance of considering the MA component in MF mixed‐data sampling (MIDAS) and unrestricted MIDAS models (MIDAS–autoregressive moving average (ARMA) and UMIDAS‐ARMA). Specifically, the simulation results indicate that the short‐term forecasting performance of MIDAS‐ARMA and UMIDAS‐ARMA are better than that of, respectively, MIDAS and UMIDAS. The empirical applications on nowcasting US gross domestic product (GDP) growth, investment growth, and GDP deflator inflation confirm this ranking. Moreover, in both simulation and empirical results, MIDAS‐ARMA is better than UMIDAS‐ARMA.     

Causality in qualitative and quantitative research

We are flooded with a wave of writings on causality in the social sciences during the last decades. The same holds for the relationship between quantitative and qualitative research in the social sciences. An enormous amount of texts appears on (causality in) qualitative research, mostly in a controversy with quantitative research. These writings induced us to develop the thesis of “unity in diversity”, i.e., that there is no difference “in principle” between causality in qualitative and quantitative research, because both are supported by what I will call an “experimental logic”. In developing this thesis a plea is being made for going back to the sources. A historical overview of theories of causality is presented, which develops into two prominent views: INUS-causation and causal realism. A historical framework is also outlined for the opposition between quantitative and qualitative research, in which French positivism and British empiricism are opposed to German neo-kantianism and neo-hegelianism. After having developed the thesis of “unity in diversity” for this historical framework, the same is being done for the recent literature: “mixed methods research”, the book DSI of KKV, the reactions of David Collier and “QCA” of Charles Ragin. At the end the question of small-n research and the case n = 1 is examined.     

Integrating Quantitative and Qualitative Analysis Using Latent and Manifest Variables

The relative virtues of quantitative and qualitative research have been vigorously debated. Several researchers recommend combining methodologies but there is little evidence in the literature to suggest how different research methods might be integrated (Bryman, 1988). The current study addresses this deficiency in the research by examining the use of latent variables in quantitative and qualitative research as a means of blending the two approaches. A study of entrepreneurial Locus of Control where quantitative and qualitative data were available illustrates the methodological issues. Analysis of quantitative data was conducted using LISREL (7.20) and qualitative data were categorised using NUD.IST (Non-numerical Unstructured Data Indexing Searching and Theorising computer software). Detailed comparisons are made between the methods described in this paper and other approaches to content analysis.     

Bayesian data augmentation methods for the synthesis of qualitative and quantitative research findings

The possible utility of Bayesian methods for the synthesis of qualitative and quantitative research has been repeatedly suggested but insufficiently investigated. In this project, we developed and used a Bayesian method for synthesis, with the goal of identifying factors that influence adherence to HIV medication regimens. We investigated the effect of 10 factors on adherence. Recognizing that not all factors were examined in all studies, we considered standard methods for dealing with missing data and chose a Bayesian data augmentation method. We were able to summarize, rank, and compare the effects of each of the 10 factors on medication adherence. This is a promising methodological development in the synthesis of qualitative and quantitative research.     

Family Formation and the Divisions of Labour in Belgian Couple Households

In this chapter the authors describe how a quantitative analysis resulted in a proposal for a qualitative research. The aim of the quantitative research was to analyse the impact of family formation on the division of labour in couple households, based on data from the Panel Study on Belgian Households.In a first step a typology was made based upon data of 1224 Belgian couple households (1995). The second step consisted of a longitudinal analysis for the period between 1992 and 1995. The analysis of the impact of birth on the division of labour gave evidence of the revocability of equal patterns of division of labour.Based on these findings the authors concluded that the classic concept of emancipation does not allow us to understand the empirical phenomena. Hence, the authors proposed a new (qualitative) investigation aimed at clarification and empirical validation of different contents of the concept of emancipation.     

Qualitative methodology and its pitfalls in educational research

There is considerable controversy in educational research with respect to the use of qualitative and quantitative data and as to what constitutes scientific research. The objective of this study is to explore the degree to which in-service teachers understand the difference between qualitative/quantitative data and methods, validity/authenticity, generalization and how these can be used to solve problems faced by the teachers. The study is based on 84 participants who had registered for a 10-week course on Methodology of Investigation in Education, as part of their Master’s degree program. The course is based on 11 readings drawing on a history and philosophy of science perspective (positivism, constructivism, Popper, Kuhn, Lakatos). Course activities included written reports, class room discussions based on participants’ presentations, and written exams. Based on the results obtained it is concluded: (a) Most participants understood that the problem to be investigated precedes the method and determines the methodology to be used; (b) As all observations are theory-laden, it is preferable that interpretations based on both qualitative and quantitative data be allowed to compete in order to provide validity to our research findings; (c) The difference between validity and authenticity was controversial and most participants considered the need for interpreting data and hence favored authenticity; (d) Discussions led to the idea of ‘degrees of validity’ as both validity in the quantitative sense and authenticity in the qualitative sense, ultimately depend on critical appraisals of the community; (e) Generalizability of results obtained from qualitative studies was a controversial topic and most participants agreed that it is not desirable to generalize; (f) Discussions suggested an alternative: In both qualitative and quantitative research generalizability is possible, provided we are willing to grant that our conceptions/theories are not entirely grounded in empirical evidence but rather on the degree to which the community can uphold such a consensus; (g) Most teachers considered the use of participant observation in qualitative research as non-controversial. Class discussions led to the understanding that emphasizing observations may lead us to the Aristotelian ideal of empirical science; (h) Formulation of hypotheses, manipulation of variables, and the quest for causal variables was considered by many teachers to be equivalent to the scientific method. Discussions facilitated the understanding that this led to idealization and thus helped to reduce the complexity of a problem.     

Alternative quality standards in qualitative research?

Qualitative researchers often use other principles for judging the quality of their study than quantitative researchers. This inhibits a straightforward assessment of the quality and comparability of different types of studies, as well as decision-making about their usefulness for further research and practice. In this article, we question the use of alternative criteria, and argue for one coherent and inclusive framework of quality criteria for both qualitative and quantitative studies. We developed such a framework, based on a comparison and operationalization of different criteria, and the elaboration of procedures to realize them. Its usability is demonstrated by the application in a qualitative case study research and a mixed-methods study. This framework enhances advancing and judging the basic quality of any type of scientific research, promoting the assessment of the quality and hence usability of studies for further research or decision-making about practice.     

HR-related facets and their relationship to organizational effectiveness: a faceted definition

Human resource (HR) professionals and employers perceive different practices as affecting organizational effectiveness. Consequently, the question is what are the components of these practices and why do they relate to organizational effectiveness. To achieve this goal, we employed the facet analytic approach and developed a faceted definition of organizational effectiveness comprising HR-related facets. Thus various facets were reviewed and incorporated into an overall faceted definition of the universe of organizational effectiveness. Future empirical research based on this faceted definition should be conducted with the objective of examining the extent to which it is tenable or alternatively, deciding upon the changes that should be introduced in order to make this definitional framework more accurate.