Effect Sizes in Qualitative Research: A Prolegomenon

The American Psychological Association Task Force recommended that researchers always report and interpret effect sizes for quantitative data. However, no such recommendation was made for qualitative data. Thus, the first objective of the present paper is to provide a rationale for reporting and interpreting effect sizes in qualitative research. Arguments are presented that effect sizes enhance the process of verstehen/hermeneutics advocated by interpretive researchers. The second objective of this paper is to provide a typology of effect sizes in qualitative research. Examples are given illustrating various applications of effect sizes. For instance, when conducting typological analyses, qualitative analysts only identify emergent themes; yet, these themes can be quantitized to ascertain the hierarchical structure of emergent themes. The final objective is to illustrate how inferential statistics can be utilized in qualitative data analyses. This can be accomplished by treating words arising from individuals, or observations emerging from a particular setting, as sample units of data that represent the total number of words/observations existing from that sample member/context. Heuristic examples are provided to demonstrate how inferential statistics can be used to provide more complex levels of verstehen than is presently undertaken in qualitative research.     

Generalization practices in qualitative research: a mixed methods case study

The purpose of this mixed methods case study was to examine the generalization practices in qualitative research published in a reputable qualitative journal. In order to accomplish this, all qualitative research articles published in Qualitative Report since its inception in 1990 (n =  273) were examined. A quantitative analysis of the all 125 empirical qualitative research articles revealed that a significant proportion (i.e., 29.6%) of studies involved generalizations beyond the underlying sample that were made inappropriately by the author(s). A qualitative analysis identified the types of over-generalizations that occurred, which included making general recommendations for future practice and providing general policy implications based only on a few cases. Thus, a significant proportion of articles published in Qualitative Report lack what we call interpretive consistency.     

A Call for Qualitative Power Analyses

The purpose of this paper is to emphasize the importance of sampling and sample size considerations in all qualitative research. Such considerations would help qualitative researchers to select sample sizes and sampling designs that are most compatible with their research purposes. First, we discuss the importance of sampling in qualitative research. Next, we outline 24 designs for selecting a sample in qualitative research. We then discuss the importance of selecting a sample size that yields data that have a realistic chance of reaching data saturation, theoretical saturation, or informational redundancy. Based on the literature, we then provide sample size guidelines for several qualitative research designs. As such, we provide a framework for making sampling and sample size considerations in interpretive research. An earlier version of this article received the 2004 Southwest Educational Research Association (SERA) Outstanding Paper Award.     

A Framework for Making Quantitative Educational Research Articles More Reader-Friendly for Practitioners

The purpose of the present article is to demonstrate how quantitative research articles can be made much more reader-friendly. In particular, we illustrate how statistical language and research terminology can be simplified in reports. Moreover, using a published article, we demonstrate how quantitative reports could be re-structured to make them more reader-friendly, without sacrificing any important statistical information. We contend that by restructuring these reports, practitioners and stakeholders would be in a much better position to read quantitative research articles, whose findings could then be utilized to improve the quality of education. As such, not only would the divide between researchers and practitioners be reduced, but also educational research studies would have a much bigger impact on schools.     

Revisiting the difference between mixed methods and multimethods: Is it all in the name?

The literature on mixed methods and multimethods has burgeoned over the last 20 years, and researchers from a growing number and diversity of fields have progressively embraced these approaches. However, rapid growth in any movement inevitably gives rise to gaps or shortcomings, such as “identity crises” or divergent conceptual views. Although some authors draw a clear and sometimes opinionated distinction between mixed methods and multimethods, for others, they are synonymous. The concepts underlying both terms therefore have become blurred and generated much confusion. The aim of this article is to explore the origins of the confusion, describe our view of mixed methods and multimethod studies, and by doing so, help to clearly delineate the two concepts. The authors have presented their opinion of how these terms and concepts should be distinguished and call for a constructive debate of the issues involved in the mixed methods and multimethod literature. This is a way truly to propel the field forward.     

A typology of mixed methods research designs

The mixed methods paradigm is still in its adolescence, and, thus, is still relatively unknown and confusing to many researchers. In general, mixed methods research represents research that involves collecting, analyzing, and interpreting quantitative and qualitative data in a single study or in a series of studies that investigate the same underlying phenomenon. Over the last several years, a plethora of research designs have been developed. However, the number of designs that currently prevail leaves the doctoral student, the beginning researcher, and even the experienced researcher who is new to the field of mixed methods research with the challenge of selecting optimal mixed methods designs. This paper presents a three-dimensional typology of mixed methods designs that represents an attempt to rise to the challenge of creating an integrated typology of mixed methods designs. An example for each design is included as well as a notation system that fits our eight-design framework. This paper won the James E. McLean outstanding paper award.     

A meta-validation model for assessing the score-validity of student teaching evaluations

Virtually every institution of higher education in the US uses some type of student teaching evaluation (STE) instrument as a means of assessing instructors’ instructional performance in courses. Unfortunately, many administrators and faculty misinterpret STE ratings. Therefore, the present article provides a comprehensive critique of STE instruments. In particular, we build on Messick’s (Educational Measurement, MacMillan, pp. 13–103, and Messick (Am. Psychol., 50, 741–749, 1995, 1989) conceptualization of validity to yield what we refer to as a meta-validity model that subdivides content-, criterion-, and construct-related validity into several areas of evidence. We use our meta-validity model to conduct a meta-validity analysis of STEs. Specifically, we assessed the score-validity of STEs based on findings from the extant literature. We conclude that strong evidence has been provided with respect to areas of criterion-related validity; however, for the most part, weak or inadequate evidence has been provided with regard to areas of both content-related and construct-related validity. This seriously calls into question both the score-validity and utility of STEs.     

Assessing legitimation in mixed research: a new framework

In this article, we have merged or intersected two typologies: Greene’s (Res Sch 13(1):93–98, 2006) four-domain typology for developing a methodological or research paradigm in the social and behavioral sciences and Onwuegbuzie and Johnson’s (Res Sch 13(1):48–63, 2006) nine-component typology for assessing mixed research legitimation. We argue that merging or interconnecting these typologies present a framework for assessing legitimation in mixed research. Specifically, we demonstrate how the nine types of legitimation map onto Greene’s (Res Sch 13(1):93–98, 2006) four methodological domains and illustrate how legitimation in mixed research, rather than being viewed as a procedure that occurs at a specific step of the mixed research process, is better conceptualized as a continuous iterative, interactive, and dynamic process. Additionally, in presenting this framework, we hope to reduce misperceptions that some researchers have voiced about mixed research.     

Validity and Qualitative Research: An Oxymoron?

Although the importance of validity has long been accepted among quantitative researchers, this concept has been an issue of contention among qualitative researchers. Thus, the first purpose of the present paper is to introduce the Qualitative Legitimation Model, which attempts to integrate many of the types of validity identified by qualitative researchers. The second purpose of this article is to describe 24 methods for assessing the truth value of qualitative research. Utilizing and documenting such techniques should prevent validity and qualitative research from being seen as an oxymoron.