Agreement and consensus in a Q-mode research design: an empirical comparison of measures,and an application

Subject of this paper is the analysis of consensus within small groups of respondents, based on a proportionally large number of variables. The target group is researchers who are interested in Q-mode research. Measures of agreement are compared, and an application from a recent project is presented. Cohen’s κ is the preferable measure, Krippendorff’s α is an alternative, which is based on a different concept of expected disagreement. At group level, along with κ and α for multiple raters, additional measures are r _wg, intraclass correlation, and κ _SC. Predictions about level differences between groups can be assessed by a t-test and θ