Factors affecting intercoder reliability: a Monte Carlo experiment |
| |
Authors: | Guangchao Charles Feng |
| |
Affiliation: | 1. School of Communication, Hong Kong Baptist University, Communication and Visual Arts Building, 5 Hereford Road, Kowloon Tong, Kowloon, Hong Kong
|
| |
Abstract: | Although it has long been a consensus that intercoder reliability is crucial to the validity of a content analysis study, the choice among them has been debated. This study reviewed and empirically tested most popular intercoder reliability indices, aiming to find the most robust index against prevalence and rater bias, by empirically testing their relationships with response surface methodology through a Monte Carlo experiment. It was found that Maxwell’s R.E is superior to Krippendorff’s α, Scott’s π, Cohen’s κ, I r of Perreault and Leigh, and Gwet’s AC 1. More nuanced relationships among prevalence, sensitivity, specificity and the intercoder reliability indices were discovered through response surface plots. Both theoretical and practical implications were also discussed in the end. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|