Factors affecting intercoder reliability: A Monte Carlo experiment

Research output: Contribution to journalJournal articlepeer-review

34 Citations (Scopus)


Although it has long been a consensus that intercoder reliability is crucial to the validity of a content analysis study, the choice among them has been debated. This study reviewed and empirically tested most popular intercoder reliability indices, aiming to find the most robust index against prevalence and rater bias, by empirically testing their relationships with response surface methodology through a Monte Carlo experiment. It was found that Maxwell’s R.E is superior to Krippendorff’s α, Scott’s π, Cohen’s κ, I r of Perreault and Leigh, and Gwet’s AC 1. More nuanced relationships among prevalence, sensitivity, specificity and the intercoder reliability indices were discovered through response surface plots. Both theoretical and practical implications were also discussed in the end.
Original languageEnglish
Pages (from-to)2959-2982
Number of pages24
JournalQuality and Quantity
Publication statusPublished - Aug 2013

User-Defined Keywords

  • Intercoder reliability
  • Prevalence
  • Simulation


Dive into the research topics of 'Factors affecting intercoder reliability: A Monte Carlo experiment'. Together they form a unique fingerprint.

Cite this