Two Liberal-Conservative Hierarchies for Indices of Intercoder Reliability

Inter-coder reliability is the most often used quantitative indicator of measurement quality in content studies. Researchers in psychology, sociology, education, medicine, marketing and other social science disciplines also use reliability to evaluate the quality of diagnosis, tests and other assessments. More than a dozen indicators of reliability have been introduced for general use.

This article analyzes 18 indices of inter-coder reliability, including the first introduced -- percent agreement (ao) and Benini’s β (1901), the most often “reinvented” -- Guttman’s ρ (1946) and Bennett et al’s S (1954), the most often used -- Scott’s π (1955) and Cohen’s κ (1960), the most strongly advocated-- Krippendorff’s α (1970), and the most recently introduced -- Gwet’s AC1 (2008).

The analysis uncovered two liberal-conservative continuum for the indices analyzed, which may be useful for content analysts who wish to better evaluate the inter-coder reliability indices of their instruments.