Assumptions Behind Inter-Coder Reliability Indices

Xinshu Zhao, Jun S. Lin, Ke Deng

Research output: Chapter in book/report/conference proceedingChapter

162 Downloads (Pure)


Intercoder reliability is the most often used quantitative indicator of measurement quality in content studies. Researchers in psychology, sociology, education, medicine, marketing and other disciplines also use reliability to evaluate the quality of diagnosis, tests and other assessments. Many indices of reliability have been recommended for general use. This article analyzes 22, which are organized into 18 chance-adjusted and four non-adjusted indices. The chance-adjusted indices are further organized into three groups, including nine category-based indices, eight distribution-based indices, and one that is double based, on category and distribution.

The main purpose of this work is to examine the assumptions behind each index. Most of the assumptions are unexamined in the literature, and yet these assumptions have implications for assessments of reliability that need to be understood, and that result in paradoxes and abnormalities. This article discusses 13 paradoxes and nine abnormalities to illustrate the 24 assumptions. To facilitate understanding, the analysis focuses on categorical scales with two coders, and further focuses on binary scales where appropriate. The discussion is situated mostly in analysis of communication content. The assumptions and patterns that we will discover will also apply to studies, evaluations and diagnoses in other disciplines with more coders, raters, diagnosticians, or judges using binary or multi-category scales.

We will argue that a new index is needed. Before the new index can be established, we need guidelines for using the existing indices. This article will recommend such guidelines.

Original languageEnglish
Title of host publicationCommunication Yearbook
EditorsCharles T. Salmon
Place of PublicationNew York ; London
Number of pages63
ISBN (Electronic)9780203113653
ISBN (Print)9780415525480
Publication statusPublished - 9 Jul 2012


Dive into the research topics of 'Assumptions Behind Inter-Coder Reliability Indices'. Together they form a unique fingerprint.

Cite this