Reliability & agreement · ~7 min read
Cohen's Kappa: Inter-Rater Agreement
Chance agreement climbs whenever one category dominates, and kappa falls with it even when the raters are reading cases identically.
In short
Cohen's kappa measures how much two raters agree on a categorical outcome after subtracting the agreement they would get by chance alone — the correction percent agreement never makes.1 Because chance agreement rises automatically whenever one category dominates, the same raw agreement can mean very different things: two raters can match on 92% of cases and still land at a kappa around 0.29, "fair" by the usual benchmarks, simply because almost every case fell into the same category regardless of who was rating.23 For ordinal categories — mild, moderate, severe grading, for example — weighted kappa applies the same chance correction while giving partial credit for near-misses instead of scoring every disagreement as equally wrong.4 Report the raw agreement, the kappa, and the category split together; none of the three tells the full story on its own.
Percent agreement flatters you
Two chart reviewers independently classify whether each patient had a surgical site infection at 30 days: yes or no. They agree on 92 of 100 charts. Ninety-two percent sounds like strong reliability, and it is the number most manuscripts report, because it is the number that is easiest to compute and easiest to defend to a reader.
It is also the number that never asks the question that actually matters: how much of that agreement would two raters get by guessing? If a complication is rare, both reviewers can call "no complication" on almost every chart without ever looking closely, and still rack up a high agreement rate purely because the majority category is easy to land on by default. Percent agreement cannot tell the difference between two raters who are genuinely reading the same charts the same way and two raters who are both defaulting to the common answer.
Cohen's kappa was built specifically to close that gap. It compares the observed agreement to the agreement expected from the raters' own marginal tendencies to pick each category, and reports what is left over once chance is subtracted out.1
The paradox: when high agreement produces a mediocre kappa
Take the 92-charts example further (the numbers here are illustrative, not a real dataset). Suppose reviewer A calls 6 of the 100 charts "infection present" and reviewer B calls 6 as well, and of those, only 2 charts are ones both reviewers flagged. The rest of the agreement — 90 of the 100 charts — is both reviewers agreeing "no infection." Observed agreement is 92%. But because both reviewers are calling the rare category on only about 6% of charts, most of that 92% agreement is two people independently landing on the easy, common answer. The expected chance agreement here works out to about 88.7%, and kappa is the gap between the two, rescaled: (0.92 − 0.887) ÷ (1 − 0.887) ≈ 0.29 — "fair," not the "almost perfect" the raw percentage implies.
This is a documented and well-described property of kappa, not a quirk of one made-up example: kappa is highly sensitive to how the outcome is distributed across categories, and skewed prevalence can produce a low kappa even when raw agreement is high.2 The reverse also happens — kappa can look artificially high when categories are split close to 50/50, even with mediocre rater agreement.
A low kappa is not automatically a rater problem. Before concluding your raters disagree badly, check how the outcome is split. A rare or very common category inflates the "chance" baseline kappa corrects for, which mechanically drags kappa down even when raters are reading cases the same way. Report the category prevalence alongside kappa so a reader can tell the two apart.
A real result: two ways of asking the same question
A 1975 study of schoolchildren's smoking beliefs asked the same 94 children "Have you ever smoked a cigarette?" twice — once by self-administered questionnaire, once by interview — and compared the two sets of yes/no answers as a rater-agreement problem.5 Observed agreement was 91.5%. Expected chance agreement, calculated from how often each method produced a "yes," was 57.2%. Kappa was (0.915 − 0.572) ÷ (1 − 0.572) ≈ 0.80 — "almost perfect" by the standard benchmark, and this time the raw agreement and the chance-corrected agreement tell roughly the same story, because neither answer category was overwhelmingly dominant.
The contrast with the SSI example above is the point: the same-looking raw agreement (91–92%) produced a kappa of 0.80 in one case and 0.29 in the other, purely as a function of how the categories were distributed.
| Kappa | Interpretation |
|---|---|
| < 0.00 | Poor |
| 0.00–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Weighted kappa: not all disagreement is equal
Standard kappa treats every disagreement identically — two raters grading a tumour as grade I versus grade III counts exactly the same as grade I versus grade II, even though one is a much bigger miss than the other. For ordinal categories, that is the wrong penalty structure. Weighted kappa, introduced by Cohen as an extension of the original statistic, applies weights (commonly linear or quadratic) so that categories one step apart cost less than categories several steps apart, while still applying the same chance correction underneath.4 It is the version to report whenever the categories being rated have a genuine order — pain scores, tumour grade, radiographic severity scales — rather than unordered labels like tissue type.
What to actually report
The Landis and Koch ranges above are the ones every reviewer recognises, but they were proposed as a convenience, not derived from data, and later authors have argued they are too lenient for health research — a kappa of 0.41, sitting at the bottom of "moderate," is not obviously an acceptable level of rater agreement for a clinical classification.6 Cite them if you use them, but do not treat them as a pass/fail line.
What actually lets a reader judge your inter-rater agreement is three numbers together: the raw percent agreement, the kappa (weighted, if the categories are ordinal, and state which weighting scheme), and the split of cases across categories. Given all three, a reviewer can tell whether a modest kappa reflects genuinely inconsistent raters or a rare outcome doing exactly what rare outcomes do to this statistic.
How to run it — SPSS vs StatsPlease
In current IBM SPSS Statistics (version 32), unweighted kappa runs through Analyze ▸ Descriptive Statistics ▸ Crosstabs. Put one rater's classifications into the Row(s) box and the other's into Column(s) — SPSS requires a square table with both variables using the same categories and the same data type, so mismatched labels or one variable coded as string and the other numeric will silently block the calculation. Click Statistics and tick Kappa. The number that matters sits in the Symmetric Measures table, in the Value column on the Kappa row — not the Crosstabs percentage table above it, which only ever shows raw agreement. The most common misread is treating the Approx. Sig. column next to kappa as a measure of how good the agreement is: it only tests whether kappa differs from zero, and with a large enough sample even a trivial kappa comes back "significant." Note too that the Crosstabs Kappa checkbox gives the standard, unweighted statistic only; weighted kappa needs IBM's separate STATS WEIGHTED KAPPA extension command, not the Crosstabs dialog.
Upload the dataset (or the relevant paired rating columns). StatsPlease's deterministic engine identifies the appropriate test from the variable type and study design, checks the relevant assumptions, computes the result using fixed, non-LLM algorithms, and drafts the Methods/Results sentence in AMA format — the same number a reader would get running the test by hand in SPSS.
You might also read
References
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46.
- Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543-549.
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.
- Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213-220.
- Bland JM, Bewley BR, Banks MH, Pollard VM. Schoolchildren's beliefs about smoking and disease. Health Educ J. 1975;34(3):71-78.
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282.
A kappa value is only as trustworthy as the arithmetic behind it, and that arithmetic depends on marginal totals most reviewers never see recomputed.
StatsPlease's deterministic engine computes both the raw percent agreement and Cohen's kappa — weighted where the categories you upload are ordinal — directly from your rater columns, and reports the category split alongside it, so the number in your Methods section is never separated from the context a reviewer needs to interpret it.
Try StatsPlease free