Intraclass Correlation Coefficient (ICC): Reporting Agreement for Continuous Measurements

Why Cohen's kappa can't score a continuous measurement

Cohen's kappa measures agreement between raters sorting cases into discrete categories: present or absent, grade 1 versus grade 2, malignant versus benign. It cannot be applied directly to a continuous measurement, degrees of knee flexion, screw length in millimeters, Cobb angle, tumor diameter, without collapsing the data into bins first, and binning throws away the size of any disagreement along with the exact numbers.¹ For continuous measurements, the ICC is the direct analog: it uses the actual values, not a rounded category.

A real result: do two blood pressure readings agree?

A cross-sectional screening cohort of 403 adults from central Virginia recorded a second systolic blood pressure reading for a subset of patients.⁴ For the 141 patients with both readings, treating "first reading" and "second reading" as two raters measuring the same 141 subjects gives a genuine repeated-measures agreement question: how closely does a second reading track the first?

Example data: central Virginia diabetes screening cohort (Willems, Saunders, Hunt, and Schorling, 1997); see references. Computed with scipy, not generated.

"Good" here follows a commonly cited convention: below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good, above 0.90 excellent.² A repeated blood pressure reading on the same patient, minutes apart, agreeing at 0.863 rather than sitting above 0.90 is a genuine, checkable finding, not a rounding artifact: it says a single systolic reading carries more measurement noise than "just take it twice and average" intuition suggests.

Ten forms, one name: why the model used has to be stated

ICC's value depends on three independent choices, first formalized as six named forms by Shrout and Fleiss and refined further since.^2,5 The model: one-way random, two-way random, or two-way mixed, depending on whether the same fixed set of raters scored every subject or a random subset did. The type: a single measurement's reliability, or the reliability of an average of several. The definition: consistency, which tolerates a systematic offset between raters (rater A always scoring two points higher than rater B still counts as perfect consistency), or absolute agreement, which does not.

On this dataset the two forms land close together, 0.863 versus 0.868, because the systematic gap between the first and second reading is small, about 2 mmHg out of a roughly 150 mmHg mean. That is not guaranteed. If two raters (or two occasions) carried a larger consistent offset, a real accommodation effect where a second reading taken minutes later reads systematically lower, the absolute-agreement and consistency versions would diverge much further, because absolute agreement penalizes that offset directly and consistency does not.³ Reporting "ICC = 0.86" with no form named leaves a reader unable to tell which question was actually answered.

Writing the AMA sentence

Name the specific form, not just "ICC": the model, and whether it reflects consistency or absolute agreement. "ICC = 0.86" is incomplete; "ICC(2,1) = 0.863, absolute agreement, 95% CI 0.812 to 0.901" tells a reviewer exactly which question the number answers and lets them judge whether that is the question your study needed answered.

Frequently asked questions

Why can't I use Cohen's kappa for a continuous measurement?

Cohen's kappa measures agreement between raters sorting cases into discrete categories. Applying it to a continuous measurement, such as a length in millimeters or an angle in degrees, requires collapsing the data into bins first, which discards the size of any disagreement. The intraclass correlation coefficient (ICC) is the direct continuous-data analog and keeps that information.

Why does ICC have so many different forms?

ICC's value depends on three independent choices: the model (one-way, two-way random, or two-way mixed), the type (a single measurement or the average of several), and the definition (consistency, which tolerates a systematic offset between raters, or absolute agreement, which does not). Each combination can produce a different number from the identical dataset, so the specific form used has to be stated.

What counts as good agreement for an ICC?

A commonly cited guideline treats values below 0.50 as poor, 0.50 to 0.75 as moderate, 0.75 to 0.90 as good, and above 0.90 as excellent reliability. These bands are a widely used convention, not a universal law, and different fields sometimes apply stricter cutoffs for measurements used in clinical decisions.

References

1. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20(1):37-46. https://doi.org/10.1177/001316446002000104
2. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016;15(2):155-163. https://doi.org/10.1016/j.jcm.2016.02.012
3. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods. 1996;1(1):30-46. https://doi.org/10.1037/1082-989X.1.1.30
4. Willems JP, Saunders JT, Hunt DE, Schorling JB. Prevalence of coronary heart disease risk factors among rural blacks: a community-based study. Southern Medical Journal. 1997;90(8):814-820.
5. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86(2):420-428. https://doi.org/10.1037/0033-2909.86.2.420