Evidence synthesis · ~7 min read
Heterogeneity in Meta-Analyses: What I² and Q Are Actually Telling You
Thirteen real trials of the same vaccine, and no two of them agree on how well it works.
Published
In short
A pooled effect from a meta-analysis is only as trustworthy as the heterogeneity statistics sitting next to it. Cochran's Q tests whether the included studies disagree more than chance alone would produce, but it has weak power when only a few trials are pooled.1 I2 reports what percentage of the total variation reflects real differences between studies rather than sampling error, and the widely cited 25/50/75% bands for low, moderate, and high heterogeneity are explicitly a tentative guide, not a rule.2 Below is a real 13-trial result where the studies disagree so sharply that the pooling method you choose changes the answer.
Two numbers, two different questions
Q asks a yes-or-no question: do these studies disagree more than sampling error alone would explain? It is a significance test, and like any significance test its power depends on how many studies are being compared. I2 asks a different question: of the total variation across studies, how much is real disagreement rather than noise? It is reported as a percentage, which is why it travels better between meta-analyses of different sizes than Q's raw chi-square statistic does.3
Neither number replaces looking at the forest plot itself. A high I2 with a handful of studies can be driven by one outlier; a low I2 with dozens of small trials can still hide a real subgroup effect. Read both statistics as a prompt to look closer, not as a verdict on their own.
A real result: does the BCG vaccine prevent tuberculosis?
Colditz and colleagues compiled 13 controlled trials of the BCG vaccine against tuberculosis, spanning studies published between 1948 and 1980 on four continents.4 Each trial reports the same structure: TB cases and non-cases in a vaccinated group, TB cases and non-cases in a control group. Converting each trial to a log odds ratio and pooling them is a real, worked meta-analysis, not a toy example.
| Trial | TB+ / N, vaccinated | TB+ / N, control | Odds ratio |
|---|---|---|---|
| Aronson 1948 | 4 / 123 | 11 / 139 | 0.39 |
| Ferguson & Simes 1949 | 6 / 306 | 29 / 303 | 0.19 |
| Rosenthal et al 1960 | 3 / 231 | 11 / 220 | 0.25 |
| Hart & Sutherland 1977 | 62 / 13,598 | 248 / 12,867 | 0.23 |
| Frimodt-Moller et al 1973 | 33 / 5,069 | 47 / 5,808 | 0.80 |
| Stein & Aronson 1953 | 180 / 1,541 | 372 / 1,451 | 0.38 |
| Vandiviere et al 1973 | 8 / 2,545 | 10 / 629 | 0.20 |
| TPT Madras 1980 | 505 / 88,391 | 499 / 88,391 | 1.01 |
| Coetzee & Berjak 1968 | 29 / 7,499 | 45 / 7,277 | 0.62 |
| Rosenthal et al 1961 | 17 / 1,716 | 65 / 1,665 | 0.25 |
| Comstock et al 1974 | 186 / 50,634 | 141 / 27,338 | 0.71 |
| Comstock & Webster 1969 | 5 / 2,498 | 3 / 2,341 | 1.56 |
| Comstock et al 1976 | 27 / 16,913 | 29 / 17,854 | 0.98 |
Pooled by inverse-variance weighting with REML-estimated between-study variance (τ2 = 0.338).
Pooled OR = 0.48 (95% CI, 0.33–0.68) · z = −4.01 · P < .001
I2 = 92.6% · Q(12) = 163.16 · P < .001
Random-effects pooling of 13 trials found the BCG vaccine associated with reduced odds of tuberculosis (OR = 0.48; 95% CI, 0.33–0.68; P < .001), with substantial heterogeneity across trials (I2 = 92.6%, Q[12] = 163.16, P < .001).
Figures computed with scipy/numpy from real trial data (k = 13 studies), matching the DerSimonian-Laird/REML and Higgins-Thompson methodology.
Example data: BCG vaccine trials (Colditz et al, JAMA, 1994); see references. Computed, not generated.
Two numbers here are worth sitting with. First, I2 = 92.6% is far above even the "high heterogeneity" end of the tentative 75% band.2 These 13 trials are not measuring one consistent effect with sampling noise around it; they are measuring genuinely different effects, later work traced much of this to the trial sites' latitude. Second, pooling the same 13 trials with a fixed-effect model instead of random-effects gives OR = 0.65, a materially different answer from the random-effects OR = 0.48. When heterogeneity is this large, the choice of pooling model is not a technicality. It changes the headline number.
Writing the AMA sentence
Report the heterogeneity statistics alongside the pooled effect, not as an afterthought below it: the pooling model (fixed or random-effects, and by what method), I2, and Q with its P value. If I2 is high, say so explicitly and note that a random-effects estimate, not a fixed-effect one, is the defensible summary. If only a handful of trials were pooled, don't lean on a non-significant Q as proof the trials agree; say what it can and cannot tell you given so few studies.
Try it yourself
Reproduce this result
The BCG counts above are public. The pooling and heterogeneity arithmetic is fully deterministic, run it yourself from the table above and the numbers match exactly, because nothing here is generated.
In StatsPlease
- Meta-analytic pooling of published summary statistics isn't yet a self-serve preset in the CSV-upload flow; today's presets analyze the patient-level data you upload directly.
- What you can do today: take any one trial's own 2×2 counts (say, Hart & Sutherland 1977, TB+ and TB− by arm), enter it as patient-level rows, select the two categorical variables as Outcome, and press Category Patterns to get that single trial's own odds ratio and chi-square, no separate run step needed.
- The pooling itself, inverse-variance weighting, REML between-study variance, Cochran's Q, and I2, runs on the log odds ratio and variance from each trial computed the same way shown above.
In SPSS
- Base SPSS Statistics has no built-in meta-analysis procedure; pooling multiple studies' summary statistics typically requires the community macros (e.g. David B. Wilson's) or exporting to a dedicated package such as R's metafor.
- For each trial, compute the log odds ratio and its variance by hand: ln(a×d / b×c) and 1/a + 1/b + 1/c + 1/d.
- Pool with inverse-variance weights, then compute Q as the weighted sum of squared deviations from the pooled mean, and I2 = max(0, (Q − df) / Q) × 100.
Frequently asked questions
What do I2 and Cochran's Q actually measure?
Cochran's Q tests whether the studies in a meta-analysis disagree more than sampling error alone would explain. I2 reports what percentage of the total variation across studies reflects real differences between studies rather than chance, expressed as a percentage rather than a P value, which makes it easier to interpret and compare across meta-analyses of different sizes.
Are the 25%, 50%, 75% I2 bands for low, moderate, high heterogeneity a strict rule?
No. Higgins et al 2003, the paper the bands trace back to, explicitly called them a tentative guide, not a rule to apply mechanically. Read the forest plot itself alongside the number before deciding how much a pooled estimate should be trusted.
Why does Cochran's Q sometimes fail to detect heterogeneity that is actually there?
Q's power to detect real heterogeneity depends heavily on the number of studies pooled. With only two or three trials, the same underlying disagreement that would be highly significant across a dozen studies can produce a non-significant Q purely because there isn't enough data to distinguish real heterogeneity from sampling noise.
You might also read
References
- Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10(1):101-129. https://doi.org/10.2307/3001666
- Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560. https://doi.org/10.1136/bmj.327.7414.557
- Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine. 2002;21(11):1539-1558. https://doi.org/10.1002/sim.1186
- Colditz GA, Brewer TF, Berkey CS, et al. Efficacy of BCG vaccine in the prevention of tuberculosis: meta-analysis of the published literature. JAMA. 1994;271(9):698-702. https://doi.org/10.1001/jama.1994.03510330076038
StatsPlease's deterministic engine computes odds ratios, confidence intervals, and chi-square tests the same reproducible way on your own uploaded data, run it on your own study and press Category Patterns for a categorical comparison, or Group Comparison for a continuous outcome; pressing the preset runs the analysis.
Try StatsPlease free