Test selection & reporting · ~5 min read
ANOVA
A significant F-test says something differs. It doesn't say what. That's the next test's job.
Published
In short
A significant one-way ANOVA tells you at least one group differs from the others. It does not tell you which one. Reporting the F-test alone and stopping there is a common, incomplete habit; a real result below shows why the omnibus test and the post-hoc test that follows it can tell two different stories about the same data.
What the omnibus F-test actually claims
A one-way ANOVA compares means across three or more groups and returns a single question's answer: is there evidence that not all the group means are equal. A significant result confirms a difference exists somewhere among the groups. It says nothing about which specific pair or pairs are driving that difference, and treating a significant F-test as if it already answered that question is the single most common way this test gets misreported.1
Getting to "which groups" by running pairwise t-tests between every combination isn't a safe fix. Each individual comparison carries a 5% chance of a false positive, and running several of them compounds that risk across the whole set of comparisons, known as the familywise error rate. A dedicated post-hoc test, Tukey's honestly significant difference (HSD) test among the most common, controls that compounded risk while still identifying which specific pairs actually differ.1
A real result: does body frame relate to blood pressure?
In 387 patients from a diabetes risk-screening dataset, diastolic blood pressure was compared across three self-reported body frame categories: small, medium, and large.2 Normality held in all three groups (Shapiro-Wilk, all P > .05) and Levene's test confirmed equal variances (P = .935), so a standard one-way ANOVA is the correctly routed test here.
| Body frame | n | Mean | SD |
|---|---|---|---|
| Small | 103 | 80.0 mmHg | 14.2 |
| Medium | 181 | 85.0 mmHg | 13.1 |
| Large | 103 | 83.4 mmHg | 13.5 |
F(2, 384) = 4.44 · P = .012 · η² = .023
Diastolic blood pressure differed significantly by body frame (F[2, 384] = 4.44, P = .012, η² = .023). Tukey HSD post-hoc comparisons showed patients with a medium frame had significantly higher diastolic blood pressure than those with a small frame (mean difference, 4.97 mmHg; 95% CI, 1.04–8.90; P = .009). Neither the large-versus-small nor the large-versus-medium comparison reached significance (P = .167 and P = .622, respectively).
Figures computed with scipy and statsmodels from real data (Vanderbilt diabetes screening dataset).
Example data: Vanderbilt University Department of Biostatistics public teaching datasets (hbiostat.org/data). Figures computed with scipy and statsmodels from real data.
That pattern is the entire reason the post-hoc step exists. The omnibus test alone confirms something is going on among the three groups. It cannot tell a reader that the signal is coming specifically from the medium-versus-small comparison, and that the large group isn't reliably distinguishable from either of the other two.
Writing the AMA sentence
Report the omnibus test first, with its effect size, then the post-hoc comparisons that actually locate the difference. Naming the specific post-hoc method matters as much as naming the omnibus test: Tukey HSD assumes equal variances, and a dataset that fails Levene's test needs a variance-robust alternative such as Games-Howell instead.
Try it yourself
Reproduce this result
The result above comes from a public dataset. Reproduce it in either tool to confirm the numbers match.
In StatsPlease
- Upload the dataset as a CSV.
- Open the Analysis tab.
- Set diastolic BP as the Outcome and frame as the Group (three levels).
- Press the Group Comparison preset button. There is no separate run step; pressing the preset runs the analysis, screening normality and variance before routing the omnibus and post-hoc tests.
- On the Results tab, the assumption checks, omnibus test, and post-hoc comparisons are shown together.
In SPSS
- Analyze → Descriptive Statistics → Explore, request Shapiro-Wilk and Levene's test by group.
- Confirm normality holds and variances are equal.
- Analyze → Compare Means → One-Way ANOVA.
- Under Post Hoc, select Tukey (or Games-Howell if Levene's test had failed).
- Report the omnibus F-test and post-hoc table together; SPSS does not compute eta-squared by default.
Frequently asked questions
Does a significant ANOVA tell you which groups are different?
No. A significant one-way ANOVA only confirms that at least one group mean differs from the others somewhere in the set. Identifying which specific groups differ requires a separate post-hoc test run after the omnibus result.
When should you use Tukey HSD versus Games-Howell for post-hoc testing?
Tukey HSD assumes equal variances across groups. If Levene's test shows variances are not equal, Games-Howell is the appropriate alternative instead of Tukey applied by default.
Is eta-squared reported automatically by statistical software?
Not always. SPSS's standard one-way ANOVA output does not compute eta-squared by default; it has to be calculated manually from the sums of squares, or read directly from software that reports it automatically.
You might also read
References
- McHugh ML. Multiple comparison analysis testing in ANOVA. Biochemia Medica. 2011;21(3):203-209. https://doi.org/10.11613/BM.2011.029
- Data obtained from http://hbiostat.org/data courtesy of the Vanderbilt University Department of Biostatistics.
StatsPlease checks normality and variance homogeneity across your own groups, then selects and runs the correctly matched post-hoc test automatically.
Upload your data, set your outcome and grouping variable, and press the Group Comparison preset; pressing it runs the analysis and returns the omnibus test with its effect size and the post-hoc comparisons, computed, not generated.
Try StatsPlease free