StatsPlease
Validation Blog Start free
← Back to Blog

Test assumptions · ~5 min read

What “Assumptions” Actually Means: The Test You Chose Has Conditions You Probably Didn’t Check

Moderate non-normality is rarely the problem. Unequal variance paired with unequal group sizes is, and it doesn't announce itself.

Published

In short

A t-test or ANOVA is a bet that four conditions hold: independent observations, similar variance between groups, a numeric outcome that can be meaningfully averaged, and approximate normality, or a sample large enough that normality stops mattering.1 Moderate non-normality alone is well tolerated at typical clinical sample sizes and is rarely the condition that actually breaks. Unequal variance combined with unequal group sizes is the one that silently changes the arithmetic, shown below on two real comparisons from the same dataset.

The four conditions a t-test is betting on

Independence: each observation isn't systematically tied to another one, so a repeated measurement on the same patient analyzed as if it came from two different patients breaks this before any other check runs. A numeric outcome on a scale where a mean is a meaningful summary. Similar variance between the groups being compared. And approximate normality, or, failing that, a sample large enough for the central limit theorem to make the shape of the underlying distribution stop mattering much.1

Violate one of these and the test doesn't refuse to run. It runs anyway and returns a number that looks exactly like a trustworthy one.

The check that gets attention, and the one that doesn't

Shapiro-Wilk gets checked, reported, and argued about. Levene's test for equal variance gets run in the background by most software and rarely gets a second look, even though the combination that actually produces a misleading result, unequal variance paired with unequal group sizes, runs through Levene's, not Shapiro-Wilk.2 A skewed distribution with adequate, similar-sized groups is usually fine. A skewed distribution is not what quietly breaks a t-test's arithmetic; a variance ratio of several-fold between two groups of very different size is.

Two real comparisons from the same dataset

Both comparisons below split 299 heart-failure patients by whether they died during follow-up (96 died, 203 survived, an unequal split in every comparison that follows).3 Both variables fail Shapiro-Wilk in both groups. Only one of them fails Levene's.

Platelet count, split the same way, has a variance ratio of 1.02 between groups and a Levene's P of .298, equal variance holds. Whether you run the pooled-variance t-test or Welch's on this comparison, the result barely moves: P = .397 pooled versus P = .399 with Welch's correction. Non-normality alone did not put this result at risk.

StatsPlease output: independent-samples t-test
GroupNMean serum creatinineSD
Survived2031.1850.654
Died961.8361.469

Levene's test: P = .0001. Variance in the "died" group is roughly five times larger than in the "survived" group, so equal variance does not hold, and Welch's correction is applied.

t(113.19) = −4.15 · P < .001 · Hedges' g = −0.66

Serum creatinine at admission was significantly higher among patients who died during follow-up than among those who survived (Welch's t-test, t[113.19] = −4.15, P < .001, Hedges' g = −0.66).

Figures computed with scipy from real cohort data (n = 299).

Example data: Heart Failure Clinical Records (see Data Sources). Computed with scipy, not generated.

Run the pooled-variance version on this same comparison anyway, ignoring what Levene's test found, and the statistic moves to t = −5.31 instead of −4.15, on 297 degrees of freedom instead of 113. Both versions clear P < .001 here, so the significance verdict happens not to flip on this particular sample. The degrees of freedom and the size of the statistic still silently changed, and on a smaller or less extreme sample the same ignored violation can move a result across the .05 line, not just shift its magnitude.

Writing the AMA sentence

When Welch's correction applies, say so in the sentence, not just in a methods-section footnote: "Welch's t-test" or "with Welch's correction," naming the test actually used rather than the one that would have run by default. That single phrase is what lets a reviewer confirm the variance-equality check happened at all.

Try it yourself

Reproduce this result

The comparison above was computed from a public cohort. Run it in either tool and the numbers match, because both are doing the same computation, not generating it.

In StatsPlease

  1. Upload the dataset as a CSV.
  2. Open the Analysis tab.
  3. Tap serum_creatinine into Outcome, and set DEATH_EVENT as Group by.
  4. Press the Group Comparison preset button. There is no separate run step; pressing the preset runs Shapiro-Wilk on each group, runs Levene's test, and selects the pooled or Welch-corrected t-test (or Mann-Whitney, if normality fails badly enough) automatically.
  5. On the Results tab, the assumption checks, the test actually used, and the AMA sentence are shown together.

In SPSS

  1. Analyze → Compare Means → Independent-Samples T Test. Test Variable: serum_creatinine; Grouping Variable: DEATH_EVENT.
  2. SPSS reports Levene's test automatically in the same output table as the t-test, both variants side by side.
  3. Read the row that matches whichever side of Levene's test your data landed on, "Equal variances assumed" or "Equal variances not assumed"; SPSS computes both but leaves picking the right one to you.

Frequently asked questions

Does a t-test require perfectly normal data?

No. Moderate non-normality is well tolerated at typical clinical sample sizes and is rarely what makes a P value wrong. The conditions that actually matter more are equal variance between groups, especially combined with unequal group sizes, and independence of observations.

What happens if group variances are unequal?

Levene's test checks this. When it rejects equal variance, the correct comparison is Welch's t-test rather than the standard pooled-variance t-test. Using the pooled version anyway does not crash or warn, it just changes the test statistic and the degrees of freedom, silently.

Why doesn't StatsPlease just always use Welch's test to be safe?

Because the choice itself is the information a reviewer expects to see disclosed. StatsPlease runs Levene's test on the data and selects Student's or Welch's based on the result, then names which one it used and why, rather than picking one variant and hiding the check that justified it.

You might also read

References

  1. Fagerland MW. t-tests, non-parametric tests, and large studies, a paradox of statistical practice? BMC Medical Research Methodology. 2012;12:78. https://doi.org/10.1186/1471-2288-12-78
  2. Blanca MJ, Alarcon R, Arnau J, Bono R, Bendayan R. Non-normal data, is ANOVA still a valid option? Psicothema. 2017;29(4):552-557. https://doi.org/10.7334/psicothema2016.383
  3. Chicco D, Jurman G. Machine learning can predict survival of patients with heart failure from serum creatinine and ejection fraction alone. BMC Medical Informatics and Decision Making. 2020;20:16. https://doi.org/10.1186/s12911-020-1023-5

StatsPlease runs Shapiro-Wilk and Levene's test on your own uploaded data before selecting the pooled or Welch-corrected comparison, so the test named on your results tab is the one your data actually supported. Select your variables and press the Group Comparison preset; pressing it runs the analysis.

Try StatsPlease free