10. Comparing more than two samples
This session is about comparing more than two groups with ANOVA (and post-hoc tests), and running that workflow in Python with pingouin.
Code set up
Statistical tests for comparing more than two samples
Choosing a Statistical Significance Test
So far we have seen the the t-test, which is specifically for comparing data that meet the following criteria:
- there is a single outcome variable that is continuous (such as percent yield),
- the input variable is categorical or discrete (such as the catalyst type),
- the data are from a normally distributed population,
- the desired comparison is between one sample to a standard or two samples.
However, there are many more statistical significance tests. The choice of statistical test depends on the types of variables involved. The decision tree below summarizes the options for data in which the following criteria are met:
- a single, continuous outcome variable and
- a normal distribution (we will discuss violations of this in session 11).
Key terms:
- Outcome variable (dependent variable): what you are measuring (e.g. reaction yield)
- Input variable (independent variable): what you are manipulating (e.g. solvent identity, catalyst identity, temperature)
- Conditions / treatments: the specific values of the input variable being compared (e.g. five different solvents = 5 conditions)
In this session, we are going to focus on the top branch that has one outcome variable, which is continuous, and input variables that are discrete or categorical. This branch includes the t-test, one-way ANOVA, and N-way ANOVA.
| Situation | Test |
|---|---|
| 1 continuous outcome, 1 input variable, 1–2 conditions | t-test |
| 1 continuous outcome, 1 input variable, >2 conditions | One-way ANOVA |
| 1 continuous outcome, >1 input variable, | N-way ANOVA |
What is ANOVA?
ANalysis Of VAriance (ANOVA) compares means across multiple conditions by decomposing variance into two components:
- Within-condition variation: the scatter among replicates within each condition, which can be attributed to measurement uncertainty and random variation (or uncontroled variables)
- Between-condition variation: the differences between condition means, which can be attributed to the factor being tested
ANOVA computes the F-statistic (analogous to the t-statistic of the t-test): the ratio of between-condition variance to within-condition variance. If this ratio is large (the conditions differ more than expected from chance), H₀ is rejected.
\[F_{stat} = \frac{\text{Variance between groups}}{\text{Variance within groups}}\]
ANOVA outputs a p-value (just like the t-test). A p-value from ANOVA indicates the probability of observing an F-statistic at least as extreme as the one obtained, if the null hypothesis were true. A small p-value means the observed group differences would be unlikely under the null hypothesis; it does not quantify the probability that the means differ.
To work with this dataset, we will have to reshape the dataframe from a wide dataframe to a long dataframe. Please see the example below. The code is melt from pandas.
Once we have our long, melted dataframe, we can perform ANOVA with pingoin.
From the ANOVA p-value, you can only learn about the group means collectively to identify if there is one condition that differs. You cannot determine which condition differs from the others.
If we obtain a statistically significant p-value from the ANOVA test, how can we identify the condition(s) that are statistically significantly different?
Post-Hoc Tests: After a Significant ANOVA
NOTE: if the ANOVA p-value is not below the threshold, you should not perform a post-hoc test!!!
If ANOVA finds a significant effect, it tells you that at least one condition is different, but not which ones.
Post-hoc tests identify which specific pair(s) of group means differ significantly, while controlling for family-wise error rate across all the comparisons.
For the ANOVA test, there are two appropriate post-hoc tests: Tukey’s and Games-Howell.
Tukey’s test: assumes similar variances and equal sample sizes.
Games-Howell test: more robust when variances or sample sizes differ between conditions.
We have not yet discussed how to test differences in variances, but this will be covered in the next session.
The output of a post-hoc test in pingouin also indicates the effect size for the pair-wise comparisons.
Below is an example of the Tukey test with pingouin.
Why Not Just Use the t-Test Repeatedly?
If you have three conditions to compare, you could run three t-tests: A vs. B, A vs. C, and B vs. C. This is incorrect for two reasons:
1. Type I error inflation: each test has a 5% chance of a false positive. Running three tests at α = 0.05 means the probability of making no false positive is 0.95³ = 86%. The chance of at least one false positive is therefore 14%, not 5%. This gets much worse with more comparisons.
2. Loss of statistical power: ANOVA uses all groups to estimate variability (pooled variance), giving more accurate estimates than individual t-tests that only use two groups at a time.
By pooling variance across all groups, ANOVA has more degrees of freedom to estimate the within-group variability. This makes the estimate of the error term more reliable and gives ANOVA higher power than repeated t-tests for detecting effects that are consistent across conditions.
Effect Sizes and Power for ANOVA
| Metric | Formula | Notes |
|---|---|---|
| η² (eta-squared) | SS_effect / SS_total | Biased; overestimates in small samples |
| ω² (omega-squared) | Corrected version of η² | Recommended for small samples |
η² is biased because it divides by total sum of squares, which includes the very effect being estimated. This makes η² appear larger in small samples than it truly is in the population. ω² applies a correction analogous to Bessel’s correction for variance, and is the preferred metric for the sample sizes typically encountered in chemistry. However, ω² calculations are not available in all statistical programs, so we will default to using η².
For a t-test, we calculated effect size and power. For ANOVA, we will use the effect size \(\eta^2\). There are other effect sizes (partial \(\eta^2\) and \(\omega^2\)), but for this entry-level course, we will stick with \(\eta^2\). Power can be calculated for ANOVA in pingouin, but its interpretation is slightly more complex, because what we derive is an omnibus power. Planning the number of replicates around an omnibus power calculation can still leave a posthoc two-way comparison under powered.
N-Way ANOVA
When there are multiple input variables, N-way ANOVA is used.
Example: you want to test the effect of two DNA systems (input 1: 4 types) and two cell types (input 2: 2 types) on protein expression. This is a 2-way ANOVA.
import pingouin as pg
pg.anova(dv="Protein_expression", between=["Cells", "DNA"], data=df)Interaction effects: an interaction is present when the effect of one input variable depends on the level of another. In this example, if the best DNA system performed differently in each cell type, that would be an interaction. When no significant interaction is found, the two factors can be treated as independent.
Bonferroni Correction
When performing many simultaneous comparisons (for example, comparing all pairwise combinations in a large N-way ANOVA), the Bonferroni correction adjusts the significance threshold:
\[\alpha_{\text{corrected}} = \frac{\alpha}{m}\]
where m is the number of comparisons. This controls the family-wise error rate but is conservative — it increases Type II error (missed true effects) as the number of comparisons grows.
- Compare >2 groups with ANOVA, not repeated t-tests (controls Type I inflation; pools variance for power).
- F = between-group / within-group variance; a significant ANOVA -> post-hoc (Tukey or Games-Howell).
- N-way ANOVA tests several factors and their interactions.
- In Python:
pg.anova(...), thenpg.pairwise_tukey(...)/pg.pairwise_gameshowell(...); data in long format viadf.melt(...).
In Person
TBD