Step by step guide ANOVA
Goal: Comparing independent group means (≤ 3) or multiple independent variables on the dependent variable.
Hypothesis:
Null-hypothesis H0 no difference between any of the group means (all means are equal)
Alternative hypothesis H1 difference between some of the group means
Goal: How unusual is our observed difference between group means in the sample, assuming H 0 is true?
Before anything else, read the question thoroughly! Think for yourself, what is asked? What is the kind of answer
they are looking for? What is H0 in this case and what would be H 1? What are the dependent and independent
variables?
Which test do I need to use?
Important remark: If the assumption of normality is violated, following central limit theorem (= the sampling
distribution of the sample means approaches a normal distribution as the sample size becomes large) you can still
run the test. Report in the conclusion.
1. One-way ANOVA (assumptions: normality, homogeneity of variance, independence)
1. We want to compare the means of different groups (≤ 3) to see if there is a significant difference.
2. Prepare the data. Recode variables when they are asked in the other direction (use ‘Transform variable’).
Check for outliers. Calculate the mean.
3. Perform reliability analysis Cronbach’s alpha reliability test under ‘Factor’ ‘Reliability test’
‘Cronbach’s α (if item is dropped)’. 0 is no reliability, 1 is perfect reliability. Rule of thumb: 0.70 is considered
acceptable.
4. Explore the data. Is normality assumed? Use QQ plot or histogram for visual data. Find z-scores of skewness
and kurtosis (divide by their standard error) between -1.96 and 1.96 = normal distribution. Check per
group. Or use Shapiro-Wilk test for smaller samples (closer to 1 = normal data).
5. Check homogeneity of variances (VR = highest variance / lowest variance), if under 2, homogeneity of
variances is met. IF variance ratio > 2, report Levene’s test (homogeneity test). If Levene’s test is significant (p
< 0.05), the data violates the assumption of homogeneity of variance.
6. Run ANOVA.
a. Issues with normality run non-parametric one-way ANOVA: Kruskal-Wallis test. Go back to ‘ANOVA’
under ‘Analysis’ and click ‘One way ANOVA Kruskal-Wallis’ under ‘non-parametric’.
b. Issues with homogeneity run one-way ANOVA, use Welch’s statistic. Go back to ‘ANOVA’ under
‘Analysis’ and click ‘One way ANOVA’.
c. Issues with normality and homogeneity run non-parametric one-way ANOVA: Kruskal-Wallis test. Go
back to ‘ANOVA’ under ‘Analysis’ and click ‘One way ANOVA Kruskal-Wallis’ under ‘non-parametric’.
7. Run follow-up tests. Do you have a hypothesis about the difference between means?
a. Non-specific hypothesis, homogeneity assumption met Post-hoc test (Tukey’s HSD)
b. Non-specific hypothesis, homogeneity assumption not met Post-hoc test (Bonferroni correction)
c. Non-specific hypothesis, normality assumption not met Post-hoc test in Kruskal-Wallis non-parametric
module. Click ‘DSCF pairwise comparisons’.
d. Specific hypothesis Planned contrast. Click ‘contrasts’. Specify which contrast [see image below]. In case
none of the contrasts match your hypothesis, you might need to recode your variables as to fit the contrast.
e. Specific hypothesis, normality assumption not met Stick to Post-hoc analysis ‘DSCF pairwise
comparison’. This effect size is found in the normal ANOVA (Cohen’s d).
8. Calculate effect sizes (η2).
a. Overall ANOVA η2
b. Post-hoc test Cohen’s d
c. Contrast r =
√ 2
t2
t +df
(calculate r yourself, t cannot be negative, df of residuals is used)
1
Goal: Comparing independent group means (≤ 3) or multiple independent variables on the dependent variable.
Hypothesis:
Null-hypothesis H0 no difference between any of the group means (all means are equal)
Alternative hypothesis H1 difference between some of the group means
Goal: How unusual is our observed difference between group means in the sample, assuming H 0 is true?
Before anything else, read the question thoroughly! Think for yourself, what is asked? What is the kind of answer
they are looking for? What is H0 in this case and what would be H 1? What are the dependent and independent
variables?
Which test do I need to use?
Important remark: If the assumption of normality is violated, following central limit theorem (= the sampling
distribution of the sample means approaches a normal distribution as the sample size becomes large) you can still
run the test. Report in the conclusion.
1. One-way ANOVA (assumptions: normality, homogeneity of variance, independence)
1. We want to compare the means of different groups (≤ 3) to see if there is a significant difference.
2. Prepare the data. Recode variables when they are asked in the other direction (use ‘Transform variable’).
Check for outliers. Calculate the mean.
3. Perform reliability analysis Cronbach’s alpha reliability test under ‘Factor’ ‘Reliability test’
‘Cronbach’s α (if item is dropped)’. 0 is no reliability, 1 is perfect reliability. Rule of thumb: 0.70 is considered
acceptable.
4. Explore the data. Is normality assumed? Use QQ plot or histogram for visual data. Find z-scores of skewness
and kurtosis (divide by their standard error) between -1.96 and 1.96 = normal distribution. Check per
group. Or use Shapiro-Wilk test for smaller samples (closer to 1 = normal data).
5. Check homogeneity of variances (VR = highest variance / lowest variance), if under 2, homogeneity of
variances is met. IF variance ratio > 2, report Levene’s test (homogeneity test). If Levene’s test is significant (p
< 0.05), the data violates the assumption of homogeneity of variance.
6. Run ANOVA.
a. Issues with normality run non-parametric one-way ANOVA: Kruskal-Wallis test. Go back to ‘ANOVA’
under ‘Analysis’ and click ‘One way ANOVA Kruskal-Wallis’ under ‘non-parametric’.
b. Issues with homogeneity run one-way ANOVA, use Welch’s statistic. Go back to ‘ANOVA’ under
‘Analysis’ and click ‘One way ANOVA’.
c. Issues with normality and homogeneity run non-parametric one-way ANOVA: Kruskal-Wallis test. Go
back to ‘ANOVA’ under ‘Analysis’ and click ‘One way ANOVA Kruskal-Wallis’ under ‘non-parametric’.
7. Run follow-up tests. Do you have a hypothesis about the difference between means?
a. Non-specific hypothesis, homogeneity assumption met Post-hoc test (Tukey’s HSD)
b. Non-specific hypothesis, homogeneity assumption not met Post-hoc test (Bonferroni correction)
c. Non-specific hypothesis, normality assumption not met Post-hoc test in Kruskal-Wallis non-parametric
module. Click ‘DSCF pairwise comparisons’.
d. Specific hypothesis Planned contrast. Click ‘contrasts’. Specify which contrast [see image below]. In case
none of the contrasts match your hypothesis, you might need to recode your variables as to fit the contrast.
e. Specific hypothesis, normality assumption not met Stick to Post-hoc analysis ‘DSCF pairwise
comparison’. This effect size is found in the normal ANOVA (Cohen’s d).
8. Calculate effect sizes (η2).
a. Overall ANOVA η2
b. Post-hoc test Cohen’s d
c. Contrast r =
√ 2
t2
t +df
(calculate r yourself, t cannot be negative, df of residuals is used)
1
