Extra / additional information about the knowledge clips: ARMS CP-part
Inhoud
Knowledge clip 1a Power and effect size ................................................................................................ 2
Knowledge clip 1b Manipulation check and randomization check ......................................................... 7
Knowledge clip 2 Interaction in ANOVA and regression + Post-hoc testing ........................................... 8
Knowledge clip 3a Mediation ................................................................................................................ 11
Knowledge clip 3b Meta-analysis .......................................................................................................... 14
,Knowledge clip 1a Power and effect size
P-value
What the p-value tells you about statistical significance
When you perform a statistical test a p-value helps you determine the significance of your results in
relation to the null hypothesis.
-> The null hypothesis (H0) states that there is no relationship between the two variables being
studied (one variable does not affect the other). It states the results are due to chance and are not
significant in terms of supporting the idea being investigated.
-> Thus, the null hypothesis assumes that whatever you are trying to prove did not happen.
-> The alternative hypothesis (Ha) is the one you would believe if the null hypothesis is concluded to
be untrue.
-> The alternative hypothesis states that the independent variable did affect the dependent variable,
and the results are significant in terms of supporting the theory being investigated (i.e. not due to
chance).
The more common an event is, the more likely it is to reach p <0.05 in a bigger dataset.
How do you know if a p-value is statistically significant?
The level of statistical significance is often expressed as a p-value between 0 and 1. The smaller the
p-value, the stronger the evidence that you should reject the null hypothesis.
• A p-value less than 0.05 (typically ≤ 0.05) is statistically significant. It indicates strong
evidence against the null hypothesis, as there is less than a 5% probability the null is correct
(and the results are random). Therefore, we reject the null hypothesis, and accept the
alternative hypothesis.
However, this does not mean that there is a 95% probability that the research hypothesis is
true. The p-value is conditional upon the null hypothesis being true is unrelated to the truth
or falsity of the research hypothesis.
• A p-value higher than 0.05 (> 0.05) is not statistically significant and indicates strong evidence
for the null hypothesis. This means we retain the null hypothesis and reject the alternative
hypothesis. You should note that you cannot accept the null hypothesis, we can only reject
the null or fail to reject it.
A statistically significant result cannot prove that a research hypothesis is correct (as this
implies 100% certainty).
Instead, we may state our results “provide support for” or “give evidence for” our research
hypothesis (as there is still a slight probability that the results occurred by chance and the
null hypothesis was correct – e.g. less than 5%).
Why is the p-value not enough?
A lower p-value is sometimes interpreted as meaning there is a stronger relationship between two
variables. However, statistical significance means that it is unlikely that the null hypothesis is true
(less than 5%).
To understand the strength of the difference between two groups (control vs. experimental) a
researcher needs to calculate the effect size.
,Effect size
= a quantitative measure of the magnitude of the experimental effect. The larger the effect size the
stronger the relationship between two variables.
You can look at the effect size when comparing any two groups to see how substantially different
they are. Typically, research studies will comprise an experimental group and a control group. The
experimental group may be an intervention or treatment which is expected to effect a specific
outcome.
For example, we might want to know the effect of a therapy on treating depression. The effect size
value will show us if the therapy as had a small, medium or large effect on depression.
So: Statistical significance is the least interesting thing about the results. You should describe the
results in terms of measures of magnitude – not just, does a treatment affect people, but how much
does it affect them (effect size).
How do we calculate the effect size?
Effect sizes either measure the sizes of associations between variables or the sizes of differences
between group means.
1. Cohen’s d: Cohen's d is an appropriate effect size for the comparison between two means. It can
be used, for example, to accompany the reporting of t-test and ANOVA results.
Cohen suggested that d = 0.2 be considered a 'small' effect size, 0.5 represents a 'medium' effect size
and 0.8 a 'large' effect size. This means that if the difference between two groups' means is less than
0.2 standard deviations, the difference is negligible, even if it is statistically significant.
2. Pearson r correlation: this parameter of effect size summarises the strength of the bivariate
relationship. The value of the effect size of Pearson r correlation varies between -1 (perfect negative
correlation) to +1 (perfect positive correlation). According to Cohen:
, Type I and Type II errors
A statistically significant result cannot prove that a research hypothesis is correct (as this implies
100% certainty). Because a p-value is based on probabilities, there is always a chance of making an
incorrect conclusion regarding accepting or rejecting the null hypothesis (H0). Anytime we make a
decision using statistics there are four possible outcomes, with two representing correct decisions
and two representing errors.
The chances of committing these two types of
errors are inversely proportional: that is,
decreasing type I error rate increases type II
error rate, and vice versa.
Type I error
A type 1 error is also known as a false positive
and occurs when a researcher incorrectly rejects
a true null hypothesis. This means that you
report that your findings are significant when in
fact they have occurred by chance.
-> The probability of making a type I error is represented by your alpha level (α), which is the p-value
below which you reject the null hypothesis. A p-value of 0.05 indicates that you are willing to accept
a 5% chance that you are wrong when you reject the null hypothesis.
-> You can reduce your risk of committing a type I error by using a lower value for p. For example,
a p-value of 0.01 would mean there is a 1% chance of committing a Type I error.
-> However, using a lower value for alpha means that you will be less likely to detect a true
difference if one really exists (thus risking a type II error).
Inhoud
Knowledge clip 1a Power and effect size ................................................................................................ 2
Knowledge clip 1b Manipulation check and randomization check ......................................................... 7
Knowledge clip 2 Interaction in ANOVA and regression + Post-hoc testing ........................................... 8
Knowledge clip 3a Mediation ................................................................................................................ 11
Knowledge clip 3b Meta-analysis .......................................................................................................... 14
,Knowledge clip 1a Power and effect size
P-value
What the p-value tells you about statistical significance
When you perform a statistical test a p-value helps you determine the significance of your results in
relation to the null hypothesis.
-> The null hypothesis (H0) states that there is no relationship between the two variables being
studied (one variable does not affect the other). It states the results are due to chance and are not
significant in terms of supporting the idea being investigated.
-> Thus, the null hypothesis assumes that whatever you are trying to prove did not happen.
-> The alternative hypothesis (Ha) is the one you would believe if the null hypothesis is concluded to
be untrue.
-> The alternative hypothesis states that the independent variable did affect the dependent variable,
and the results are significant in terms of supporting the theory being investigated (i.e. not due to
chance).
The more common an event is, the more likely it is to reach p <0.05 in a bigger dataset.
How do you know if a p-value is statistically significant?
The level of statistical significance is often expressed as a p-value between 0 and 1. The smaller the
p-value, the stronger the evidence that you should reject the null hypothesis.
• A p-value less than 0.05 (typically ≤ 0.05) is statistically significant. It indicates strong
evidence against the null hypothesis, as there is less than a 5% probability the null is correct
(and the results are random). Therefore, we reject the null hypothesis, and accept the
alternative hypothesis.
However, this does not mean that there is a 95% probability that the research hypothesis is
true. The p-value is conditional upon the null hypothesis being true is unrelated to the truth
or falsity of the research hypothesis.
• A p-value higher than 0.05 (> 0.05) is not statistically significant and indicates strong evidence
for the null hypothesis. This means we retain the null hypothesis and reject the alternative
hypothesis. You should note that you cannot accept the null hypothesis, we can only reject
the null or fail to reject it.
A statistically significant result cannot prove that a research hypothesis is correct (as this
implies 100% certainty).
Instead, we may state our results “provide support for” or “give evidence for” our research
hypothesis (as there is still a slight probability that the results occurred by chance and the
null hypothesis was correct – e.g. less than 5%).
Why is the p-value not enough?
A lower p-value is sometimes interpreted as meaning there is a stronger relationship between two
variables. However, statistical significance means that it is unlikely that the null hypothesis is true
(less than 5%).
To understand the strength of the difference between two groups (control vs. experimental) a
researcher needs to calculate the effect size.
,Effect size
= a quantitative measure of the magnitude of the experimental effect. The larger the effect size the
stronger the relationship between two variables.
You can look at the effect size when comparing any two groups to see how substantially different
they are. Typically, research studies will comprise an experimental group and a control group. The
experimental group may be an intervention or treatment which is expected to effect a specific
outcome.
For example, we might want to know the effect of a therapy on treating depression. The effect size
value will show us if the therapy as had a small, medium or large effect on depression.
So: Statistical significance is the least interesting thing about the results. You should describe the
results in terms of measures of magnitude – not just, does a treatment affect people, but how much
does it affect them (effect size).
How do we calculate the effect size?
Effect sizes either measure the sizes of associations between variables or the sizes of differences
between group means.
1. Cohen’s d: Cohen's d is an appropriate effect size for the comparison between two means. It can
be used, for example, to accompany the reporting of t-test and ANOVA results.
Cohen suggested that d = 0.2 be considered a 'small' effect size, 0.5 represents a 'medium' effect size
and 0.8 a 'large' effect size. This means that if the difference between two groups' means is less than
0.2 standard deviations, the difference is negligible, even if it is statistically significant.
2. Pearson r correlation: this parameter of effect size summarises the strength of the bivariate
relationship. The value of the effect size of Pearson r correlation varies between -1 (perfect negative
correlation) to +1 (perfect positive correlation). According to Cohen:
, Type I and Type II errors
A statistically significant result cannot prove that a research hypothesis is correct (as this implies
100% certainty). Because a p-value is based on probabilities, there is always a chance of making an
incorrect conclusion regarding accepting or rejecting the null hypothesis (H0). Anytime we make a
decision using statistics there are four possible outcomes, with two representing correct decisions
and two representing errors.
The chances of committing these two types of
errors are inversely proportional: that is,
decreasing type I error rate increases type II
error rate, and vice versa.
Type I error
A type 1 error is also known as a false positive
and occurs when a researcher incorrectly rejects
a true null hypothesis. This means that you
report that your findings are significant when in
fact they have occurred by chance.
-> The probability of making a type I error is represented by your alpha level (α), which is the p-value
below which you reject the null hypothesis. A p-value of 0.05 indicates that you are willing to accept
a 5% chance that you are wrong when you reject the null hypothesis.
-> You can reduce your risk of committing a type I error by using a lower value for p. For example,
a p-value of 0.01 would mean there is a 1% chance of committing a Type I error.
-> However, using a lower value for alpha means that you will be less likely to detect a true
difference if one really exists (thus risking a type II error).
