MOVIECLIP. Power and effect size (practical 1).
Effect size
an objective and standardized measure of the size of an observed effect.
It measures the proportion of total variance in the dependent variable, that is associated with
the membership of different groups as defined in the independent variable.
Partial eta-squared (pn2) (ANOVA)
Small effect: .01
Medium effect: .06
Large effect: >.14
Pearson r (correlation) or regression B
Small effect: .10
Medium effect: .30
Large effect: >.50
Power and effect size
Statistical power has an influence on the possibility of detecting an existing effect of a
particular size, therefore the opportunity to correctly reject the null hypothesis.
The power is 1 – B, where B is the probability of a type II error.
Note: This B is a different B then the B in a regression analysis.
Goal power of at least .80
True statements about power and effect size.
1. If your effect size is large and your sample is large,
you can assume that your power is large.
2. With a small or large number of participants, it is important to look at the effect size
before you draw conclusions about your findings.
Type I error
Type I error (alpha) is the probability that an effect will be detected, where in fact no effect
exists (false positive): the null hypothesis (H0) is rejected when in fact it is true.
Type II error
Type II error (beta) is the probability that no effect will be detected where an effect does in
fact exist (false negative): the null hypothesis (H0) is not rejected when in fact it is false.
The possibility to detect an effect, the power, depends on:
The p-value/alpha (type I error)
The beta (type II error). Since the power is 1-beta.
Effect size
Sample size
, Power and effect size
Overlap between the 2
distributions is the type I
error. When there is no
overlap, this will be zero.
When there is overlap,
there is a change there will
be a false negative (type II
error)
The type II error
represents the overlap
between the two
distributions, where the p-
value is higher than chance.
Example 2. Larger sample size leads to less variance in spread (more accurate
estimate) and therefore a steeper curve. This also results in a smaller overlap between
both curves. The alpha becomes smaller (more likely to be significant) and also the
beta declines. This means
that the power increases.
Example 3. When the
means are further apart (or
the SD reduces) the effect
size is bigger. This leads to
further distancing of the
distributions and reducing
the overlap. This increases
the power.
Example.
1. 1000 participants
F(986)= 23.45, p<.001.
n2= .02
P-value is large, but the effect size is small. This indicates a less relevant effect.
2. 50 participants
F(46)=23.45, p<.05. n2=.20
P-value is smaller, but still significant. The effect size is larger and more relevant.
Effect sizes have great value to drive clinical decision making.
MOVIECLIP. Manipulation and randomization checks.
Experimental research.
Objective= establish causal relationships
You want to know whether variable A causes changes in variable B.
There are three relevant comparisons:
1. Baseline comparison. Determine whether the control group and experimental groups
differ at baseline. For this comparison you can use a randomization check.
With a randomization test you check whether the participants in your study are
allocated to a group at random or whether this wat determined by the researcher.
Effect size
an objective and standardized measure of the size of an observed effect.
It measures the proportion of total variance in the dependent variable, that is associated with
the membership of different groups as defined in the independent variable.
Partial eta-squared (pn2) (ANOVA)
Small effect: .01
Medium effect: .06
Large effect: >.14
Pearson r (correlation) or regression B
Small effect: .10
Medium effect: .30
Large effect: >.50
Power and effect size
Statistical power has an influence on the possibility of detecting an existing effect of a
particular size, therefore the opportunity to correctly reject the null hypothesis.
The power is 1 – B, where B is the probability of a type II error.
Note: This B is a different B then the B in a regression analysis.
Goal power of at least .80
True statements about power and effect size.
1. If your effect size is large and your sample is large,
you can assume that your power is large.
2. With a small or large number of participants, it is important to look at the effect size
before you draw conclusions about your findings.
Type I error
Type I error (alpha) is the probability that an effect will be detected, where in fact no effect
exists (false positive): the null hypothesis (H0) is rejected when in fact it is true.
Type II error
Type II error (beta) is the probability that no effect will be detected where an effect does in
fact exist (false negative): the null hypothesis (H0) is not rejected when in fact it is false.
The possibility to detect an effect, the power, depends on:
The p-value/alpha (type I error)
The beta (type II error). Since the power is 1-beta.
Effect size
Sample size
, Power and effect size
Overlap between the 2
distributions is the type I
error. When there is no
overlap, this will be zero.
When there is overlap,
there is a change there will
be a false negative (type II
error)
The type II error
represents the overlap
between the two
distributions, where the p-
value is higher than chance.
Example 2. Larger sample size leads to less variance in spread (more accurate
estimate) and therefore a steeper curve. This also results in a smaller overlap between
both curves. The alpha becomes smaller (more likely to be significant) and also the
beta declines. This means
that the power increases.
Example 3. When the
means are further apart (or
the SD reduces) the effect
size is bigger. This leads to
further distancing of the
distributions and reducing
the overlap. This increases
the power.
Example.
1. 1000 participants
F(986)= 23.45, p<.001.
n2= .02
P-value is large, but the effect size is small. This indicates a less relevant effect.
2. 50 participants
F(46)=23.45, p<.05. n2=.20
P-value is smaller, but still significant. The effect size is larger and more relevant.
Effect sizes have great value to drive clinical decision making.
MOVIECLIP. Manipulation and randomization checks.
Experimental research.
Objective= establish causal relationships
You want to know whether variable A causes changes in variable B.
There are three relevant comparisons:
1. Baseline comparison. Determine whether the control group and experimental groups
differ at baseline. For this comparison you can use a randomization check.
With a randomization test you check whether the participants in your study are
allocated to a group at random or whether this wat determined by the researcher.
