APPLIED RESEARCH METHODS: D&H
Lecture 1 – a power primer (Cohen, 1992)
Power: the probability that a test rejects H0 and H1 is true.
Type 1 error: rejecting H0 when it should not have been rejected.
Type 2 error: not rejecting H0 when it should have been rejected.
When power increases, chance of type 2 error decreases.
Beta: probability of making a type 2 error.
The mean power to detect medium effect sizes: .48
Fisherian null hypothesis: clear yes or no decision-making with probability level of p=0.05
Why does research often ignore the power analysis?
Low level of consciousness about effect size.
Only the statistical test result with p-value is looked at.
Reference material for power analysis is too complicated.
The 4 variables of statistical inference:
Sample size (N)
Significance criteria (alpha)
Population effect size (ES)
Statistical power
Sample size (N)
Number of participants for study.
Larger sample size = more certainty in results.
If you expect a big difference, you can use a smaller sample size.
Significance criteria (alpha)
The rule to decide how sure we want to be before claiming something is true.
Alpha set at 0.05 5% chance of being wrong is accepted.
If a study examines multiple things, 0.01 could be used.
Population effect size (ES)
The size of a real difference.
Most difficult part of analyses.
Low level of consciousness about the magnitude phenomena: to estimate how big or
small the difference is.
Neyman-Pearson method: the degree to which H0 is false, indicated by the
discrepancy between H0 and H1.
H0 = ES: 0
, What is a small ES?
d= .20
What is a medium ES?
d= .50
What is a large ES?
d= .80
Statistical power
The test’s long-term ability to correctly detect a true difference or effect.
Other terms: the likelihood of saying there is an effect when there really is one.
Power = 1-beta: correctly rejecting a false hypothesis.
Example: power of test is 80% = 80% chance that the test will correctly identify a real
difference of effect.
What does it mean when the power of a test is too low?
There’s a high risk of missing true effects.
What does it mean when the power of a test is too high?
The test might require more resources than is available.
What is the suggested power level and why?
80% because this strikes a balance between the risks of type 1 and type 2 errors.
Statistical tests
T-test for difference between two independent means.
o df = 2(N-1).
T-test for significance of a product moment correlation coefficient r
o df = N-2.
Difference between two independent r’s
o Fisher z transformation of r.
Binomial distribution/large curve test (for large samples).
o Population proportion (P) = .50.
o Also used in nonparametric significance test for differences between paired
observations
Normal curve test for the difference between two independent proportions.
o Arcsine transformation Φ.
o Results are the same when the test is using chi-square with df = 1.
Chi-square test for goodness of fit or association (one way) with two-way
contingency tables
o Goodness-of-fit: df = k-1.
o Contingency tables: df = (a-1) (b-1).
One way analysis of variance.
o df = g-1.
Lecture 1 – a power primer (Cohen, 1992)
Power: the probability that a test rejects H0 and H1 is true.
Type 1 error: rejecting H0 when it should not have been rejected.
Type 2 error: not rejecting H0 when it should have been rejected.
When power increases, chance of type 2 error decreases.
Beta: probability of making a type 2 error.
The mean power to detect medium effect sizes: .48
Fisherian null hypothesis: clear yes or no decision-making with probability level of p=0.05
Why does research often ignore the power analysis?
Low level of consciousness about effect size.
Only the statistical test result with p-value is looked at.
Reference material for power analysis is too complicated.
The 4 variables of statistical inference:
Sample size (N)
Significance criteria (alpha)
Population effect size (ES)
Statistical power
Sample size (N)
Number of participants for study.
Larger sample size = more certainty in results.
If you expect a big difference, you can use a smaller sample size.
Significance criteria (alpha)
The rule to decide how sure we want to be before claiming something is true.
Alpha set at 0.05 5% chance of being wrong is accepted.
If a study examines multiple things, 0.01 could be used.
Population effect size (ES)
The size of a real difference.
Most difficult part of analyses.
Low level of consciousness about the magnitude phenomena: to estimate how big or
small the difference is.
Neyman-Pearson method: the degree to which H0 is false, indicated by the
discrepancy between H0 and H1.
H0 = ES: 0
, What is a small ES?
d= .20
What is a medium ES?
d= .50
What is a large ES?
d= .80
Statistical power
The test’s long-term ability to correctly detect a true difference or effect.
Other terms: the likelihood of saying there is an effect when there really is one.
Power = 1-beta: correctly rejecting a false hypothesis.
Example: power of test is 80% = 80% chance that the test will correctly identify a real
difference of effect.
What does it mean when the power of a test is too low?
There’s a high risk of missing true effects.
What does it mean when the power of a test is too high?
The test might require more resources than is available.
What is the suggested power level and why?
80% because this strikes a balance between the risks of type 1 and type 2 errors.
Statistical tests
T-test for difference between two independent means.
o df = 2(N-1).
T-test for significance of a product moment correlation coefficient r
o df = N-2.
Difference between two independent r’s
o Fisher z transformation of r.
Binomial distribution/large curve test (for large samples).
o Population proportion (P) = .50.
o Also used in nonparametric significance test for differences between paired
observations
Normal curve test for the difference between two independent proportions.
o Arcsine transformation Φ.
o Results are the same when the test is using chi-square with df = 1.
Chi-square test for goodness of fit or association (one way) with two-way
contingency tables
o Goodness-of-fit: df = k-1.
o Contingency tables: df = (a-1) (b-1).
One way analysis of variance.
o df = g-1.
