Week 5
NHST → null hypothesis significance testing
● type I error → false positive
○ we reject a true null hypothesis
● type II error → false negative
○ we accept a false null hypothesis
What does the p value tell us?
● p = Pr(F | H0 is true)
● the p-value is the probability of the finding if the null hypothesis were true
● the p value does not measure the probability that the studied hypothesis is true, or
the probability that the data were produced by a random chance alone
● Statistical significance depends upon both sample size and effect size
○ for this reason, p values are considered to be confounded because of their
dependence on sample size
○ sometimes a statistical significant result means only that a huge sample size
was used
Effect size
● we need a different measure for the magnitude of an observed difference → effect
size
● (unstandardised) effect size → difference
○ raw effect measurement
○ uses study-specific units
○ difficult to compare across studies
● standardised effect size → mean difference relative to variability in values
○ variability across population
○ magnitude of difference (“d family”) or variance accounted for (“r family”)
without units
○ Allows comparison across studies
, Thresholds
● Have been defined for (standardised) effect sizes
● Keep in mind these are arbitrary
● Compare to other effects found in related literature as well
● r2 as a regression effect size measure, ranges from 0 to 1
● here: r2=0.33 converted to Cohen’s d: d = 1.41, i.e,. a large effect
● (this still means 67% of height is predicted by factors other than
gender)
Confidence intervals
● range of estimates for a parameter (e.g., population mean, effect
size, …), calculated at a designated confidence level
● e.g., “95% CI [<lower limit>,<upper limit> ]“
● = 95% chance your parameter will fall between the calculated
interval if sampled repeatedly
● Estimates are more precise when the sample size is large
What is statistical power?
● α → probability of a type I error scenario
● β → probability of a type II error scenario
● 1 - β → power
○ pr(we will correctly reject H0 | H0 is false)
What is power analysis
● Investigates the relationship between:
○ Sample size,
■ estimate sample size for a given effect size, alpha, and power: a priori
power analysis
○ True effect size in population,
■ estimate effect size for a given sample size, alpha, and power:
sensitivity analysis
○ Alpha, and
NHST → null hypothesis significance testing
● type I error → false positive
○ we reject a true null hypothesis
● type II error → false negative
○ we accept a false null hypothesis
What does the p value tell us?
● p = Pr(F | H0 is true)
● the p-value is the probability of the finding if the null hypothesis were true
● the p value does not measure the probability that the studied hypothesis is true, or
the probability that the data were produced by a random chance alone
● Statistical significance depends upon both sample size and effect size
○ for this reason, p values are considered to be confounded because of their
dependence on sample size
○ sometimes a statistical significant result means only that a huge sample size
was used
Effect size
● we need a different measure for the magnitude of an observed difference → effect
size
● (unstandardised) effect size → difference
○ raw effect measurement
○ uses study-specific units
○ difficult to compare across studies
● standardised effect size → mean difference relative to variability in values
○ variability across population
○ magnitude of difference (“d family”) or variance accounted for (“r family”)
without units
○ Allows comparison across studies
, Thresholds
● Have been defined for (standardised) effect sizes
● Keep in mind these are arbitrary
● Compare to other effects found in related literature as well
● r2 as a regression effect size measure, ranges from 0 to 1
● here: r2=0.33 converted to Cohen’s d: d = 1.41, i.e,. a large effect
● (this still means 67% of height is predicted by factors other than
gender)
Confidence intervals
● range of estimates for a parameter (e.g., population mean, effect
size, …), calculated at a designated confidence level
● e.g., “95% CI [<lower limit>,<upper limit> ]“
● = 95% chance your parameter will fall between the calculated
interval if sampled repeatedly
● Estimates are more precise when the sample size is large
What is statistical power?
● α → probability of a type I error scenario
● β → probability of a type II error scenario
● 1 - β → power
○ pr(we will correctly reject H0 | H0 is false)
What is power analysis
● Investigates the relationship between:
○ Sample size,
■ estimate sample size for a given effect size, alpha, and power: a priori
power analysis
○ True effect size in population,
■ estimate effect size for a given sample size, alpha, and power:
sensitivity analysis
○ Alpha, and
