ARM artikelen
Lecture 1
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not
WEIRD. Nature, 466(7302), 29-29.
WEIRD: Western, Educated, Industrialized, Rich and Democratic. 96% of subjects were
from Western industrialized countries – which house just 12% of the world’s
population.
Four suggestions:
• Editors and reviewers should push researchers to support any generalizations
with evidence.
• Granting agencies, reviewers and editors should give researchers credit for
comparing diverse and inconvenient subject pools.
• Granting agencies should prioritize cross-disciplinary, cross-cultural research.
• Researchers must strive to evaluate how their findings apply to other
populations.
Low-cost ways to approach this in short term:
• Select a few judiciously chosen populations that provide a ‘tough test’ of
universality in some domain, such as societies with limited counting systems for
testing theories about numerical cognition.
• Long term: establish a set of principles that researchers can use to distinguish
variable from universal aspects of psychology.
Shaughnessy et al. (2015). Research methods in psychology: Sampling in survey
research (p. 138-144).
Careful selection of a survey sample allows researchers to generalize findings from the
sample to the population.
Selection bias: when there is overrepresentation or underrepresentation of some
segments of the population > the characteristics of the sample are systematically
different from the characteristics of the population.
Population: set of all cases of interest
Sampling frame: “list” of members of the population in order to select a subset of that
population
Sample: the subset of the population actually drawn from the sampling frame
The “power” of samples to describe the larger population is based on the assumption
that survey responses in a sample can be applied to the population from which the
sample was drawn.
,Representativeness: ability to generalize from a sample to the population. A sample is
representative of the population to the extent that it exhibits the same distribution of
characteristics as the population.
Biased sample: distribution of characteristics of the sample is systematically different
from the population
- Selection bias: when procedures used to select the sample result in
overrepresentation/underrepresentation of a segment of the population. Zie
hierboven.
- Response rate bias
Approaches to selecting a survey sample
- Probability sampling: the method of choice for obtaining a representative
sample; two types of PS:
o Simple random sampling: each element of the population had an equal
chance of being included in the sample
o Stratified random sampling: the population is divided into
subpopulations (strata) and random samples are drawn from the strata
- Nonprobability sampling: (convenience sampling) does not guarantee that
every element in the population had an equal chance of being included.
o Rule with convenience sampling > you should consider that convenience
sampling will result in a biased sample unless you have strong evidence
confirming the representativeness of the sample.
Sample depends on the degree of variability in the population
> heterogeneous <> homogeneous.
Most samples are somewhere in between.
Perugini, Galluci, & Constantini (2018). A practical primer to power analysis for
simple experimental designs (p. 1-9).
Interest in power has increased → recent replicability crisis. Systematically performing
studies lacked the power to detect effect sizes of interest results in a prevalence of false-
positive findings in the literature. One of main benefits of power analysis is that
researchers become aware of their chances of finding an effect of interest.
• Type I error: rejecting the null hypothesis when it is true (false positive, )
• Type II error: failing to reject it when it is false (false negative, )
• Power of a statistical test = the probability of successfully rejecting the null
hypothesis when it is false (1-). Power depends on sample size, effect size, and
the decision criterion (-level).
• Power increases with increasing sample size, increasing effect size, and more
lenient decision criteria (=0.10 instead of =.01).
→ Statistical power matters not only because it directly increases the likelihood of
finding an effect if it exists, but also because it contributes indirectly to reducing the
overall rate of data interference errors.
, Cohen’s d = effect size as the standardized mean difference between two conditions
And with its conventional values of 0.20, 0.50 and 0.80 to indicate a small, medium and
large effect size.
AUC is the effect size as the probability that a person picked at random from one group
will have a higher score than a person picked at random from the other group. 0.50 =
null (no improvement from a random selection device).
• A priori (prospective) power analysis: goal is to achieve a given desirable power
level (0.8) given a certain -level. Once power and -level are fixed, it is required
to estimate an expected effect size and then calculate how many participants are
needed to achieve the desired power. Problem: the expected size is one’s best
guess, and its inaccuracy has substantial implications for the actual sample size
needed to achieve the desired level of power.
o Consider different scenarios by varying the expected effect size
o Consider the uncertainty in the estimate, which is reflected in its
confidence interval, and then settle on a sample size that takes into
account the desired level of protection against overestimating the effect
size and consequently running an underpowered study.
• Sensitivity analysis: When researchers do not have much leeway for increasing
sample size, they have a relatively fixed maximum sample size.) Useful for
determining the strength of an effect that can be reliably detected. Requires fixing
a certain -level, the available sample size and a desired level of power to identify
the minimum size of the effect that can be reliably detected.
Aim of power analysis in following examples is prospective to estimate the minimum
sample size N necessary to obtain a statistically significant test with a certain likelihood
applied to the expected effect size index.
G*power (dit is hetzelfde als a priori)
• Select the appropriate test in the Test Family menu by choosing t-test.
• Statistical test menu: select “Means, Difference between two independent means
(two groups)
• Select “A priori”
Sensitivity analysis:
• Select “Sensitivity” in “Type of Analysis”
• Plugging in results we just obtained
Lecture 1
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not
WEIRD. Nature, 466(7302), 29-29.
WEIRD: Western, Educated, Industrialized, Rich and Democratic. 96% of subjects were
from Western industrialized countries – which house just 12% of the world’s
population.
Four suggestions:
• Editors and reviewers should push researchers to support any generalizations
with evidence.
• Granting agencies, reviewers and editors should give researchers credit for
comparing diverse and inconvenient subject pools.
• Granting agencies should prioritize cross-disciplinary, cross-cultural research.
• Researchers must strive to evaluate how their findings apply to other
populations.
Low-cost ways to approach this in short term:
• Select a few judiciously chosen populations that provide a ‘tough test’ of
universality in some domain, such as societies with limited counting systems for
testing theories about numerical cognition.
• Long term: establish a set of principles that researchers can use to distinguish
variable from universal aspects of psychology.
Shaughnessy et al. (2015). Research methods in psychology: Sampling in survey
research (p. 138-144).
Careful selection of a survey sample allows researchers to generalize findings from the
sample to the population.
Selection bias: when there is overrepresentation or underrepresentation of some
segments of the population > the characteristics of the sample are systematically
different from the characteristics of the population.
Population: set of all cases of interest
Sampling frame: “list” of members of the population in order to select a subset of that
population
Sample: the subset of the population actually drawn from the sampling frame
The “power” of samples to describe the larger population is based on the assumption
that survey responses in a sample can be applied to the population from which the
sample was drawn.
,Representativeness: ability to generalize from a sample to the population. A sample is
representative of the population to the extent that it exhibits the same distribution of
characteristics as the population.
Biased sample: distribution of characteristics of the sample is systematically different
from the population
- Selection bias: when procedures used to select the sample result in
overrepresentation/underrepresentation of a segment of the population. Zie
hierboven.
- Response rate bias
Approaches to selecting a survey sample
- Probability sampling: the method of choice for obtaining a representative
sample; two types of PS:
o Simple random sampling: each element of the population had an equal
chance of being included in the sample
o Stratified random sampling: the population is divided into
subpopulations (strata) and random samples are drawn from the strata
- Nonprobability sampling: (convenience sampling) does not guarantee that
every element in the population had an equal chance of being included.
o Rule with convenience sampling > you should consider that convenience
sampling will result in a biased sample unless you have strong evidence
confirming the representativeness of the sample.
Sample depends on the degree of variability in the population
> heterogeneous <> homogeneous.
Most samples are somewhere in between.
Perugini, Galluci, & Constantini (2018). A practical primer to power analysis for
simple experimental designs (p. 1-9).
Interest in power has increased → recent replicability crisis. Systematically performing
studies lacked the power to detect effect sizes of interest results in a prevalence of false-
positive findings in the literature. One of main benefits of power analysis is that
researchers become aware of their chances of finding an effect of interest.
• Type I error: rejecting the null hypothesis when it is true (false positive, )
• Type II error: failing to reject it when it is false (false negative, )
• Power of a statistical test = the probability of successfully rejecting the null
hypothesis when it is false (1-). Power depends on sample size, effect size, and
the decision criterion (-level).
• Power increases with increasing sample size, increasing effect size, and more
lenient decision criteria (=0.10 instead of =.01).
→ Statistical power matters not only because it directly increases the likelihood of
finding an effect if it exists, but also because it contributes indirectly to reducing the
overall rate of data interference errors.
, Cohen’s d = effect size as the standardized mean difference between two conditions
And with its conventional values of 0.20, 0.50 and 0.80 to indicate a small, medium and
large effect size.
AUC is the effect size as the probability that a person picked at random from one group
will have a higher score than a person picked at random from the other group. 0.50 =
null (no improvement from a random selection device).
• A priori (prospective) power analysis: goal is to achieve a given desirable power
level (0.8) given a certain -level. Once power and -level are fixed, it is required
to estimate an expected effect size and then calculate how many participants are
needed to achieve the desired power. Problem: the expected size is one’s best
guess, and its inaccuracy has substantial implications for the actual sample size
needed to achieve the desired level of power.
o Consider different scenarios by varying the expected effect size
o Consider the uncertainty in the estimate, which is reflected in its
confidence interval, and then settle on a sample size that takes into
account the desired level of protection against overestimating the effect
size and consequently running an underpowered study.
• Sensitivity analysis: When researchers do not have much leeway for increasing
sample size, they have a relatively fixed maximum sample size.) Useful for
determining the strength of an effect that can be reliably detected. Requires fixing
a certain -level, the available sample size and a desired level of power to identify
the minimum size of the effect that can be reliably detected.
Aim of power analysis in following examples is prospective to estimate the minimum
sample size N necessary to obtain a statistically significant test with a certain likelihood
applied to the expected effect size index.
G*power (dit is hetzelfde als a priori)
• Select the appropriate test in the Test Family menu by choosing t-test.
• Statistical test menu: select “Means, Difference between two independent means
(two groups)
• Select “A priori”
Sensitivity analysis:
• Select “Sensitivity” in “Type of Analysis”
• Plugging in results we just obtained
