Building Body of Research
Putting it All Together
➔ scientific literature: series of related studies, by various researchers, conducted on the same
topic
◆ aka ‘body of research’
➔ theory testing and replication provide foundation for building a body of research on a topic
➔ theory building vs theory testing
◆ theory building = observation → data collection → pattern →theory
◆ theory testing (hypothesis testing) =theory→ hypothesis→ data collection → theory
testing
Replication Is Key
➔ direct replication: conducting the exact same study
◆ using a sample from thesamepopulation
◆ using thesamemethods
◆ establishesstatistical validity
◆ goal is to control for type 1 error
➔ conceptual replication (across diff populations):re-testing the same theoretical
idea/hypothesis using adifferent population
◆ establishesexternal validity
● seeing whether the same pattern of results generalize to other populations
➔ conceptual replication (across diff variable operationalizations):re-testing the same
theoretical idea/hypothesis usingdifferent operationalizationsof thesame constructs
◆ establishesconstruct validity
● seeing if the same pattern of results occur even when a construct is measured
or manipulated in a diff way
➔ pattern and parsimony: internal validity is strengthenedwhen patterns of causality can be
established across multiple experimental studies
➔ goals of replication can be combined and established in a single study
Summarizing Body of Research
➔ review article:summarizes the key trendsthat emergefrom the scientific literature
◆ variety of kinds with diff focuses (i.e. some focus on the research, some focus on the
methods)
◆ a more qualitative form of analysis
➔ meta-analysis: averaging the statistical results ofmultiple studies toexamine the overall
effect size
◆ a more quantitative form of analysis
, Conducting a Literature Review
➔ the steps
◆ formulate question
◆ identify criterion
◆ kew word search
◆ scan abstracts (prescreening the articles)
◆ choose articles
◆ extract patterns
◆ organize info
◆ write paper on present results
➔ flows similar to a regular study
Conducting A Meta-Analysis
➔ the steps
◆ formulate a question
◆ identify criterion
◆ key word search
◆ scan abstracts
◆ choose articles
◆ extract effects
◆ run analyses
◆ report results
Publication Bias
➔ null findings are less likely to be published
◆ “the file drawer” problem
◆ focusing on only published work could pose a issue with bias for conducting a review
or a meta-analysis
➔ solution: including both published and unpublished studies
Interpreting Forest Plots
➔ meta-analyses average the effect size for each study in order to calculate the overall effect
size
➔ sometimes diff studies give diff weights in calculations based on sample size, quality of the
design, etc
◆ accounted for in the plot’s size
● smaller plot = lower weight
● bigger plot = higher weight
➔ the distribution of effect size can be compared to a line of ‘no effect’
◆ the line defines the null hypothesis
Putting it All Together
➔ scientific literature: series of related studies, by various researchers, conducted on the same
topic
◆ aka ‘body of research’
➔ theory testing and replication provide foundation for building a body of research on a topic
➔ theory building vs theory testing
◆ theory building = observation → data collection → pattern →theory
◆ theory testing (hypothesis testing) =theory→ hypothesis→ data collection → theory
testing
Replication Is Key
➔ direct replication: conducting the exact same study
◆ using a sample from thesamepopulation
◆ using thesamemethods
◆ establishesstatistical validity
◆ goal is to control for type 1 error
➔ conceptual replication (across diff populations):re-testing the same theoretical
idea/hypothesis using adifferent population
◆ establishesexternal validity
● seeing whether the same pattern of results generalize to other populations
➔ conceptual replication (across diff variable operationalizations):re-testing the same
theoretical idea/hypothesis usingdifferent operationalizationsof thesame constructs
◆ establishesconstruct validity
● seeing if the same pattern of results occur even when a construct is measured
or manipulated in a diff way
➔ pattern and parsimony: internal validity is strengthenedwhen patterns of causality can be
established across multiple experimental studies
➔ goals of replication can be combined and established in a single study
Summarizing Body of Research
➔ review article:summarizes the key trendsthat emergefrom the scientific literature
◆ variety of kinds with diff focuses (i.e. some focus on the research, some focus on the
methods)
◆ a more qualitative form of analysis
➔ meta-analysis: averaging the statistical results ofmultiple studies toexamine the overall
effect size
◆ a more quantitative form of analysis
, Conducting a Literature Review
➔ the steps
◆ formulate question
◆ identify criterion
◆ kew word search
◆ scan abstracts (prescreening the articles)
◆ choose articles
◆ extract patterns
◆ organize info
◆ write paper on present results
➔ flows similar to a regular study
Conducting A Meta-Analysis
➔ the steps
◆ formulate a question
◆ identify criterion
◆ key word search
◆ scan abstracts
◆ choose articles
◆ extract effects
◆ run analyses
◆ report results
Publication Bias
➔ null findings are less likely to be published
◆ “the file drawer” problem
◆ focusing on only published work could pose a issue with bias for conducting a review
or a meta-analysis
➔ solution: including both published and unpublished studies
Interpreting Forest Plots
➔ meta-analyses average the effect size for each study in order to calculate the overall effect
size
➔ sometimes diff studies give diff weights in calculations based on sample size, quality of the
design, etc
◆ accounted for in the plot’s size
● smaller plot = lower weight
● bigger plot = higher weight
➔ the distribution of effect size can be compared to a line of ‘no effect’
◆ the line defines the null hypothesis
