Survey research Experimental lab research Archival research Experimental field research Qualitative research
Measurement reliability 1. Missing data in cross-sectional data (solve by: 1. listwise
threats deletion; 2. mean substitution)
2. Missing data in longitudinal data (solve by interpolation)
2. Inaccurately recorded data
3. Fake data
Measurement reliability For multi-item measures: Cronbach's Alpha For single-item measures: concrete measure 1. When executing survey
For multi-item measures, Cronbach's Alpha measures (like NPS) you can use
Cronbach's Alpha to demonstrate
internal consistency.
Measurement validity- 1. Social desirability bias (alchohol & tobacco, healty eating) 1. Social desirability bias can threaten the measurement validity of the dependent 1. Wrongly used proxies. (R&D funding for firm innovativeness)
threats 2. Survey mode effects (lonely people reacting to telephone variable.
surveys)
To solve social desirability bias: use leading & loading
questions like: 'everybody does it', authorities recommend it,
assume the behaviour, 'reasons for doing it'.
How to improve The validity of the measured (& manipulated) variables can The validity of the measured (& manipulated) variables can be demonstrated by: 1. Provide precedence (has the proxy been used before?) 1. Providing precedence (has it
measurement validity? be demonstrated by: 1. Providing precedence (has this been used before) 2. Provide sound logic been measured before?) AND 2.
1. Providing precedence but high-quality evidence (has this 2. Sound logic (why does this measure capture the variable?) 3. Show that there is a high correlation between a proxy and a using sound logic (why does this
been used before) 3. Manipulation checks can be used to help ensure that respondents really survey measure for a small subsample of your data (larger than measure capture the variable?)
2. Sound logic (Overlap between variables) understand the manipulation. A manipulation check is used to determine the 0.3 is substantial)
effectiveness of a manipulation in an experimental design. 4. Nomological validity (provide evidence of substantial
1. Expert input correlations (r > 0.3) with related constructs.
2. Pilot testing (minimum of 10 people)
Threat to internal validity 1. History effect: events/factors outside the experiment have an impact on the DV 1. Unexpected factors (changes in 1. Researcher bias (influence of a researcher his
during the experiment. (road obstruction) environment) prior knowledge and assumptions on his/her
2. Maturation effect: biological/psychological changes over time (hunger, tiredness) 2. Poor timing (A/B testing on a study, examples are selective perception and
3. Testing effect: respondents are influenced by premeasurement (people watch wrong day) interpretation)
commercials different because they know they are being researched) 3. Failure to randomize (sending 2. Respondent bias (participants not providing
4. Instrumentation effect: setup changes (different interviewer) catalogs to previous-buyers) honest responses but pleasing the interviewer
5. Selection bias effect: incorrect selection of respondents. 4. Failure-to-treat (subjects that = authority bias; conformity bias = doing what
6. Mortality effect: drop out of respondents during experiment are supposed to receive the majority does/thinks)
treatment do not receive it)
5. Spillovers (participant affected
by treatment of other
participants) and side effects (no
consideration of unexpected side
effect)
6. Insuffiecient sample size
To improve internal 1. Include questions relating to control variables 1. Randomization (prevents selection bias, instrumentation, history and mortality Include control variables in the statistical analysis. By including 1. Prior and post-hoc 1. Triangulation (researching from different
validity effects) control variables you can "filter out" or "isolate" the control Randomization checks (check angles, using several moderators, locations or
2. Design control (include control group that does NOT receive treatment & extra variables effect from the relationship between the variables of randomization procedure) multiple individuals)
group without pre-test but with experimental manipulation) (prevents history, interest. 2. Decide on a unit (decide on a 2. Peer debriefing: getting feedback from
maturation and testing effect) unit that matches the study goal people at different stages of the research
3. Statistical control: measure extraneuous variables and include these in the and outcome measures) (reduces research bias)
statistical analysis. (prevent history & selection effect) 3. Power calculations (how many 3. Member checking: testing emergent findings
participants do you need?) with participants (reduces researcher bias,
4. Outcome measures (needs to respondent bias and reactivity)
capture all relevant metrics) 4. Negative case analysis: investigating further
in cases that stand out from the rest of the
data.
Measurement reliability 1. Missing data in cross-sectional data (solve by: 1. listwise
threats deletion; 2. mean substitution)
2. Missing data in longitudinal data (solve by interpolation)
2. Inaccurately recorded data
3. Fake data
Measurement reliability For multi-item measures: Cronbach's Alpha For single-item measures: concrete measure 1. When executing survey
For multi-item measures, Cronbach's Alpha measures (like NPS) you can use
Cronbach's Alpha to demonstrate
internal consistency.
Measurement validity- 1. Social desirability bias (alchohol & tobacco, healty eating) 1. Social desirability bias can threaten the measurement validity of the dependent 1. Wrongly used proxies. (R&D funding for firm innovativeness)
threats 2. Survey mode effects (lonely people reacting to telephone variable.
surveys)
To solve social desirability bias: use leading & loading
questions like: 'everybody does it', authorities recommend it,
assume the behaviour, 'reasons for doing it'.
How to improve The validity of the measured (& manipulated) variables can The validity of the measured (& manipulated) variables can be demonstrated by: 1. Provide precedence (has the proxy been used before?) 1. Providing precedence (has it
measurement validity? be demonstrated by: 1. Providing precedence (has this been used before) 2. Provide sound logic been measured before?) AND 2.
1. Providing precedence but high-quality evidence (has this 2. Sound logic (why does this measure capture the variable?) 3. Show that there is a high correlation between a proxy and a using sound logic (why does this
been used before) 3. Manipulation checks can be used to help ensure that respondents really survey measure for a small subsample of your data (larger than measure capture the variable?)
2. Sound logic (Overlap between variables) understand the manipulation. A manipulation check is used to determine the 0.3 is substantial)
effectiveness of a manipulation in an experimental design. 4. Nomological validity (provide evidence of substantial
1. Expert input correlations (r > 0.3) with related constructs.
2. Pilot testing (minimum of 10 people)
Threat to internal validity 1. History effect: events/factors outside the experiment have an impact on the DV 1. Unexpected factors (changes in 1. Researcher bias (influence of a researcher his
during the experiment. (road obstruction) environment) prior knowledge and assumptions on his/her
2. Maturation effect: biological/psychological changes over time (hunger, tiredness) 2. Poor timing (A/B testing on a study, examples are selective perception and
3. Testing effect: respondents are influenced by premeasurement (people watch wrong day) interpretation)
commercials different because they know they are being researched) 3. Failure to randomize (sending 2. Respondent bias (participants not providing
4. Instrumentation effect: setup changes (different interviewer) catalogs to previous-buyers) honest responses but pleasing the interviewer
5. Selection bias effect: incorrect selection of respondents. 4. Failure-to-treat (subjects that = authority bias; conformity bias = doing what
6. Mortality effect: drop out of respondents during experiment are supposed to receive the majority does/thinks)
treatment do not receive it)
5. Spillovers (participant affected
by treatment of other
participants) and side effects (no
consideration of unexpected side
effect)
6. Insuffiecient sample size
To improve internal 1. Include questions relating to control variables 1. Randomization (prevents selection bias, instrumentation, history and mortality Include control variables in the statistical analysis. By including 1. Prior and post-hoc 1. Triangulation (researching from different
validity effects) control variables you can "filter out" or "isolate" the control Randomization checks (check angles, using several moderators, locations or
2. Design control (include control group that does NOT receive treatment & extra variables effect from the relationship between the variables of randomization procedure) multiple individuals)
group without pre-test but with experimental manipulation) (prevents history, interest. 2. Decide on a unit (decide on a 2. Peer debriefing: getting feedback from
maturation and testing effect) unit that matches the study goal people at different stages of the research
3. Statistical control: measure extraneuous variables and include these in the and outcome measures) (reduces research bias)
statistical analysis. (prevent history & selection effect) 3. Power calculations (how many 3. Member checking: testing emergent findings
participants do you need?) with participants (reduces researcher bias,
4. Outcome measures (needs to respondent bias and reactivity)
capture all relevant metrics) 4. Negative case analysis: investigating further
in cases that stand out from the rest of the
data.
