Experimental Clinical Psychology (ECP) – Leiden University 2021-2022
Lecture 1: The validity concept
Clinical Psychology: the study of understanding mental, emotional and behavioural disorders
or problems. Including prevention, assessment, diagnosis, and treatment of mental, emotional
and behavioural disorders/problems. Niko Tinbergen (1907-1988): a behaviour/emotion is
understood when you understand:
1. Its development
2. The underlying mechanisms
3. Its function(s): the role that it serves.
4. Its evolution
For understanding we need scientific experimentation, because it has proven the best option
available. It can eliminate crazy ideas, but it is not flawless. The validity concept will help
you recognize flaws in scientific work: in the ECP course we will point out typical flaws.
VALIDITY…?
Oxford dictionary: the quality of being logically or factually sound: soundness or cogency (=
clear, logical, convincing). Validity starts with the theory and is evident in all steps of science.
Theory Hypothesis Data* Analysis** Reporting.
*Incl. recruitment, measurement – for all types of data.
**Incl. interpretation.
The validity of the end product
How does it work? What is the underlying mechanism? = MEDIATON
For whom does it work? Who is affected by … this? = MODERATION
It influences the association.
MEDIATION
Below: Article to show how it works: Problem-
Solving Education (PSE) to Prevent Depression
Among Low-Income Mothers.
MODERATION: is always a 3-variable issue, for instance:
- Stress exposure alcohol consumption – moderator: gender.
- Healthy diet subjective depression score – moderator: expectancy.
A moderator is already present and a mediator changes over the course of for example the
treatment.
1
, Experimental Clinical Psychology (ECP) – Leiden University 2021-2022
What is moderation? (Fournier et al. 2010, JAMA Psychiatry)
ADM work quite well when people are severe
depressed when they enter the trial. Not so
severe depressed the difference between
placebo and anti-depressants is not that big. The
effects of anti-depressants on average only
appear when depression in intake is high.
Conclusion: people should only be treated with
anti-depressants when they are severe depressed.
So, this association is moderated by severity.
VALIDITY:
Oxford dictionary: the quality of being logically or factually sound: soundness or cogency.
Kazdin: … only per sub-type of validity.
Validity: a multi-dimensional construct: 4 types of experimental validity used to
evaluate the methodology of a study and the question they address.
1. Internal validity:
= to what extent can the intervention rather than extraneous influences be considered to
account for the results, changes, or differences among conditions (e.g., baseline,
intervention)?
In other words: e.g., to what extent does the intervention [or what researcher manipulates /
does] account for change …
2. Construct validity:
= given that the experimental manipulation or intervention was responsible for change,
what specific aspect of the manipulation was the mechanism, process, or causal agent?
What is the conceptual basis (construct) underlying the effect? In other words: what is the
conceptual basis [i.e., construct] of the phenomenon under study …
3. External validity:
= to what extent can the results be generalized or extended to people, settings, times,
measures/outcomes, and characteristics other than those included in this particular
demonstration?
4. Data-evaluation validity:
= to what extent is a relation shown, demonstrated, or evident between the experimental
manipulation or intervention and the outcome? What about the data and methods used for
evaluation that could mislead or obscure demonstrating or failing to demonstrate an
experimental effect?
In other words: to what extent does the data + analyses demonstrate that what is said …
NOT yes versus no and dependent.
1. INTERNAL VALIDITY: threats to internal validity …
Maturation: within-subject change over time not due to the intervention: e.g., growing
older, stronger, growing tired, healthier, smarter, and more tired or bored.
Testing: within-subjects over time not due to the intervention but to tests: e.g., participants
get good at it …
Instrumentation: within-researcher/study e.g., a change in criteria …
2
, Experimental Clinical Psychology (ECP) – Leiden University 2021-2022
In general, internal validity may be threatened …(I) poor design, (II) good design, sloppy
conduct, (III) good design, but hard to control (e.g., attrition, non-blinding)
How to safeguard against threats to internal validity? Randomization (maturation,
confounding, …) PLUS think of the threats beforehand and eliminate/minimize their effect.
2. CONSTRUCT VALIDITY … is causality due to the construct
How to safeguard against threats to construct validity?
Think of the threats beforehand and eliminate/minimize
their effect.
3. EXTERNAL VALIDITY: to what extent can the results be generalized / extended to other
people, …, settings …
The International Society for Nutritional Psychiatry Research (ISNPR) (Lancet Psychiatry,
2015)
Consensus statement [review-like]: “… Diet and nutrition are crucial factors in the high
incidence of depression …”
External validity --- cont.
Creating a timeline and outcome
definition.
In general, external validity may be threatened …
Lack of generalizability of measures (e.g., symptoms vs. disorder).
Selected samples (students)
Narrow stimulus sampling (e.g., a single therapist in a trial)
Reactivity, responsiveness, sensitization (+ novelty) …
How to safeguard against threats to external validity? Think of the threats beforehand and
eliminate/minimize their effect PLUS fairly acknowledge to whom findings may apply and
why.
4. DATA-EVALUATION VALIDITY: to what extent does the data + analyses demonstrate
that what is said … [Erich von Wolf (Germany, 1870): Zero data-evaluation validity Zero
construct- and external validity]
Lack of data-evaluation validity – cont.
How to safeguard against threats to validity?
Think of the threats beforehand: eliminate their effect.
Use the strengths of science and follow its rules.
In the article for the next workgroup: Threats to validity [mostly internal]
3
Lecture 1: The validity concept
Clinical Psychology: the study of understanding mental, emotional and behavioural disorders
or problems. Including prevention, assessment, diagnosis, and treatment of mental, emotional
and behavioural disorders/problems. Niko Tinbergen (1907-1988): a behaviour/emotion is
understood when you understand:
1. Its development
2. The underlying mechanisms
3. Its function(s): the role that it serves.
4. Its evolution
For understanding we need scientific experimentation, because it has proven the best option
available. It can eliminate crazy ideas, but it is not flawless. The validity concept will help
you recognize flaws in scientific work: in the ECP course we will point out typical flaws.
VALIDITY…?
Oxford dictionary: the quality of being logically or factually sound: soundness or cogency (=
clear, logical, convincing). Validity starts with the theory and is evident in all steps of science.
Theory Hypothesis Data* Analysis** Reporting.
*Incl. recruitment, measurement – for all types of data.
**Incl. interpretation.
The validity of the end product
How does it work? What is the underlying mechanism? = MEDIATON
For whom does it work? Who is affected by … this? = MODERATION
It influences the association.
MEDIATION
Below: Article to show how it works: Problem-
Solving Education (PSE) to Prevent Depression
Among Low-Income Mothers.
MODERATION: is always a 3-variable issue, for instance:
- Stress exposure alcohol consumption – moderator: gender.
- Healthy diet subjective depression score – moderator: expectancy.
A moderator is already present and a mediator changes over the course of for example the
treatment.
1
, Experimental Clinical Psychology (ECP) – Leiden University 2021-2022
What is moderation? (Fournier et al. 2010, JAMA Psychiatry)
ADM work quite well when people are severe
depressed when they enter the trial. Not so
severe depressed the difference between
placebo and anti-depressants is not that big. The
effects of anti-depressants on average only
appear when depression in intake is high.
Conclusion: people should only be treated with
anti-depressants when they are severe depressed.
So, this association is moderated by severity.
VALIDITY:
Oxford dictionary: the quality of being logically or factually sound: soundness or cogency.
Kazdin: … only per sub-type of validity.
Validity: a multi-dimensional construct: 4 types of experimental validity used to
evaluate the methodology of a study and the question they address.
1. Internal validity:
= to what extent can the intervention rather than extraneous influences be considered to
account for the results, changes, or differences among conditions (e.g., baseline,
intervention)?
In other words: e.g., to what extent does the intervention [or what researcher manipulates /
does] account for change …
2. Construct validity:
= given that the experimental manipulation or intervention was responsible for change,
what specific aspect of the manipulation was the mechanism, process, or causal agent?
What is the conceptual basis (construct) underlying the effect? In other words: what is the
conceptual basis [i.e., construct] of the phenomenon under study …
3. External validity:
= to what extent can the results be generalized or extended to people, settings, times,
measures/outcomes, and characteristics other than those included in this particular
demonstration?
4. Data-evaluation validity:
= to what extent is a relation shown, demonstrated, or evident between the experimental
manipulation or intervention and the outcome? What about the data and methods used for
evaluation that could mislead or obscure demonstrating or failing to demonstrate an
experimental effect?
In other words: to what extent does the data + analyses demonstrate that what is said …
NOT yes versus no and dependent.
1. INTERNAL VALIDITY: threats to internal validity …
Maturation: within-subject change over time not due to the intervention: e.g., growing
older, stronger, growing tired, healthier, smarter, and more tired or bored.
Testing: within-subjects over time not due to the intervention but to tests: e.g., participants
get good at it …
Instrumentation: within-researcher/study e.g., a change in criteria …
2
, Experimental Clinical Psychology (ECP) – Leiden University 2021-2022
In general, internal validity may be threatened …(I) poor design, (II) good design, sloppy
conduct, (III) good design, but hard to control (e.g., attrition, non-blinding)
How to safeguard against threats to internal validity? Randomization (maturation,
confounding, …) PLUS think of the threats beforehand and eliminate/minimize their effect.
2. CONSTRUCT VALIDITY … is causality due to the construct
How to safeguard against threats to construct validity?
Think of the threats beforehand and eliminate/minimize
their effect.
3. EXTERNAL VALIDITY: to what extent can the results be generalized / extended to other
people, …, settings …
The International Society for Nutritional Psychiatry Research (ISNPR) (Lancet Psychiatry,
2015)
Consensus statement [review-like]: “… Diet and nutrition are crucial factors in the high
incidence of depression …”
External validity --- cont.
Creating a timeline and outcome
definition.
In general, external validity may be threatened …
Lack of generalizability of measures (e.g., symptoms vs. disorder).
Selected samples (students)
Narrow stimulus sampling (e.g., a single therapist in a trial)
Reactivity, responsiveness, sensitization (+ novelty) …
How to safeguard against threats to external validity? Think of the threats beforehand and
eliminate/minimize their effect PLUS fairly acknowledge to whom findings may apply and
why.
4. DATA-EVALUATION VALIDITY: to what extent does the data + analyses demonstrate
that what is said … [Erich von Wolf (Germany, 1870): Zero data-evaluation validity Zero
construct- and external validity]
Lack of data-evaluation validity – cont.
How to safeguard against threats to validity?
Think of the threats beforehand: eliminate their effect.
Use the strengths of science and follow its rules.
In the article for the next workgroup: Threats to validity [mostly internal]
3
