Week 1 - DESIGN:
The Scientific Method:
Scientific Theory:
Falsifiable: Testable predictions,
supported by independent evidence.
Complete (accounts for most available
data) Parsimonious (simplest
explanation without redundant factors)
Theory > Testable Hypotheses:
- Translate theory into a research
question (testable), define variables
you want to measure
- Operationalise variables –
defined set of procedures for
manipulation &/or measurement of variables
of interest. Detailed enough to allow
replication
Research Design: The quality of the answer
we obtain to our research question depends on
method
Design/Methodology: Conclusions, affects,
comparisons (significance of t-test)
1. Manipulation of IV
2. Control of all other variables (e.g.,
randomly assigning, keeping variable
constant)
Non-Experimental Design: IV is not manipulated, but measured (looks at relationship between variables)
Limitations: IV not being manipulated > cannot infer that relationship is causal
Quasi-Experimental Designs: Manipulation of IV (2+ levels), control all possible, no randomisation
Limitations: No causality can be inferred & systematic differences between groups may remain
Experimental Designs: How participants are allocated to different groups within experiment
Limitations: May not be feasible to randomly assign participants (practical & ethical considerations)
Between Subjects Within Subjects
- P randomly allocated to conditions - All P goes through all conditions
- Comparisons of between groups performance - Compare every P to themselves
- Less powerful, more error larger N - Issues: maturation effects, order effects etc
Types of Factorial Designs: Allow the determination of main effect of each variable and an interaction.
Between Groups – Each
combination of levels gets a
different group of participants
Within Groups (Repeated
Measures) – Every participant
is exposed to every
combination of levels
Mixed Design – At least one of the variables is manipulated within and
one between subjects
Issues: Complexity of interpretation, power issues, complex to
counterbalance
More Than 1 Continuous IV: Test complex models overall & compare models of increasing complexity
, Week 2 – VALIDITY:
Issue of Causality: Focus on distinction between non-experimental, quasi-exp & experimental designs
- Research Q guides everything, design follows more/less naturally & aim for exp whenever possible
- Correlational when: establishing a relationship, identifying relevant parameters, ethical/practical issues
Evaluating Research: Reliability (measure accurately) Validity (measure what we are saying we measure)
Validity: The quality of being logically or factually sound (Does the conclusion follow from the premises?)
Construct Validity: Quality of measurement. The degree to which a test or instrument is capable of
measuring a concept, trait or other theoretical entity. A test has construct validity if it demonstrates an
association between the test scores and the prediction of a theoretical trait. Intelligence tests can be used
Internal Validity: Quality of research (conclusions) Does the conclusion of causality follow from data?
Internal validity refers to the degree of confidence that the causal relationship being tested is trustworthy and
not influenced by other factors or variables.
Threats to Internal Validity:
1. History: When repeated measure & single group - extraneous event/variable may have affected results.
2. Maturation: Time-related changes in a repeated measure study. (t1 to t2 change occur w/o intervention)
3. Testing: Taking one measurement of DV affects the DV itself – Practice effects, reactive measurement
4. Instrumentation: Observed changes in DV may be due to measurement device (instrument or person)
5. Statistical Regression: Repeated measure of a person, they will not always get the exact same score.
Observed difference in score is attributable to the regression to the mean score within the experiment
6. Selection: The selection method is such that groups are not equal to begin with
7. Mortality: Participants in different groups drop out at different rates – groups are not the same anymore
8. Interactions with Selection: The effects observed are the result of an interaction between the selected
groups and another threat – observed effects cannot be attributed to IV but are the result of the interaction
9. Diffusion (Imitation): results of one group contaminated by information about the other group
External Validity: Quality of research (generalisation) Do results extend to situations, population &/or
times out that being studied? External validity refers to the extent to which results from a study can be
applied to other situations, groups or events.
- Environmental (ecological validity) – Does the observed results apply to other situations (lab vs class)
- Population Validity – Does the observed result apply to other individuals (sample VS population)
- Temporal Validity – Does the observed results apply to other times?
Threats to External Validity:
1. Interaction of Testing and Treatment: Repeated measures studies, results are experimental specific
because they would only happen under pre-post-test conditions. Effect is observed > internal validity is high
2. Interaction of Selection and Treatment: Is the effect of treatment only applicable to selected groups?
3. Reactive Arrangements: Are the findings observed the result of the experimental situation? Can the
result be a reaction to the experimental situation (not the independent variable manipulation?)
4. Demand Characteristics: Characteristics
of the experiment that bias responding, P
determines demands. When looking at the
particular DV outside the experiment > no
effect. Effect is experiment specific
5. Multiple Treatment Interference: When
findings only apply to multiple testing –
learning & forgetting
The Scientific Method:
Scientific Theory:
Falsifiable: Testable predictions,
supported by independent evidence.
Complete (accounts for most available
data) Parsimonious (simplest
explanation without redundant factors)
Theory > Testable Hypotheses:
- Translate theory into a research
question (testable), define variables
you want to measure
- Operationalise variables –
defined set of procedures for
manipulation &/or measurement of variables
of interest. Detailed enough to allow
replication
Research Design: The quality of the answer
we obtain to our research question depends on
method
Design/Methodology: Conclusions, affects,
comparisons (significance of t-test)
1. Manipulation of IV
2. Control of all other variables (e.g.,
randomly assigning, keeping variable
constant)
Non-Experimental Design: IV is not manipulated, but measured (looks at relationship between variables)
Limitations: IV not being manipulated > cannot infer that relationship is causal
Quasi-Experimental Designs: Manipulation of IV (2+ levels), control all possible, no randomisation
Limitations: No causality can be inferred & systematic differences between groups may remain
Experimental Designs: How participants are allocated to different groups within experiment
Limitations: May not be feasible to randomly assign participants (practical & ethical considerations)
Between Subjects Within Subjects
- P randomly allocated to conditions - All P goes through all conditions
- Comparisons of between groups performance - Compare every P to themselves
- Less powerful, more error larger N - Issues: maturation effects, order effects etc
Types of Factorial Designs: Allow the determination of main effect of each variable and an interaction.
Between Groups – Each
combination of levels gets a
different group of participants
Within Groups (Repeated
Measures) – Every participant
is exposed to every
combination of levels
Mixed Design – At least one of the variables is manipulated within and
one between subjects
Issues: Complexity of interpretation, power issues, complex to
counterbalance
More Than 1 Continuous IV: Test complex models overall & compare models of increasing complexity
, Week 2 – VALIDITY:
Issue of Causality: Focus on distinction between non-experimental, quasi-exp & experimental designs
- Research Q guides everything, design follows more/less naturally & aim for exp whenever possible
- Correlational when: establishing a relationship, identifying relevant parameters, ethical/practical issues
Evaluating Research: Reliability (measure accurately) Validity (measure what we are saying we measure)
Validity: The quality of being logically or factually sound (Does the conclusion follow from the premises?)
Construct Validity: Quality of measurement. The degree to which a test or instrument is capable of
measuring a concept, trait or other theoretical entity. A test has construct validity if it demonstrates an
association between the test scores and the prediction of a theoretical trait. Intelligence tests can be used
Internal Validity: Quality of research (conclusions) Does the conclusion of causality follow from data?
Internal validity refers to the degree of confidence that the causal relationship being tested is trustworthy and
not influenced by other factors or variables.
Threats to Internal Validity:
1. History: When repeated measure & single group - extraneous event/variable may have affected results.
2. Maturation: Time-related changes in a repeated measure study. (t1 to t2 change occur w/o intervention)
3. Testing: Taking one measurement of DV affects the DV itself – Practice effects, reactive measurement
4. Instrumentation: Observed changes in DV may be due to measurement device (instrument or person)
5. Statistical Regression: Repeated measure of a person, they will not always get the exact same score.
Observed difference in score is attributable to the regression to the mean score within the experiment
6. Selection: The selection method is such that groups are not equal to begin with
7. Mortality: Participants in different groups drop out at different rates – groups are not the same anymore
8. Interactions with Selection: The effects observed are the result of an interaction between the selected
groups and another threat – observed effects cannot be attributed to IV but are the result of the interaction
9. Diffusion (Imitation): results of one group contaminated by information about the other group
External Validity: Quality of research (generalisation) Do results extend to situations, population &/or
times out that being studied? External validity refers to the extent to which results from a study can be
applied to other situations, groups or events.
- Environmental (ecological validity) – Does the observed results apply to other situations (lab vs class)
- Population Validity – Does the observed result apply to other individuals (sample VS population)
- Temporal Validity – Does the observed results apply to other times?
Threats to External Validity:
1. Interaction of Testing and Treatment: Repeated measures studies, results are experimental specific
because they would only happen under pre-post-test conditions. Effect is observed > internal validity is high
2. Interaction of Selection and Treatment: Is the effect of treatment only applicable to selected groups?
3. Reactive Arrangements: Are the findings observed the result of the experimental situation? Can the
result be a reaction to the experimental situation (not the independent variable manipulation?)
4. Demand Characteristics: Characteristics
of the experiment that bias responding, P
determines demands. When looking at the
particular DV outside the experiment > no
effect. Effect is experiment specific
5. Multiple Treatment Interference: When
findings only apply to multiple testing –
learning & forgetting
