WEEK 6
CHAPTER 9 - EXPERIMENTAL DESIGN
9.1 FOUNDATIONS OF EXPERIMENTAL DESIGN
Experimental designs are labeled as the most “rigorous” of all research designs
– All other designs are judged against it
From a theoretical perspective: strongest design (implementing it well to establish internal
validity)
+ best to establish cause-effect relationship
9.2 INTRODUCTION: THE ORIGINS OF EXPERIMENTAL DESIGN
– Considered to have its roots in the agricultural experiments of the early twentieth century
– Trying to figure out combinations of newly developed artificial fertiliser treatments
that improve agricultural yield
– Ronald Fisher credited with developing the random experimental design (1919)
– Suggested that researchers divide the large farm into smaller plots and randomly
assign different fertilisers to different areas
– Demonstrated that randomness had the ability to untangle the effect of a particular
intervention from all other potential confounding factors
– But as early as 1901, Edward L. Thorndike and Robert S. Woodworth had identified the
need for using a control group in experiments to study outcomes like “learning” and
“intelligence”
Random assignment = assigning your sample into two or more subgroups by chance
(9.2a) Distinguishing features of experimental design
Best to establish internal validity
– Causal, cause-effect relationships
But you need to address that:
If X, then Y
And
If not X, then not Y
(in a comparative research e.g. experimental or quasi-experimental design)
Program/Treatment group = receives program of interest and usually contrasted with control
group or group receiving another treatment
Comparison group = compared to contrasted with group that receives program or intervention
of interest
Probabilistically equivalent = the notion that two groups, if measured indefinitely, would on
average perform identically. (But doesn’t mean they would obtain the exact same average
score
,(9.2b) Experimental design and threats to internal validity
Significant advantages:
. Relies on having two groups (program & control)
. Random assignment ensures that the two groups are equivalent to each other before a
treatment is administered to any one group
– It is strong against all the single group threats to internal validity
– Also strong against multiple group threats, except selection mortality
– e.g. strong against selection-history and selection-maturation threats, because the
two groups are created to be probabilistically equivalent to begin with
– e.g. no pretest, so also strong against selection-testing and selection-instrumentation
threats when it doesn’t use repeated (pre-post) measurement
– Has all social threats to internal validity
– Requires random assignment, so in institutional settings like schools it is more likely
that participants will be aware of each other and of the conditions assigned to them
, (9.2c) Design notation of a two-group experimental design
Simplest: two-group posttest-only randomised experiment
– Best to establish cause-effect relationships
Most interested in determining whether the two groups are different after the program
– You measure the groups on one or more measures
– Compare them by testing for the differences between the means, using a t-test or one-
way analysis of variance (ANOVA)
– Enables you to make a decision about whether any difference you observed between
the groups is likely to be due to chance or not
(9.2d) Difference between random selection and assignment
– Random selection is how you draw the sample of people for your study from a
population
– Random assignment is how you assign the sample that you draw to different groups or
treatments in your study
Possible to have both or one of them or neither
9.3 CLASSIFYING EXPERIMENTAL DESIGNS
Can be classified using a simple signal-to-noise ratio metaphor
– Assume that what you observe or see in a research study can be divided in: signal and
noise
Variability = the extent to which the values measured or observed for a variable differ
Signal: related to the key variable of interest (the contrast you’re trying to measure or the
effect of the program)
Noise: consists of all random factors in the environment that make it hard to se the signal
(e.g. local distractions, how people felt that day etc)
– In order to see true effect: signal needs to be high relative to noise
– e.g. powerful treatment (strong signal) and good measurement (low noise) = higher
chance of seeing true effects
So this helps us classify experimental design into two categories:
. Signal enhancers
○ Signal enhancing designs are called factorial designs
◆ Focus on the program or treatment, its components and its major dimensions
and enable you to determine whether the program has an effect, whether
different subcomponents are effective and whether there are interaction in its
effects caused by subcomponents
○ You would examine several different variations of the treatment
. Noise reducers
○ Covariance designs
◆ Help minimise noise through the inclusion of one or more variables (covariates)
that account for some of the variability in the outcome measure or dependent
CHAPTER 9 - EXPERIMENTAL DESIGN
9.1 FOUNDATIONS OF EXPERIMENTAL DESIGN
Experimental designs are labeled as the most “rigorous” of all research designs
– All other designs are judged against it
From a theoretical perspective: strongest design (implementing it well to establish internal
validity)
+ best to establish cause-effect relationship
9.2 INTRODUCTION: THE ORIGINS OF EXPERIMENTAL DESIGN
– Considered to have its roots in the agricultural experiments of the early twentieth century
– Trying to figure out combinations of newly developed artificial fertiliser treatments
that improve agricultural yield
– Ronald Fisher credited with developing the random experimental design (1919)
– Suggested that researchers divide the large farm into smaller plots and randomly
assign different fertilisers to different areas
– Demonstrated that randomness had the ability to untangle the effect of a particular
intervention from all other potential confounding factors
– But as early as 1901, Edward L. Thorndike and Robert S. Woodworth had identified the
need for using a control group in experiments to study outcomes like “learning” and
“intelligence”
Random assignment = assigning your sample into two or more subgroups by chance
(9.2a) Distinguishing features of experimental design
Best to establish internal validity
– Causal, cause-effect relationships
But you need to address that:
If X, then Y
And
If not X, then not Y
(in a comparative research e.g. experimental or quasi-experimental design)
Program/Treatment group = receives program of interest and usually contrasted with control
group or group receiving another treatment
Comparison group = compared to contrasted with group that receives program or intervention
of interest
Probabilistically equivalent = the notion that two groups, if measured indefinitely, would on
average perform identically. (But doesn’t mean they would obtain the exact same average
score
,(9.2b) Experimental design and threats to internal validity
Significant advantages:
. Relies on having two groups (program & control)
. Random assignment ensures that the two groups are equivalent to each other before a
treatment is administered to any one group
– It is strong against all the single group threats to internal validity
– Also strong against multiple group threats, except selection mortality
– e.g. strong against selection-history and selection-maturation threats, because the
two groups are created to be probabilistically equivalent to begin with
– e.g. no pretest, so also strong against selection-testing and selection-instrumentation
threats when it doesn’t use repeated (pre-post) measurement
– Has all social threats to internal validity
– Requires random assignment, so in institutional settings like schools it is more likely
that participants will be aware of each other and of the conditions assigned to them
, (9.2c) Design notation of a two-group experimental design
Simplest: two-group posttest-only randomised experiment
– Best to establish cause-effect relationships
Most interested in determining whether the two groups are different after the program
– You measure the groups on one or more measures
– Compare them by testing for the differences between the means, using a t-test or one-
way analysis of variance (ANOVA)
– Enables you to make a decision about whether any difference you observed between
the groups is likely to be due to chance or not
(9.2d) Difference between random selection and assignment
– Random selection is how you draw the sample of people for your study from a
population
– Random assignment is how you assign the sample that you draw to different groups or
treatments in your study
Possible to have both or one of them or neither
9.3 CLASSIFYING EXPERIMENTAL DESIGNS
Can be classified using a simple signal-to-noise ratio metaphor
– Assume that what you observe or see in a research study can be divided in: signal and
noise
Variability = the extent to which the values measured or observed for a variable differ
Signal: related to the key variable of interest (the contrast you’re trying to measure or the
effect of the program)
Noise: consists of all random factors in the environment that make it hard to se the signal
(e.g. local distractions, how people felt that day etc)
– In order to see true effect: signal needs to be high relative to noise
– e.g. powerful treatment (strong signal) and good measurement (low noise) = higher
chance of seeing true effects
So this helps us classify experimental design into two categories:
. Signal enhancers
○ Signal enhancing designs are called factorial designs
◆ Focus on the program or treatment, its components and its major dimensions
and enable you to determine whether the program has an effect, whether
different subcomponents are effective and whether there are interaction in its
effects caused by subcomponents
○ You would examine several different variations of the treatment
. Noise reducers
○ Covariance designs
◆ Help minimise noise through the inclusion of one or more variables (covariates)
that account for some of the variability in the outcome measure or dependent
