Summary Experimental Research (lectures/web lectures)

CONTENT
Lecture 1 Introduction ............................................................................................................................................ 2
Lecture 2 ANOVA .................................................................................................................................................... 6
Lecture 3 Manipulations, controlling for variables & mediation analysis & binary dependent variables ............ 17
Lecture 4 Within-subjects designs, field experiments, quasi-experiments........................................................... 27
Lecture 5 Measures, models, and manipulating people ....................................................................................... 38
Lecture 6 Power and problems ............................................................................................................................. 46
Weblectures .......................................................................................................................................................... 55
Exam tips ............................................................................................................................................................... 91




1

,EXPERIMENTAL RESEARCH
LECTURE 1 INTRODUCTION

EXPERIMENTAL RESEARCH CONSISTS OF FOUR PHASES

 Phase 1: this phase deals with the formulation of the problem statement and hypotheses about the
relationship between (an) independent variable(s) and (a) dependent variable(s). We will learn how to
generate a researchable problem statement together with the specific hypotheses that will be tested in
an experiment.
 Phase 2: this phase deals with the design of an experiment in which you (1) manipulate (an) independent
variable(s) to observe effects on dependent variable(s), and (2) control for confounding factors. We will
discuss (a) how the independent and dependent variables are operationalized in experiments, and (b)
how to control for confounding variables.
 Phase 3: this phase deals with conducting the experiment and collecting data. We will discuss different
types of experimental designs (e.g., completely randomized designs, single factor designs, factorial
designs, mixed designs, etc.) and understand the differences between main and interaction effects.
 Phase 4: this final phase deals with the data analysis and interpretation of the experimental findings.
We will discuss (a) how to statistically analyze experimental design using ANOVA (analysis of variance)
techniques, (b) how to interpret results obtained from an experiment, and (c) how these results can
lead to derive new hypotheses to be tested in a follow-up experiment.

BEHAVIORAL RESEARCH (DIFFERENT RESEARCH TYPES)




DESCRIPTIVE :
Describes behavior, thoughts or feelings.

 Public opinion polls = most common example
 Survey research
 Changes can be measured, if respondents fill out the survey at different points in time (longitudinal or
panel design)

Examples:

 40% of site visitors who don’t convert to consumers, indicate that the shipping costs were too high;
 For males this was 30%, for females this is 50%.

CORRELATIONAL:
 Investigates the relationships among various psychological variables
 Its aim is to discover correlations between variables
 Used to describe the relationship between two or more naturally occurring variables

Example “Does the weather affect our mood?”(Howard and Hoffman, 2984). They found a significant effect on
mood correlated with the weather (by a mood questionnaire).




2

,But this experiment has only a few respondents (24 students). And we ask ourselves: is it the weather that has
an effect or is it the activities which you do during some weather? Does it matter? It does not matter for
predicting, but it does matter for explaining. And a better explanations helps again to make better predictions.

New experiment “Does temperature affect consumers’ evaluation of products?” Intention to purchase online vs.
average temperature for each day (based on 24 months of available data). Result: the higher the temperature,
the higher the ‘purchases’ on the site (correlational). But is it really the temperature that drives the effect?

 Problem = it cannot establish CAUSALITY. You measure 2 things (at same time, go together) but how
can you be sure that the first variable influences the other one, or the other way around.

Importantly: we determine whether one variable is related to another by seeing whether scores on the two
variables covary (whether they vary and change together).

Correlation coefficient = indicates the degree to which two variables are related; -1 to +1. Coefficient scores do
not always accurately reflect the relationship between two variables!

Spurious correlations: High temperature  eating ice cream and aggression.

Correlation between how much ice people eat and aggression. Aggression leading to ice cream eating, or ice
cream eating leading to aggression?  Temperature. People eat more ice cream when it is warm outside, more
aggressive when it is warm outside. Ice cream eating and aggression seem to correlate, but it is because
something else…

Three requirements:

1. Correlation is only one of the necessary conditions for causality
2. Directionality (logical in time) – it makes sense that something is before another thing.
3. Elimination of extraneous variables.

Example ‘Does temperature affect consumers’ evaluation of products?’  In this case, there is a correlation and
the directionality is clear (temperature cannot caused by purchases), but other variables might play a lore.

Descriptive, or correlational? Correlation research very close to descriptive research. Any correlation data is
descriptive, but better than only describing (any relation is a description of your data).


QUASI-EXPERIMENTAL:
In some cases it might not be possible to manipulate (change) the independent variable, quasi-experimental
research is used. Example = safety belt in car. Use control condition.


EXPERIMENTAL:
 Its aim is to find whether certain variables cause changes in behavior, thought, or emotions (in consumer
research)
 It involves manipulating (changing) and independent variable (X) and assessing potential changes in an
outcome variable (e.g., behavior)
 The key is RANDOMIZATION of subjects to treatments
 If behavioural change occurs, we can infer that X causes Y

Randomization = Arbitrarily assigning each participant one condition of the experiment

If I assign person A to the control group, and assign person B to a treatment group, person A and B are not the
same. However, if I do this for every subject, the average person in the control group is the same as the average
person in the treatment group. With large samples, true randomization creates balance in for example age,
gender, preference, etc. of participants. The groups are thus the same on average. Any difference we thus later
find must have occurred because of the treatment.  If I randomly assign my participants, and I see an effect,
it cannot be because individual differences (but because 2 groups get different videos for example). With 2 people:
individual difference. But with larger samples, you become more likely that the effect is true.


3

, As the groups are the same on average on all aspects (again, if your sample is large enough), it is better than for
example matching techniques that economists/marketing modelers tend to use. WHY? Because for matching we
have to a priori predict the possible confounds (alternative explanations), and match people based on those
confounds. However, there could well be confounds that we do not yet know about.  If you match participants
(not randomization), you have to predict on beforehand where you should match on (= problem).

Allocation to treatments needs to be random, otherwise, it might affect the results. Example in slides (p. 37)

GOALS OF SCIENCE

1. Finding regularities, patterns
2. To predict outcomes

We do so by investigating and explaining. A good explanation of why something happens is a THEORY. Every day
use of the word theory: a guess, a hunch. Scientists use of the word theory: well-substantiated, documented,
supported explanation of observations.

GOOD THEORY

WHAT (defines constructs) – HOW (proposition about the relationship between constructs (how they related to
each other)) – WHY (arguments that justify the propositions (why you think this is the case))

WHAT  conceptual vs. operational definition

Use this in phase 1 and 2 of experimental research. Example in slides (p. 42-47)

THE BASE (STATISTICS)

Use this in phase 4 of experimental research.

If we observe a difference on a measure between two groups, what makes us more likely to believe that there
actually is a difference?

 A larger difference between the group
 Less variation between individuals in each group
 A larger sample size

These latter two help to determine the precision of the estimate.

There is a mean, but there is always a variation (uncertainty around it). Statistical test: test whether means are
different.

 Mean: average of everyone we tested in each condition
 Standard deviation is a measure of variability in the data: do people’s individual responses differ a lot
from the mean, or only a little bit?

We need both the mean and a measure of variance to test how likely it is that a difference is due to error. Example
in slides (p. 53-54).




Formula t-test:

In other words, all else being equal:

 A larger difference between means gives a larger t-value
 A smaller variance gives a larger t-value
 A larger sample size gives a larger t-value



4