Research methods and techniques 2
HOC 1: Introduction to scientific reasoning
Overview lecture:
1. Psychology is a way of thinking (chapter 1)
2. Sources of information (chapter 2)
1. Psychology is a way of thinking
Producing versus consuming research
When you work as a psychologist, you are a consumer of
science to help a patient, to look up a therapy,… —>
requires you to think critically about research so you can
use it in an appropriate way
When you run scientific experiments yourself, you are a
producer of research. This could happen when you do a
PHD
• Examples to illustrate the importance of the research consumer role:
o Facilitated communication treatment
➢ for example when someone can’t talk, a health care worker could assist
this person and through a keyboard, this person could select letters by
hand and form words
➢ It might be a good idea to check the reliability of this method before
actually using it on patients —> it actually seems that this method of
communication treatment is not very great, because different people see
different things and communicate different things (which results in the
patient not being able to really express his feelings/thoughts) —> it’s
more the health care workers who influence the communication of the
person
➢ ➔ something that might look good at first glance is not always as great as
it looked when you do research
o Scared-straight approach
➢ This is a method where they try getting people back on the right path by
scaring them ➔ for example, there are shows where young adults are
being sent to prison and treated terribly
➢ After a few days, the young people are so scared that they promise to be a
better person
➢ The effectiveness of this method does not seem to be good when
compared to other kids who were treated with another method. The
group that went through the scared-straight approach, was more likely to
be arrested, in criminal environments,… ➔ didn’t work
1
, o Mindfulness and academic performance
➢ Children who received mindfulness training had better academic
performances than the control group.
➢ It seems that using mindfulness in school results in better academic
performances, so it might be a good idea to integrate mindfulness
• Critical mindset is essential!!!
• Not all published research is correct or robust
A good example of this is the study that was done by ‘Bem’ and led to what is now
called ‘the Replication crisis in psychology’ = he did 9 experiments where he
‘proved’ that there is such a thing as ‘premonition’ (≈ predicting the future)
o Example of 1 experiment: Students were given a list of words (words were shown
one by one), after a certain point he stopped showing the words and the students
had to write down as many words as possible (which they remembered) →
recollection task
o Second part of the experiment was that the students could see some of the
words again (subset) + had to copy them again by writing them down —> strange,
normally it’s the other way around in an experiment. He did this to achieve
training effects!
o ➔ The conclusion was that the words people most likely remembered were the
words from that subset that they were training after writing them down. It’s like
people were already predicting which words they would see next and they
remembered them better!
o How to explain this? —> led to researchers questioning their way of researching
o Problems! ➔ Bem only reported the results of 9 experiments, but he actually ran
hundreds. He only reported experiments where he proved that people could
predict it! He did hundreds of experiments were this was not proven, but he
didn’t report these results ➔ very important to know = shows you have to be
critical!
o Also led to replication crisis = most studies that were done before, could not be
replicated because most experiments were lucky shots.
How scientists work
• Science is based on empiricism
• Scientists test theories
• Scientists work on fundamental and applied problems
• Science is continuously evolving
• Scientists publish their findings in scientific journals
• Scientists communicate with the general public via journalists
Empiricism
• We use an empirical method in scientific research and we use data
• Empirical method is based on data obtained through:
o Our senses (sight, hearing, touch)
o Instruments that assist our senses (thermometer, questionnaires, timer)
• Empiricists aim to do research in a systematic, rigorous, and replicable manner
2
,• Empiricism is not based on own experiences, intuition, or authority
Scientists test theories
We use an empirical approach to test theories. To do so, we use the
empirical research cycle as shown on the left. We start with a theory. From
this theory, we form a few research questions which we will try to resolve in
an experiment. Before we do that, we produce a research design and some
hypotheses. After this, we obtain data and we can accept/change the
theory
Experiment Harry Harlow: there were a few contradictory
theories about the bond between mother and child:
1) This bond exists because the mom feeds the baby
2) This bond exists because the mom provides comfort,
warmth and safety for the baby
➔ They made 2 surrogate moms, one with a hard ‘body’, but with
food and one who was soft and warm, but didn’t have ability to
feed
This experiment led to the ‘contact comfort theory’ —> the soft, warm and comforting
mom was more important for the baby than the feeding mother
• Characteristics of good theories
o Supported by data
o Falsifiable —> must be constructed in such a way that you could create a study
that could disprove a theory. If it is not falsifiable, it is pseudoscience
o Parsimonious (“Occam’s razor”) = if we have 2 explanations for a phenomenon
and they are both qualitative, the easiest explanation will be the one we use
• Examples of non-falsifiable theories:
o Facilitated communication treatment believers
o See book “De ongelovige Thomas heeft een punt”
• Theories are evaluated based on all available evidence
o A theory can never be “proven” (we always say we ‘support’ a theory), but it can
be falsified (when the theory is not correct)
➢ For example = ‘all swans are white’ —> if you go check this in a few lakes
and you only find white swans, you can ‘support’ the theory. This doesn’t
mean that there are ONLY white swans in the whole wide world (because
you did not check them all), so you cannot ‘prove’ the theory. Once you
see 1 black swan, the theory is immediately falsified!
o Replication is crucial!
3
, Scientists form a community
• Merton’s scientific norms:
1. Universalism = everyone, regardless of their background, should be able to do
science —> it’s not who you are that matters, but how you do science!
2. Communality —> you should share the results you became with the scientific
community
3. Disinterestedness —> scientists should pursue research not because they are
interested in money, fame,… but it should be done because people want to find
out something
4. Organized skepticism —> we should have a system that is in place to check
each others work, so that we can evaluate it and see to what extent we can
accept claims
Fundamental VS applied research
• There is a difference between the type of research
1) Fundamental research is being done because we want to advance theoretical
insights, learn more about a phenomenon,… —> more theoretical
2) Applied research means that we are not really interested in advancing theories,
but we want to use this theory to apply in practical situations —> we will be
testing in a specific setting whether a specific theory works and can be used
Both kinds of research are being used, depending on the domain of science
Overall, the general focus in psychology is more on fundamental research
Continuously evolving
• Theories are continuously tested, modified, and falsified
• Research triggers follow-up research (example: impact of color on approach-and
avoidance-motivation in context (Meier et al., 2012))
o Example = experiment: ‘how does color affect motivation’
➢ People were divided into 2 groups. The people in both groups got a
booklet with anagrams. Both booklets were the same qua exercises
1) one group had a booklet with a red cover
2) the other group had a booklet with a blue cover
➢ They tested how many exercises were solved by both groups and in what
time → Group with red cover = slower, Group with blue cover = faster
➢ ➔ lea to the thought that color could have an effect on motivation. Red
would lead people to be more careful (because it is associated with
danger), try to avoid mistakes which results in being slower ➔ red
triggers avoidance motivation (slower). Blue triggers approach
motivation (faster)
4
, ➢ ➔ led to follow-up research where they tried to test this theory, but also
extend this theory (could context also play a role?). They asked people to
come to the lab and they were told that in a few minutes, they had to
interview someone else. The people saw a picture of the person they
were going to interview.
1) One group saw the picture of the person they had to interview in
a blue shirt (right)
2) the other one saw the person in a red shirt (left)
➢ next, they told the participants that the interview was a particular type of
interview. They again divided the participants into 2 groups
1) Group was told it was a dating interview
2) Group was told it was an intelligence interview
➢ After this, they allowed the participants to walk to the other room (where
they would do the interview) and they measured how fast the participants
walked the hallway to go to the other room (high score = more time =
slow).
❖ ➔ In the dating interview, they saw that participants who saw a
picture with a red shirt were walking faster compared to the
group who saw a picture with blue shirts. This meant that (in this
context), red was related to approach motivation and not
avoidance motivation (because red is associated with love,…)!
(other conclusion than before)
❖ In the intelligence interview, they saw that people who saw the
picture with red walked slower than the people who saw the blue
picture
Colors impacts motivation, but the effect differs depending on the context!
Couple of years later, replication study shows that these effects were not easy to
find again ➔ falsified
Publishing
• Publishers have to go through a whole process to be able to publish in a journal
• Manuscripts are submitted to scientific journals (e.g., Nature, Science, Psychological
Bulletin)
• Peer-review process
o Role of editor = manages submissions and makes the first decision whether or
not the article could be published
o Role of reviewers = these people are invited to read the manuscript because
they have a certain expertise of the topic; they will evaluate the strengths of the
articles and the weaknesses
5
, o Reject, revise and resubmit, accept ➔ scientists careers rely on this —> if you
have a lot of publications in scientific journals, you will have a higher chance of
being hired, getting a promotion,… ➔ so there is a motivation for scientists to
publish (which may change their thoughts and intentions)
Acceptance rate is about 10% for most of the journals ➔
you have a very small chance to be accepted to publish in
these journals —> brings a lot of stress
You can see the submission rates (how many articles are
being sent to the journal) and the final number of accepted
articles that got published
Higher numbers from +- 2007 —> because of the huge mass
of scientists that is still growing and the easier way of doing
research due to access to computers,…
Important! = the number of submissions keeps growing, but
the acceptance rate is still the same —> pressure is
increasing for scientists
Communicating results from a scientific study can go through a
scientific journal, but also via other ways.
The final statements are often different from the original
findings because editors and people who reproduce results,
can sometimes change words,… —> you have to be critical!
2. Sources of information
• Researchers:
o Use a comparison group
o Control for third variables
o Try to evaluate information without bias
• Research is better than:
o Experience
o Intuition
o Authority arguments
6
HOC 1: Introduction to scientific reasoning
Overview lecture:
1. Psychology is a way of thinking (chapter 1)
2. Sources of information (chapter 2)
1. Psychology is a way of thinking
Producing versus consuming research
When you work as a psychologist, you are a consumer of
science to help a patient, to look up a therapy,… —>
requires you to think critically about research so you can
use it in an appropriate way
When you run scientific experiments yourself, you are a
producer of research. This could happen when you do a
PHD
• Examples to illustrate the importance of the research consumer role:
o Facilitated communication treatment
➢ for example when someone can’t talk, a health care worker could assist
this person and through a keyboard, this person could select letters by
hand and form words
➢ It might be a good idea to check the reliability of this method before
actually using it on patients —> it actually seems that this method of
communication treatment is not very great, because different people see
different things and communicate different things (which results in the
patient not being able to really express his feelings/thoughts) —> it’s
more the health care workers who influence the communication of the
person
➢ ➔ something that might look good at first glance is not always as great as
it looked when you do research
o Scared-straight approach
➢ This is a method where they try getting people back on the right path by
scaring them ➔ for example, there are shows where young adults are
being sent to prison and treated terribly
➢ After a few days, the young people are so scared that they promise to be a
better person
➢ The effectiveness of this method does not seem to be good when
compared to other kids who were treated with another method. The
group that went through the scared-straight approach, was more likely to
be arrested, in criminal environments,… ➔ didn’t work
1
, o Mindfulness and academic performance
➢ Children who received mindfulness training had better academic
performances than the control group.
➢ It seems that using mindfulness in school results in better academic
performances, so it might be a good idea to integrate mindfulness
• Critical mindset is essential!!!
• Not all published research is correct or robust
A good example of this is the study that was done by ‘Bem’ and led to what is now
called ‘the Replication crisis in psychology’ = he did 9 experiments where he
‘proved’ that there is such a thing as ‘premonition’ (≈ predicting the future)
o Example of 1 experiment: Students were given a list of words (words were shown
one by one), after a certain point he stopped showing the words and the students
had to write down as many words as possible (which they remembered) →
recollection task
o Second part of the experiment was that the students could see some of the
words again (subset) + had to copy them again by writing them down —> strange,
normally it’s the other way around in an experiment. He did this to achieve
training effects!
o ➔ The conclusion was that the words people most likely remembered were the
words from that subset that they were training after writing them down. It’s like
people were already predicting which words they would see next and they
remembered them better!
o How to explain this? —> led to researchers questioning their way of researching
o Problems! ➔ Bem only reported the results of 9 experiments, but he actually ran
hundreds. He only reported experiments where he proved that people could
predict it! He did hundreds of experiments were this was not proven, but he
didn’t report these results ➔ very important to know = shows you have to be
critical!
o Also led to replication crisis = most studies that were done before, could not be
replicated because most experiments were lucky shots.
How scientists work
• Science is based on empiricism
• Scientists test theories
• Scientists work on fundamental and applied problems
• Science is continuously evolving
• Scientists publish their findings in scientific journals
• Scientists communicate with the general public via journalists
Empiricism
• We use an empirical method in scientific research and we use data
• Empirical method is based on data obtained through:
o Our senses (sight, hearing, touch)
o Instruments that assist our senses (thermometer, questionnaires, timer)
• Empiricists aim to do research in a systematic, rigorous, and replicable manner
2
,• Empiricism is not based on own experiences, intuition, or authority
Scientists test theories
We use an empirical approach to test theories. To do so, we use the
empirical research cycle as shown on the left. We start with a theory. From
this theory, we form a few research questions which we will try to resolve in
an experiment. Before we do that, we produce a research design and some
hypotheses. After this, we obtain data and we can accept/change the
theory
Experiment Harry Harlow: there were a few contradictory
theories about the bond between mother and child:
1) This bond exists because the mom feeds the baby
2) This bond exists because the mom provides comfort,
warmth and safety for the baby
➔ They made 2 surrogate moms, one with a hard ‘body’, but with
food and one who was soft and warm, but didn’t have ability to
feed
This experiment led to the ‘contact comfort theory’ —> the soft, warm and comforting
mom was more important for the baby than the feeding mother
• Characteristics of good theories
o Supported by data
o Falsifiable —> must be constructed in such a way that you could create a study
that could disprove a theory. If it is not falsifiable, it is pseudoscience
o Parsimonious (“Occam’s razor”) = if we have 2 explanations for a phenomenon
and they are both qualitative, the easiest explanation will be the one we use
• Examples of non-falsifiable theories:
o Facilitated communication treatment believers
o See book “De ongelovige Thomas heeft een punt”
• Theories are evaluated based on all available evidence
o A theory can never be “proven” (we always say we ‘support’ a theory), but it can
be falsified (when the theory is not correct)
➢ For example = ‘all swans are white’ —> if you go check this in a few lakes
and you only find white swans, you can ‘support’ the theory. This doesn’t
mean that there are ONLY white swans in the whole wide world (because
you did not check them all), so you cannot ‘prove’ the theory. Once you
see 1 black swan, the theory is immediately falsified!
o Replication is crucial!
3
, Scientists form a community
• Merton’s scientific norms:
1. Universalism = everyone, regardless of their background, should be able to do
science —> it’s not who you are that matters, but how you do science!
2. Communality —> you should share the results you became with the scientific
community
3. Disinterestedness —> scientists should pursue research not because they are
interested in money, fame,… but it should be done because people want to find
out something
4. Organized skepticism —> we should have a system that is in place to check
each others work, so that we can evaluate it and see to what extent we can
accept claims
Fundamental VS applied research
• There is a difference between the type of research
1) Fundamental research is being done because we want to advance theoretical
insights, learn more about a phenomenon,… —> more theoretical
2) Applied research means that we are not really interested in advancing theories,
but we want to use this theory to apply in practical situations —> we will be
testing in a specific setting whether a specific theory works and can be used
Both kinds of research are being used, depending on the domain of science
Overall, the general focus in psychology is more on fundamental research
Continuously evolving
• Theories are continuously tested, modified, and falsified
• Research triggers follow-up research (example: impact of color on approach-and
avoidance-motivation in context (Meier et al., 2012))
o Example = experiment: ‘how does color affect motivation’
➢ People were divided into 2 groups. The people in both groups got a
booklet with anagrams. Both booklets were the same qua exercises
1) one group had a booklet with a red cover
2) the other group had a booklet with a blue cover
➢ They tested how many exercises were solved by both groups and in what
time → Group with red cover = slower, Group with blue cover = faster
➢ ➔ lea to the thought that color could have an effect on motivation. Red
would lead people to be more careful (because it is associated with
danger), try to avoid mistakes which results in being slower ➔ red
triggers avoidance motivation (slower). Blue triggers approach
motivation (faster)
4
, ➢ ➔ led to follow-up research where they tried to test this theory, but also
extend this theory (could context also play a role?). They asked people to
come to the lab and they were told that in a few minutes, they had to
interview someone else. The people saw a picture of the person they
were going to interview.
1) One group saw the picture of the person they had to interview in
a blue shirt (right)
2) the other one saw the person in a red shirt (left)
➢ next, they told the participants that the interview was a particular type of
interview. They again divided the participants into 2 groups
1) Group was told it was a dating interview
2) Group was told it was an intelligence interview
➢ After this, they allowed the participants to walk to the other room (where
they would do the interview) and they measured how fast the participants
walked the hallway to go to the other room (high score = more time =
slow).
❖ ➔ In the dating interview, they saw that participants who saw a
picture with a red shirt were walking faster compared to the
group who saw a picture with blue shirts. This meant that (in this
context), red was related to approach motivation and not
avoidance motivation (because red is associated with love,…)!
(other conclusion than before)
❖ In the intelligence interview, they saw that people who saw the
picture with red walked slower than the people who saw the blue
picture
Colors impacts motivation, but the effect differs depending on the context!
Couple of years later, replication study shows that these effects were not easy to
find again ➔ falsified
Publishing
• Publishers have to go through a whole process to be able to publish in a journal
• Manuscripts are submitted to scientific journals (e.g., Nature, Science, Psychological
Bulletin)
• Peer-review process
o Role of editor = manages submissions and makes the first decision whether or
not the article could be published
o Role of reviewers = these people are invited to read the manuscript because
they have a certain expertise of the topic; they will evaluate the strengths of the
articles and the weaknesses
5
, o Reject, revise and resubmit, accept ➔ scientists careers rely on this —> if you
have a lot of publications in scientific journals, you will have a higher chance of
being hired, getting a promotion,… ➔ so there is a motivation for scientists to
publish (which may change their thoughts and intentions)
Acceptance rate is about 10% for most of the journals ➔
you have a very small chance to be accepted to publish in
these journals —> brings a lot of stress
You can see the submission rates (how many articles are
being sent to the journal) and the final number of accepted
articles that got published
Higher numbers from +- 2007 —> because of the huge mass
of scientists that is still growing and the easier way of doing
research due to access to computers,…
Important! = the number of submissions keeps growing, but
the acceptance rate is still the same —> pressure is
increasing for scientists
Communicating results from a scientific study can go through a
scientific journal, but also via other ways.
The final statements are often different from the original
findings because editors and people who reproduce results,
can sometimes change words,… —> you have to be critical!
2. Sources of information
• Researchers:
o Use a comparison group
o Control for third variables
o Try to evaluate information without bias
• Research is better than:
o Experience
o Intuition
o Authority arguments
6
