Scrutiny and validation proper science
Meta-science = relatively new kind of science focused on scientific research itself
H1 how science works
Science is social construct: need to show and convince fellow scientists
Stuart Mill: we understand grounds of our opinion only when we can show that other
theories than ours are false
Claims need to survive communal filter process to separate false from true, this process of
filtering gives science power because standard is set high
Getting peer support should not be the goal but a means to goal: finding truth
Oldenburg Royal Society: first scientific journal thing 1665 P15
Hooke first observation Jupiter P16
Finding funding is hard
Study design P17/18 and process of science from scratch: fund, research, publishing
When editor of journal accepts paper, it gets peer reviewed
Since 1970s all journals modern model of sending out submissions to independent experts
for peer review
Peer reviewers mostly anonymous: blessing and a curse
Sometimes researcher has to redo part of research to be accepted by journal
Who ensures integrity and honesty? First attempt to write down unwritten rules: Robert
Merton: 4 values = Mertonian Norms P21:
- Universalism: regardless of race, sex, age, gender, income etc.
- Disinterestedness: not in it for fame or money
- Communality: share knowledge
- Scepticism: nothing is sacred, never trust face value
Mill, Dawkins, Planck P22
Data needs to be replicable
Popper: repetition or recurrence events in accordance with rules of regularities exclude
coincidence
Boyle: focused lot on replicability
H2 the replication crisis
Kahneman: thinking, fast and slow (shows mistakes, flaws and biases human thinking)
For example priming alters behavior unconsciously, example: Macbeth effect = when asking
someone to recall unethical deed bigger chance they’ll clean themselves shortly after =
unconscious cleanliness and virtue
Image of money primes as well more self-sufficient and egocentric
Replication crisis started when priming of word old walking slower couldn’t be replicated
due to better technology
Amy Cuddy most watched TED about power pose in stressful situations: couldn’t be
replicated
Zimbardo: prison experiment ‘guards’ and ‘prisoners’ was flawed because he instructed
guards to commit atrocities
, Replicability rates 38-77% = replication crisis
Might not be that bad because sometimes not replicable due to:
- Bad luck
- Failed replication due to being run with slight changes of methodology
Replicability cognitive psychology > social psychology
Within psychology lot of bullshit been debunked, so crisis is real
Psychology hard to replicate because lots of abstract phenomena
Similar problems in economic, neuroscience, evolutionary biology, ecology, marine biology
and chemistry P33
Sometimes data even can’t be reproduced (as opposed to replicated) P35
Lots of papers insufficient in providing full method in order to be reproducible for someone
else
Medical reversal = when treatment is canceled due to change in scientific revelation that
original research that lead to original treatment was wrong P39
Misconceptions about: childbirth, allergy, heart-attack and stroke P40
Cochrane Collaboration: 45% medical treatments insufficient evidence
H4 bias
Morton: measurements human skulls in 1840s Europeans bigger brains so smarter
Jay Gould: showed Morton’s inconsistencies and faults
Bias can lead to data massaging or leaving out results
Biases appear in every stage of scientific process
70-90% studies in meta-analysis positive result
This is unrealistic and due to publication bias = file-drawer problem
Always degree of error in numbers because numbers are chaotic
p-value gives us criterium
Sampling error = discrepancy sample vs. population
Also measurement error
Meta-analysis gives more weight to studies with larger sample sizes
Graph with sample size on y-axis and effect size on x-axis will look like triangle because larger
sample size closer to true effect size
Confirmation bias = interpreting evidence so it fits our beliefs and desires, is root of
publication bias
Small effects often not published meta-analysis isn’t valid
Even when you account for publication bias during meta-analysis you can’t tell how much
you should correct found effect size
Besides for practical and scientifical reasons, null results should also be published for ethical
reasons as obligation to the world and the people that partook and invested in the research
P97
File drawing your null effect research has downside no extra thing on CV, solution: data
manipulation: data massaging: unconscious/semiconscious manipulation
p-hacking = nudging or hacking their p-values to get below .05
HARKing form of p-hacking in which you hypothesize after results are known
When nulhypothesis is true, increasing the number of statistical tests snowballs our chances
of obtaining false-positive result
Meta-science = relatively new kind of science focused on scientific research itself
H1 how science works
Science is social construct: need to show and convince fellow scientists
Stuart Mill: we understand grounds of our opinion only when we can show that other
theories than ours are false
Claims need to survive communal filter process to separate false from true, this process of
filtering gives science power because standard is set high
Getting peer support should not be the goal but a means to goal: finding truth
Oldenburg Royal Society: first scientific journal thing 1665 P15
Hooke first observation Jupiter P16
Finding funding is hard
Study design P17/18 and process of science from scratch: fund, research, publishing
When editor of journal accepts paper, it gets peer reviewed
Since 1970s all journals modern model of sending out submissions to independent experts
for peer review
Peer reviewers mostly anonymous: blessing and a curse
Sometimes researcher has to redo part of research to be accepted by journal
Who ensures integrity and honesty? First attempt to write down unwritten rules: Robert
Merton: 4 values = Mertonian Norms P21:
- Universalism: regardless of race, sex, age, gender, income etc.
- Disinterestedness: not in it for fame or money
- Communality: share knowledge
- Scepticism: nothing is sacred, never trust face value
Mill, Dawkins, Planck P22
Data needs to be replicable
Popper: repetition or recurrence events in accordance with rules of regularities exclude
coincidence
Boyle: focused lot on replicability
H2 the replication crisis
Kahneman: thinking, fast and slow (shows mistakes, flaws and biases human thinking)
For example priming alters behavior unconsciously, example: Macbeth effect = when asking
someone to recall unethical deed bigger chance they’ll clean themselves shortly after =
unconscious cleanliness and virtue
Image of money primes as well more self-sufficient and egocentric
Replication crisis started when priming of word old walking slower couldn’t be replicated
due to better technology
Amy Cuddy most watched TED about power pose in stressful situations: couldn’t be
replicated
Zimbardo: prison experiment ‘guards’ and ‘prisoners’ was flawed because he instructed
guards to commit atrocities
, Replicability rates 38-77% = replication crisis
Might not be that bad because sometimes not replicable due to:
- Bad luck
- Failed replication due to being run with slight changes of methodology
Replicability cognitive psychology > social psychology
Within psychology lot of bullshit been debunked, so crisis is real
Psychology hard to replicate because lots of abstract phenomena
Similar problems in economic, neuroscience, evolutionary biology, ecology, marine biology
and chemistry P33
Sometimes data even can’t be reproduced (as opposed to replicated) P35
Lots of papers insufficient in providing full method in order to be reproducible for someone
else
Medical reversal = when treatment is canceled due to change in scientific revelation that
original research that lead to original treatment was wrong P39
Misconceptions about: childbirth, allergy, heart-attack and stroke P40
Cochrane Collaboration: 45% medical treatments insufficient evidence
H4 bias
Morton: measurements human skulls in 1840s Europeans bigger brains so smarter
Jay Gould: showed Morton’s inconsistencies and faults
Bias can lead to data massaging or leaving out results
Biases appear in every stage of scientific process
70-90% studies in meta-analysis positive result
This is unrealistic and due to publication bias = file-drawer problem
Always degree of error in numbers because numbers are chaotic
p-value gives us criterium
Sampling error = discrepancy sample vs. population
Also measurement error
Meta-analysis gives more weight to studies with larger sample sizes
Graph with sample size on y-axis and effect size on x-axis will look like triangle because larger
sample size closer to true effect size
Confirmation bias = interpreting evidence so it fits our beliefs and desires, is root of
publication bias
Small effects often not published meta-analysis isn’t valid
Even when you account for publication bias during meta-analysis you can’t tell how much
you should correct found effect size
Besides for practical and scientifical reasons, null results should also be published for ethical
reasons as obligation to the world and the people that partook and invested in the research
P97
File drawing your null effect research has downside no extra thing on CV, solution: data
manipulation: data massaging: unconscious/semiconscious manipulation
p-hacking = nudging or hacking their p-values to get below .05
HARKing form of p-hacking in which you hypothesize after results are known
When nulhypothesis is true, increasing the number of statistical tests snowballs our chances
of obtaining false-positive result
