Download presentation - 12th Alps

12th Alps-Adria Psychology Conference
Rijeka, September 29th 2016
The credibility of psychological science at stake:
Lessons to be learned from low reproducibility of
psychological studies
Valentin Bucik
Department of Psychology, University of Ljubljana, Slovenia
[email protected]
Personal & professional frustration
We usually hear:
„Science often serve(d) the politics or sometimes even ideology …“
More accurate to say:
„Scientists often serve(d) the politics or sometimes even ideology …“
Many cases urge us to ask: „Is the credibility of our science at risk?“
Diederik Stapel is a Dutch former professor of social psychology
at Tilburg University and before that at the Universities of
Groningen and Amsterdam.
In 2011 Tilburg University suspended Stapel for fabricating and
manipulating data for his research publications.
This scientific misconduct took place over a number of years and
fraud was found in at least 55 (!) scientific journal papers from
2006 to 2011 (+book chapters,+ dissertations …)
»The Radan Case«: dr. Radan has been charged with 20 criminal counts, including
seven murders by arbitrarily cuting short the lives of several patients by administering a
lethal dose of potassium to their veins. Radan claimed that he just simulated the
situations to show the instability of the control mechanisms in the settings at the clinic.
2 judicial experts, both clinical psychologists (using similar diagnostic tools):
A (for the public prosecutor): Radan suffers from the narcissistic personality disorder
and superiority complex; his self-esteem is split into irrational external image and inner
self concept. His inner feelings of hostility, destructiveness and sadism can be
transformed into nice and compliant person on the outside;»The God complex«.
B (for the defence): Radan doesn't show personality disorder or any other form of
psychological distress. His narcissism is within the borders of normality. In the time of
the »potassium provocation« he was burned out and consequently showing impulsive
behaviour.
„Why APA Leadership Should Resign Over Torture Scandal, And
Why You Should Care“
Science Aug. 2015 report:
(270 authors led by Brian Nosek – University of Virginia)
97 % vs 36 %
(a)
(b)
(c)
low statistical power
a surprising result
a p value only slightly less than .05
This study was rather brutally misused by SLO conservative circles in
December 2015 in public debate about the change of „Marriage and
Family Relations Act“ – which wanted to make same-sex parents equal to
oposite-sex parents in raising a child
• by erroneous and malicious interpretations of the results from Science
that “more than two thirds of psychological studies are falsified, faked,
or fraudulent”,
• with clear intention to discredit psychological science and profession as
not trustworthy and credible enough to be taken seriously.
Namely, official groups of psychologists, advocating equality, passed a
common public statement, based on several international studies and
meta-analyses.
Slovene Psychologists Society,
3 departments of psychology,
Chamber of Clinical Psycologists,
psychology students…)
F.D., social worker & psychotherapist: „… so
please, don‘t get stranded by sweet stories
and rhetoric of psychologists; they don‘t
have a single expert argument and every
research data can clearly reject all claims by
the colleagues psychologists, who – sadly –
are selling their expertise to politics. …“
Proti = Against
Dr. J.C., chemist and father of five: „Let me tell
you about psychological science: Science, one of
the most respectful scientific journals, clearly
stated that 68 % of all studies in psychology are
adapted, falsified; just fraud, scam, fabrication …
So this is a first-hand testimony…“
„24Kul“, Clerical Slovene website:
„Unprecedented scandal in science!
The group of the best scientists published a study in Science, that caused
panic within the pleaders of psychological theories. More than 75%
scientific studies published in psychological journals in 2008 could not be
repeated.
Why? Because researchers were adapting them and distorting the data for
the needs of the daily politics just to delude the layman public.
International group of experts demonstrated that the majority of
psychologists furnish their points of view with false, untrue and fabricated
data and conclusions.“
„Družina“ [„Family“], central Slovene catolic newspaper:
„Non-scientific research!
It was proven for all studies, that was cited by the Slovene Psychologists
Society in their statement, to be non-scientific. On the other hand, all studies
performed on the scientific sample confirm that it is much better for children
to grow up in family with mother and father. „
All these opinions referred to (misinterpreted) the Science 2015 study …
…and also quoted the …
„Research report – review of empirical data about same sex relationships
and marriages and the welfare of the children in these relationships“
(December 2015)
6 authors: theologists, sociologists, 1 BA student of psychology (1st year)
With all the statistical and methodological textbooks in psychology and
other social sciences, dealing with Numerus, sampling etc., they relied on
Nock, 2001…
…and they focused only on two criteria for trustworthy study („golden rule“?):
- Random (probability) sampling
- Number of participants (N>400)
According to this (concluding remarks in the „report“):
• „From 31 studies in our meta-analysis, only two based on random sampling
and in all studies the N was smaller than 400. In various studies there was no
control group.
• Based on these facts our conclusions are the same as in other studies (Nock,
2001):
• the majority of studies, supposedly proving that children in the samesex families do not differ in development from children in different-sex
families, are not credible and trustworthy
• we think that such studies should not serve as means for final, decisive
and proper answer to the questions about the development of
children in same-sex parental relationships
• therefore we take promoting the results of these studies to be false,
misleading and deceptive.“
The same „methodology“of „meta-analysing“ and discourse as in Nock, (2001)
• The respondents are homosexual couples who wish to be issued marriage
certificates. The province of Ontario will not do so, and cites the common
law definition of marriage.
• Appeal dismissed; definition changed immediately and the province
ordered to issue marriage certificates to qualifying same-sex couples.
Who is S.L.Nock?
… and Nock‘s „methodology“ is similar to „Brief of Amici Curiae“ (2015) for the
Supreme Court of the United States
During a speech at Franciscan University of Steubenville in 2014
entitled "What Sexual Behavior Patterns Reveal about the Mating
Market and Catholic Thought," Regnerus' views on same-sex
relationships continued to spread controversy when he claimed that
"normalization of gay men's sexual behavior in society will contribute
to a surge in the "practice of heterosexual anal sex.„
Regarding the connection between one's faith and the activities of
Christian professors, Regnerus noted in an alumni profile that
"I believe that if your faith matters, it should inform what you teach
and what you research".
Are the data really non-conclusive?
- Random or probabilistic sampling is ok, but not always possible and not
always the best solution (e.g. stratified sampling or quota sampling gives
much more precise prediction of election results in political marketing
research).
- N over 400 is extremely rare in social science studies, especially when
doing research in marginalized groups; and credibility of the conclusions is
not in linear proportion to the sample size (i.e. „the bigger the sample the better“).
- Are we really in a position to expect final, decisive, proper, answers in
science?
- Science is probabilistic and is trying to be skeptic, nomothetical and
falsifiable in Karl Popper‘s sense in validating theories and research
outcomes.
- There are no final answers in science nor absolute truth.
- According to different criteria of research methodology in
social sciences all studies, criticised by Slovenian „experts“
and Nock, Mark, Regnerus, Sullins … can actually be
recognized as valid and reliable – in the extend of
generalizability that is possible according to size and
representitiveness of the sample…
…as being clearly shown in the following examples:
2005
„Brief of Amici Curiae“ (2012)
to the US District Court for the Northern District of California
What did the Science 2015 study really say?
• Modern science understands scientific findings as reproducible, replicable,
and generalizable. The results of the Science study (surprisingly low
reproducibility) were sobering.
• In general, reproducibility seems undervalued because scientists prioritize
novelty over replication.
• Innovation is the engine of discovery; researchers are usually driven by
searching for the barriers of science; when a topic seems covered, they
tend to rush forward but forget to stop and check the stability of the
outcomes.
• But replication and cross-validation also help to establish a firm
nomological network and high validity of scientific theories.
• Science can learn from replication studies, critically pointing to important
issues in planning and performing research of good quality.
• It will also help prevent the manipulations and the poorly supported
reproaches of psychology.
According to a 2016 poll of 1,500 scientists, 70% of them failed to reproduce
another scientist's experiments (50% failed to reproduce their own
experiment; Baker, 2016).
These numbers differ among disciplines:
• chemistry:
90% (60%)
• biology:
80% (60%)
• physics and engineering:
70% (50%)
• medicine:
70% (60%)
• Earth & environment science:
60% (40%).
In 2009, 2% of scientists admitted to falsifying studies at least once and 14%
admitted to personally know someone who did. Misconducts were reported
more frequently by medical researchers than others (Fanelli, 2009).
Causes of the „Reproducibility Crisis“ (Begley &
Ioannidis, 2015):
• Generation of new data / publications at an unprecedented rate.
• Compelling evidence that the majority of these discoveries will
not stand the test of time.
• Causes: failure to adhere to good scientific practice & the
desperation to „publish or perish“.
• This is a multifaceted, multistakeholder problem.
• No single party is solely responsible, and no single solution will
suffice.
Why replication in psychology is so often discouraged (Earp & Everett, 2013):
• Independent, direct replications of others’ findings can be timeconsuming for the replicating researcher.
• Replications are likely to take energy and resources directly away from
other projects that reflect one’s own original thinking.
• Replications are generally harder to publish (in large part because they are
viewed as being unoriginal.
• Even if replications are published, they are likely to be seen as 'bricklaying'
exercises, rather than as major contributions to the field.
• Replications bring less recognition and reward, and even basic career
security, to their authors.
Some after-effects of the Science study (reflected also in TOP - below):
• expect that journals will most likely publish more replications in the future. They
are already launching new policies that will encourage authors, editors, and
reviewers to re-examine and recalibrate the basic notions about what constitutes
a good research.
• Editorial boards will advance the “acceptance culture” of the submitted articles
such as sharing data, analysis code, and study materials, disclosing all data
exclusions, requiring authors to discuss sample sizes and statistical power, report
effect size, etc.
At least two lessons that can be learned from the Science study:
• that the project was conducted with concern about the health of the discipline,
believing in its promise for accumulating knowledge about human behaviour that
can advance the quality of the human condition and
• that many will be tempted to conclude that psychology is a bad apple in the
basket. However, this is not the case: this is a problem shared with natural
sciences, medical sciences, and biomedicine, as well as behavioural or social
sciences, because the replication efforts in other fields are similarly low.
(TOP-Transparency & Openness Promotion)
Replication is not a silver bullet.
Even carefully-designed replications, carried out in good faith by expert
investigators, will never be conclusive on their own (Earp & Trafimow, 2015).
But (Tsang and Kwan, 1999) :
If replication is interpreted in a strict sense, conclusive replications or
experiments are also impossible in the natural sciences.… So, even in the
“hardest” science (i.e., physics) complete closure is not possible. The best
we can do is control for conditions that are plausibly regarded to be
relevant.
→
→
→
→
→
„Crises in psychology are not caused by methodological flaws but by the way
people talk about them (Kruglanski & Stroebe,2012)
• There is an epistemological misunderstanding that emphasizes the phenomenon instead
of its underlying mechanisms.
• Theoretical hypothesis (and construct) is what matters (conceptual vs. direct replications)
• For the meaningful replications, attempts at reinstating the original circumstances are not
sufficient.
• Instead, replicators must ascertanin that conditions are realized that reflect the
teheoretical variables manipulated (measured) in the original study.
„Publihs or persih“ is a curse in a way…
"Science, I had come to learn, is as political,
competitive, and fierce a career as you can
find, full of the temptation to find easy paths."
— Paul Kalanithi, neurosurgeon and writer (1977–2015)
The 7 biggest problems facing science, according to 270 scientists
(Belluz, Plumer & Resnick, September 7, 2016):
1.
2.
3.
4.
5.
6.
7.
Academia has a huge money problem
Too many studies are poorly designed
Replicating results is crucial — and rare
Peer review is broken
Too much science is locked behind paywalls
Science is poorly communicated
Life as a young academic is incredibly stressful
Faking Science: A True Story of Academic Fraud
by Diedrik Stapel
Translated by Nicholas J.L. Brown [from Dutch “Ontsporing” (“Derailment”)]
Responding to the interim report, Stapel stated:
„I failed as a scientist. I adapted research data and fabricated research. Not
once, but several times, not for a short period, but over a longer period of
time. I realize that I shocked and angered my colleagues, because of my
behavior. I put my field, social psychology in a bad light. I am ashamed of it
and I deeply regret it.“
Thank you!