Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
12th Alps-Adria Psychology Conference Rijeka, September 29th 2016 The credibility of psychological science at stake: Lessons to be learned from low reproducibility of psychological studies Valentin Bucik Department of Psychology, University of Ljubljana, Slovenia [email protected] Personal & professional frustration We usually hear: „Science often serve(d) the politics or sometimes even ideology …“ More accurate to say: „Scientists often serve(d) the politics or sometimes even ideology …“ Many cases urge us to ask: „Is the credibility of our science at risk?“ Diederik Stapel is a Dutch former professor of social psychology at Tilburg University and before that at the Universities of Groningen and Amsterdam. In 2011 Tilburg University suspended Stapel for fabricating and manipulating data for his research publications. This scientific misconduct took place over a number of years and fraud was found in at least 55 (!) scientific journal papers from 2006 to 2011 (+book chapters,+ dissertations …) »The Radan Case«: dr. Radan has been charged with 20 criminal counts, including seven murders by arbitrarily cuting short the lives of several patients by administering a lethal dose of potassium to their veins. Radan claimed that he just simulated the situations to show the instability of the control mechanisms in the settings at the clinic. 2 judicial experts, both clinical psychologists (using similar diagnostic tools): A (for the public prosecutor): Radan suffers from the narcissistic personality disorder and superiority complex; his self-esteem is split into irrational external image and inner self concept. His inner feelings of hostility, destructiveness and sadism can be transformed into nice and compliant person on the outside;»The God complex«. B (for the defence): Radan doesn't show personality disorder or any other form of psychological distress. His narcissism is within the borders of normality. In the time of the »potassium provocation« he was burned out and consequently showing impulsive behaviour. „Why APA Leadership Should Resign Over Torture Scandal, And Why You Should Care“ Science Aug. 2015 report: (270 authors led by Brian Nosek – University of Virginia) 97 % vs 36 % (a) (b) (c) low statistical power a surprising result a p value only slightly less than .05 This study was rather brutally misused by SLO conservative circles in December 2015 in public debate about the change of „Marriage and Family Relations Act“ – which wanted to make same-sex parents equal to oposite-sex parents in raising a child • by erroneous and malicious interpretations of the results from Science that “more than two thirds of psychological studies are falsified, faked, or fraudulent”, • with clear intention to discredit psychological science and profession as not trustworthy and credible enough to be taken seriously. Namely, official groups of psychologists, advocating equality, passed a common public statement, based on several international studies and meta-analyses. Slovene Psychologists Society, 3 departments of psychology, Chamber of Clinical Psycologists, psychology students…) F.D., social worker & psychotherapist: „… so please, don‘t get stranded by sweet stories and rhetoric of psychologists; they don‘t have a single expert argument and every research data can clearly reject all claims by the colleagues psychologists, who – sadly – are selling their expertise to politics. …“ Proti = Against Dr. J.C., chemist and father of five: „Let me tell you about psychological science: Science, one of the most respectful scientific journals, clearly stated that 68 % of all studies in psychology are adapted, falsified; just fraud, scam, fabrication … So this is a first-hand testimony…“ „24Kul“, Clerical Slovene website: „Unprecedented scandal in science! The group of the best scientists published a study in Science, that caused panic within the pleaders of psychological theories. More than 75% scientific studies published in psychological journals in 2008 could not be repeated. Why? Because researchers were adapting them and distorting the data for the needs of the daily politics just to delude the layman public. International group of experts demonstrated that the majority of psychologists furnish their points of view with false, untrue and fabricated data and conclusions.“ „Družina“ [„Family“], central Slovene catolic newspaper: „Non-scientific research! It was proven for all studies, that was cited by the Slovene Psychologists Society in their statement, to be non-scientific. On the other hand, all studies performed on the scientific sample confirm that it is much better for children to grow up in family with mother and father. „ All these opinions referred to (misinterpreted) the Science 2015 study … …and also quoted the … „Research report – review of empirical data about same sex relationships and marriages and the welfare of the children in these relationships“ (December 2015) 6 authors: theologists, sociologists, 1 BA student of psychology (1st year) With all the statistical and methodological textbooks in psychology and other social sciences, dealing with Numerus, sampling etc., they relied on Nock, 2001… …and they focused only on two criteria for trustworthy study („golden rule“?): - Random (probability) sampling - Number of participants (N>400) According to this (concluding remarks in the „report“): • „From 31 studies in our meta-analysis, only two based on random sampling and in all studies the N was smaller than 400. In various studies there was no control group. • Based on these facts our conclusions are the same as in other studies (Nock, 2001): • the majority of studies, supposedly proving that children in the samesex families do not differ in development from children in different-sex families, are not credible and trustworthy • we think that such studies should not serve as means for final, decisive and proper answer to the questions about the development of children in same-sex parental relationships • therefore we take promoting the results of these studies to be false, misleading and deceptive.“ The same „methodology“of „meta-analysing“ and discourse as in Nock, (2001) • The respondents are homosexual couples who wish to be issued marriage certificates. The province of Ontario will not do so, and cites the common law definition of marriage. • Appeal dismissed; definition changed immediately and the province ordered to issue marriage certificates to qualifying same-sex couples. Who is S.L.Nock? … and Nock‘s „methodology“ is similar to „Brief of Amici Curiae“ (2015) for the Supreme Court of the United States During a speech at Franciscan University of Steubenville in 2014 entitled "What Sexual Behavior Patterns Reveal about the Mating Market and Catholic Thought," Regnerus' views on same-sex relationships continued to spread controversy when he claimed that "normalization of gay men's sexual behavior in society will contribute to a surge in the "practice of heterosexual anal sex.„ Regarding the connection between one's faith and the activities of Christian professors, Regnerus noted in an alumni profile that "I believe that if your faith matters, it should inform what you teach and what you research". Are the data really non-conclusive? - Random or probabilistic sampling is ok, but not always possible and not always the best solution (e.g. stratified sampling or quota sampling gives much more precise prediction of election results in political marketing research). - N over 400 is extremely rare in social science studies, especially when doing research in marginalized groups; and credibility of the conclusions is not in linear proportion to the sample size (i.e. „the bigger the sample the better“). - Are we really in a position to expect final, decisive, proper, answers in science? - Science is probabilistic and is trying to be skeptic, nomothetical and falsifiable in Karl Popper‘s sense in validating theories and research outcomes. - There are no final answers in science nor absolute truth. - According to different criteria of research methodology in social sciences all studies, criticised by Slovenian „experts“ and Nock, Mark, Regnerus, Sullins … can actually be recognized as valid and reliable – in the extend of generalizability that is possible according to size and representitiveness of the sample… …as being clearly shown in the following examples: 2005 „Brief of Amici Curiae“ (2012) to the US District Court for the Northern District of California What did the Science 2015 study really say? • Modern science understands scientific findings as reproducible, replicable, and generalizable. The results of the Science study (surprisingly low reproducibility) were sobering. • In general, reproducibility seems undervalued because scientists prioritize novelty over replication. • Innovation is the engine of discovery; researchers are usually driven by searching for the barriers of science; when a topic seems covered, they tend to rush forward but forget to stop and check the stability of the outcomes. • But replication and cross-validation also help to establish a firm nomological network and high validity of scientific theories. • Science can learn from replication studies, critically pointing to important issues in planning and performing research of good quality. • It will also help prevent the manipulations and the poorly supported reproaches of psychology. According to a 2016 poll of 1,500 scientists, 70% of them failed to reproduce another scientist's experiments (50% failed to reproduce their own experiment; Baker, 2016). These numbers differ among disciplines: • chemistry: 90% (60%) • biology: 80% (60%) • physics and engineering: 70% (50%) • medicine: 70% (60%) • Earth & environment science: 60% (40%). In 2009, 2% of scientists admitted to falsifying studies at least once and 14% admitted to personally know someone who did. Misconducts were reported more frequently by medical researchers than others (Fanelli, 2009). Causes of the „Reproducibility Crisis“ (Begley & Ioannidis, 2015): • Generation of new data / publications at an unprecedented rate. • Compelling evidence that the majority of these discoveries will not stand the test of time. • Causes: failure to adhere to good scientific practice & the desperation to „publish or perish“. • This is a multifaceted, multistakeholder problem. • No single party is solely responsible, and no single solution will suffice. Why replication in psychology is so often discouraged (Earp & Everett, 2013): • Independent, direct replications of others’ findings can be timeconsuming for the replicating researcher. • Replications are likely to take energy and resources directly away from other projects that reflect one’s own original thinking. • Replications are generally harder to publish (in large part because they are viewed as being unoriginal. • Even if replications are published, they are likely to be seen as 'bricklaying' exercises, rather than as major contributions to the field. • Replications bring less recognition and reward, and even basic career security, to their authors. Some after-effects of the Science study (reflected also in TOP - below): • expect that journals will most likely publish more replications in the future. They are already launching new policies that will encourage authors, editors, and reviewers to re-examine and recalibrate the basic notions about what constitutes a good research. • Editorial boards will advance the “acceptance culture” of the submitted articles such as sharing data, analysis code, and study materials, disclosing all data exclusions, requiring authors to discuss sample sizes and statistical power, report effect size, etc. At least two lessons that can be learned from the Science study: • that the project was conducted with concern about the health of the discipline, believing in its promise for accumulating knowledge about human behaviour that can advance the quality of the human condition and • that many will be tempted to conclude that psychology is a bad apple in the basket. However, this is not the case: this is a problem shared with natural sciences, medical sciences, and biomedicine, as well as behavioural or social sciences, because the replication efforts in other fields are similarly low. (TOP-Transparency & Openness Promotion) Replication is not a silver bullet. Even carefully-designed replications, carried out in good faith by expert investigators, will never be conclusive on their own (Earp & Trafimow, 2015). But (Tsang and Kwan, 1999) : If replication is interpreted in a strict sense, conclusive replications or experiments are also impossible in the natural sciences.… So, even in the “hardest” science (i.e., physics) complete closure is not possible. The best we can do is control for conditions that are plausibly regarded to be relevant. → → → → → „Crises in psychology are not caused by methodological flaws but by the way people talk about them (Kruglanski & Stroebe,2012) • There is an epistemological misunderstanding that emphasizes the phenomenon instead of its underlying mechanisms. • Theoretical hypothesis (and construct) is what matters (conceptual vs. direct replications) • For the meaningful replications, attempts at reinstating the original circumstances are not sufficient. • Instead, replicators must ascertanin that conditions are realized that reflect the teheoretical variables manipulated (measured) in the original study. „Publihs or persih“ is a curse in a way… "Science, I had come to learn, is as political, competitive, and fierce a career as you can find, full of the temptation to find easy paths." — Paul Kalanithi, neurosurgeon and writer (1977–2015) The 7 biggest problems facing science, according to 270 scientists (Belluz, Plumer & Resnick, September 7, 2016): 1. 2. 3. 4. 5. 6. 7. Academia has a huge money problem Too many studies are poorly designed Replicating results is crucial — and rare Peer review is broken Too much science is locked behind paywalls Science is poorly communicated Life as a young academic is incredibly stressful Faking Science: A True Story of Academic Fraud by Diedrik Stapel Translated by Nicholas J.L. Brown [from Dutch “Ontsporing” (“Derailment”)] Responding to the interim report, Stapel stated: „I failed as a scientist. I adapted research data and fabricated research. Not once, but several times, not for a short period, but over a longer period of time. I realize that I shocked and angered my colleagues, because of my behavior. I put my field, social psychology in a bad light. I am ashamed of it and I deeply regret it.“ Thank you!