This study was hailed as a win for science reform. Now it's being retracted.
A paper that shows how far the reproducibility movement has come may also show how far it has to go.
How far has the open-science movement come in the last decade? It’s something I thought about a lot while working on my latest story, which is about a very meta saga in none other than the metascience world.
In short, a study that purportedly quantified the benefits of adhering to rigorous scientific practices (including preregistration), led by champions of the science-reform movement (including a preregistration pioneer), is getting retracted from Nature Human Behaviour for — among other things — not being preregistered.
From today’s story in The Chronicle of Higher Education (free to read if you make an account with your email):
…This project’s core premise — that it was testing whether rigor-enhancing practices like preregistration increase replicability — did not itself appear to have been preregistered. Other documents that have been posted online since publication make clear that the study was initially conceived to explore another subject altogether. And when their hypothesis did not pan out, the scholars appeared to overhaul their paper’s focus while playing down their original intentions.
In explaining the retraction, the journal cited a “lack of transparency and misstatement of the hypotheses and predictions the reported meta-study was designed to test; lack of preregistration for measures and analyses supporting the titular claim (against statements asserting preregistration in the published article); selection of outcome measures and analyses with knowledge of the data; and incomplete reporting of data and analyses.” As a result, the journal’s editors stated that they “no longer have confidence in the reliability of the findings and conclusions reported in this article.”
Critics say that those issues cut against the paper’s message — and some of its authors’ reputations as champions of scientific rigor. Brian A. Nosek, a psychologist at the University of Virginia, conducted some of the seminal studies about the reproducibility crisis and directs the Center for Open Science, a nonprofit that advocates for transparency in science and operates a database of preregistrations, a concept he pioneered. Another collaborator, the University of California at Berkeley business professor Leif Nelson, is part of a trio of scholars whose blog, Data Colada, has rooted out problems in the work of behavioral-science stars like Amy Cuddy, Dan Ariely, and Francesca Gino.
The authors don’t agree with most of the journal’s reasons for retracting; here’s what they have to say.
Some of the paper’s critics told me that, in their view, the episode demonstrates the dangers of putting individuals on a pedestal. “Here, in the best-case scenario — in a paper about the importance of embracing these reforms, by the experts who developed these reforms — the reforms themselves haven’t been well-embraced,” said Joseph Bak-Coleman, a computational social scientist affiliated with the University of Konstanz. Berna Devezer, a meta-scientist at the University of Idaho, suspects that members of the open-science community have been afraid to weigh in because of who authored the paper. “Some people just have an a priori confidence and trust in the authors, and they expect that they cannot do any wrong or anything, and some people just don’t want to hear any criticism of open-science practices,” she said.