The replication crisis was an important reason to initiate a grant for replication studies in The Netherlands (NWO, 2017-2019). In their call for replication proposals the Dutch research funding organization NWO writes:
“Replication lies at the heart of the scientific method and makes it possible to build upon previously demonstrated and confirmed scientific findings. Many studies, however, have proved not to be reproducible. If research is not reproducible then this is often attributed to chance, or unintended errors, but p-hacking, publication bias and especially selective reporting will undoubtedly play a major role in this as well. By encouraging the realisation of replication research, NWO wants to make a contribution to increasing the transparency of research and the quality and completeness of the reporting of results.”
How do the researchers who received such grants and do the actual replication work engage with this crisis in their own work? In this blog post, we will dive into these questions.
From the interviews, observations and collected documents of these replicators it became obvious that most of the medical researchers we interviewed used the grants as a vehicle to perform certain studies they intended to perform anyway. They did not participate in the funding call because of a sense of crisis per se (“for us, it was just a possibility to do the study that had to be done”). Some humanities researchers did indeed refer to the crisis, but then typically as one occurring outside of their own field that could be used as a vehicle to reflect upon one’s own work: “my idea is that the humanities can profit from (…) the things learned from the replication crisis, in, say, the biomedical sciences and social psychology”.
Perhaps not surprisingly, replicators of the 17 replication studies in the social sciences engaged most with the replication crisis. Some of them directly referred to the crisis when explaining the reasons for undertaking their replication research. In their preregistrations or published papers some of them for example wrote: “In light of the current replication crisis in the social sciences (…)” or “given the developed awareness regarding the replication crisis in psychology (…)”. Some also used the replication crisis as a motivational argument when being interviewed:
“We embraced the initiative because we all are supporters of open science and the initiative that emerged in the aftermath of the replication crisis in social psychology. And we’re also concerned after the falsification scandal around Diederik Stapel… all this triggered the initiative by NWO, but also our support for such initiatives. And at the same time, and it is the same thing basically – since this is a result that we also embraced within our discipline and think this is an important result. It allows us to contribute to our field, by making sure that this result is robust and can relied on in future research.”
This researcher articulates two motives for doing a replication study: He engages with the replication crisis and supports open science, and he wants to contribute to his field. We recognized these two motives also in many of our other interviews with psychologists. Some researchers want to replicate a specific study because they suspect the result is not reproducible, and hence a part of the replication crisis: “Several labs in the world have tried [to reproduce the effect], with different paradigms. None of these were successful, except for the [original] lab itself.” Other researchers replicate a study because it has such a great impact in a specific research field that they want additional verification for the effect: “This study was cited a lot in my field (…). But of course, then you’ll have to be sure the study is correct.”
Interestingly, some social science researchers express outright hesitation towards the current discussions and ideas of a crisis. Some emphasize that we should rather learn something positive from doing replication studies:
“We select the most prominent studies now (in this replication crisis); it is a really negative debate. Not very helpful. I would like to turn that around: what do we want to learn from this? That is the value if the effect is not there.”
And some think an exclusive focus on replications may even have a questionable effect:
“There are certain psychological theories where hundreds of studies have contributed to. But then there is one group of people who attempted to replicate it, and they decide: no, the effect is zero. End of discussion. I find that a pity, too. And then I start wondering. How is it then possible that those 200 people did find anything at all? Is that then all publication bias? Sure, that could be the case. But just figure out why, how it really works, you know, that we really learn something.”
In short, we see a variety of relations regarding “the replication crisis”. Some researchers clearly experience a replication crisis in their field and aim to use replications to uncover bad science. Some use the crisis to reflect on their own research practices. Some use the sense of a crisis to perform certain studies or verify specific results (but without suspicions of previous wrongdoing). Others think the replication debate is very negative, which can have harmful effects.
Replication in Action 