How are the replication researchers we are studying doing in these corona times? In our first round of interviews, we noticed a strong impact from Corona restrictions and/or work overload in most of the NWO replication studies. Many of the studies are delayed and thus still running.
To study “Replication in Action”, we first try to get an overview of the participating studies (23 projects – see previous blog post for examples). We approached the PI’s of these studies for an interview, and so far we had meetings with one or several researchers of 15 of the participating replication projects. In these interviews, we noticed that more studies than expected were still running. Some had not even started yet. Some studies are still in the set-up stage of their replication projects; others have just started collecting data; again others are still in the analyzing or writing up phase; and only about 25% are done. It turned out that, as in most parts of our society, replication researchers suffered from the Corona crisis too.
Standstill
The impact of Corona restrictions was visible for example in those studies that work with group settings. As one of the PI’s reported: “One of the reasons why we’re really delayed… is because the intended group size is of course just not possible due to Corona. That really posed a lot of problems.” Someone else told us “and then there was the corona pandemic, and yes… that brought everything to a standstill. Because we were supposed to test in groups. That was really crazy.” But also in studies where group settings were not an issue, replication researchers ran into problems. Some depended on the lab work of collaborators, which was delayed either because those collaborators were not allowed to do lab work or because they needed to focus on Corona-related work instead. Especially in the medical sciences, replication research suffered from the impact that Corona had on the work of the replication researchers themselves: many were involved in helping to fight Corona, for example by performing patient care on the IC or by giving epidemiological advice.
Re-starting research
As a result of Corona policies, some of the replication studies were shut down for about 1.5 year, and some researchers are now (re-)starting their experiments or projects. Re-starting can give some annoyances because of bureaucratic paperwork and other problems. Ethical approval has expired; trained students and PhD’s are gone; or contact with collaborators is lost. Running experiments sometimes require extra measures: labs need ventilation; there have to be protocols for extra cleaning; or more or bigger labrooms are required. Also striking was that quite some researchers aimed to do their study in 2022 instead, probably with the idea that “hopefully the Corona measures will be [better] than this year.” On the other hand, some researchers decided to finish projects before they had all the data that they originally had planned to collect – because their postdocs and PhD’s need to finish in time.
Adaptation
Some study designs had to be adapted because of Corona. One researcher explains: “We first wanted to have labs overseas, but then Corona came and we were not able to do that. And we ended up with labs in Europe only.” Other researchers had collaborations with other labs or research sites, but because of the lockdown or because they were simply overworked due to Corona, these other labs/sites withdrew their participation. Communication in general went differently. While some researchers would have liked to meet the original researchers in person, they now had to get all information by email or (sporadically) online meetings.
What does replication mean?
It is interesting to relate our above findings to our primary study question – what does replication mean? It seems that the somewhat different circumstances in which our replication researchers find themselves nowadays do have an impact on how replications are being done in practice. We found for example that group and sample sizes are being reduced, lab settings are being changed, and communication goes via different channels. But then, how special is the Corona situation? Can conditions from the past ever be exactly replicated in other circumstances?
Replication in Action 