Time and budget restrictions as well as other funding criteria can shape how science is being done in practice. Here, we describe the impact of NWO funding specifications on “our” replication research projects.
Funder stage: What kinds of replications should receive funding?
In 2017, the Dutch science funding organisation NWO earmarked three million euros for replication projects in the Social Sciences, Medical Sciences and, in the last round, also in the Humanities. The replication calls (2017-2019) were explicitly considered to be a pilot for the funders in order to learn more about whether and how to eventually fund more replication research.1 NWO decided that researchers could either submit proposals for a replication study with existing data (replication type 1), asking for a budget up to 75.000 euros, or submit a proposal for a replication with new data (replication type 2), asking for up to 150.000 euros. In both situations, the maximum duration for projects was 2 years. And because NWO expected that with so little budget and time one could typically not perform a medical clinical trial study, they explicitly excluded these from the start.
In their call for proposals, NWO also wrote that they expect researchers to replicate a landmark study. In addition, the replications should be conducted “as precise as possible” and if not, to specify where and why the study deviated from the original study. Researchers were also asked to “demonstrate that the sample size is large enough for the proposed research” – if necessary with the help of a statistician.2
Grant application stage: What kind of replications were applied for?
The funder criteria outlined above shaped the kind of grant applications submitted at least to some degree. On the positive side, many researchers emphasized that the funding enabled them to finally do certain types of replications at all: those depending on at least some kind of funding. On the negative side, the funding also seemed restrictive. For example, there is only one medical RCT study generating new data in the NWO sample. And this specific project could only get of ground because external other funding was added. With regards to RCT’s, one researcher emphasized that 150.000 euros is in principle not enough budget, and that NWO would need to provide more funding should they be interested to receive more applications of this kind. We also know of other applications that were rejected because the original study was not perceived to be sufficiently important (landmark) in its field.
It is interesting to note that the most-funded type of replication is the one generating new data (type 2), although some researchers also added some analysis of existing data (type 1) to this form of replication. In general, there seemed to be more interest in the bigger bag of money, and thus wherever possible researchers applied for a type 2 replication. In addition, most researchers wrote applications promising to do experiments with much more power than the original studies had, or to generate/analyse high-powered data sets. Many replicators also stated that they had sufficient information regarding how to do the study as exact as possible.
Project in practice: How do funding criteria impact running research?
It is one thing to promise great things in a grant proposal, it is another thing to do what one promised in research practice. From the start of our project, it became clear for example that 2 years is often not sufficient to really conduct replications in practice – and extra difficulties due to corona exacerbated the situation (see blog 3).
Many problems in running research were a direct consequence of trying to adhere to funder criteria: performing a landmark study; a precise replication; and having a large enough sample size. Doing all of this turned out to be doable within the allotted time and budget only for some of the researchers. For others, it was quite a hazard. As one of the replicators explained: “It is about those spectacular things, Nature-, Science like. Yes. These studies are actually almost never… [replicated]. It is about small sample sizes, no pre-registration etc., but yes, very spectacular. But it is so expensive, and it is done with 20 or 30 people in an fMRI. Well, you should actually check that with 200 people or 300 people, or whatever.”
In psychology for example, some of the replicators ended up with participant samples which they felt were at (or even beyond) the limit of what is feasible in practice. “I think this is about the limit of human data we can collect. 300 people is just a lot.” And not only conducting the experiments was an issue, also the recruitment was often more problematic than anticipated. While it may not be too difficult to find 20 people with a specific psychopathologic diagnosis or health issue willing and being able to participate in a study, this is different for 100, 200 or even 300 subjects. Hence, some researchers expressed that they are really worried that they will simply not find enough participants for their study, within the expected time: “I totally understand the value of direct replication. But we should realize that it is harder to do in some fields then in others.” Also some humanities researchers and medical researchers ended up generating or analysing data sets of a very big size, eventually running into troubles with time due to unforeseen difficulties, such as cleaning of data taking much more time than expected.
The criterion of exact replication was another complication for some of the researchers, in particular in psychology. While some could simply re-use some or all of the material from the original project to conduct the experiment, others had to reconstruct/ rebuild them from scratch. (“He told us literally: ‘all materials have been relinquished’.”) On top of that, even in those situations where all materials could indeed be reused, there was still no exact knowledge of how the experiment had actually been conducted in practice. Not everything can be put on paper, after all, or expressed in dialogue. “We could perfectly rebuild everything. But the data, and how he defined everything about the inclusion criteria etcetera, which were very vague; in every article he did that a little differently. (…) and he did his best to answer my questions, but he also said: ‘it is so many years ago’.” Interestingly, we noticed that not all researchers felt as obliged to fulfil the criterion of exactness, in particular in the medical research and in the humanities. Those researchers felt more at ease to change some of the material, techniques etcetera if it seemed useful for doing good research (see Blog 4). Consequently, these more flexible researchers ran into much less trouble when performing their study.
All in all, doing the promised replications in practice often turned out to be more difficult than expected. Not surprisingly, then, we also observed that often, replication researchers had (and some still have) to make trade-offs to finish their research within the expected period and budget. Some researchers, for example, stopped prematurely because PhD’s needed to finish, or stopped shortly after their first publication. This restriction also left no time to work with collected additional information, for example with added questionnaires, extra experiments, or archival work. (We haven’t touched that data. (…) Perhaps that is for a secondary analysis. Now that we have finished the paper, which was delayed a lot of course because of corona. And we have a lot of other projects too of course, so this extra, that eh … well we haven’t planned yet when we will start with that.) Some medical and humanities researchers tried or still try to find further funding sources to expand their original project, but this often turns out to be difficult. In general, one can say that funders have a strong hand in shaping replication research, and that the “projectification” of science leaves some findings unexplored.
1 See: https://www.nwo.nl/en/researchprogrammes/replication-studies
2 Netherlands Organisation for Scientific Research (2016) Call for proposals replication studies, 1th round, The Hague.
Replication in Action 