In this blog post we report first observations and impressions from our study following 24 recently funded replication studies in practice.
NWO funding calls for replication studies: what were the characteristics?
The aim of the three replication calls, according to the NWO website, was to encourage researchers to carry out replication research and thus to increase insight into the reproducibility of results. But not only that: The funding call also had the aim to gain knowledge of whether and how replications could effectively be included more broadly in NWO research programs. This would also help NWO to reflect on which requirements should be set for funded research in terms of methodology and transparency.
NWO distinguished between three different types of replication studies:
- Reproduction – replication with existing data. This entails repeated analysis of the datasets from the original study.
- Replication with new data: new data collection with the same research protocol as the original study.
- Replication with the same research question: new data collection with a different research protocol than the original study but with the same research question as in the original research.
The NWO pilot program only funded replication research that is “pure”, with which NWO meant replications of the first two types.
The NWO pilot program only funded replication research that is “pure”, with which NWO meant replications of the first two types.
In which research fields are the replications being carried out?
In 2017 and 2018, the NWO calls for proposals were aimed at the social and medical sciences. In 2019, the call was extended to include the humanities as well. In total, NWO has funded 24 replication studies in three consecutive years (2017 – 2019): 18 social science studies, 5 medical studies and only one single replication study in the humanities.
Overview replication studies (and how many of those are participating in our study):
| Social sciences | Medicine | Humanities | |
| 2017 | 6 (6) | 3 (2) | n/a |
| 2018 | 7 (6) | 1 (1) | n/a |
| 2019 | 5 (4) | 1 (1) | 1 (1) |
Observing the funded replication studies in action: what will we do?
Twenty-one researchers doing replication studies (‘replicators’) agreed to participate in our ‘replication in action’ ethnographic study. For comparative reasons (because NWO only funded one single humanities replication study), we also include two replication studies in the humanities that had recently been funded by the Templeton foundation in 2020.
Of all 23 studies, 14 studies are still running and 9 are already finished. We will actively observe practices of the running studies and do interviews and analyze documents and literature to learn about all replication studies. We will also gather available literature regarding the three studies of which the PI’s have been unresponsive.
Different replication study types: first impressions
We already detected a couple of interesting differences and similarities between the different replication studies. Perhaps most striking is that 3/4 of the funded replication studies are being done in the social sciences, mostly in psychology. But perhaps this finding is not so surprising, given that the replication crisis has gotten most traction within this discipline (e.g. Open Science Collaboration 2015). Within psychology, a whole range of fields is represented, such as clinical psychology, neuropsychology and educational psychology. Interestingly, all three funded humanities studies work with replications involving historical material and sources. This is an interesting finding when considering that there had been quite a debate regarding which humanities studies could be replicated (Peels & Bouter 2018; Penders, Holbrook & De Rijcke 2019). The medical studies are varied in their topics. Two are in epidemiology, one performs a very large, randomized control trial (RCT), and another one re-analyzed existing data from a large RCT.
From what we know so far, there seem to be at least eight multi lab/ center studies that generate new data. This means that the replication does not happen locally at one site, but that a specific study is being replicated across multiple sites. In some cases, this includes even multi-national sites. For example, one RCT medical study is being done at several intensive care units in The Netherlands and one in Italy, and includes hundreds of patients. There are also seven social science replication studies that replicate across (national/ international) labs. One of these replicated a study in six different European countries.
There are, as far as we know, very few studies that reanalyze existing data (NWO replication type 1). For example, one medical replication study reanalyzed existing data from a multi-center RCT. Other studies generate new data, but they can also include re-analysis of existing data or include so far unpublished data. In addition, epidemiology studies typically do not exclusively generate entirely “new” data but often also use already existing data in new ways (e.g. from cohort studies). Nevertheless, by far most studies will generate completely new data (NWO replication type 2). Interestingly, as far as we can see this applies to the majority (or even all) of social science studies.
Most studies replicate rather recent landmark studies. At least four medical studies replicate studies published after 2010. Most landmark psychology studies also stem from after 2005; only some are older (e.g. from the 70s). In certain medical and humanities studies, several landmark papers are replicated in one single study. One humanities study for example replicates four studies, with a range from 1965 to 2014.
Conclusion: characteristics of NWO-funded replication studies
Our first impressions let us conclude that replication studies do vary, but that most NWO-funded replications are being carried out in various psychology research fields. In addition, the majority of projects seems to replicate rather recent landmark papers (after 2005). Finally, it seems important to more closely investigate whether replication practices can be fit into the NWO replication typology or whether they might be more complex.
References
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Peels, R., & Bouter, L. (2018). Humanities need a replication drive too. Nature. https://doi.org/10.1038/d41586-018-05454-w
Penders, B., Holbrook, J. B., & de Rijcke, S. (2019). Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing. Publications, 7(3), 52. https://doi.org/10.3390/publications7030052
Replication in Action 