What the Replication in Action project is about
In the last decade, concerns have been raised in several fields of research about the reproducibility of research findings. In psychology, for example, a large-scale effort to replicate 100 experiments could only reproduce about 40% of the original results (Open Science Collaboration 2015). A systematic review of the medical literature showed that the reproducibility of pre-clinical research findings was even lower (Begley & Ioannidis, 2015). Researchers declared a replication crisis in psychology and in medicine, and questions are increasingly being raised in other fields as well (e.g. Mueller-Langer et al., 2019).
Our project, funded by the Dutch Research Council (NWO) aims to increase our understanding of the practical and epistemological issues at stake in replication studies. If a result is not reproducible in a replication study, does that mean that the original claim was false? And if not, how may we explain the failure to reproduce? Reproducibility of results by different researchers is generally considered an important touchstone of the validity of scientific claims, but there are a number of complexities associated with replication and reproducibility.
Firstly, a replication study is never identical to the original, if only because it was done at a different time. When a replication fails to reproduce the result of the original study, the question therefore inevitably arises whether this is due to problems with the original study or to problems with the replication. The results of replication studies are hence difficult to interpret, and controversy often ensues. There is also discussion about the different kinds of replication and their respective value in different situations and discplines. For example, instead of closely replicating the procedures of the original study in so-called ‘direct replications’, some researchers argue that it would make more sense to test the theory about these mechanisms in novel ways: so-called ‘conceptual replications’. Other researchers have warned that confirmation bias might affect conceptual replications, because unsuccessful replications can be discarded by arguing that they do not properly operationalize the theory (Pashler & Harris, 2012; Nosek, Spies & Motyl, 2012).
Finally, it is a matter of debate when replication studies are necessary. Several authors have proposed criteria for selecting studies most in need of replication, arguing that scarce resources (money, time) must be efficiently used (e.g. Field et al., 2019). A more fundamental question is whether replication is a worthwhile goal in all fields of research. For example, in response to a call from Peels & Bouter (2018) for replication studies in the humanities Penders, Holbrook & De Rijcke (2019) have argued that reproducibility is not always a reasonable expectation in the humanities
(see also Leonelli, 2018). Studies in the humanities often concern unique, non-repeatable events, and/or they rely on interpretation rather than statistical analysis or formal modeling.
What we are going to do
The Replication in Action project aims to study how these and other issues are discussed and dealt with in practice: by funders deciding on the allocation of resources, by researchers planning and executing replication studies, and by the disciplines involved more generally. To better understand the practical and epistemological complexities of replication studies, we will study ethnographically how replication studies are conducted and how their results are discussed in the research teams and beyond. We focus on 24 individual replication studies that have been funded by the Dutch research funder NWO in three consecutive years (2017, 2018, 2019). We complement these studies with two additional replication studies in the humanities that recently have been funded by the Templeton World Charity Foundation, and which are being conducted at VU University in Amsterdam. The reason for this addition is that only one single humanities replication study has as yet been funded by NWO.
How we will do it
We will closely follow the researchers in their everyday practices of conducting replication research in order to understand how replication studies are conducted, and how researchers relate the replication effort and results to the original study in order to establish if it was a good replication. While such observations of ‘science in action’ are a standard approach in science and technology studies, it has not been utilized for investigating and analysing replication studies since the eighties (see Collins 1985; Mulkay & Gilbert 1986; Travis 1981). Our science in action approach will entail a multi-sited ethnography, meaning that we will observe replication practices at various sites (universities, labs) and levels (experiments, research meetings, lectures, online discussions), and that we also intervene in the replication debate (e.g. with a workshop, blog posts, a commentary). Thus, we do not aim to ‘objectively’ or ‘distantly’ describe replication practices, but to witness and document the complexities of such studies, including the influence we may have on them (see also Hine, 2007).
We will study the complexities of replication ‘in action’ with an empirical ethics approach. This entails that we will explicitly investigate what researchers themselves think would be the right thing to do in a replication. What does each of them understand by ‘good’ or ‘bad’ replication studies and practices, and why? Does what is seen as a good replication result or practice vary across the fields, studies, materials or methods used? We analyse notions of ‘good’ or ‘bad’ replications empirically, in each unique situation (Pols 2015; 2016). This empirical ethics approach has its precursors in ethnographic studies of care practices (where the ethnographer investigates what good care means under various circumstances (see e.g. Pols 2014; 2015; Mesman 2015, Jerak-Zuiderent 2012; Garfinkel 1967). Arguably, doing good research also depends on the specificity of different research settings and circumstances (see e.g. Mol et al, 2015; Puig de la Bellacasa 2017). In this project, our novel aim is to analyse how ‘good replication studies’ are
practically conducted. A comparison of the unique study settings and circumstances of replication studies ‘in action’ will help to understand the epistemological claims that come with replication efforts across research settings (Leonelli, 2018).
Who we are
Prof. dr. Jeannette Pols, project leader. Amsterdam UMC, Department of Medical Ethics, and University of Amsterdam, Department of Anthropology
Dr. Maarten Derksen, project leader. University of Groningen, Faculty of Behavioural and Social Sciences, Unit of Theory & History of Psychology
Dr. Jonna Brenninkmeijer, post doctoral researcher. Amsterdam UMC, Department of Medical Ethics, and University of Groningen, Faculty of Behavioural and Social Sciences, Unit of Theory & History of Psychology
Dr. Stephanie Meirmans, post doctoral researcher. Amsterdam UMC, Department of Medical Ethics
Replication in Action 