In March this year, we organised a replication expert workshop with all PI’s and/or executive researchers from the replication studies in our project. The workshop took place at the Hortus Zuidas in Amsterdam with (besides the 4 of us) 20 participating researchers from humanities (6), medical science (2), and social science (12) together representing 16 different replication studies. The aim of the workshop was to specify the concerns and ideas about what a good replication study is about, and to think about policy advice and guidelines for future replication practices. In this blog post we report about some of the experiences and ideas that were brought up in this workshop.
In a first inventory of the concerns and interest of replication researchers, several people expressed their worries or negative experiences with the impact of their replication study, and some explicitly discussed the (vulnerable) position of PhD’s and postdocs doing replication studies. It appeared that doing a replication takes a lot of time and effort and does not always result in highly valued publications. Replication “is not a very sexy topic” in all research fields. The results are also not always very easy to publish, and when published, replicators have the feeling that the older studies are still seen as more important. However, there were also replicators who had opposite experiences. In some fields it seems that replication studies are valued and published very well, and have high impact.
Another topic that was brought up in the discussion was how replication studies do or should relate to the original. Some of the replicators for example mentioned the issue of historical changes. Theories they want to replicate can be very familiar nowadays, the original materials do perhaps not have the same effect, and the original methods are perhaps not up to current standards. How should replicators deal with these differences? One solution that was brought up, and practised by several replicators, was to do it all: according to the original protocol, and according to their own ideas of how it should be done. However, this is not very easy to write down, and it also does not reduce the amount of work.
In general, the replicators seemed to agree that they learned a lot from doing a replication. They learned for example that transparency is very important but simultaneously also difficult (lots of details are missing); that concepts can be interpreted differently; that decisions of the original researchers are not always very easy to follow; and that doing replication research is not easy but perhaps even more difficult than ordinary research. Hence, they all agreed that replication research is an effort that is very valuable for the researcher, and should also be supported and rewarded by science policy makers, research organizations, journals and funders.
All replication researchers seem to agree that replications should become mainstream. It gives new knowledge and makes better researchers. Replication leads to more open science because researchers become very aware of the importance (and difficulties) of writing up and sharing what exactly they did in their research. It reveals misunderstandings about concepts, methods and analyses. Replicators were also very enthusiastic about the value of replication for teaching. Doing a replication reveals a lot about methods, transparency etc, and since students are the future scientists this is seen as an indirect approach to make researchers better researchers.
Replication in Action 