Illustration by Roberto Bia, shared under a Creative Commons CC BY license.
Post by ASAPbio Fellow Laura Celotto
There has been much debate for over a decade about reproducibility in science (Ioannidis). By “reproducibility”, we mean the ability of a scientist to reproduce their own results or others scientists’ published work. In a survey conducted by Nature, more than 70% of researchers reported that they had failed to reproduce the work of other scientists, and more than half had failed to reproduce their own work (Baker). Projects that attempted to replicate published findings reported that they failed to reproduce up to 20% of the results from preclinical cancer studies and 40% of the results in psychology research (Begley & Ellis, Open Science Collaboration). The reasons for such challenges around reproducibility are several and include issues related to study design, reporting, publication bias – and occasionally scientific misconduct -, but what are the options available to tackle reproducibility?
During our journey through the ASAPbio Fellows program, we wondered whether sharing research work as a preprint could help in tackling the issues of science reproducibility, due to the early availability of the work to the community. As part of our project exploring journal editors’ perspectives on preprints, we approached this topic with the editors we interviewed.
Preprints offer potential for improving manuscripts prior to journal submission but this is not yet common in practice
We asked Katherine Brown, Executive Editor at Development, whether she thought that preprinting might help in detecting potential issues in a manuscript. She argued that people who preprint are cautious about the accuracy of the studies they post on the servers, because their work would stay there, publicly available, potentially in perpetuity. Moreover, Katherine noted that one of the advantages of preprints is their exposure to many more eyes than those of the two or three referees and an editor involved in the traditional journal peer review of a manuscript – that is, the preprint brings exposure to a larger research community who can scrutinize the work. Such community feedback could help authors correct potential mistakes in the manuscript prior to submission to a journal. On the other hand, it is not uncommon for preprints to be posted when the manuscript has already been submitted to a journal, hence preventing the benefit of mistake correction afforded when preprinting earlier. Moreover, public commenting of preprints is rare and does not always take place before the journal publication for the paper. Mario Malički, Editor-in-Chief of Research Integrity and Peer Review noted:
“It takes roughly 100 days to see a public comment on bioRxiv, by then, the manuscript has often already been published.“
Mario shared that currently public commenting on preprints is not a widespread practice, and usually includes praise or suggestions for the authors of the study, and less often a full critical scrutiny of the work (Malički et al.).
What elements signal that a preprint is reliable?
When asked about the potential of preprints in detecting poor scientific practice and/or misconduct, Mario noted that it is not the scope of a preprint, nor that of the “traditional” reviewers of a paper, to detect data manipulation and/or fabrication. The task of peer reviewing a preprint, and a research work in general, is to evaluate the robustness of the presented data and of their interpretation, but peer review relies on a basic trust in the goodness and integrity of the submitted/posted work. In this context, we asked Mario whether there are any signals for the reliability of a preprint. Mario noted that there is a lack of consensus about what makes a preprint reliable, but in his view, a major indicator of preprint reliability is whether the data are accessible. As Editor-in-Chief, he makes sure that the data of manuscripts submitted to his journal are always accessible for scrutiny. In addition, he ensures that the authors clearly report the limitations of their work, for the sake of transparency. Verification of authors’ identities, for instance by checking their ORCID as well as their institutional affiliation, is a further indicator of preprint reliability, even though Mario acknowledges that this might introduce a bias towards researchers affiliated with institutes based in Western countries.
An additional indicator of preprint reliability is the transparency in reporting the methods used in the study. Alejandra Clark, Division Editor at PLOS ONE, noted that a clearly defined scientific question and a transparent account of the methods help make a preprint more reliable. Accuracy and integrity of methods are important parameters also for Beth Osia, who is part of the editorial team at Open Biology, and previously Proceedings B, that solicits preprints for submission to the journals. Beth added that, for her, a reliable preprint should also show consistency between the data presented in the figures and the data reported in the related text.
Based on these conversations, the following seem to be important elements signaling the reliability of a preprint:
- Data accessibility
- Transparency in reporting of methods
- Clear definition of the research question
- Consistency between figures and text
- Verified identity of the authors
Preprints and reproducibility: where next?
As part of the effort to make research publications more transparent, Mario Malički envisions a future where artificial intelligence will assist editors and the whole research community in the early assessment of preprints. He wishes for a future where automated tools will scan preprints with respect to language quality, methods, ethics and conflict of interests, likely speeding up the editorial process and reducing the burden on editors, authors, and additional stakeholders.
Whether this prediction will become true is most likely a matter of time. For the moment, the initial question of this blog post “Are preprints a useful tool to address issues of science reproducibility and integrity?” does not yet have a definitive answer. It is likely that researchers as well as publishers will need to shape their policies as preprinting practices evolve across disciplines. The future of preprints is open, and the research community has an opportunity to shape what reproducible practices entail in the context of preprints.
