Open pre-print peer review: a call for greater transparency in the evaluation of manuscripts

Lachlan Coin, Institute for Molecular Bioscience, The University of Queensland, Brisbane, Australia
Darya Vanichkina, Centenary Institute, University of Sydney, Sydney, Australia
Alicia Oshlack, Murdoch Childrens Research Institute, Royal Children’s Hospital, Melbourne, Australia

Transparency and openness are extremely beneficial for science.  The immediate and open publication of findings via preprint servers results in rapid dissemination of scientific discovery amongst researchers.   However, under the status quo, preprints are peer-reviewed by journals in a manner which is completely non-transparent to the reader;  and are commonly published in a subscription journal.  As a result, only the draft submission is openly available.  The reviews and the authors’ response are locked away forever.  Readers are asked to trust the journals implicit guarantee that each manuscript has gone through the same robust peer-review process.   However, the reliability of the peer-review process at high profile journals is somewhat of  a myth, as is demonstrated by a trend towards a higher retraction index with higher impact factor [1]. This practice of hiding reviews is damaging to scientific progress and to the general trust in the academic publishing system overall, since readers cannot see which aspects of the study were questioned and whether some aspects – which the reader may consider critical  – were ever questioned at all.

We therefore call for a greater transparency in the evaluation of scientific manuscripts in the life sciences. In particular, we call for the submission of all manuscript revisions in bioRxiv, not just the original draft; and for a switch to an open peer review process linked to bioRxiv.

An open peer review layer for bioRxiv

We argue that the current peer review process should be replaced with an open bioRxiv-based peer review layer.  Signed open reviews (which may still be convened by a journal) would be viewable from the bioRxiv manuscript page.  Reviewers would – similarly to the currently existing peer review process – consider all aspects of a manuscript in their review and would comment on reliability, quality, novelty and impact.   The author would be able to update their manuscript and to openly respond to the reviews.   We believe that open reviews would thus be more likely to be framed and viewed as constructive criticism, which could be used to improve the work in order to get it to an acceptable standard for publication.  An open, signed peer review is automatically portable between journals, allowing authors to rapidly shift papers rejected on ‘impact’ to a different journal.  A database of open, signed reviews will also help to overcome the issue that papers rejected for good reason in one journal can often end up published unchanged in another journal:  under the model we propose the authors would be required to either openly respond to or acknowledge the shortcomings of their work.

We advocate for journals, especially academic run society journals, to allow authors to ask for open peer review of their submitted preprints. Ten journals now allow direct submission from bioRxiv[2]  but the peer review process remains closed.  We ask these forward-thinking journals to lead the way in facilitating open peer review linked to bioRxiv manuscript versions.

 

The importance of curation and the role of the journals

 

Despite advocating a shift of peer-review from journal-bound to an open preprint-layer, we still see an important role for journals in continuing to curate the scientific literature.  This curation will take the form of journals choosing to print papers which it deems important to its field and of interest to its readers. High-profile journals such as Cell, Nature and Science  still have an important role in highlighting the best research, of interest to researchers across all fields and disciplines; although we agree that measuring journal impact factors, and judging candidates for grants or promotion on this basis is very distorting.   We anticipate that other ways of highlighting interesting research to a particular community will emerge [3].

How do we make it happen?

There is already a groundswell of support for posting to preprint servers at the same time as submission to a journal. The Analytic and Translational Genetics Unit lab at MGH, for example has recently mandated that all papers led by an author from ATGU should be posted to bioRxiv at submission. We would like to encourage other departments, institutes and universities to follow this lead and mandate bioRxiv posting, both at the submission, and at the point of re-submission after addressing reviewers comments.

Peer-review activism:  access to peer review is a critical requirement in getting a paper published.  Scientists who want to support a move to open peer review of preprints should consider only agreeing to review preprints, and should consider posting and signing all of their reviews to an open peer review database, or to their blog.

Funder-mandates:  we would like to call on funders to mandate submission to preprint servers on submission and revision; and to also mandate open peer review of preprints.

Journal support:  we call on journals to shift to an open peer review model, and to encourage reviewers to deposit these reviews in an open-peer review database which is cross-referenced to preprint servers. We call on journals to change their preprint policies to explicitly state that they allow submission of manuscript revisions to bioRxiv.

Technical hurdles:  The technical hurdles in implementing an open peer-review layer proposed here are not substantial. Blog posts referencing bioRxiv posts are already linked at bioRxiv, so reviewers can already post open reviews to personal, or community blogs such as Haldane’s sieve [3].  There has also been substantial effort recently in development of open peer review databases [4 – 7] which could be rapidly adapted to provide a peer-review layer for bioRxiv.   As long as each of these are linked at the bioRxiv preprint page, there is no need to pick a ‘winner’, but to allow each to evolve to address this need as it emerges.

Acknowledgements:

This commentary grew out of a discussion between Kat Holt, Alicia Oshlack, Mike Inouye, Daniel Macarthur, Darya Vanichkina and Lachlan Coin about optimising the presentation, reporting and review of findings in the life sciences, with a specific focus on  preprints and new approaches to peer review at the Lorne Genome Conference 2016.

  1. Brembs, Björn, and Marcus Munafò. “Deep impact: unintended consequences of journal rank.” arXiv preprint arXiv:1301.3748 (2013).
  2. http://www.cshl.edu/news-and-features/biorxiv-preprints-can-now-be-submitted-directly-to-leading-research-journals.html
  3. Haldane’s sieve: http://haldanessieve.org/.
  4. Publons: https://publons.com/
  5. Academic Karma: http://academickarma.org
  6. Pubmed Commons: http://www.ncbi.nlm.nih.gov/pubmedcommons/
  7. is http://hypothes.is/

Conflicts of interest:

Lachlan Coin is the founder of Academic Karma, which is an open journal-independent  preprint peer review platform.

  • Lachlan Coin

    In this post I called for signed, open peer review of preprints. I have now changed my position slightly after a lot of conversations with colleagues, and now I think that the decision to sign the review or not can only be made by the reviewer. The most important feature of what we propose is a system for content-open peer-review of preprints.