Preprint Journal Clubs: building a community of PREreviewers

By Samantha Hindle and Daniela Saderi, PREreview

The image above (DOI) is CC-BY 4.0 licensed and is available for download on Figshare.

Preprints are freely available scientific manuscripts that have not yet undergone editorial peer review. They provide data and knowledge that is current, accessible by all, and at a stage where community peer review can contribute to scientific progression. Rather than restricting feedback to two or three journal-selected reviewers, preprints can be read and evaluated by a diverse population of interested scientists at different career stages. Theoretically, the advantages of opening up scientific evaluation to a larger pool of scientists should be straightforward: the more reviewers, the fewer mistakes – or to quote Linus’ Law, “Given enough eyeballs, all bugs are shallow.” Practically speaking, this can be more complicated as scientists have limited free time, are not well-incentivized for their reviewing activities, and some may argue that “too many cooks spoil the broth.”

So, how do we harness the power of community-driven review, whilst reducing the burden on the community and maximizing efficiency? We, amongst others, are encouraging the peer review of preprints during scientific journal club discussions. As these discussion groups are already ongoing in research institutions all over the world, and are often supported by existing infrastructure, why not direct that energy to something that can be used both by the authors and the community? We believe that preprint journal clubs can provide the peer-to-peer support and mentor-driven guidance to help train a diverse community of peer reviewers and in particular, early-career researchers (ECRs). This idea is not a new one; what’s new is the way we are implementing it and the scale to which we want to see it happen.

Is this what the community wants?

In the summer of 2017, we conducted a survey and collected responses from 102 people, primarily academics [1], in order to assess the community’s attitudes regarding preprint commenting and peer-review training. Most of those surveyed had not yet participated in a preprint journal club, but were positive about the idea. When we probed about peer-review experience, even though the majority of the respondents had contributed to peer review, very few had received any training. Many recognized the positive impact that can derive from reviewing preprints at journal clubs, especially in regards to ECRs receiving peer-review training.

In response to these results, and after discussions with various interest groups, we launched PREreview in September, 2017.

PREreview today

PREreview.org is a free online platform for the collaborative writing of preprint reviews after discussion at journal clubs. The website is supported by Authorea’s infrastructure, allowing reviewers to write and publish (with a free DOI, Digital Object Identifier) their feedback directly on PREreview. Each review can also be tagged with the name of the preprint journal club group, as well as with specific keywords that are consistent with the science reviewed, to facilitate discoverability and categorization.

Our goal was to create a collaborative space for a community of preprint reviewers to grow and flourish in a supportive “environment,” perhaps lifting some from the pressure that might arise from posting the feedback directly on the preprint server. In addition, by developing easy-to-use resources, we want to encourage the uptake of preprint journal clubs in institutions all over the world.

We also created email templates to help researchers initiate preprint journal clubs, information sheets and slide decks to introduce the concept of preprints to the group, and guidelines to encourage constructive rather than destructive feedback. Although these resources are designed for preprint journal clubs, they can also be used by individuals to guide them in traditional peer review.

PREreview tomorrow

We are currently in our beta testing phase and, as part of this process, we have been surveying both the preprint journal club participants and the preprint authors who received feedback. So far, we have been overwhelmed with the level of enthusiasm with which the community has embraced this trial run. When we asked participants if they would be likely to attend a preprint journal club in the future, we heard a resounding “Yes!” And the preprint authors? All of the authors said they were happy to receive the preprint review, that the feedback was useful, and that they would likely incorporate the feedback into their manuscripts. Also, almost all of them were positive about the review being openly available online, a response we did not necessarily expect.

“The feedback we received from being reviewed during journal club was of comparable quality to that we received from journal reviewers. Addressing the points of concern most certainly produced a better, more effectively communicated, paper.”

Preprint author’s response to a PREreview

What’s next? One of the biggest barriers remains the fact that it takes time to write a review and scientists are busy. While we do not yet have a definitive solution to this problem, we have taken several actions to try and address this. For example, we provide two kinds of guidelines for preprint review writing: a longer and more formal version, elaborating on tips and strategies to provide constructive feedback, and a quick version, in the form of a short questionnaire. We also encourage a 2-person approach, in which one presents the preprint and the other writes the notes. To further maximize time efficiency, we have experimented with the use of collaborative annotation and commenting tools like Hypothes.is to promote live capture of the journal club discussions.

A recent analysis by eLife [2] showed that most ECRs prefer to receive peer-review training by working through a manuscript with their mentor. Soon, we hope to make available a PREreview resource that will build on the fantastic work of Dr. Prachee Avasthi, Assistant Professor at the University of Kansas Medical Center and member of our PREreview Advisory Board. We hope to develop and share a full curriculum that can be used by university programs in different fields to teach students how to critically evaluate a scientific manuscript and write a peer review using preprints as primary material. Encouragingly, this is already happening within the walls of preprint journal clubs, as one of our beta testers has remarked:

“…it was also a class on “how to write a review” for many of the participants”

Feedback from a PREreview beta tester

Lastly, features that we look forward to implement and add to PREreview include ways by which authors can solicit reviews at the time of preprint submission, the requirement of an ORCiD iD authentication to write and comment on PREreview, functionality that will make it easier for authors to reply to the reviews should they want to, and, most importantly, creating a system to directly link the reviews and eventual author’s responses to the preprints themselves as Linked Data.

Conclusions

In summary, reviewing preprints benefits many stakeholders in the scientific community. Authors receive early feedback on their manuscript, while early-career researchers benefit from the opportunity to develop their critical thinking and peer-review skills, an act that can itself further their networking and career opportunities. The whole scientific community also benefits from access to scientists’ discussions about the latest discoveries.

The emergence of several initiatives that aim to improve peer review gives us confidence that we can look forward to a brighter future for scientific evaluation and dissemination. The success of these efforts, including ours, will depend on the level of engagement and contribution coming from the whole community. We invite anyone who would like to help with PREreview to contact us at preprintjc@gmail.com or leave a comment below.