Testing a new preprint review modality to foster participation

While there is increasing interest in preprint review activities, the level of public commenting and reviewing on preprints remains low overall. To explore review modalities that may foster participation in preprint review, we are adapting the crowd review model pioneered by the journal Synlett to preprints, and are running a trial to learn whether this approach provides an engaging format for researchers to participate in preprint feedback and contribute to public preprint reviews. 

The Crowd preprint review trial involves bioRxiv preprints in cell biology which authors have requested feedback on. We will coordinate a group of researchers in cell biology interested in contributing feedback – the Crowd. During the trial, we will circulate a preprint or two weekly to the Crowd, and will invite members to provide comments via a Hypothes.is group; Crowd members can comment on the full paper or parts of the study according to their interest. After seven days, we will close the commenting period and generate a collective synthesis of the comments to be posted as a public review.

You can view a list of preprints with Crowd reviews on Sciety or access the reviews on bioRxiv.

Crowd preprint review trial FAQ

Why is ASAPbio running this trial?

Interest in preprint review activities is growing and there are different platforms available for posting comments and reviews (e.g. comments on the preprint server, PREreview, preLights, PubPeer etc). However, the level of public commenting and reviewing on preprints remains low (a recent analysis of bioRxiv preprints reported that less than 10% of preprints received comments), which suggests there are barriers to participating in preprint review, possibly related to concerns over whether the authors would welcome feedback on the preprint, whether the commenter feels qualified to review the full paper, and potential vulnerability for early career researchers if they comment openly on a preprint by a more senior researcher.

We want to learn whether the crowd review approach provides a framework to address those barriers, by providing a forum for commenting on preprints where authors have explicitly requested feedback, in a format that gives flexibility on what comments to provide, and by generating public reviews that will not necessarily link the commenter’s identity to specific comments in the review.

What does the trial involve?

We will include bioRxiv preprints in cell biology which authors have requested feedback on. ASAPbio will approach preprint authors to request confirmation of their interest in receiving feedback and consent for inclusion in the trial. ASAPbio will circulate a preprint or two to the Crowd at weekly intervals, and request comments within the following seven days. Crowd members will provide comments via a private Hypothes.is group. After the commenting period, a collective synthesis of the comments will be posted as a public review via bioRxiv’s TRiP framework.

A summary of the trial workflow is outlined below.

How long does the trial last?

The trial will run for three months, from 26 August to the end of November 2021. Those interested in participating can sign up at any stage prior to the trial start or during the trial period.

If I join the trial, am I expected to provide comments every week?

No, Crowd members do not have to contribute comments every week. They can choose whether to comment on the paper circulated in a specific week according to their expertise and their availability on the particular week. We hope that 5-8 crowd members will provide comments each week. 

How do the Crowd reviewers provide comments?

The Crowd reviewers will be part of a private Hypothes.is group and will comment on preprints via Hypothes.is annotations. Crowd members can comment on the full paper or on specific parts of it.

Will the comments be anonymous?

The Crowd reviewers are part of a private Hypothes.is group. Their comments will be visible by any member of the Hypothes.is group but not to the public. We should however note that the url for the Hypothes.is group is not password protected and thus if any members shared it with a colleague, they would be able to join the group and see the comments. With this in mind, we recommend that reviewers write comments in a format where they’d be comfortable being seen by others.

For the synthesized review that will be publicly posted, we will aggregate the comments and list contributors as a list, without linking specific comments to the commenter.

How will the synthesized review be created?

A Crowd member (or an ASAPbio team member where needed) will collect the comments on the preprint at the end of the commenting period for a paper, and generate a synthesis. This will consolidate comments related to different sections of the paper for ease of reading. There will not be any edits done for content, except if any concerns arise about the tone or nature of the comment, in which case, ASAPbio can decide to exclude specific comments from the synthesis.

The synthesized review will be posted publicly on bioRxiv via the TRiP framework. Those who contributed comments will be acknowledged in the synthesized review, without linking specific comments to the commenter.

Crowd review trial participants