Fostering public preprint feedback through crowd preprint review

While there is increasing interest in preprint review activities, the level of public commenting and reviewing on preprints remains low overall. To explore review modalities that may foster participation in preprint review, in 2021 ASAPbio started activities to facilitate public reviews on preprints inspired in the crowd review model pioneered by the journal Synlett. We coordinated a group of researchers who commented on cell biology preprints resulting in 14 public reviews – you can read more about last year’s trial here. We saw a high level of reviewer engagement, so we continued the activities in 2022, when we involved three groups focused on cell biology (bioRxiv preprints), biochemistry (bioRxiv preprints) and infectious diseases preprints in Portuguese from SciELO Preprints. Our activities resulted in 27 public reviews for bioRxiv preprints and 13 reviews for SciELO Preprints.

Motivated by the positive response, we will pursue further crowd preprint review activities in 2023, involving several groups lead by 2023 ASAPbio Fellows.

Sign up as a crowd preprint reviewer!

We invite cancer biologists, neuroscientists and meta-researchers to sign up as crowd preprint reviewers. Participating as a crowd preprint reviewer is a great opportunity to develop your reviewing skills and to support trust in preprints.

The ASAPbio Fellows will coordinate ‘crowds’ of preprint reviewers – each crowd will include a group of researchers with expertise in the designated discipline who agree to provide feedback on preprints. Each crowd will receive one new preprint to review every two weeks. Crowd members will be invited to provide comments via a collaborative Google Doc; Crowd members can comment on the full paper or parts of the study according to their interest. After 14 days, the commenting period for the paper will be closed and the collective comments will be synthesized into a single report, to be posted as a public review.

Crowd preprint review FAQ

Why does ASAPbio run crowd preprint review?

Interest in preprint review activities is growing and there are different platforms available for posting comments and reviews (e.g. comments on the preprint server, PREreview, preLights, PubPeer etc). However, the level of public commenting and reviewing on preprints remains low (an analysis of bioRxiv preprints reported that less than 10% of preprints received comments), which suggests there are barriers to participating in preprint review, possibly related to concerns over whether the commenter feels qualified to review the full paper, and potential vulnerability for early career researchers if they comment openly on a preprint by a more senior researcher.

Based on our activities over the last couple of years, we feel that the crowd preprint review approach provides a way to engage in public preprint feedback while mitigating those concerns, as it provides a format that gives flexibility on what comments to provide and generates public reviews that will not link the commenter’s identity to specific comments in the review.

What does this type of review involve?

There will be different crowds with specific scopes. Each crowd will circulate a new preprint for comments every two weeks, and crowd members will be asked to contribute comments over the following 14 days. Crowd members will provide comments via a collaborative Google Doc. After the commenting period, a collective synthesis of the comments will be posted as a public review on the preprint. Those public reviews will also be listed on Sciety.

A summary of the workflow is outlined below.

How long will the activities last?

The activities will start in June 2023 and will run until the end of the year. Those interested in participating can sign up at any stage prior or during that period.

If I join as a crowd reviewer, am I expected to provide comments every week?

No, Crowd members do not have to contribute comments every week. They can choose whether to comment on the paper circulated in a specific week according to their expertise and their availability on the particular week. We hope that 5-8 crowd members will provide comments for each preprint. 

How do the Crowd reviewers provide comments?

The Crowd reviewers will comment on preprints via a collaborative Google Doc. Crowd members can comment on the full paper or on specific parts of it.

Will the comments be anonymous?

The Crowd reviewers are part of a Google group. Their comments will be visible by any member of that group but not to the public. For the synthesized review that will be publicly posted, we will aggregate the comments and list contributors as a list, without linking specific comments to the commenter.

How will the synthesized review be created?

A Crowd member will collect the comments on the preprint at the end of the commenting period for a paper, and generate a synthesis. This will consolidate comments related to different sections of the paper for ease of reading. There will not be any edits done for content, except if any concerns arise about the tone or nature of the comment, in which case, ASAPbio can decide to exclude specific comments from the synthesis.

The synthesized review will be posted publicly on the preprint. Those who contributed comments will be acknowledged in the synthesized review, without linking specific comments to the commenter.