Fostering public preprint feedback through crowd preprint review

While there is increasing interest in preprint review activities, the level of public commenting and reviewing on preprints remains low overall. To explore review modalities that may foster participation in preprint review, last year we completed a trial to facilitate public reviews on preprints inspired in the crowd review model pioneered by the journal Synlett. We coordinated a group of researchers who commented on cell biology preprints resulting in 14 public reviews – you can read more about last year’s trial here.

We saw a high level of reviewer engagement and in line with our goal to support preprint review, we will run further crowd preprint activities in 2022. This year we want to include additional disciplines and groups to see if this review modality can expand across different communities. In addition, we are pleased to collaborate with SciELO Preprints to also include the review of preprints in Portuguese.

The activities will involve the review of preprints posted in bioRxiv or SciELO Preprints, in the disciplines and languages outlined below:

  • Cell biology preprints from bioRxiv (English)
  • Biochemistry preprints from bioRxiv (English)
  • Infectious diseases preprints from SciELO Preprints (Portuguese)

Sign up as a crowd preprint reviewer!

We invite cell biologists, biochemists and researchers in infectious diseases to sign up as crowd preprint reviewers. Participating as a crowd preprint reviewer is a great opportunity to develop your reviewing skills and to support trust in preprints.

Sign up as crowd preprint reviewer

ASAPbio will coordinate three ‘crowds’ of preprint reviewers – each crowd will include a group of researchers with expertise in the designated discipline who agree to provide feedback on preprints. We will circulate a preprint weekly to each crowd, and will invite members to provide comments via a collaborative Google Doc; Crowd members can comment on the full paper or parts of the study according to their interest. After 14 days, we will close the commenting period and generate a collective synthesis of the comments to be posted as a public review.

Crowd preprint review FAQ

Why does ASAPbio run crowd preprint review?

Interest in preprint review activities is growing and there are different platforms available for posting comments and reviews (e.g. comments on the preprint server, PREreview, preLights, PubPeer etc). However, the level of public commenting and reviewing on preprints remains low (an analysis of bioRxiv preprints reported that less than 10% of preprints received comments), which suggests there are barriers to participating in preprint review, possibly related to concerns over whether the commenter feels qualified to review the full paper, and potential vulnerability for early career researchers if they comment openly on a preprint by a more senior researcher.

Based on our trial last year, we feel that the crowd preprint review approach provides a way to engage in public preprint feedback while mitigating those concerns, as it provides a format that gives flexibility on what comments to provide and generates public reviews that will not link the commenter’s identity to specific comments in the review.

What does this type of review involve?

We will include bioRxiv preprints in cell biology and biochemistry, and preprints in Portuguese posted in SciELO Preprints in the field of infectious diseases.

ASAPbio will circulate a preprint to each Crowd at weekly intervals, and request comments within the following 14 days. Crowd members will provide comments via a collaborative Google Doc. After the commenting period, a collective synthesis of the comments will be posted as a public review on the preprint – as a comment bioRxiv’s and via Hypothes.is on SciELO Preprints. Those public reviews will also be listed on Sciety.

A summary of the workflow is outlined below.

How long will the activities last?

The activities will run for three months, from May to August 2022. Those interested in participating can sign up at any stage prior or during that period.

If I join as a crowd reviewer, am I expected to provide comments every week?

No, Crowd members do not have to contribute comments every week. They can choose whether to comment on the paper circulated in a specific week according to their expertise and their availability on the particular week. We hope that 5-8 crowd members will provide comments for each preprint. 

How do the Crowd reviewers provide comments?

The Crowd reviewers will comment on preprints via a collaborative Google Doc. Crowd members can comment on the full paper or on specific parts of it.

Will the comments be anonymous?

The Crowd reviewers are part of a Google group. Their comments will be visible by any member of that group but not to the public. For the synthesized review that will be publicly posted, we will aggregate the comments and list contributors as a list, without linking specific comments to the commenter.

How will the synthesized review be created?

A Crowd member (or an ASAPbio team member where needed) will collect the comments on the preprint at the end of the commenting period for a paper, and generate a synthesis. This will consolidate comments related to different sections of the paper for ease of reading. There will not be any edits done for content, except if any concerns arise about the tone or nature of the comment, in which case, ASAPbio can decide to exclude specific comments from the synthesis.

The synthesized review will be posted publicly on the preprint – as a comment bioRxiv’s and via Hypothes.is on SciELO Preprints. Those who contributed comments will be acknowledged in the synthesized review, without linking specific comments to the commenter.