Last year we ran a trial where we experimented with translating the crowd review model pioneered by the journal Synlett to preprints, and we coordinated activities to develop public reviews on cell biology preprints. The activities resulted in sign up by over 100 researchers, and in 14 public preprint reviews developed through comments contributed by 31 crowd members. Encouraged by this experience, we decided to run a new cycle of activities in 2022.
We reflected on the lessons from last year’s trial and so we incorporated several updates into our activities for 2022:
- We expanded the activities to have three groups -’crowds’- instead of one
- We incorporated a group that would review preprints in Portuguese. We wanted to allow broader diversity and participation in preprint review activities and we were delighted to start a collaboration with SciELO Preprints to include preprints in Portuguese posted in that server.
- We wanted to facilitate commenting and to streamline some of the steps, so we updated the workflow in several ways: the preprint selection would be handled by the crowd and we would not seek author consent for inclusion, commenting would take place via Google Docs, we extended the commenting period to 14 days, and we engaged community members, ASAPbio Fellows in particular, in synthesizing the reviews.
The workflow we used this year is summarized in the diagram below.
Our 2022 program ran from 1 June until the middle of September, and we are pleased to share some of the highlights from our activities this year.
Program outcomes
We had a great response to our call for participation with almost 200 researchers signing up for participation across the three groups. Over 50 crowd members provided comments on at least one preprint, with many crowd members commenting on several preprints. The contributions have led to 40 public reviews on preprints: 27 posted on bioRxiv preprints (a list is available via Sciety here) and 13 on SciELO Preprints (available via this Sciety group).
We surveyed the crowd reviewers to learn about their experience in the program and we received 59 responses across the three groups. 47 of the respondents rated the experience in the trial as positive or very positive, and a majority (54) indicated they would be interested in participating in this type of activity in the future.
The reviewers indicated that they particularly appreciated the possibility of commenting on parts of the preprint rather than having to complete a full review of the full paper. The reviewers were also positive about the commenting format that allowed them to see other commenter’s feedback and even interact with others and discuss specific points where the groups members had different opinions or felt further clarity was needed.
We also asked the reviewers for their thoughts on what may have added to their experience. The participants noted time constraints as one of the main barriers regarding contributions to the review, and also suggested the possibility of hosting calls to allow the participants in specific groups to discuss the preprints and their feedback on the paper.
We also surveyed the authors of the preprints for their views on the feedback they had received. The sample of responses is small (n=5) and thus limits generalization, but all the authors who completed the survey indicated that the feedback for their preprint had been useful or very useful, and 4 out of 5 indicated they would incorporate changes to the manuscript in response to the comments received.
Some additional highlights
Like last year, most of the participants in the trial were early-career researchers. Thanks to the expansion in groups, we had a broader geographical representation across the participants. It has been positive to see the continued interest in the community to develop reviewing skills and contribute to public feedback on preprints. We thank all the reviewers for their many contributions and their feedback on their experience as participants.
Last year, only a third of the preprint authors we approached agreed to have their paper reviewed in the trial. In 2022 we decided to select the preprints ourselves without requesting author consent first, and thus, we wondered whether we may receive any concerns from the authors of included preprints. We notified authors of our intention to review the papers upon selection, and also shared our feedback with them when the synthesis of the review was ready. We did not receive any concerns from authors and those who were in touch were positive about the opportunity of receiving feedback on their work.
This year we had several 2022 Fellows and community members involved in coordinating activities in the program. This has been invaluable at all stages of the project, and we thank the ASAPbio Fellows João Victor Cabral Costa, Ruchika Bajaj, Richa Arya, Michael Robichaux, Vasanthanarayan Murugesan, Ehssan Meglad and Bianca Trovò, as well as ASAPbio Fellow Vanessa Bortoluzzi, the editor of Memorias do Instituto Oswaldo Cruz Adeilton Brandão, and Alex Mendonça from SciELO Preprints, for their support synthesizing the feedback on different preprints and their generous contributions to the preparations and running of activities.
Where next?
Momentum is growing around the value of public preprint dialogue on preprints and we are happy to have contributed public reviews on preprints. We are encouraged to see the engagement with public preprint feedback across the different groups focused on different disciplines and languages. We are exploring ways of facilitating further public reviews of preprints in 2023 and very much welcome collaborations with groups interested in exploring this review modality – if you would like to learn more or are interested in trying this out, do get in touch!
In the meantime, if you want to keep up-to-date with developments and conversations around preprint review, do sign up to attend the livestream sessions from our upcoming meeting on ‘Recognizing preprint review’.
