This blog post was co-written by Jonny Coates and 2024 ASAPbio Fellows; Josie, Jade & Lamis
It’s peer review week and to celebrate, we are releasing a preview of one of the 2024 ASAPbio Fellows projects focused on investigating the ASAPbio Crowd Preprint Review activities.
ASAPbio Crowd Preprint Review
The theme of peer review week this year is “Innovation and Technology in Peer Review”. Since 2021, ASAPbio has been running its own version of crowd preprint review. This approach involves using a crowd of reviewers to collaboratively review a manuscript and was developed by the journal Synlett in 2017. The use of a crowd is the innovative aspect to this approach of performing peer review as it enables real-time discussion amongst reviewers and allows individual reviewers to only comment on aspects that fall within their expertise. This means that a large number of small comments can be combined into a traditional-looking peer review. Additionally, ASAPbio posts all reviews publicly which are subsequently linked back to the preprint, providing transparency and greater context for readers.
The ASAPbio approach involves crowd leads choosing preprints and disseminating them to the crowd members for review. Once crowd members add their comments, crowd leads synthesise the comments and discussions into a traditional-looking peer review. They then upload the synthesised reviews so that the reviews are publicly available (Fig 1).
ASAPbio began with a pilot in 2021 focused on cell biology preprints. The following year this expanded to three disciplines, including one in collaboration with SciELO that covered Portuguese-language preprints. In 2023, there were 4 crowds covering preprints across bioRxiv. Four crowds were again launched in 2024 and will run for an entire 12 month period, in contrast to the more tightly defined periods of previous years. Between 2021 and September 2024, this initiative has produced 97 public reviews on preprints (Table 1).
| Year | Number of fields | Number of participants who signed up to receive review info | Total number preprints reviewed |
| 2021 | 1 | 113 | 14 |
| 2022 | 3 | 160 | 40 |
| 2023 | 4 | 205 | 36 |
| 2024* | 4 | 140 | 7 |
Experience of crowd members
A group of Fellows has been conducting a survey of current and past crowd members to reflect on the experience of being a crowd member. As a preview of our work, we present some data on the survey respondents and the reasons that they joined the program.
The majority of respondents indicated that they had a background in biology or meta-research with remaining disciplines being collated into “other”. Given the historical roots of ASAPbio as a life science organisation, it is unsurprising that biology is so well represented. There has also been a focus on meta-research as a crowd which explains the number of respondents who identified this as being their scientific area of interest (Fig 2A). In terms of career stages, most respondents were post-doctoral researchers (post-docs) or Principle Investigators (PIs) (Fig 2B). There were also 7 students who responded to the survey. Of those who stated that they had never participated in a crowd review, the majority were students or PIs whereas those most actively engaged were the post-docs. Those who identified as “other” job titles were also actively participating.
Why do people participate (or not) in crowd review?
Respondents indicated varied reasons for why they participate in crowd review activities (Fig 2C). Engaging in Open Science practices was most cited, followed by developing peer review skills, and contributing to the science community. Less frequently, respondents mentioned participating in crowd reviews for personal gains, such as increasing their visibility, showing evidence of their productivity, and keeping up-to-date with the latest research in their field. The top reasons for participating collectively represent 3 levels of impact: 1) Society, 2) Self, and 3) The scientific community. This strong emphasis on contributing to open science and self-directed training highlights the two key benefits of crowd preprint review. Indeed, the collaborative nature of the review process was highlighted in several free-text responses, with one responder communicating how it enables “sharing reviewers concerns, coming to an agreement, [and] sharing the workload.”
The two biggest reasons that respondents stated for not reviewing a given preprint were a lack of relevant expertise and a lack of time (Fig 2D). Given the size of the crowds, it is not unexpected that some will not have the specific expertise for any component of a given preprint. However, the benefit of a crowd is that this “expertise” burden is reduced as reviewers do not need to comment on the entire preprint. It is possible that this aspect could be better communicated to encourage people to review more granularly and to accept that as a valid contribution. Academics are under heavy workloads and are time-poor. Additionally, we have noticed in previous iterations of the crowd review program that in the busier months in the academic calendar, there is an anecdotal reduction in reviewing activities. A lack of interest in the topic was the third most common reason for not reviewing a given preprint. Thus far, crowds have been designed to cover entire fields (such as “cell biology” or “immunology”), so it cannot be expected that every chosen preprint will be of interest to the entire crowd.
This preview of our on-going efforts highlights altruism and a desire to participate in better practices as the primary motivator for participating in the crowd review program.
Conclusion
We’re still performing a full data analysis and expect to post a full preprint before the end of the year! But if you would like to join the crowd preprint review efforts for 2024-2025 you can sign up by clicking this link. If you have any questions about crowd review then please contact Jonny Coates (jonny.coates@asapbio.org)
