In preparation for the Recognizing Preprint Peer Review workshop, ASAPbio integrated input from two working groups to prepare a survey for researchers, funders, and journal editors and publishing organization employees. The survey sought to gather views and experience with preprint feedback and review from a broad range of stakeholders, to help inform the conversations at the workshop.
The survey garnered 230 responses, and we share here summaries of the two largest categories of respondents: 161 responses from researchers and 51 responses from journal editors and publishing organization employees. You can view the results on Google Sheets and on Zenodo.
Researchers would be incentivized to participate in preprint review by funder & journal recognition
Most researcher respondents were group leaders, and ⅔ described their primary discipline as biology. Our sample displays a strong geographic bias toward North America and Europe, with only 20% of respondents coming from outside of these continents.
Most respondents had received no feedback on preprints, which, for the purpose of this survey, we defined as any public commentary on preprints. Of those who had received some feedback, only a small fraction indicated that the feedback came in the form of detailed reviews.
With few researchers having received feedback, perhaps it’s unsurprising that a significant number of them expressed concerns with the prospect: the most significant concerns related to hesitancy about the quality or fairness of the feedback and about the commenter’s motivations for providing it.
However, more than half of respondents said they’d be likely or very likely to request feedback on their preprints if journals incorporated preprint reviews into editorial decisions or treated them like reviews transferred from another journal. Other potential incentives, such as funders recognizing preprint peer reviews in various ways, were not far behind.
Similar to their (sparse) experience receiving public feedback, relatively few researchers had posted comments or reviews on preprints themselves. The most common reason given for not engaging was “lack of time,” which we speculate means that researchers see providing preprint feedback as an activity that may yield small payoffs in terms of recognition or utility to others, compared to other research contributions. As above, many researchers indicated that formal recognition for preprint review by both journals and funders would encourage them to participate.
Journal editors and publishing organization employees cite trust and discovery as key challenges; favor reviews created by organized platforms
Among journal editors and publishing organization employees who took our survey, many identified as in-house or academic editors. 60% were from biology journals with the majority based in Europe.
Nearly half of this group reported that their journal had previously incorporated preprint reviews produced by a platform such as Peer Community In or Review Commons. Given that the number of journals offering these integrations is small, our sample represents community members at journals who are relatively progressive with regards to incorporating preprint feedback. Indeed, many respondents indicated that they were motivated to engage with preprint feedback for a variety of reasons, most commonly to check for potential concerns with the paper that should be considered, reuse feedback to streamline their processes, and identify new reviewers.
Regarding challenges to engage with preprint feedback, more than half of respondents reported being discouraged by the small number of preprints with feedback, the difficulty of finding that feedback across different platforms, and doubts about feedback integrity or commenter motivations.
Given the challenges with discovery and trust, it’s unsurprising that many more respondents said their journal would be willing to reuse preprint reviews that originated from a platform that organizes and oversees the peer review process versus reviews posted independently by individual reviewers. The majority of respondents said that this coordination by a platform was important or very important for enabling their journal to reuse preprint reviews. However, even more indicated that a mechanism to verify the reviewer’s identity was a key feature of the review.
In the coming weeks, we’ll share additional outputs of the Recognizing Preprint Peer Review workshop that delve deeper into strategies for different stakeholders to support preprint review and tackle these key emerging challenges of recognition and trust.
