By Victoria Yan
2022-01-27 update: This work has now been completed. Please see our blog post announcing PReF.
Why develop a preprint review taxonomy?
Dozens of projects organizing peer review of preprints are active or being developed. In this landscape full of new possibilities, differentiating among innovative forms of preprint review is challenging. Furthermore, these reviews are often hard to discover and not directly linked to the preprint. In order to tackle the challenges of preprint review terminology and enhancing preprint review discovery, the ASAPbio Taxonomy working group is working on creating a simple, reader-friendly system to classify preprint review processes.
The working group, made up of 14 members, represents a diverse group of preprint servers representatives, publishers, and technologists. The next steps of the working group are to finalize the taxonomy language intelligible to a larger audience and to develop a simple visual display for readers.
FeedbackASAP breakout session
We presented the draft preprint taxonomy at the ASAPbio #FeedbackASAP meeting on July 21st, 2021. The draft taxonomy consists of 8 terms, with their associated help text description and options (see table below). First, we stress-tested the draft taxonomy for intelligibility by asking participants to apply the classification categories to Review Commons (an ASAPbio-EMBO project that organizes journal-independent peer review). Through this exercise, we received suggestions that for the term “Reviewer selected by,” the process by which reviewers are selected should also be captured. For instance, reviewers maybe are first nominated by authors prior to editor selection. As a group, we noted that how “Competing interest” is handled at various preprint review projects is diverse and unclear. Participants noted that conflict of interest can be extremely important in a small number of cases. On the other hand, in most situations, colleagues or friends should not be precluded from providing feedback on preprints. However, it is important that it becomes a norm to declare any potential competing interest.
| Name of Field | Help text | Options |
| Review requested by | Who submits or initiates the feedback process? | Authors Non-authors |
| Reviewer selected by | Who selects the feedback providers? | Authors Editor, service, or community Self-nominated |
| Review coverage | Does the feedback cover the entire paper or only a certain section or aspect? | Full paper Specific aspects |
| Reviewer identity known to | Are the identities of reviewers known to everyone (public), editors or service, or noone? | Public Editor or service No one |
| Competing interests | Is a declaration of competing interest required? | Displayed Checked Not included |
| Public interaction | Is there an opportunity for the public to engage as an integral part of the process? | Included Not included |
| Opportunity for author response | Is the author’s response included as an integral part of the process? | Included Not included |
| Recommendation | Does the service provide a binary decision (accept/reject or recommendation) or a scalar rating after the review process? | Binary decision Other scale or rating None |
Where can the taxonomy be useful?
In our next exercise, the group was asked to consider which terms displayed in which locations would be useful from the perspective of readers, authors, reviewers, and publishers. We found that for readers, most terms in the taxonomy are helpful when displayed adjacent to individual preprints and their linked reviews. On the other hand, authors and reviewers might find it helpful to have “Author response”, “Public interaction”, “Review coverage”, and “Review requested by” displayed on descriptions of a given preprint review service. Specifically, “Reviewer identity known to” is important for the reviewers. Lastly, from the perspective of publishers, “Reviewer identity known to” was found to be most important on preprints and linked reviews.
Which aspects are important in preprint review?
Finally, we asked the group to rank the terms from most important to least important. Some of the terms such as “Review coverage” and “Author response,” were unanimously considered important. Interestingly, the terms “Recommendation” and “Review requested by” were more polarized and found on both ends of the scale. The group noted that peer review of preprints can be used for much more than providing an editorial decision for the publisher. For example, it can be an early form of feedback aimed at improving the work, or it can provide additional context for the preprint. The target audience of review in the case of preprints is shifting away from editors and towards authors and readers. Therefore, less emphasis is placed on an editorial decision as the primary endpoint of review.
Developing a simple and useful taxonomy
We had several additional suggestions for taxonomy, both new terms (for example, the public availability of the review and the process of reviewer selection) and additional levels of detail. As a group, we came to the conclusion that an essential aspect of the development of the preprint review taxonomy is maintaining focus on what problems can the taxonomy solve. For improving the clarity and discoverability of review processes, we will be mindful of the level of granularity that can be pragmatically implemented and develop a simple taxonomy that is well-suited for widespread adoption. If you have additional feedback on the taxonomy, we welcome your input in the comments below!
Photo by UX Indonesia on Unsplash
