Dozens of projects organizing peer review of preprints are active or being developed. In this landscape full of new possibilities, differentiating among innovative forms of preprint review is challenging. Furthermore, these reviews are often hard to discover and not directly linked to the preprint. In order to tackle the challenges of preprint review terminology and enhancing preprint review discovery, the ASAPbio Taxonomy working group is working on creating a simple, reader-friendly system to classify preprint review processes.
The working group, made up of 14 members, represents a diverse group of preprint servers representatives, publishers, and technologists. The next steps of the working group are to finalize the taxonomy language intelligible to a larger audience and to develop a simple visual display for readers.
FeedbackASAP breakout session
We presented the draft preprint taxonomy at the ASAPbio #FeedbackASAP meeting on July 21st, 2021. The draft taxonomy consists of 8 terms, with their associated help text description and options (see table below). First, we stress-tested the draft taxonomy for intelligibility by asking participants to apply the classification categories to Review Commons (an ASAPbio-EMBO project that organizes journal-independent peer review). Through this exercise, we received suggestions that for the term “Reviewer selected by,” the process by which reviewers are selected should also be captured. For instance, reviewers maybe are first nominated by authors prior to editor selection. As a group, we noted that how “Competing interest” is handled at various preprint review projects is diverse and unclear. Participants noted that conflict of interest can be extremely important in a small number of cases. On the other hand, in most situations, colleagues or friends should not be precluded from providing feedback on preprints. However, it is important that it becomes a norm to declare any potential competing interest.
Name of Field
Help text
Options
Review requested by
Who submits or initiates the feedback process?
Authors Non-authors
Reviewer selected by
Who selects the feedback providers?
Authors Editor, service, or community Self-nominated
Review coverage
Does the feedback cover the entire paper or only a certain section or aspect?
Full paper Specific aspects
Reviewer identity known to
Are the identities of reviewers known to everyone (public), editors or service, or noone?
Public Editor or service No one
Competing interests
Is a declaration of competing interest required?
Displayed Checked Not included
Public interaction
Is there an opportunity for the public to engage as an integral part of the process?
Included Not included
Opportunity for author response
Is the author’s response included as an integral part of the process?
Included Not included
Recommendation
Does the service provide a binary decision (accept/reject or recommendation) or a scalar rating after the review process?
Binary decision Other scale or rating None
The draft version of the preprint feedback taxonomy as of the breakout session on July 21, 2021
Where can the taxonomy be useful?
In our next exercise, the group was asked to consider which terms displayed in which locations would be useful from the perspective of readers, authors, reviewers, and publishers. We found that for readers, most terms in the taxonomy are helpful when displayed adjacent to individual preprints and their linked reviews. On the other hand, authors and reviewers might find it helpful to have “Author response”, “Public interaction”, “Review coverage”, and “Review requested by” displayed on descriptions of a given preprint review service. Specifically, “Reviewer identity known to” is important for the reviewers. Lastly, from the perspective of publishers, “Reviewer identity known to” was found to be most important on preprints and linked reviews.
Which aspects are important in preprint review?
Finally, we asked the group to rank the terms from most important to least important. Some of the terms such as “Review coverage” and “Author response,” were unanimously considered important. Interestingly, the terms “Recommendation” and “Review requested by” were more polarized and found on both ends of the scale. The group noted that peer review of preprints can be used for much more than providing an editorial decision for the publisher. For example, it can be an early form of feedback aimed at improving the work, or it can provide additional context for the preprint. The target audience of review in the case of preprints is shifting away from editors and towards authors and readers. Therefore, less emphasis is placed on an editorial decision as the primary endpoint of review.
Developing a simple and useful taxonomy
We had several additional suggestions for taxonomy, both new terms (for example, the public availability of the review and the process of reviewer selection) and additional levels of detail. As a group, we came to the conclusion that an essential aspect of the development of the preprint review taxonomy is maintaining focus on what problems can the taxonomy solve. For improving the clarity and discoverability of review processes, we will be mindful of the level of granularity that can be pragmatically implemented and develop a simple taxonomy that is well-suited for widespread adoption. If you have additional feedback on the taxonomy, we welcome your input in the comments below!
By Victoria Yan
2022-01-27 update: This work has now been completed. Please see our blog post announcing PReF.
Why develop a preprint review taxonomy?
Dozens of projects organizing peer review of preprints are active or being developed. In this landscape full of new possibilities, differentiating among innovative forms of preprint review is challenging. Furthermore, these reviews are often hard to discover and not directly linked to the preprint. In order to tackle the challenges of preprint review terminology and enhancing preprint review discovery, the ASAPbio Taxonomy working group is working on creating a simple, reader-friendly system to classify preprint review processes.
The working group, made up of 14 members, represents a diverse group of preprint servers representatives, publishers, and technologists. The next steps of the working group are to finalize the taxonomy language intelligible to a larger audience and to develop a simple visual display for readers.
FeedbackASAP breakout session
We presented the draft preprint taxonomy at the ASAPbio #FeedbackASAP meeting on July 21st, 2021. The draft taxonomy consists of 8 terms, with their associated help text description and options (see table below). First, we stress-tested the draft taxonomy for intelligibility by asking participants to apply the classification categories to Review Commons (an ASAPbio-EMBO project that organizes journal-independent peer review). Through this exercise, we received suggestions that for the term “Reviewer selected by,” the process by which reviewers are selected should also be captured. For instance, reviewers maybe are first nominated by authors prior to editor selection. As a group, we noted that how “Competing interest” is handled at various preprint review projects is diverse and unclear. Participants noted that conflict of interest can be extremely important in a small number of cases. On the other hand, in most situations, colleagues or friends should not be precluded from providing feedback on preprints. However, it is important that it becomes a norm to declare any potential competing interest.
Non-authors
Editor, service, or community
Self-nominated
Specific aspects
Editor or service
No one
Checked
Not included
Not included
Not included
Other scale or rating
None
Where can the taxonomy be useful?
In our next exercise, the group was asked to consider which terms displayed in which locations would be useful from the perspective of readers, authors, reviewers, and publishers. We found that for readers, most terms in the taxonomy are helpful when displayed adjacent to individual preprints and their linked reviews. On the other hand, authors and reviewers might find it helpful to have “Author response”, “Public interaction”, “Review coverage”, and “Review requested by” displayed on descriptions of a given preprint review service. Specifically, “Reviewer identity known to” is important for the reviewers. Lastly, from the perspective of publishers, “Reviewer identity known to” was found to be most important on preprints and linked reviews.
Which aspects are important in preprint review?
Finally, we asked the group to rank the terms from most important to least important. Some of the terms such as “Review coverage” and “Author response,” were unanimously considered important. Interestingly, the terms “Recommendation” and “Review requested by” were more polarized and found on both ends of the scale. The group noted that peer review of preprints can be used for much more than providing an editorial decision for the publisher. For example, it can be an early form of feedback aimed at improving the work, or it can provide additional context for the preprint. The target audience of review in the case of preprints is shifting away from editors and towards authors and readers. Therefore, less emphasis is placed on an editorial decision as the primary endpoint of review.
Developing a simple and useful taxonomy
We had several additional suggestions for taxonomy, both new terms (for example, the public availability of the review and the process of reviewer selection) and additional levels of detail. As a group, we came to the conclusion that an essential aspect of the development of the preprint review taxonomy is maintaining focus on what problems can the taxonomy solve. For improving the clarity and discoverability of review processes, we will be mindful of the level of granularity that can be pragmatically implemented and develop a simple taxonomy that is well-suited for widespread adoption. If you have additional feedback on the taxonomy, we welcome your input in the comments below!
Photo by UX Indonesia on Unsplash