In advance of the Recognizing Preprint Peer Review meeting in December 2022, ASAPbio convened two Working Groups of different stakeholders to articulate the added value of incorporating preprint review into assessment processes, catalog potential uses of preprint review as part of assessment processes, develop working definitions for preprint review according to the needs of different stakeholders and discuss how policies and processes may be updated to incorporate preprint review. This report outlines the recommendations developed by the Working Group involving researchers and representatives of funders and institutions.
Download PDF and cite as: Avissar-Whiting, Michele, Belliard, Frederique, Dumanis, Sonya, Eldon Whylly, Kristin, Farley, Ashley, Franko, Maryrose, Hazlett, Haley, Neves, Kleber, Puebla, Iratxe, Rooryck, Johan, Royle, Stephen, Spiro, John, Stern, Bodo, Strinzel, Michaela, & Polka, Jessica. (2023). Recommendations on Recognizing Preprint Review from the ASAPbio Funder, Researcher, and Institution Working Group. Zenodo. https://doi.org/10.5281/zenodo.7584319
The broader use of preprints has led to a wide range of activities involving reactions, comments and review of preprints. This means that in the current ecosystem, peer review also happens outside of journals and can take place on research outputs shared as preprints, facilitating a more open and diverse discourse on research works. As this type of activity flourishes, it is important for funders and institutions to consider those forms of review as valid scholarship within their research assessment processes.
Why incorporate preprint reviews into funder and institutional research assessment processes?
The traditional journal publication framework imposes limits on the types of research outputs considered valid for evaluation and funding decisions by funders and institutions, and it has created known inequalities. The digital publishing ecosystem now has the tools to not only allow researchers to share a wide range of research outputs, but also to decouple publication from review. It is thus relevant to update research assessment frameworks to include a broader view of a researcher’s contributions. Preprint review provides the following benefits to funders and institutions:
- Opportunities for faster scientific scrutiny and corrections – science can be assessed and corrected faster, increasing failure efficiency and avoiding redundancy or work on research avenues that are not productive
- Catalyzes culture change, normalizing open participation in scientific dialogue
- Signals the value of reviews, both as scholarly contributions in their own right and as further context that adds to the understanding of the paper by different audiences
- Makes better use of researcher time and intellectual energy, as reviews are not used only as a curation mechanism but are preserved as part of the scientific dialogue
- Provides an opportunity to address bias in the evaluation and non-collegial feedback
- Increases transparency in review and opens up the scholarly communication process
Working definition for preprint review
While there can be many types of feedback on a preprint (ranging from brief comments on social media to full in-depth reviews), for the purpose of these recommendations, the Working Group used the following definition:
A preprint review is an assessment of a preprint that includes:
- Rigor and validity of the research discussed
- Consideration of the reviewer’s competing interests, either through a check by an editor/coordinator, if available, or via a public declaration of competing interests by the reviewer
- Mechanism to confirm the reviewer identity, either via verification by an editor/coordinator, if available, or via a public declaration by the reviewer (i.e. the reviewer signs the review) along with ORCID verification
A preprint with reviews is a preprint that has two or more reviews as defined above. Table 1 lists examples of practices by services and/or platforms that provide these reviews, including several that provide reviews fulfilling all of the criteria outlined.
Table 1: Practices of example preprint review services and/or platforms
|F1000 Research example||eLife example||Review Commons example||PCI example||PREreview|
|preLights example||Preprint server comment |
|Reviewer competing interests – checked by editor or publicly declared||✅|
|Not included by default||Not included by default|
|Reviewer identity – verified or signed (w/ORCID notification)||✅|
verified & ORCID
verified & ORCID
|Can be signed, but no ORCID notification||✅|
|Can be signed, but no ORCID notification|
Incorporating preprint feedback and reviews into research assessment processes
Many funders have policies that recognize preprints as valid research outputs for the purposes of grant applications and reports (asapbio.org/funder-policies), and many institutions also encourage preprints as part of hiring and promotion (asapbio.org/university-policies). While the use of preprint reviews is less common, there are some examples of funders and institutions that already endorse the use of preprints with reviews and preprint reviews in research assessment. As noted in Table 2, some organizations have already signaled that preprints with reviews fulfill their requirements for peer-reviewed publications.
Table 2: Factors in Preprint Review Recognition
|Passive endorsement||Active endorsement||Requirement||Future considerations|
|Recognition of preprints in research assessment||Preprints may be cited as sign of productivity|
Example: NIH, ERC, etc
|All cited articles must be publicly available (i.e., articles that are submitted, in revision or in press can only be included if they are preprinted)|
Example: HHMI, Arnold Ventures
|All research articles must be preprintedExample: ASAP, CZI||Review of open sciences practices (open data, etc) in parallel|
|Recognition of refereed preprints in research assessment||Refereed preprints may be listed in addition to journal articles|
Example: cOAlition S statement (~30 funders), EMBO
|Refereed preprints considered superior to journal articlesCitation instructions ask for DOIs of preprint, final version and preprint peer reviews, if they existScientists are rewarded if their citations include preprints and preprint reviews||Contributions must be publicly available and scholarly in nature in order to be classified as a ‘peer-reviewed article’ (e.g., preprints and preprint reviews)Standard journal articles (without open peer review) are in a separate category of ‘endorsed articles’.|
|Recognition of preprint reviews in research assessment||Preprint reviews may be listed as scholarly contributions in addition to traditional reviews||Organization rewards scientists who adopt preprint review as their default peer review activityExample: Fraser & Malnoë lab policy||Organization requires preprint review history to select outside reviewers for funding and hiring panels|
|Recognition of preprint and preprint review outside of research assessment||Organization covers preprints and refereed preprints in news releases||Tools/forms explicitly built to encourage preprint outputsOrganization pays for preprint review of its articles||Organization limits news releases to preprints and refereed preprints (when the science is news)Organization contracts with service providers to conduct peer review on all its preprints.|
Example: Arcadia Science
Funders and institutions seeking to incorporate preprints with reviews or preprint reviews into their processes can consider the following areas and policies for implementation:
- Requirements for PhD graduation or postdoctoral fellowships: Include refereed preprints as fulfilling graduation or fellowship eligibility requirements where a peer-reviewed publication is required. As an example, EMBO’s Postdoctoral Fellowships require that candidates have a first-author peer-reviewed article (which may be a refereed preprint), and graduate programs in France accept preprints recommended by Peer Community In as fulfilling graduation criteria.
- Grant applications and reports, hiring, promotion: Indicate as part of guidelines for grant applications and reports, and in job applications, that preprints with reviews may be included, e.g. cOAlition S organizations have indicated that they consider preprints with reviews as equivalent to peer-reviewed articles published in a journal, and 9 funders and research organizations supporting eLife’s editorial model have affirmed they will consider reviewed preprints in their evaluation processes. As part of the bibliography of candidates’ work, include sections where applicants can highlight open peer reviews and endorsements of their preprints, as well their own output of public reviews.
- Recruitment of grant reviewers: When recruiting reviewers for grant applications, assess the researcher’s output of public reviews (via their ORCiD, CV, etc) to identify suitable reviewers.
- Reporting, monitoring/evaluating program impact: Include preprints with reviews and public review output when evaluating a department/unit/program’s research output.
Challenges and limitations
While the Working Group believes there are many benefits to preprint review, we also acknowledged potential considerations and challenges around the implementation of preprint review in research assessment:
- Place value on open scholarly discourse, rather than using preprint review as a proxy for quality: Recognition for preprint review must be based on assigning value to openness when engaging on scholarly discourse around research works. We should nurture a culture of openness and of engagement with public discourse, while recognizing that the assessment within reviews does not necessarily determine the quality of the preprint.
- Peer review is not incentivized: Many academics are already overwhelmed with requests to review from journals, which itself carries few concrete benefits. This makes it challenging to devote additional hours to new forms of reviewing.
- Change is slow: Scholarly publishing, and academic culture in general, can be slow to change. Policies and initiatives must work to evolve entrenched practices.
- Undesirable hierarchies: Recognition for preprint review should not create an artificial hierarchy in which preprints with reviews are valued above those without; it will be important to preserve the value of a standalone preprint as a research output, rather than ratcheting expectations that they should all be reviewed.
- Recognition for preprint review must build on recognition for preprints: Preprint review is layered on preprints and thus requires a robust pool of preprints to evaluate. It is thus necessary to support further recognition of preprints themselves as part of research assessment.
- Scaling: While there are different preprint-review initiatives, they are currently operating at a relatively small scale. To increase the visibility and value of preprint reviews for researchers, it is important to scale up preprint review platforms and/or services, so that they can meet demand as this grows.
- Infrastructure: Preprint reviews currently exist across a wide range of platforms, and at the same time readers are unlikely to seek out reviews independently of their respective preprints. It will be important to continue to develop infrastructure that facilitates linking and connections across the distributed network of preprint reviews and improves their visibility in association with individual preprints and related scholarly outputs.
- Most reviews are still closed: The majority of peer review still takes place at journals within a closed/confidential system available only to a select few, thus keeping reviews inaccessible to the broader community. It will be important to support transparency in review not only on preprints but also for journal articles.
Working group participants
- Michele Avissar-Whiting, HHMI, 0000-0003-0030-3135
- Frederique Belliard, TU Delft, 0000-0001-8750-7207
- Sonya Dumanis, Aligning Science Across Parkinsons, 0000-0002-3345-9497
- Kristin Eldon Whylly, Templeton World Charity Foundation, 0000-0002-4677-148X
- Ashley Farley, Bill & Melinda Gates Foundation, 0000-0001-9310-6944
- Maryrose Franko, HRA, 0000-0003-1870-8472
- Haley Hazlett, DORA, 0000-0001-9937-435X
- Kleber Neves, Serrapilheira Institute, 0000-0001-9519-4909
- Iratxe Puebla, ASAPbio, 0000-0003-1258-0746
- Johan Rooryck, cOAlition S, 0000-0001-7214-7405
- Steve Royle, Warwick, 0000-0001-8927-6967
- John Spiro, Simons Foundation, 0000-0003-2820-5844
- Bodo Stern, HHMI, 0000-0002-4900-936X
- Michaela Strinzel, SNSF, 0000-0003-3181-0623
- Jessica Polka, ASAPbio, 0000-0001-6610-9293