Recommendations on Recognizing Preprint Review from the ASAPbio Funder, Researcher, and Institution Working Group

In advance of the Recognizing Preprint Peer Review meeting in December 2022, ASAPbio convened two Working Groups of different stakeholders to articulate the added value of incorporating preprint review into assessment processes, catalog potential uses of preprint review as part of assessment processes, develop working definitions for preprint review according to the needs of different stakeholders and discuss how policies and processes may be updated to incorporate preprint review. This report outlines the recommendations developed by the Working Group involving researchers and representatives of funders and institutions.

Download PDF and cite as: Avissar-Whiting, Michele, Belliard, Frederique, Dumanis, Sonya, Eldon Whylly, Kristin, Farley, Ashley, Franko, Maryrose, Hazlett, Haley, Neves, Kleber, Puebla, Iratxe, Rooryck, Johan, Royle, Stephen, Spiro, John, Stern, Bodo, Strinzel, Michaela, & Polka, Jessica. (2023). Recommendations on Recognizing Preprint Review from the ASAPbio Funder, Researcher, and Institution Working Group. Zenodo. https://doi.org/10.5281/zenodo.7584319

The broader use of preprints has led to a wide range of activities involving reactions, comments and review of preprints. This means that in the current ecosystem, peer review also happens outside of journals and can take place on research outputs shared as preprints, facilitating a more open and diverse discourse on research works. As this type of activity flourishes, it is important for funders and institutions to consider those forms of review as valid scholarship within their research assessment processes.

Why incorporate preprint reviews into funder and institutional research assessment processes?

The traditional journal publication framework imposes limits on the types of research outputs considered valid for evaluation and funding decisions by funders and institutions, and it has created known inequalities. The digital publishing ecosystem now has the tools to not only allow researchers to share a wide range of research outputs, but also to decouple publication from review. It is thus relevant to update research assessment frameworks to include a broader view of a researcher’s contributions. Preprint review provides the following benefits to funders and institutions:

Opportunities for faster scientific scrutiny and corrections – science can be assessed and corrected faster, increasing failure efficiency and avoiding redundancy or work on research avenues that are not productive
Catalyzes culture change, normalizing open participation in scientific dialogue
Signals the value of reviews, both as scholarly contributions in their own right and as further context that adds to the understanding of the paper by different audiences
Makes better use of researcher time and intellectual energy, as reviews are not used only as a curation mechanism but are preserved as part of the scientific dialogue
Provides an opportunity to address bias in the evaluation and non-collegial feedback
Increases transparency in review and opens up the scholarly communication process

Working definition for preprint review

While there can be many types of feedback on a preprint (ranging from brief comments on social media to full in-depth reviews), for the purpose of these recommendations, the Working Group used the following definition:

A preprint review is an assessment of a preprint that includes:

Rigor and validity of the research discussed
Consideration of the reviewer’s competing interests, either through a check by an editor/coordinator, if available, or via a public declaration of competing interests by the reviewer
Mechanism to confirm the reviewer identity, either via verification by an editor/coordinator, if available, or via a public declaration by the reviewer (i.e. the reviewer signs the review) along with ORCID verification

A preprint with reviews is a preprint that has two or more reviews as defined above. Table 1 lists examples of practices by services and/or platforms that provide these reviews, including several that provide reviews fulfilling all of the criteria outlined.

Table 1: Practices of example preprint review services and/or platforms

	F1000 Research example	eLife example	Review Commons example	PCI example	PREreview example	preLights example	Preprint server comment example
Rigor/validity addressed	✅	✅	✅	✅	✅	possibly	possibly
Reviewer competing interests – checked by editor or publicly declared	✅ declared	✅ checked	✅ checked	✅ checked	✅ declared	Not included by default	Not included by default
Reviewer identity – verified or signed (w/ORCID notification)	✅ verified & ORCID	✅ verified & ORCID	✅verified	✅ verified	Can be signed, but no ORCID notification	✅ verified	Can be signed, but no ORCID notification

Incorporating preprint feedback and reviews into research assessment processes

Many funders have policies that recognize preprints as valid research outputs for the purposes of grant applications and reports (asapbio.org/funder-policies), and many institutions also encourage preprints as part of hiring and promotion (asapbio.org/university-policies). While the use of preprint reviews is less common, there are some examples of funders and institutions that already endorse the use of preprints with reviews and preprint reviews in research assessment. As noted in Table 2, some organizations have already signaled that preprints with reviews fulfill their requirements for peer-reviewed publications.

Table 2: Factors in Preprint Review Recognition

	Passive endorsement	Active endorsement	Requirement	Future considerations
Recognition of preprints in research assessment	Preprints may be cited as sign of productivity Example: NIH, ERC, etc	All cited articles must be publicly available (i.e., articles that are submitted, in revision or in press can only be included if they are preprinted) Example: HHMI, Arnold Ventures	All research articles must be preprintedExample: ASAP, CZI	Review of open sciences practices (open data, etc) in parallel
Recognition of refereed preprints in research assessment	Refereed preprints may be listed in addition to journal articles Example: cOAlition S statement (~30 funders), EMBO	Refereed preprints considered superior to journal articlesCitation instructions ask for DOIs of preprint, final version and preprint peer reviews, if they existScientists are rewarded if their citations include preprints and preprint reviews	Contributions must be publicly available and scholarly in nature in order to be classified as a ‘peer-reviewed article’ (e.g., preprints and preprint reviews)Standard journal articles (without open peer review) are in a separate category of ‘endorsed articles’.
Recognition of preprint reviews in research assessment	Preprint reviews may be listed as scholarly contributions in addition to traditional reviews	Organization rewards scientists who adopt preprint review as their default peer review activityExample: Fraser & Malnoë lab policy	Organization requires preprint review history to select outside reviewers for funding and hiring panels
Recognition of preprint and preprint review outside of research assessment	Organization covers preprints and refereed preprints in news releases	Tools/forms explicitly built to encourage preprint outputsOrganization pays for preprint review of its articles	Organization limits news releases to preprints and refereed preprints (when the science is news)Organization contracts with service providers to conduct peer review on all its preprints. Example: Arcadia Science

Funders and institutions seeking to incorporate preprints with reviews or preprint reviews into their processes can consider the following areas and policies for implementation:

Requirements for PhD graduation or postdoctoral fellowships: Include refereed preprints as fulfilling graduation or fellowship eligibility requirements where a peer-reviewed publication is required. As an example, EMBO’s Postdoctoral Fellowships require that candidates have a first-author peer-reviewed article (which may be a refereed preprint), and graduate programs in France accept preprints recommended by Peer Community In as fulfilling graduation criteria.
Grant applications and reports, hiring, promotion: Indicate as part of guidelines for grant applications and reports, and in job applications, that preprints with reviews may be included, e.g. cOAlition S organizations have indicated that they consider preprints with reviews as equivalent to peer-reviewed articles published in a journal, and 9 funders and research organizations supporting eLife’s editorial model have affirmed they will consider reviewed preprints in their evaluation processes. As part of the bibliography of candidates’ work, include sections where applicants can highlight open peer reviews and endorsements of their preprints, as well their own output of public reviews.
Recruitment of grant reviewers: When recruiting reviewers for grant applications, assess the researcher’s output of public reviews (via their ORCiD, CV, etc) to identify suitable reviewers.
Reporting, monitoring/evaluating program impact: Include preprints with reviews and public review output when evaluating a department/unit/program’s research output.

Challenges and limitations

While the Working Group believes there are many benefits to preprint review, we also acknowledged potential considerations and challenges around the implementation of preprint review in research assessment:

Place value on open scholarly discourse, rather than using preprint review as a proxy for quality: Recognition for preprint review must be based on assigning value to openness when engaging on scholarly discourse around research works. We should nurture a culture of openness and of engagement with public discourse, while recognizing that the assessment within reviews does not necessarily determine the quality of the preprint.
Peer review is not incentivized: Many academics are already overwhelmed with requests to review from journals, which itself carries few concrete benefits. This makes it challenging to devote additional hours to new forms of reviewing.
Change is slow: Scholarly publishing, and academic culture in general, can be slow to change. Policies and initiatives must work to evolve entrenched practices.
Undesirable hierarchies: Recognition for preprint review should not create an artificial hierarchy in which preprints with reviews are valued above those without; it will be important to preserve the value of a standalone preprint as a research output, rather than ratcheting expectations that they should all be reviewed.
Recognition for preprint review must build on recognition for preprints: Preprint review is layered on preprints and thus requires a robust pool of preprints to evaluate. It is thus necessary to support further recognition of preprints themselves as part of research assessment.
Scaling: While there are different preprint-review initiatives, they are currently operating at a relatively small scale. To increase the visibility and value of preprint reviews for researchers, it is important to scale up preprint review platforms and/or services, so that they can meet demand as this grows.
Infrastructure: Preprint reviews currently exist across a wide range of platforms, and at the same time readers are unlikely to seek out reviews independently of their respective preprints. It will be important to continue to develop infrastructure that facilitates linking and connections across the distributed network of preprint reviews and improves their visibility in association with individual preprints and related scholarly outputs.
Most reviews are still closed: The majority of peer review still takes place at journals within a closed/confidential system available only to a select few, thus keeping reviews inaccessible to the broader community. It will be important to support transparency in review not only on preprints but also for journal articles.