ASAPbio supports NIH CSR proposal to notify study sections about NIH preprint policy

Update, July 1, 2019: The NIH CSR has responded to this feedback: read the response at https://asapbio.org/nih-csr-response.

Through policy notice NOT-OD-17-050, the NIH “encourages investigators to use interim research products, such as preprints, to speed the dissemination and enhance the rigor of their work”, and allows preprints to be cited “anywhere other research products are cited” such as fellowship and grant applications (but excluding post-submission materials) — for more information, see funder policies.

Earlier this year, reports on social media suggested that preprints were not always being seen as valid evidence of productivity or rigor during grant review by NIH CSR study sections. We surveyed researchers to understand how commonplace this was and received several responses. In April, the NIH Center for Scientific Review (CSR) shared an open invitation to provide feedback about peer review at the NIH (Twitter, April 24 2019: https://twitter.com/NIH/status/1121119130164301831).

Today, we have submitted a statement of feedback to the NIH CSR in support of their proposal that NIH Scientific Review Officers (SROs) notify study sections about policy notice NOT-OD-17-050 during preparatory discussions before the study section begins to review submissions (Twitter, March 20 2019: https://twitter.com/CSRpeerreview/status/1108403793794879488). This statement includes a summary of the responses we received, including details of reported instances where study sections have not acted in line with this policy and/or may not have been sufficiently aware of it.

A public version of the submitted statement is provided below.

Response to Call for Feedback to NIH CSR, May 2019

ASAPbio (https://asapbio.org/) is a scientist-driven nonprofit working to promote innovation and transparency in the life sciences. Details of our staff, board members, and funding sources are available at https://asapbio.org/about-us.

We have prepared this statement in response to an open call for feedback (Twitter, April 24 2019: https://twitter.com/NIH/status/1121119130164301831) while the NIH prepares materials to inform study sections about peer review at NIH. Based on the reports shared on social media and those we have received directly about this issue, we support and endorse the proposed action that SROs notify study sections about the NIH policy notice NOT-OD-17-050 on ‘Reporting Preprints and Other Interim Research Products’ during preparatory discussions before the study section begins to review submissions, as suggested on March 20 (Twitter, March 20 2019: https://twitter.com/CSRpeerreview/status/1108403793794879488).

Please contact Naomi Penfold, Associate Director, ASAPbio on naomi.penfold@asapbio.org (primary) or Jessica Polka, Executive Director, ASAPbio on jessica.polka@asapbio.org (secondary) for further information. This feedback was submitted on May 30 2019; all reporters named in this feedback have provided consent.

Not all study sections are treating preprints in accordance with NOT-OD-17-050 and NOT-OD-18-228

In recent months, we have learned about several instances during NIH study section reviews where preprints may not have been counted as evidence of productivity or have been flagged as evidence that detracts from an applicant’s record due to the as-yet-unreviewed nature of the preprint.

Following concerns raised on social media in March 2019 (Twitter, March 10 2019: https://twitter.com/hollyingrahamSF/status/1104813237109850112), we surveyed a group of over 100 international researchers to better understand their experiences regarding citation of preprints in NIH applications and proposals. We asked researchers who had cited preprints in applications or who had been involved in study section review discussions of applications in which preprints were cited to share experiences since March 2017 (when the NIH policy came into effect). We have also received information from beyond this initial survey group.

So far, we have received 14 individual reports, with some respondents agreeing to be identified as detailed below. These reports reflect experiences of submitting and reviewing applications across a range of career stages —including and not limited to F32 postdoctoral fellowships, U19 project proposals and R01 grants — and to various study sections — including and not limited to Behavioral Neuroscience (with NIMH), Genetic Variation and Evolution (GVE), Macromolecular Structure and Function C (MSFC), Molecular Neurogenetics (MNG), Pathogenic Eukaryotes (PTHE), Synapses, Cytoskeleton and Trafficking (SYN) and a ZRG1 special emphasis panel.

Applicants citing preprints in their applications

We received multiple reports from researchers who had cited preprints in their application, proposal or grant submission to the NIH: while the majority of respondents reported receiving no explicit feedback with regard to their cited preprint(s), there were cases where preprinting had been perceived negatively. One report that preprints have “not counted” as evidence for student productivity in training grants is particularly concerning. In another case, it was explicitly stated in the study section feedback that the lack of peer-reviewed outputs was a weakness:

“In a review of one of our NIH grant submissions, the unreviewed nature of the preprints we cited was identified as a weakness. Specifically, under weaknesses, the reviewer stated, “Many of the innovative methods are not necessarily peer reviewed. ~20 of the references that describe key technology are only deposited as non-peer reviewed preprints on the preprint server for biology, operated by Cold Spring Harbor Laboratory (bioRxiv.org). Given the excessive amount of referenced, unpublished documents, it is slightly difficult to assess the feasibility of the various methods/technologies. Reference 37 (bioRxiv) that is a central innovation item regarding the role of the HUSH complex in the control of latent HIV-1 infection is not peer reviewed.” This reviewer feedback does not seem to be in line with NIH’s policy to support the inclusion of preprinted materials as evidence in grant submissions. The study section was ZRG1 AARR-K (56); 08/06/2018; in response to PAR-17-340: Collaborative Program Grant for Multidisciplinary Teams.” — Jeremy Luban, M.D., David J. Freelander Chair in AIDS Research and Professor, Program in Molecular Medicine and Biochemistry & Molecular Pharmacology, University of Massachusetts Medical School

No researcher reported receiving positive feedback about citing preprints in their application.

Study section reviewers assessing cited preprints

We also received several independent reports from researchers who have sat on study sections who recall preprints having been discussed both as a positive and a negative factor. Reports indicate this disparity is mainly due to differences in reviewer opinion as to whether preprints count as valid evidence of productivity or not (due to unreviewed status).

“During study section reviews in June 2018, we didn’t see many submissions that cited preprints but I do remember the discussion of one application that cited preliminary data and a couple of bioRxiv preprints. Another reviewer felt the applicant’s recent productivity was insufficient. I argued against this based on the preliminary data and preprints. In the end, all three primary reviewers scored that application well, but it was hard to tell if the members of the study section were convinced by my argument that preprints should count towards productivity. The study section was SYN June 7, 2018.” — Michael Ailion, PhD, Associate Professor of Biochemistry, University of Washington

<This report is redacted for the online version, no permission to share publicly.>

Another reporter highlighted that guidance received from the Scientific Review Officer (SRO) influences how they evaluate submissions, and without guidance, the assumption has been that preprints are counted negatively. From multiple reports, we understand that none of these study section reviewers recalled the SRO of their study section providing guidance about how to treat preprints in accordance with NOT-OD-17-050 (we did not survey for SRO guidance with respect to NOT-OD-18-228). Worryingly, one reporter recalls an SRO saying to a panel member that it is CSR practice to disregard preprints, despite the panel member pointing out NOT-OD-17-050.

Potential actions

We support and endorse the proposed action that SROs notify study sections about the NIH policy notice NOT-OD-17-050 during preparatory discussions before the study section begins to review submissions, as mentioned above. We propose that the inclusion of a brief statement about the NIH’s support for preprints would help to clarify and reaffirm the NIH’s policy support for preprints as evidence of productivity and rigor, and empower SROs to ensure that preprints are treated in line with NIH policy during study section discussions and reports.

This information could include:

a description of what a preprint is and is not: “A preprint is a complete scientific manuscript that is uploaded by the authors to a public server without peer review and can be viewed without charge by anyone in the world”; it is not the version of a manuscript that has passed journal-led peer review, although the preprint may be updated in response to peer feedback (see https://asapbio.org/preprint-info for more information);
the reasons why the NIH encourages researchers to post preprints and cite them in their applications;
and how reviewers can assess preprints for rigor, scientific impact and quality. Reviewers can read the preprinted manuscript in order to evaluate directly the rigor of any scientific work supporting the application that has not yet passed the journal peer-review process.

We think it is key to emphasize that preprints are permitted as evidence of productivity and that reviewers are asked to evaluate based on scientific rigor not peer-review status in accordance with the abovementioned NIH policy notices. While preprints are not yet peer-reviewed for a journal, they can be assessed in the same manner as other preliminary data. This information could be recalled by the SRO during the study section review process whenever an intervention is appropriate, for example to remind reviewers about the policy when a reviewer disregards a cited preprint due only to its unreviewed status.

We are available to support the NIH with the preparation of materials to communicate about preprints.