ASAPbio response to FASEB’s statement on the NIH RFI on preprints

1/19/2017 update: We will close the signature drive at 9pm EST on Sunday, 1/22/2017.

Summary

The NIH released a RFI (request for information) on “including preprints and interim research products in the NIH applications and reports.”  ASAPbio, and many individual scientists, responded with arguments in favor of providing scientists with the option, not requirement, of citing preprints in NIH applications/reports as public evidence of their most recent work and productivity. FASEB, a scientific society claiming to be the voice of 125,000 scientists, issued a strong negative response to allowing preprints to be used in NIH grant review.  We, junior and senior scientists of the ASAPbio Board of Directors feel that there are many deeply problematic issues with FASEB’s arguments including 1) an unfamiliarity with preprints and even an articulation of incorrect information, 2) a lack of transparency of how they derived their decision, and 3) a view that there is “no need to read” original scientific papers, which we feel is not the type of culture that the funding agencies should foster in order to promote excellence in grant review. Because FASEB claims to speak for many societies and many scientists, their letter (signed only by the FASEB President) could be given disproportionate weight by the NIH (for a past historical example, see this article).  ASAPbio therefore has written this detailed response to FASEB (see below). We are also collecting signatures of scientists who support the option to cite preprints in NIH grant applications and reports here until 9pm EST on 1/22/2017.

To whom it may concern:

On December 15th, FASEB released the public version of their response to the NIH RFI on “Interim Research Products,” which closed for public comment on December 9th. The conclusion of this public statement was that FASEB does “not support the inclusion of preprints or interim research products in NIH grant applications and reports at this time” and states five arguments backing this statement. ASAPbio, a scientist-led organization promoting the value of preprints in improving the speed, transparency, and openness of scientific communication, wishes to respond to this FASEB statement for the following reasons:

First, the arguments made by FASEB reveal a lack of familiarity with multiple aspects of preprints, preprint servers and the current use of preprints in the exchange of scientific information, which warrants a point-by-point counter argument.

Second, as former members and chairs of NIH study sections, we find the FASEB response problematic with regard to its implications for the culture of study sections and the responsibilities of its members in making well-informed decisions on grants. The argument of increased workload and burden (point 2) extends beyond preprints to peer-reviewed literature, sending an overarching message that there is “no need to read”.  Study section members are peer reviewers and encouraging them to read papers and directly evaluate data and arguments is better for high quality grant review than relying on a list of journal names or impact factors.

Third, the mechanism by which FASEB derived its statement, which is signed only by the FASEB President, is not transparent, and it is not clear who is represented by their response. For example, individual FASEB members, including the lead author of this letter, were not, to our knowledge, consulted for the purpose of gathering opinions for drafting their response. Some of the inaccuracies in FASEB’s response noted below could have been addressed in pre-publication review by engaging its members, such as ourselves.

Fourth, at stake is the broader issue is whether preprints are viewed as a respectable form of scholarly communication. Inclusion of preprints in NIH grant applications, as well as grants from other agencies (see below), would allow preprints, along with journal publications, to be used as evidence of productivity. Grants are critical for career advancement. If not allowed, then the incentives for using preprints are greatly diminished. This issue has been deliberated at two meetings organized by ASAPbio with many stakeholders (our February 2016 meeting at HHMI (report in Science) and our May 2016 Funders’ Workshop at NIH), and attendees concluded that preprints should be used as evidence of productivity in the review of grants and academic promotions in the life sciences.

Fifth, looking beyond these current points of disagreement, we hope to use this response to express our desire to engage scientific societies in open and productive conversations. The future of scholarly communication will require practicing scientists, academic institutions, national or international scientific societies, and funding agencies working together for the common good of scientists and society as a whole. The desires of individual scientists and scientific societies have not always been in sync regarding scholarly communication, but such problems can be minimized by direct conversations rather than volleys of policy statements. At the close of this response, ASAPbio wishes to engage FASEB and other societies in open and productive conversations on how to advance preprints and other forms of open scientist-to-scientist exchange of knowledge.

As a preamble to our point-by-point response on the issue of preprints and grant review, we note that the physics, computer science and mathematics communities have used preprints in grant review for decades without the negative consequences cited by FASEB. Preprint use is now rapidly growing in biology, including by leading laboratories in the life sciences. The FASEB response lacks a depth of understanding of how preprints and journal articles complement one another. FASEB frames their arguments in terms of preprints versus peer-reviewed journal articles, not recognizing that both are valuable at different times and for different reasons. Preprints serve a useful role in providing a public document of recent data and productivity, which otherwise would not be available during the time of the journal submission and review process, which often is prolonged. Recent research results are often the most pertinent to the aims of a grant application. Thus, preprints provide information on an applicant’s most recent work, while journal articles provide the final publication of record. Both are valuable for scholarly communication; both are valuable to funding agencies whose overarching mission is to advance scientific progress; both are valuable to review committees who are tasked with making well informed decisions on grant applications. Recognizing the usefulness of both preprint and journal communication mechanisms, the Wellcome Trust, the Medical Research Council of the UK, the Helmsley Trust, the Simons Foundation, and the Human Frontiers Science Program have issued public statements within the last six months encouraging the use of preprints in their grant applications.

Our detailed response to the five concerns outlined in the FASEB letter follows:

Concern #1:   “The lack of a clear definition of “preprint”.”

A preprint is a scholarly manuscript that has been prepared for publication but has not yet been formally peer-reviewed or published by a journal. This is not a new concept and, as the FASEB letter notes, has been long used in the physics and mathematics community. Saying that the life sciences community has “a core challenge… in defining what constitutes a preprint” is thus a straw man. The FASEB statement then states, “To be referenced as part of an NIH grant application, preprint and interim research products should already have a permanent digital object identifier (DOI) and be accessible throughout the application and funding process to standardize these entities.” We are happy to correct the impression that this issue has not already been dealt with: all preprint archives for the biological sciences assign a DOI to each preprint. (The physics ArXiv, established before the advent of DOIs, invented their own permanent identifier.)

Concern # 2: “The risk of an increased workload and burden for reviewers”

The letter states: “While we acknowledge that typical grant application reviewing practice does not obligate a reviewer to critique supplementary data, reliance on materials that have not been rigorously peer reviewed, such as preprints, may put additional pressure on study sections when considering elements of the proposed research.” Most NIH grants contain extensive preliminary data. Therefore, the NIH already allows applicants to include preliminary data in grant applications. Data in preprints are no different but have some important added advantages. First, preprints, which take the form of a complete scientific manuscript and are usually the same as a version submitted to a journal, help a reviewer better evaluate recent data by providing detailed methods and the full context of the results. Second, the data provided in a preprint is a public record, indicating that the authors stand behind their work and are willing to have it scrutinized by the entire scientific community. This indicates that the authors have reached a stage of completeness beyond preliminary data that might be included in a grant. Third, being a public document with a time stamp, a preprint does not pose risk to the applicant of their results being used in unintended ways by a reviewer who might not indicate a conflict of interest. Importantly, implicit in this FASEB statement is notion that reviewers would need to read a preprint but do not need to read published articles cited in grants. The imprimatur of a journal or its impact factor is no substitute for critical reading of scientific papers, which can reveal flaws in experimental design or lack of support for a particular statement in a grant. We also disagree with the statement that “… preprints and interim research products could be used … to bypass grant application page limits when describing scientific premise. .. allowing them to be included in grant applications will burden reviewers with the additional responsibility of assessing the merits of a potentially unlimited volume and heterogeneous works in progress submitted in support of a research proposal.” In physics and biology, the vast majority of preprints take the form of a journal article and are ultimately published as journal articles. By this FASEB argument, it would be best if scientists produce fewer journal articles so as not to burden review committees. Again, the implication of the FASEB remarks is that journal articles cited by an applicant are not to be read, as they must be valid and support the applicant’s hypothesis. We realize that the FASEB statement might reflect an unfortunate current reality of how many reviewers perceive their work, with many using journal name as indicator of quality for rapidly scoring applications. However, we feel that the “no need to read” viewpoint of grant reviewing is not serving our community well and is not the best method of assessing grant applications.

Concern #3: Negative effect on rigor and reproducibility of research

Response: The NIH now requires applicants to address the strength and weaknesses of prior research, including peer-reviewed studies. This requirement recognizes that the fact that both peer-reviewed as well as non-peer-reviewed literature need to be read and evaluated. This intent is at odds with the FASEB statement that “relying on non-peer reviewed material as a source of data supporting rigor and reproducibility would put the onus on the reviewer to gauge the authenticity and quality of such data.” Reviewers of grants are also reviewers of journal papers, and it is well within their capability to assess the validity of data and arguments contained in a preprint. If reviewers cannot evaluate the data pertaining to grants to which they have been assigned, then there is a deeper problem. Again, as expressed in our response to concern #2, there is a disturbing undertone to the FASEB response concerning expectations of grant reviewers to read scientific literature. We unfortunately agree that FASEB is correct that some grant reviewers would prefer to save time and not read papers related to an application, simply skimming journal names as a proxy of evidence of rigor, evidence, and achievement. However, we share the view of many of our colleagues that it is better to read and evaluate the recent work of an applicant in the form of a preprint or a published article in order to render a better judgment on the application. Prohibiting reviewers to access and use such information only hinders their ability to execute their jobs as conscientious grant reviewers. Just because some reviewers do not want to be burdened by the possibility of reading scientific literature does not mean that the system and culture of grant reviewing should be reduced to this lowest common denominator.

Concern #4: Challenges of balancing application criteria to ensure sound science while supporting early career investigators

Response: The statement, “the only appeal of the citation of preprints in grant applications is that it may allow early career investigators to demonstrate current progress in support of a broader research proposal” has no basis in fact or understanding of how preprints could be used in reviewing a grant. Many leading senior scientists support the use of preprints. The inclusion of a preprint citation in the grant application of a senior scientist provides information of recent work of relevance to the grant reviewers, just as is the case with a junior scientist. The notion of senior scientists being “above providing evidence” is a hierarchical idea that should not be part of the grant review system. At the same time, we enthusiastically embrace any development that would help junior scientists, especially those submitting their first proposals to the NIH. For junior scientists, preprints can be particularly advantageous in providing evidence of productivity and independence, which are often issues of concern to grant reviewers. Given the problem of the steadily increasing age of first-time NIH grant recipients, efforts that promote fairness in the evaluation of junior scientists in the grant review system deserve serious consideration.

Concern #5: The need to provide stakeholders with ample time to accommodate potential changes in application policies.

Response:  It takes no more time to insert a preprint citation in EndNote or in a biosketch than it takes to insert a citation to a published manuscript. We therefore disagree with the tone of the FASEB response that permitting citation of preprints is equivalent to other, more substantial changes to NIH policy such as shortening applications from 25 to 12 pages and dramatically changing the format of the biosketch, both of which required time for community to adapt. Since citation of preprints will be optional, a change in policy regarding preprints will not create a burden for grant applicants. Preprint citation will only be used if there is relevant work that has not advanced to a journal publication of record and is relevant to the application. While this may only apply to a subset of applications, it nonetheless will be valuable for the fair review of those applications. We, however, agree with FASEB that instructions on the use of preprints in applications should be provided to reviewers of grants, largely because they are still relatively new to many life scientists.  

Concluding remarks

We recognize that the NIH has issued this RFI to evaluate whether preprints have a useful role in grant review as well as to gauge the community interest in preprints overall. It is instructive to review the disconnect between the interests of individual scientists and those of scientific societies that occurred in the evaluation of Harold Varmus’ E-Biomed proposal in 1999. The article “The real stakes of virtual publishing: the transformation of E-Biomed into PubMed Central” by Kling, Spector and Fortuna (DOI: 10.1002/asi.10352open access version) outlines how original E-Biomed proposal, which included a preprint and post publication repository of scientific papers, was supported by individual scientists but opposed by scientific societies. The “highly visible and highly influential position statements made by scientific societies against the proposal” played a major role in diluting E-Biomed to what we now know as PubMed Central. It would be unfortunate if we, as a scientific community, do not learn from this history and repeat mistakes of the past. ASAPbio therefore seeks to engage FASEB and other societies in the future of scholarly scientific communication for the benefit of scientists and society at large. We would be happy to present to your executive committees and engage with your members. Scientific societies have long served researchers by providing them with opportunities to share unpublished data through poster and oral presentations at meetings and conferences. Today’s technology enables this sharing to flourish in the digital space as well as the physical. Societies therefore have much to gain by recognizing and supporting digital platforms for sharing information. At the same time, societies also have much to offer in terms of peer review, which we, the coauthors of this response, recognize as crucial to the improvement of scientific work. Society journals play an important role in providing these services. There is a concern that the growth of preprints will threaten the existence of journals, which provide much of the revenue of scientific societies. However, the facts do not support this concern. After thirty years of arXiv, however, physicists still submit the vast majority of their preprints to journals (~73% of older preprints can be matched to journal articles, and probably more if one excludes conference proceedings, PhD thesis, etc) (DOI: 10.1002/asi.23044preprint available). There is every reason to believe that biologists will do the same and that society journals, if they provide useful services, will continue to thrive. In addition, we feel that preprints offer new opportunities for scientific societies such as FASEB to participate in scholarly communication and offer services and support to its members.

Authored by:

Cynthia Wolberger
Ron Vale
James Fraser
Jessica Polka