Michael Eisen1,2 and Leslie B. Vosshall 3,4
1 Department of Molecular and Cell Biology and 2 Howard Hughes Medical Institute, University of California, Berkeley, CA. 3 Laboratory of Neurogenetics and Behavior and 4 Howard Hughes Medical Institute, The Rockefeller University, New York, NY.
Research papers are the primary tangible and lasting output of science. It is how we communicate our discoveries, and how we are evaluated for hiring, promotion, and prizes. The current system by which most scientific papers are published predates the Internet by several hundred years, has changed little over centuries, and has largely failed to take advantage of new technologies enabled by the internet.
We believe that this system no longer serves the needs of scientists.
- It is slow. Manuscripts spend an average of nine months in peer review prior to publication, In part this delay is due to the fact that reviewers increasingly demand more data and more experiments before endorsing a paper for publication. This massively slows the dissemination of scientific knowledge.
- It is expensive. We spend $10 billion a year on science, technical and medical journal publishing, over $6,000 per article, and increasingly these costs are coming directly from research grants.
- It is arbitrary. The system of peer review we use to determine which papers are published and where is flawed. Excellent papers are rejected, and flawed papers are accepted. Despite this, journal name continues to be used as a proxy for the quality of the paper.
- It is inaccessible. Even with the significant efforts of the open-access publishing movement, the vast majority of scientific literature is not accessible without a subscription.
These four major pathologies of science publishing can be cured if researchers, not scientific publishers, decide when a work is ready to be published. Authors would publish manuscripts outside of the current publisher-driven infrastructure by uploading work to pre-print servers prior to conventional peer review. Pre-print publishing is the norm in in physics, mathematics, economics, and other disciplines, but has not been adopted to the same extent in biomedicine. We therefore strongly support the goal of ASAP Bio to accelerate the online availability of biomedical research manuscripts by getting funders and other stakeholders to endorse the adoption of pre-prints.
However, without fundamental reform in the way that peer review is carried out, the push for pre-prints will not succeed. Therefore, an important additional goal for the upcoming ASAP Bio meeting must be for funders to endorse alternative mechanisms for carrying out peer review. Such mechanisms would operate outside of the traditional journal-based system and focus on assessing the quality, audience, and impact of work published exclusively as “pre-prints”. If structured properly, we anticipate that a new system of pre-prints coupled with post-publication peer review will replace traditional scientific publishing much as online user-driven reviews (Amazon, Yelp, Trip Advisor) have replaced publisher-driven metrics to assess quality (Consumer Reports, Zagat, Fodor’s).
In this white paper, we explain why the adoption of pre-prints and peer review reform are inseparable, outline possible alternative peer review systems, and suggest concrete steps that research funders can take to leverage changes in peer review to successfully promote the adoption of pre-prints.
Pre-prints and journal-based peer review cannot coexist
The essay by Ron Vale that led to the ASAP Bio meeting is premised on the idea that we should use pre-prints to augment the existing, journal-based system for peer review. Under this model, biomedical researchers would post papers on pre-print servers and then submit them to traditional journals, which would review them as they do today, and re-publish those works that they deem suitable for their journal.
There are many reasons why such a system would be undesirable – it would leave intact a journal system that is inefficient, ineffective, inaccessible, and expensive. But more proximally, there is simply no way for such a symbiosis between pre-prints and the existing journal system to work.
Why are biomedical scientists not disseminating their work via pre-prints, when arXiv, a pre-print server for physics, mathematics, and other quantitative sciences, is thriving, with over 1,000,000 papers published since it launched in 1994? The answer lies in the uniquely strong power journals have over careers in biomedicine. Authors have no incentive to pursue alternative venues for publishing their work, and indeed have incentives not to. If we as a biomedical community want to promote the universal adoption of pre-prints, we have to tackle this issue. We have to do more than pay lip-service to the potential of pre-prints, we have to change the incentives that drive people away from them.
Scientists publish for two reasons: to communicate their work to their colleagues, and to get credit for it in hiring, promotion, and funding. If publishing behavior were primarily driven by a desire to communicate, biomedical scientists would leap at the opportunity to publish pre-prints, which make their work available to the widest possible audience at the earliest possible time at virtually no cost. That they do not underscores the reality that, for most biomedical researchers, decisions about how they publish are driven by the impact of these decisions on their careers.
A corollary of this reality is that pre-prints will be not be embraced by biomedical scientists until they are accepted as formally published works, and if they get full credit for the contributions these publication describe in funding, hiring and promotion. This can only happen if we create and embrace systems of post-publication peer review to evaluate the quality and impact of, and appropriate audience for papers published as preprints prior to conventional peer review.
Even if we are wrong, and pre-prints become widely adopted by biomedical scientists without new systems for peer review, the system that Vale envisions in which preprints are reviewed by traditional journals is unsustainable. If all, or even most, papers are available immediately for free online, it is all but certain that libraries would begin to cut subscriptions. As a consequence, traditional journal publishing, which still relies almost exclusively on revenue from subscriptions, would no longer be economically viable.
Thus, no matter how you see it happening, a belief in the importance of pre-print use in biomedicine requires the creation of an alternative system for assessing papers. We therefore suggest that the most important act for funders, universities, and other stakeholders is not to just endorse the use of pre-prints in biomedicine, but to endorse the development and use of a viable alternative to journal titles in the assessment of the quality, impact, and audience of works published exclusively as “pre-prints”. Peer review for the Internet Age
The current journal-based peer review system attempts to assure the quality of published works; help readers find articles of import and interest to them; and assign value to individual works and the researchers who created them. Post-publication peer review of works initially published as pre-prints can not only replicate these services, but do it faster, cheaper, and more effectively.
The primary justification for carrying out peer review prior to publication is that this prevents flawed works from being widely disseminated. Inviting a panel of two or three experts to assess the methods, reasoning, and presentation of the science in the paper, undoubtedly leads to many flaws being identified and corrected before publication.
But pre-publication review is actually very ineffective at this. Any practicing scientist can easily point to deeply flawed papers that have made it through peer review in their field, even in supposedly high-quality journals. And even when flaws are identified, it rarely matters. In a world where journal title is the accepted currency of quality, a deeply flawed Science or Nature paper is still a Science or Nature paper, meaning that there is little incentive for authors or readers to attempt to correct the published record.
Pre-publication review was developed and optimized for printed journals, which needed to catch errors before they were permanently rendered in a printed paper journal. Moreover, because printing and shipping is an expensive act, space in each issue had to be carefully rationed. Peer-review served a gate-keeping function to select only those papers that a small group of reviewers and editors deemed “appropriate” for the imputed quality of the journal. But today it is absurd to rely solely on the opinions of two or three reviewers, who may or may not be the best qualified to assess a paper, who often did not want to read the paper in the first place, who are acting under intense time pressure, and who are casting judgment at a fixed point in time, to be to sole arbiters of the validity and value of a work. Because it is far more inclusive and takes place throughout the useful lifetime of a paper, post-publication peer review of pre-prints is scientific peer review optimized for the Internet Age.
Beginning to experiment with systems for post-publication review now will hasten its development and acceptance, and is the quickest path to the universal pre-prints publishing. In the spirit of experimentation, we propose a possible system below.
A system for post-publication peer review
The link between pre-prints and post-publication review has long been recognized (c.f. Harold Varmus’s e-Biomed proposal from 1999), but there is considerable confusion and even anxiety about how such systems could work. To focus discussion, we propose here one possible system for post-publication peer review that addresses many concerns that have arisen from earlier discussion.
First, authors would publish un-reviewed papers on pre-print servers that screen them to remove spam and papers that fail to meet technical and ethical specifications, before making them freely available online. At this point peer review begins, proceeding along two parallel tracks.
Track 1: Organized review in which groups, such as scientific societies or self-assembling sets of researchers, representing fields or areas of interest arrange for the review of papers they believe to be relevant to researchers in their field. They could either directly solicit reviewers or invite members of their group to submit reviews, and would publish the results of these reviews in a standardized format. These groups would be evaluated by a coalition of funding agencies, libraries, universities, and other parties according to a set of commonly agreed upon standards, akin to the screening that is done for traditional journals at PubMed.
Track 2: Individually submitted reviews from anyone who has read the paper. These reviews would use the same format as organized reviews, and would become part of the permanent record of the paper. Ideally, we want everyone who reads a paper carefully to offer their view of its validity, audience, and impact. To ensure that the system is not corrupted, individually submitted reviews would be screened for appropriateness, conflicts of interest, and other problems, and there would be mechanisms to adjudicate complaints about submitted reviews.
Authors would have the ability at any time to respond to reviews and to submit revised versions of their manuscript.
Such a system has many immediate advantages over our current system of pre-publication peer review. The amount of scrutiny a paper receives would scale with the level of interest in the paper. If a paper is read by thousands of people, many more than the three reviewers chosen by a journal are in a position to weigh in on its validity, audience, and importance. And instead of only evaluating papers at a single fixed point in time, the process of peer review would continue for the useful lifespan of the paper.
What about concerns about anonymity for reviewers? We believe that peer review works best when it is completely open and reviewers are identified. This both provides a disincentive to various forms of abuse, and allows readers to put the review in perspective. We also recognize that there are many scientists who would not feel comfortable expressing their honest opinions without the protection of anonymity. We therefore propose that reviews be allowed to remain anonymous provided that one of the groups defined in Track 1 above vouch for their lack of conflict and appropriate expertise. This strikes the right balance between providing anonymity to reviewers while protecting authors from anonymous attacks.
What about the concern of flawed papers being published, or being subject to misuse and misinterpretation while they are being reviewed? We do not consider this to be a serious problem. The people in the best position to make use of immediate access to published papers – practicing scientists in the field of the paper – are in the best position to judge the validity of the work themselves and to share their impressions with others. Readers who need external assessment of the quality of a work can wait until it comes in, and are those no worse off than they are in the current system. If implemented properly, such a system would ensure the best of both worlds – rapid access for those who want and need it, and quality control over time for a wider audience.
Assessing quality and audience without journal names
The primary reason the traditional journal-based peer review system persists despite its anachronistic nature is that the title of the journal in which a scientific paper appears reflects the reviewers’ assessment of the appropriate audience for the paper and their valuation of its contributions to science. There is obviously value in having people make judgments about the potential audience and impact of a paper, and there are many circumstances where having an external assessment of a scientist’s work can be of use. But there is no reason we have to use journal titles to convey this information.
It would be relatively simple to give reviewers of published pre-prints a set of tools to specify the most appropriate audience for the paper, to anticipate their expected level of interest in the work, and to gauge the impact of the work. We can also take advantage of various automated methods to suggest papers to readers, and for such readers to rate the quality of paper by a set of useful metrics. Systems that use the Internet to harness collective expertise have fundamentally changed nearly every other area human society – it’s time for them to do the same for science.
A commitment to promoting pre-prints in biomedicine requires a commitment to promoting a new system for reviewing works published initially as un-reviewed pre-prints. Such systems are practical and a dramatic improvement over the current system. We call on funders and other stakeholders to endorse the universal publication of pre-prints and post-publication peer review as inseparable steps that would dramatically improve the way scientists communicate their ideas and discoveries. We recognize that such a system requires standards, and propose that a major outcome of the ASAP Bio meeting be the creation of an “International Peer Review Standards Organization” to work with funders and other stakeholders to establish these criteria and to work through many of the important issues, and then serve as a sanctioning body for groups of reviewers who wish to participate in this system. We are prepared to take the lead in assembling an international group of leading scientist to launch such an organization.