Michael Eisen1,2 and Leslie B. Vosshall 3,4
1 Department of Molecular and Cell Biology and 2 Howard Hughes Medical Institute, University of California, Berkeley, CA. 3 Laboratory of Neurogenetics and Behavior and 4 Howard Hughes Medical Institute, The Rockefeller University, New York, NY.
Research papers are the primary tangible and lasting output of science. It is how we communicate our discoveries, and how we are evaluated for hiring, promotion, and prizes. The current system by which most scientific papers are published predates the Internet by several hundred years, has changed little over centuries, and has largely failed to take advantage of new technologies enabled by the internet.
We believe that this system no longer serves the needs of scientists.
- It is slow. Manuscripts spend an average of nine months in peer review prior to publication, In part this delay is due to the fact that reviewers increasingly demand more data and more experiments before endorsing a paper for publication. This massively slows the dissemination of scientific knowledge.
- It is expensive. We spend $10 billion a year on science, technical and medical journal publishing, over $6,000 per article, and increasingly these costs are coming directly from research grants.
- It is arbitrary. The system of peer review we use to determine which papers are published and where is flawed. Excellent papers are rejected, and flawed papers are accepted. Despite this, journal name continues to be used as a proxy for the quality of the paper.
- It is inaccessible. Even with the significant efforts of the open-access publishing movement, the vast majority of scientific literature is not accessible without a subscription.
These four major pathologies of science publishing can be cured if researchers, not scientific publishers, decide when a work is ready to be published. Authors would publish manuscripts outside of the current publisher-driven infrastructure by uploading work to pre-print servers prior to conventional peer review. Pre-print publishing is the norm in in physics, mathematics, economics, and other disciplines, but has not been adopted to the same extent in biomedicine. We therefore strongly support the goal of ASAP Bio to accelerate the online availability of biomedical research manuscripts by getting funders and other stakeholders to endorse the adoption of pre-prints.
However, without fundamental reform in the way that peer review is carried out, the push for pre-prints will not succeed. Therefore, an important additional goal for the upcoming ASAP Bio meeting must be for funders to endorse alternative mechanisms for carrying out peer review. Such mechanisms would operate outside of the traditional journal-based system and focus on assessing the quality, audience, and impact of work published exclusively as “pre-prints”. If structured properly, we anticipate that a new system of pre-prints coupled with post-publication peer review will replace traditional scientific publishing much as online user-driven reviews (Amazon, Yelp, Trip Advisor) have replaced publisher-driven metrics to assess quality (Consumer Reports, Zagat, Fodor’s).
In this white paper, we explain why the adoption of pre-prints and peer review reform are inseparable, outline possible alternative peer review systems, and suggest concrete steps that research funders can take to leverage changes in peer review to successfully promote the adoption of pre-prints.
Pre-prints and journal-based peer review cannot coexist
The essay by Ron Vale that led to the ASAP Bio meeting is premised on the idea that we should use pre-prints to augment the existing, journal-based system for peer review. Under this model, biomedical researchers would post papers on pre-print servers and then submit them to traditional journals, which would review them as they do today, and re-publish those works that they deem suitable for their journal.
There are many reasons why such a system would be undesirable – it would leave intact a journal system that is inefficient, ineffective, inaccessible, and expensive. But more proximally, there is simply no way for such a symbiosis between pre-prints and the existing journal system to work.
Why are biomedical scientists not disseminating their work via pre-prints, when arXiv, a pre-print server for physics, mathematics, and other quantitative sciences, is thriving, with over 1,000,000 papers published since it launched in 1994? The answer lies in the uniquely strong power journals have over careers in biomedicine. Authors have no incentive to pursue alternative venues for publishing their work, and indeed have incentives not to. If we as a biomedical community want to promote the universal adoption of pre-prints, we have to tackle this issue. We have to do more than pay lip-service to the potential of pre-prints, we have to change the incentives that drive people away from them.
Scientists publish for two reasons: to communicate their work to their colleagues, and to get credit for it in hiring, promotion, and funding. If publishing behavior were primarily driven by a desire to communicate, biomedical scientists would leap at the opportunity to publish pre-prints, which make their work available to the widest possible audience at the earliest possible time at virtually no cost. That they do not underscores the reality that, for most biomedical researchers, decisions about how they publish are driven by the impact of these decisions on their careers.
A corollary of this reality is that pre-prints will be not be embraced by biomedical scientists until they are accepted as formally published works, and if they get full credit for the contributions these publication describe in funding, hiring and promotion. This can only happen if we create and embrace systems of post-publication peer review to evaluate the quality and impact of, and appropriate audience for papers published as preprints prior to conventional peer review.
Even if we are wrong, and pre-prints become widely adopted by biomedical scientists without new systems for peer review, the system that Vale envisions in which preprints are reviewed by traditional journals is unsustainable. If all, or even most, papers are available immediately for free online, it is all but certain that libraries would begin to cut subscriptions. As a consequence, traditional journal publishing, which still relies almost exclusively on revenue from subscriptions, would no longer be economically viable.
Thus, no matter how you see it happening, a belief in the importance of pre-print use in biomedicine requires the creation of an alternative system for assessing papers. We therefore suggest that the most important act for funders, universities, and other stakeholders is not to just endorse the use of pre-prints in biomedicine, but to endorse the development and use of a viable alternative to journal titles in the assessment of the quality, impact, and audience of works published exclusively as “pre-prints”. Peer review for the Internet Age
The current journal-based peer review system attempts to assure the quality of published works; help readers find articles of import and interest to them; and assign value to individual works and the researchers who created them. Post-publication peer review of works initially published as pre-prints can not only replicate these services, but do it faster, cheaper, and more effectively.
The primary justification for carrying out peer review prior to publication is that this prevents flawed works from being widely disseminated. Inviting a panel of two or three experts to assess the methods, reasoning, and presentation of the science in the paper, undoubtedly leads to many flaws being identified and corrected before publication.
But pre-publication review is actually very ineffective at this. Any practicing scientist can easily point to deeply flawed papers that have made it through peer review in their field, even in supposedly high-quality journals. And even when flaws are identified, it rarely matters. In a world where journal title is the accepted currency of quality, a deeply flawed Science or Nature paper is still a Science or Nature paper, meaning that there is little incentive for authors or readers to attempt to correct the published record.
Pre-publication review was developed and optimized for printed journals, which needed to catch errors before they were permanently rendered in a printed paper journal. Moreover, because printing and shipping is an expensive act, space in each issue had to be carefully rationed. Peer-review served a gate-keeping function to select only those papers that a small group of reviewers and editors deemed “appropriate” for the imputed quality of the journal. But today it is absurd to rely solely on the opinions of two or three reviewers, who may or may not be the best qualified to assess a paper, who often did not want to read the paper in the first place, who are acting under intense time pressure, and who are casting judgment at a fixed point in time, to be to sole arbiters of the validity and value of a work. Because it is far more inclusive and takes place throughout the useful lifetime of a paper, post-publication peer review of pre-prints is scientific peer review optimized for the Internet Age.
Beginning to experiment with systems for post-publication review now will hasten its development and acceptance, and is the quickest path to the universal pre-prints publishing. In the spirit of experimentation, we propose a possible system below.
A system for post-publication peer review
The link between pre-prints and post-publication review has long been recognized (c.f. Harold Varmus’s e-Biomed proposal from 1999), but there is considerable confusion and even anxiety about how such systems could work. To focus discussion, we propose here one possible system for post-publication peer review that addresses many concerns that have arisen from earlier discussion.
First, authors would publish un-reviewed papers on pre-print servers that screen them to remove spam and papers that fail to meet technical and ethical specifications, before making them freely available online. At this point peer review begins, proceeding along two parallel tracks.
Track 1: Organized review in which groups, such as scientific societies or self-assembling sets of researchers, representing fields or areas of interest arrange for the review of papers they believe to be relevant to researchers in their field. They could either directly solicit reviewers or invite members of their group to submit reviews, and would publish the results of these reviews in a standardized format. These groups would be evaluated by a coalition of funding agencies, libraries, universities, and other parties according to a set of commonly agreed upon standards, akin to the screening that is done for traditional journals at PubMed.
Track 2: Individually submitted reviews from anyone who has read the paper. These reviews would use the same format as organized reviews, and would become part of the permanent record of the paper. Ideally, we want everyone who reads a paper carefully to offer their view of its validity, audience, and impact. To ensure that the system is not corrupted, individually submitted reviews would be screened for appropriateness, conflicts of interest, and other problems, and there would be mechanisms to adjudicate complaints about submitted reviews.
Authors would have the ability at any time to respond to reviews and to submit revised versions of their manuscript.
Such a system has many immediate advantages over our current system of pre-publication peer review. The amount of scrutiny a paper receives would scale with the level of interest in the paper. If a paper is read by thousands of people, many more than the three reviewers chosen by a journal are in a position to weigh in on its validity, audience, and importance. And instead of only evaluating papers at a single fixed point in time, the process of peer review would continue for the useful lifespan of the paper.
What about concerns about anonymity for reviewers? We believe that peer review works best when it is completely open and reviewers are identified. This both provides a disincentive to various forms of abuse, and allows readers to put the review in perspective. We also recognize that there are many scientists who would not feel comfortable expressing their honest opinions without the protection of anonymity. We therefore propose that reviews be allowed to remain anonymous provided that one of the groups defined in Track 1 above vouch for their lack of conflict and appropriate expertise. This strikes the right balance between providing anonymity to reviewers while protecting authors from anonymous attacks.
What about the concern of flawed papers being published, or being subject to misuse and misinterpretation while they are being reviewed? We do not consider this to be a serious problem. The people in the best position to make use of immediate access to published papers – practicing scientists in the field of the paper – are in the best position to judge the validity of the work themselves and to share their impressions with others. Readers who need external assessment of the quality of a work can wait until it comes in, and are those no worse off than they are in the current system. If implemented properly, such a system would ensure the best of both worlds – rapid access for those who want and need it, and quality control over time for a wider audience.
Assessing quality and audience without journal names
The primary reason the traditional journal-based peer review system persists despite its anachronistic nature is that the title of the journal in which a scientific paper appears reflects the reviewers’ assessment of the appropriate audience for the paper and their valuation of its contributions to science. There is obviously value in having people make judgments about the potential audience and impact of a paper, and there are many circumstances where having an external assessment of a scientist’s work can be of use. But there is no reason we have to use journal titles to convey this information.
It would be relatively simple to give reviewers of published pre-prints a set of tools to specify the most appropriate audience for the paper, to anticipate their expected level of interest in the work, and to gauge the impact of the work. We can also take advantage of various automated methods to suggest papers to readers, and for such readers to rate the quality of paper by a set of useful metrics. Systems that use the Internet to harness collective expertise have fundamentally changed nearly every other area human society – it’s time for them to do the same for science.
A commitment to promoting pre-prints in biomedicine requires a commitment to promoting a new system for reviewing works published initially as un-reviewed pre-prints. Such systems are practical and a dramatic improvement over the current system. We call on funders and other stakeholders to endorse the universal publication of pre-prints and post-publication peer review as inseparable steps that would dramatically improve the way scientists communicate their ideas and discoveries. We recognize that such a system requires standards, and propose that a major outcome of the ASAP Bio meeting be the creation of an “International Peer Review Standards Organization” to work with funders and other stakeholders to establish these criteria and to work through many of the important issues, and then serve as a sanctioning body for groups of reviewers who wish to participate in this system. We are prepared to take the lead in assembling an international group of leading scientist to launch such an organization.
[aio_button align=”none” animation=”none” color=”gray” size=”small” icon=”none” text=”View PDF” relationship=”dofollow” url=”https://asapbio.org/wp-content/uploads/2016/02/Eisen_Vosshall_ASAPBio_WhitePaper.pdf”]
An interesting and provocative paper and a great starting point. But I have a few questions that I hope will help the discussion dig into some important details. To draw more people in to that discussion, I’ve blogged them here: http://occamstypewriter.org/scurry/2016/02/06/combining-pre-prints-and-post-publication-peer-review-a-new-big-deal/
Interested to see responses either here or on the blog.
After we wrote this white paper, Eisen and I decided that “posting” a “pre-print” on a “server” misses the point entirely. We suggest that scientists will be “publishing” their “papers” on a “” that is a place where the sciences lives and post-publication peer review begins. We just don’t know what that “” will be or what it should be called. Lots of ideas being bounced around on Twitter. Feedback desperately needed.
Yes, please! This has always been my favourite solution (Which I humbly proposed in 2013 http://lab.gilest.ro/blog/what-is-wrong-with-scientific-publishing-and-how-to-fix-it/ )
@leslievosshall:disqus , in the open software world the is called a .
Hi Leslie, great call to action! We have been working on a open preprint peer review system: http://academickarma.org/ . The idea is that an author submits their work to a preprint server, then uses our platform to find an appropriate editor, who invites reviewers. All the reviews are open. We use a non monetary ‘karma’ currency to try to make sure that academics contribute as much as they use from this system.
Hi Leslie, I’m going to work back from your point about publishing on a ‘thing’ to briefly present something outside the box. This would be an alternative to our outdated system, that incorporates some of the advantages of preprints, but goes a lot further. I think we can and need to take this further. The main premise is to remove the ‘story-telling’ from science, and go back to the data. I’ll try to be brief – this has been several years on the back burner, takes many other people’s ideas and inputs into account, and been run by a few ‘influential’ people who didn’t make me think it was too insane! It’s great to see the names going to this meeting and I do hope this is the start of a new era. Please forgive this random presentation of essentially why I think we need to go further than preprints.
Completely restructure biological research by eliminating traditional journal publications, enabling rapid, open and online dissemination of peer-reviewed results on one platform, empowering all scientists to work together on global projects.
The current system:
– Scientists required to produce their own ‘complete’ scientific story for publication preferably in a high impact journal before they can get tenure/grants.
Some of the problems:
– There is an every-man-for-himself culture, with many labs competing and keeping results secret, wasting time and resources.
– Science is never complete, but manuscripts in journals are static and mostly independent, and even when conclusions are later disqualified (sometimes by unpublished work) as knowledge inevitably moves on, they remain in the literature.
– The real value of a manuscript is difficult to judge at the time of publication.
– There is enormous pressure to publish in the top journals, which can be counterproductive.
– Papers often go to multiple journals before they are published, causing delays (sometimes years) and leading to repetitive lengthy peer review.
– Journals are competing for the top stories/authors, and have interests that don’t always align with publishing the most solid scientific discoveries.
– A lot of manuscripts are not accessible without journal subscriptions.
The current system actually harms science. We need a new one.
How about this?:
– We build a web-based infrastructure with open access to publish all results (big and small), incorporating a simple, open peer review system, and ways to link related results.
– The new ‘scientific literature’ is dynamic, and grows with time as results can be connected and interpreted to make novel discoveries.
– We incorporate mechanisms to acknowledge individual scientists, also over time, that can be easily mined by governing bodies.
– We develop ways to communicate results to the wider research community and the public.
What we gain:
– Accelerated scientific discoveries as ‘publication’ occurs in real time, bypassing journal-related delays, and making it less likely for two labs to be working for years on the same problem. Instead we have more minds (essentially anyone who is interested) working together on the same existing problems.
– Avoid lengthy review processes that can steer research into artificial journal-style packaging.
– We record all the data (after it is deemed solid by peer review), even that which doesn’t fit in the current system (i.e., an incomplete story or negative data) or whose value may be realized later. This can lead to a new field of science whereby scientists mine the published data to make new discoveries.
– Fidelity of research is easier to ensure as everything is linked on one platform. And sharing work sooner means it can be corroborated or disqualified sooner, avoiding costly diversions. And you still get acknowledged for repeating results, which is critical but mostly absent from the current system as novelty is a key factor for publication.
– We keep the young, smart PIs because they can still get tenure when they are just unable to turn a high-risk promising project into a full scientific story within 5 years.
How do we succeed?:
– We get the top scientists/technologists on side.
– We build a kick-ass platform – this should be possible – we are a community of the brightest minds, and we generate some of the most valuable data.
– We get major funding bodies on side to require publication on the platform. Particularly biomedical funders want good results published fast.
– We charge (preferably a minimum amount) for publication, as do all open access journals. The sheer quantity should hopefully boost revenue (like PLoS ONE). Hosting all the scientific data that is produced should somehow open up revenue options, right?
The new system mimics a cell’s signalling network: it is dynamic and massively interconnected, containing feedback loops for adjustments as technology and knowledge evolves. This also makes it a more natural process, which is what scientific discovery is.
Thanks for all the feedback, everyone. In reading all the comments here and on Twitter, your blog links, and all the other white papers, I conclude that everyone sees the same problems. And everyone has variations on the same theme to correct those problems. While many scientists express understandable anxiety about change, I have not heard many clamoring to keep the current system in place indefinitely. So what should we do next?. Now we need to stop discussing our agreement to the idea and actually act on it.
https://uploads.disquscdn.com/images/1488719f317139306054e5da858a168cd0733f3056036952b4fc80e9722b2ea7.png A solution was recently proposed: “Peer Community in” is a non-profit scientific organization aimed at creating specific communities of researchers freely reviewing and recommending preprints and postprints in their field.
The first community is Peer Community in Evolutionary Biology. It has been launched in January 2017 and already gathers 269 ‘recommenders’.
Until now only postprints have been recommended however preprints are currently being evaluated and peer reviewed. Visit https://evolbiol.peercommunityin.org and https://peercommunityin.org