In Defence of Peer Review

By Tony Hyman and Ron Vale

Max Planck Institute of Molecular Cell Biology and Genetics, Dresden Germany (hyman@mpi-cbg.de) and the University of California, San Francisco, United States (ron.vale@ucsf.edu)

Rapid changes in communication technology have led to sea changes in publication. The days when John Maddox (1) joined Nature and found submitted manuscripts sitting in piles forgotten on the floor have long gone, and no longer do we receive publications as bound printed volumes. Those lucky enough to be part of a rich institution that can pay subscription fees have instant access to most of the world’s literature at their fingertips. However, there is one aspect of publication that has remained essentially unchanged over the past half-century: peer review. Journals send out scientific papers to two or three experts who provide critiques of the work. These critiques are used to alert authors and editors to possible mistakes, flaws in interpretation, or lack of clarity in presentation; meanwhile, editors use these critiques to judge whether a paper is suitable for publication in their journal. Granted, papers are now transferred to referees by web interfaces rather than by the post, but why in these days of the internet, with pervasive online, crowd-sourced commentary, has technology not had more impact on peer review? Is the premise that expert peer review is an essential part of scientific dissemination still a valid one, or does peer review persist for reasons of inertia alone? Here, we argue for the former, while acknowledging the need for fresh thinking in the peer review paradigm.

Everyone complains about peer review. We all curse the dreaded third reviewer who we feel has deliberately misunderstood our work; his/her comments can be unprofessional and aggressive. We feel impotent to respond as we would like, because the editor holds the power to require a clean bill of health before the paper is accepted, often necessitating that we address all of the points raised by the referees. We are rejected soundly by one journal and yet receive positive comments from a comparable competitor journal. This seemingly arbitrary nature of peer review has led to calls for reform. But on the other hand, for all of its inevitable flaws, most of which are intrinsic to human nature, peer review still works remarkably well. It aspires to be rigorous, and few researchers would argue that their articles have not benefited from honest critique through the peer review system. We complain about reviewer 3, but we often forget to acknowledge the excellent critiques of referees 1 and 2 who got up early on a Sunday morning to carefully read through and comment on our paper.

Peer review is part of a wider form of self-governance that we practice as scientists. We review each other’s grants, site-visit institutions, and decide who should receive jobs or be promoted within an institution. Despite the fact that governments pour billions of dollars into research, they have been largely content to allow scientists to take responsibility for their own evaluation and self-review. And scientists take this responsibility seriously, as anyone who has ever served on a grant review panel can attest. Indeed, when asked, the majority of scientists are very supportive of pre-publication peer review (3), suggesting that we need to strengthen and improve peer review as a cornerstone of the scientific profession.

In considering how the internet could impact peer review, it is a salutary lesson to analyse how the internet has impacted another aspect of knowledge distribution, which is the dissemination of national and global news. As traditionally practiced, journalism has strict standards and codes of ethics. Copy is prepared by journalists, checked by sub-editors and other fact-checkers, before being subject to editorial decision higher up the chain. Failure to abide by journalistic standards can lead to sanctions by professional oversight bodies, such as the Independent Press Standards Organisation in the UK (4). The trustworthiness and independence of a free press is so critical that it has been called the ‘fourth estate’. The internet has opened up more portals for receiving news and made it accessible on mobile devices. However, the internet has altered classic journalism so quickly and drastically that it has led to unanticipated consequences and dramatic shifts in elections and global politics. Most problematically, much of our news is no longer subject to the rigours of fact checking as power shifts from news organisations to individual journalists. Many local newspapers have disappeared or operate with diminished staff (5), while low quality on-line news sites enjoy increased impact and some are being manipulated for political gain. Who has not watched with disquiet the way that our recent elections around the world that have been unduly influenced by unsubstantiated rumour? Indeed, some have argued that democracy, as it was originally envisioned, may not survive the assault of unchecked and unsubstantiated news (6).

There are parallels between our publication system and journalism. As scientists, we are the journalists who prepare the copy; the peer reviewers are the sub-editors who check the data, and the editors decide on whether the sub-editors have done their job properly, how interesting the work is, and where it should appear in the newspaper. Competition among journalists for where their articles appear can be as fierce inside a newspaper as it is for scientists to publish their work in top journals. While it is unlikely that scientific literature will fall prey to the same political and economic pressures that are at play in the news industry, it would be unwise suddenly to abandon, or even downgrade, the central importance of pre-publication peer review. Unregulated dissemination of scientific knowledge could become subject to gaming-the-system and other abuses that might be hard to anticipate at the onset, just as we have witnessed for the dissemination of news. Pre-publication peer review is also a mechanism, or brand, that is understood by the general public, many of whom are knowledgeable of the difference between a peer-reviewed publication and an unsubstantiated post on the internet. Indeed, it was the fact that the falsified peer reviewed manuscript on vaccine and autism was published in the Lancet that made it so damaging. In this case, neither the editors nor the peer reviewers did their job properly, and the lack of rapid post-publication correction had terrible consequences.

One could argue that, in the end, good science will be recognised without pre-publication peer review, using various post-publication review mechanisms (7). Indeed, the internet provides a myriad of opportunities for post-publication peer review that could become a powerful new mechanism for evaluating work. However, we would argue that such experiments should be developed and tested on a strong foundation of pre-publication peer review. Data on the efficacy and utility of pre- and post-publication review will be essential for deciding how best to orchestrate the resources of the scientific community in evaluating research studies.

When thinking about how pre-publication peer review might be strengthened, we can return to our example of newspapers. Scientists, like journalists, are governed by a code of ethics. Editors, in both newspapers and journals, are accountable to the owners of the journals. However, unlike sub-editors in a reputable newspaper, peer review is at best ad hoc, and peer reviewers have little vested interest in the journal (with some exceptions: eLife, for example, has given peer review a more central role in the journal through its board of reviewing editors (8)). Editors, or grant review bodies, often have difficulty in finding reviewers. One proposal to strengthen peer review would be to formalise its different roles, by separating the roles of pre-publication peer review of data and interpretation from the role of evaluating the suitability of the work for a particular journal. If pre-publication technical review were separated from the decision by an editor as to the paper’s appropriateness for their journal, it would allow peer review to best perform its most important role in improving scientific work and examining whether the paper is rigorous and well-presented. This role could be established by, for instance, using professional, paid review services. This could still be performed by practicing scientists, but they would have been recruited specifically to such services and paid for their contribution. The use of professional peer review services also would allow the editors to dedicate more time in selecting the most interesting papers for their journals and commissioning commentaries on the work to make them broadly accessible.

Somewhat surprisingly, pre-publication peer review is relatively new in the history of science. For instance, Science did not request outside reviewers until the late 1940s (9) and this practice only became routine in Nature in the late 1960s (10). However, science was quite different in that era and perhaps pre-publication peer review was less vital. At that time, the entire field of molecular biology could attend a Cold Spring Harbor Meeting, most people knew what others were doing in their field, manuscripts were mailed to colleagues before publication, and editors could easily keep up with the in-flow of content. Those days are long past, and biomedical science has become a much larger enterprise. It is difficult to envisage what the future may hold. But just as we see in the problems of internet news today, racing to the future without considering the consequences may leave us blindsided. The recent history of newspapers makes the case that while we should use the internet to incorporate novel mechanisms to disseminate scientific information and review post-publication, we must support and strengthen good pre-publication peer review, as the only current mechanism to ensure the publication of reproducible results that stand the test of time.

Of course, much can still be done now to improve the transparency and quality of pre-publication peer review, as discussed in other blogs on the ASAPbio website (13). Furthermore, innovations in post-publication evaluation also are needed to allow the rapid correction of mistakes and recognize work that grows in its importance. Most importantly, scientists must carefully consider how best to improve and evaluate their work and guide these systems forward. This responsibility must not be outsourced to publishers, governments, or institutions. The public has placed their trust in us, the scientific community, and we need to deliver.

References

In Defence of Peer Review

Related Posts

Leave a Reply Cancel reply