It’s time to open the black box of peer review

By Jessica Polka and Ron Vale, ASAPbio

Photo CC BY-NC by Premnath Thirumalaisamy

Opening the content of peer review reports—whether they are anonymous or not—will improve their quality, ensure that ideas that emerge through review are accessible to other researchers, and enable innovation and reform.

Peer review is considered an essential standard of scientific publishing. Despite complaints, most scientists feel that peer review remains a valuable part of the scientific process. Yet, for something deemed so essential and valuable to the process of establishing scientific credibility, it is surprising that it is disposed of once a paper has been accepted for publication. Some journals (EMBO, BMJ, eLife, PeerJ, and F1000Research, among others) are displaying the content of peer reviews alongside the published paper (termed “open reports” using Tony Ross-Hellauer’s taxonomy). However, they are in the minority; only 2.2% of life science journals publish open reports. Furthermore, even in cases where peer reports are open, they are not currently not easily searchable.

Here we present the benefits and disadvantages of open reports. In comparing the two, we believe that the benefits outweigh the disadvantages and that the practice of releasing open reports should expand to become an industry-wide standard.

Why open peer review reports?

Opening up peer review reports would have many benefits, such as:

1. Allowing other researchers to benefit from the ideas, critiques, arguments, and references contained in peer review reports. We have all received peer reviews that contained excellent ideas that could be of interest to other scientists. Thus, peer review is a form of scholarship that can be useful to readers beyond its role in certifying a manuscript for journal publication.
2. At a system-wide level, allowing an analysis of the strengths and weaknesses of peer review. Reviewers can be biased for or against individual authors, types of institutions, or work that agrees or disagrees with their own ideas. The reviews themselves can be unconstructive or of low quality. Unfortunately, we have little idea how pervasive these problems actually are. Initiatives such as PRE, Project Cupcake, and a recent Peer Review Transparency workshop encourage journals to release partial information about peer review, such as the type of review (double blind, single blind, number of reviewers, etc). However, to assess content and quality, the peer review reports themselves must assessed firsthand (or at least by a third party independent of the journal). Peer review cannot be improved if it is hidden from view.
3. Making it clear to readers which parts of a manuscript have been scrutinized (or whether the manuscript has been scrutinized at all). What passes for peer review likely varies widely from journal to journal and even from manuscript to manuscript. Making reports open will enhance trust in the publication system. Furthermore, if post-publication peer review of papers is to grow, the availability of the pre-publication open reports also will likely provide valuable information about how potential weaknesses or issues have been addressed.
4. Prompt referees to be more careful and thoughtful with their commentary and encourage a much needed a tone of scholarship that is associated with work that will be preserved. In this regard, a randomized-controlled trial at the BMJ showed that while making peer review reports both open and signed did not change reviewer quality (as rated by editors) or the likelihood of acceptance/rejection, it did increase the amount of time reviewers spent on their reviews—perhaps indicating they were composed with greater care. Further studies of this sort are needed.
5. Enhancing peer review training by providing early career researchers with valuable examples of high quality reports.

What are the downsides to open reports?

One criticism of open reports is that it will flood the scholarly literature with additional material, much of which will have little value. Many argue that they have little time to read publications, let alone read peer review. There is no denying that most open reports will go unread and unanalyzed. However, in our view, this is not a valid reason to hide them. The solution to “information overload” is not to repress information but to search for it in better ways. Even if one person picks up a relevant point from an open report, this could spark new thinking that leads to a scientific discovery.

Authors and reviewers (even if unnamed) also might feel uneasy about open reports because of possible embarrassment associated with public exposure of mistakes or oversights. However, science advances overall through imperfections and corrections. No scientist is immune to mistakes. The only way to have problems corrected is through constructive feedback from our peers. We need to acknowledge and celebrate this process rather than hide it from view.

Do names matter?

Some individuals strongly argue that reviewer names should be attached to open reports, which would enable reviewers to be held directly accountable for their words. However, attaching names to peer review activity (open identities) was an unpopular concept among researchers surveyed by Ross-Hellauer. This is not surprising given that many reviewers fear that criticizing authors under their own names will damage relationships and invite retribution. The concern is likely greater for early career researchers or others who feel vulnerable. Authors also may be more inclined to discount the criticisms of junior reviewers, either viewing them as less competent than well-established investigators or feeling less obligated to follow their recommendations because they have little power and influence.

Given these concerns, it seems most sensible at this time to make signing reviews optional. Insisting that reviewers’ names must be published along with the reports could undermine the important proposition of broad adoption of open reports.

Next steps

How should the community advance the cause of open report? Technically, reviewers and authors can make it clear that they will not adhere to an “expectation of confidentiality” (PDF) that shrouds the peer review process and seek to publish reports, for example as comments on a preprint (if the article in question appears on a server).

But a far better solution would be for journals themselves instigate change themselves. We would advocate that journals publish peer review reports alongside the article and register them with Crossref’s new schema to ensure they become a part of the scholarly record. We also would like to ultimately see peer reviews readily linked to search portals such as PubMed and Google Scholar. Importantly, peer reviews should also be openly licensed and accessible to humans and machines from the onset to facilitate system-wide analysis. These reforms will finally allow the scientific community to have access to the vast amount of labor they have invested in peer review. The visibility associated with open reports also will help us to understand, validate, and enhance our trust in the peer review system.