The publication of peer review is not a new idea, but it has yet to become widespread among journals in the life sciences. We’ve compiled here reports from journals with experience with this practice. If you have any additional questions or contributions, please email firstname.lastname@example.org.
How is the publication of peer review reports related to "open peer review?"
The term can mean many different things. In a review of the use of the term, Tony Ross-Hellauer identified seven orthogonal traits that can be present to varying degrees in different implementations of “open peer review:”
- Open identities: Authors and reviewers are aware of each other’s identity
- Open reports: Review reports are published alongside the relevant article.
- Open participation: The wider community are able to contribute to the review process.
- Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
- Open pre-review manuscripts: Manuscripts are made immediately available (e.g., via pre-print servers like arXiv) in advance of any formal peer review procedures.
- Open final-version commenting: Review or commenting on final “version of record” publications.
- Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.
Ross-Hellauer T. What is open peer review? A systematic review [version 2; referees: 4 approved]. F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.2) is licensed under CC BY 4.0
In this FAQ, we will focus on open reports, meaning that the contents of peer review are published alongside the article, with or without the reviewers’ names.
Which journals publish peer review reports?
Publons maintains a list of journals that allow them to display peer review information, though what information can be displayed varies. However, this is not a complete list of all journals publishing peer review reports.
TRANSPOSE is a new initiative that provides a framework for collecting journal policies on open peer review, co-reviewing, and preprinting. Contributions from anyone are welcome.
Will authors be reluctant to submit to a journal that publishes peer review?
The EMBO Journal
At The EMBO Journal, which publishes peer reviews anonymously, editors did not see a decline in the submission rate after introduction of this policy; furthermore, only 5.3% of authors opt out of publishing the peer review process file.
At eLife, articles are published along with the decision letter and the authors’ response letter. Initially, authors were able to opt out of publishing the decision letter and response, but so few chose the opt out that the letters are now always published. The journal receives over 600 submissions each month.
Nature Communications reports “no drop in submissions on introduction of transparent peer review in 2016. We offer author opt-in so the authors know they will have a choice before acceptance of the manuscript. Feedback from authors shows that they are supportive of the initiative. They recognise that the transparency of publishing peer review reports will create a fairer peer-review process and may even result in a better study.”
Forty-four of the sixty-eight BMC series journals operate open peer review as standard (not optional). The journals have not received any negative feedback from authors about this.
A study published in BMJ Open in 2015 analysed peer review reports in BMC Infectious Diseases compared with BMC Microbiology, as well as in the Journal of Inflammation that had changed from signed, published peer review to single blind peer review. Author satisfaction was higher for BMC Infectious Diseases compared with BMC Microbiology, including the response to the question regarding the helpfulness of comments from peer reviewers. In the Journal of Inflammation, there was no difference in author satisfaction between the periods of time when the journal operated on open or single-blind peer review.
In a trial of 5 Elsevier journals that piloted published peer review (with optional signing), all journals taking part in the pilot showed an upward submission trend.
In a pilot in which authors were allowed to opt-in to the publication of peer review reports, “83% of authors opt[ed in] for transparent peer review (278 of 336 submissions),” however, “The percentage of peer reviewers signing their reports is low at 19% (of 86 reports posted, 16 had identities).”
Will reviewers be reluctant to perform peer review if it will be published?
The EMBO Journal
The EMBO Journal reports no difference in reviewer acceptance rates.
European Journal of Neuroscience
European Journal of Neuroscience found that only 18 out of 3293 invited reviewers declined because of their (signed) transparent process.
In a trial of 5 Elsevier journals (published review, optional signing), the rate of reviewers’ acceptance to review declined from 43.6% to 30.9% during the pilot across all the journals. However, the global rate of acceptance of review invitations is also declining; the study authors’ modeling & comparison to other journals suggest this is independent of review openness.
Survey results associate with the same trial of 5 Elsevier journals (published review, optional signing): “Of the 440 reviewers who declined to review, 100 (22.7%) responded to the survey. Of these 100 respondents, 91 (91.0%) indicated that publication of the review report was not a reason for declining…Sixteen of 40 editors responded to the survey (40.0%). Five editors indicated that the pilot study made it more difficult for them to find a reviewer, 2 indicated that the pilot study did not have any influence, and 9 were undecided..”
Nature Communications reports: “Some reviewers declined to review manuscripts due to this scheme. The small increase in reviewer declines did not impact on an Editor’s ability to find reviewers for a manuscript or cause delays to manuscripts in peer review.”
From a Nature Communications editorial, “Transparent peer review one year on:” We were curious to see how our reviewers would react to this approach. However, this has only been a problem for a small number of reviewers, who declined to review under this circumstance. For example, we had instances where reviewers objected because they feared that for their small research area it would be easy to deduce their identity from the published reports.”
For the BMC series journals there is some evidence to suggest that reviewer uptake is lower due to the open peer review policy of the journals. A very small number of reviewers actually cite open peer review as the reason for declining, but editorial board members for some journals have said that they sometimes have difficulty recruiting reviewers for this reason. In a study comparing acceptance of peer reviewer invitations by peer review model: (open, single-blind, and double-blind) the proportion of accepted invitations was lower for open peer review journals (42% compared to 53% for single-blind peer review journals). As a result, for journals with open peer review models, a higher number of reviewers may need to be invited.
Will the quality or rigor of the reviews change?
The EMBO Journal
The EMBO Journal reports: “We have not noticed any systematic change, although a few referees have noted that they are taking extra care with writing their reports. Importantly, neither referees nor authors have become more timid in putting forthright arguments to paper.”
In a randomized controlled trial conducted in 1999 to 2000, The BMJ found no difference in article acceptance rates or review quality for papers with signed and published peer reviews (versus those undergoing a traditional closed process). However, potential reviewers were likely to refuse participation in the study, and reviewers whose work would be signed and published took longer to write their reviews.
European Journal of Neuroscience
European Journal of Neuroscience, which publishes signed peer review, reports: “The reviews that we receive are just as rigorous and fair as they were before the change. If anything, the quality of the reviews has improved as the reviewers know that they will be publicly accountable for their comments.”
In a trial of published, optionally signed peer review in 5 Elsevier journals, sentiment analysis of reviews conducted before and during the pilot suggests no change in the negativity/positivity or subjectivity/objectivity of the language.
In a survey associated with the same trial of 5 Elsevier journals (published, optionally signed): “Ten [out of 16] journal editors said that since the pilot study, they noticed that the reports were more in depth and constructive for authors, 5 did not notice any difference, and 1 did not reply to this question.”
Nature Communications reports: “Anecdotal evidence from reviewers […] indicates that they will make a greater effort to ensure that their report is thorough and appropriate for publication. Thus, we should see more thoughtfully written and comprehensive reviews in our published articles.”
In this study published in BMJ Open, the main findings were that the quality of peer review reports was slightly higher in BMC Infectious Diseases (publication of signed peer review) compared with BMC Microbiology (single-blind peer review), but there was no difference for the signed & published versus single-blind review in the Journal of Inflammation. These results suggest that it may be advantageous to use open peer review but they do not undermine the validity of using the single-blind approach.
Will publishing peer review make peer reviewers easier to identify (if they are anonymous?)
Nature Communications reports: “Feedback […] is mixed in this regard – some believe that confidentiality is a key part of peer review, while others have chosen to sign their reports due to the new process (see here for example). In particular, reviewers in small research fields had concerns that their identity could be deduced from their report. We have not heard substantiated reports of this in any field in which we publish to date.”
From a Nature Communications editorial, “Transparent peer review one year on:” “For example, we had instances where reviewers objected because they feared that for their small research area it would be easy to deduce their identity from the published reports.”
Has publishing peer reviews impacted editorial decisions?
Nature Communications reports that they have seen “no impact on our editorial decisions due to our transparent peer review policy.”
A study published in BMJ Open in 2015 found that overall reviewer recommendations were similar in BMC Infectious Diseases (signed, published peer review) compared with BMC Microbiology (single blind peer review), suggesting no difference between these two approaches. However, in the Journal of Inflammation, it was found that reviewers were more likely to recommend acceptance under signed, published peer review as compared with under single-blind peer review
What are reader interest levels in published peer review reports?
At The EMBO Journal, peer review process files receive about 10% of the hits as the papers themselves.
In the Elsevier pilot, ⅓ of visitors clicked through to the peer review reports.
Nature Communications reports that “reader engagement in the reviewer comments […] is difficult to assess. We have received some technical comments on published articles that refer to the Peer Review File, indicating that they find it useful to gain a deeper understanding of the paper and stimulate discussion (see here for example).
Have peer review reports been used as “ammunition” to criticize individual authors?
European Journal of Neuroscience (signed, published peer review) addresses some fears:
“We are now 10 months into the new transparent system, so what are our experiences, have our fears been born out, have our hopes been realized? […]
- The careers of young scientists will be destroyed by evil senior scientists on whose paper they have commented negatively. As far as we know this has proved not to be the case. This was a fallacious fear anyway, as most scientists at every level, are supportive of accountable review and transparency.”
- Battles and vendettas arise between authors and reviewers. Again, as far as we know this has proved not to be the case. And again, a fallacious fear, as rigorous, fair and unemotional discussion is part of the scientific method. Battles and vendettas have no place in science!”
Nature Communications reports that “Any criticism that we have seen at Nature Communications as a result of the reviewer reports has been in the context of constructive and scientific discussions. We have no reports of reviewer comments being used as ‘ammunition’ in an unprofessional manner.”
Have political opponents of a particular type of research used peer review reports to undermine trust in science?
Peer review reports have been referenced in a criticism of a published paper: “It is therefore salutary to read the pre-publication peer reviews by Alan Emond and Katherine M Keyes  of this paper by Brent Taylor, Hershel Jick and Dean MacLaughlin  and ask why it was ever published.”
Nature Communications reports that this has not been seen at the journal.
How have authors benefited from published peer review?
The European Journal of Neuroscience
The European Journal of Neuroscience reports that “Reviewers are more prompt in returning their comments and our impression is that their reviews are more carefully constructed, more thoughtful and of a higher quality.”
In a trial of 5 Elsevier journals that published anonymous review, “The percentage of authors who indicated that they like the publication of the peer review reports ranged from 48.7% (Engineering Fracture Mechanics) to 60.5% (Annals of Medicine and Surgery), and 51.6% to 63.9% of authors indicated that this policy would not influence their decision where to publish.”
Nature Communications reports that “60% of […] authors opt to publish the peer review reports accompanying their papers. Their feedback mainly focuses on the added transparency from the process and the ability to provide more context and background to the study.”
From a Nature Communications editorial called “Transparent peer review one year on”: “On average, around 60% of our authors are voluntarily opting in to publishing the peer review history of their paper. We believe that this is a very encouraging result, given also the diversity of our content across so many research areas… However, in almost all research areas the majority of papers that we publish will now contain a peer review file.”
How have readers benefited from published peer review?
Jonathan Klassen, an Assistant Professor at University of Connecticut, writes: “I’ve followed the journal Biology Direct (https://biologydirect.biomedcentral.com/) for some time and have very much appreciated the dialog between the authors and experts in the field. Certainly the quality of reviews there is typically quite high, and gave me something to model my own reviews after as a starting scientist. And my perception is that these reviews tend to add quite significant scientific insight, which the reviewers get at least some credit for.”
“Perhaps the most influential aspect of the exercise, called the Reproducibility Project: Cancer Biology, has nothing to do with those arguments. It lies beneath the surface, in the peer reviews of the project teams’ replication plans, which were published before the studies began. These reviews can be read as part of the editorial decision letters linked to each replication plan, or ‘registered report’ (see go.nature.com/2jte08a).
From a Nature editorial entitled “Replication studies offer much more than technical details”: “What, one might ask, could be less interesting? What insights can emerge from technical feedback on plans to repeat what other groups of scientists have done? Plenty. The decision letters reveal the practice of science at its best: probing whether an experiment truly tests a particular idea; identifying shortcomings in the original set-ups; and proposing further, sounder tests. To those committed to improving science, reading these insights — the fruit of voluntary donations of time and effort — will prove a moving experience. Journal clubs would do well to read the original papers, and then compare their own analyses with those of the scholars who have reviewed the replication reports.”
Nature Communications reports: “One of the aims of transparent peer review at Nature Communications is to explore ways of making research publishing less of a “black-box”, allowing the research community to assess the thoroughness to which a paper has been scrutinised prior to publication. Early career researchers will also be able to gain a deeper understanding of the review process.”
From a Nature Communications editorial called “Transparent peer review one year on”: “Our intention is to open up the peer review process at the journal, and to provide information on the expert reports that led to our editorial decision to publish a paper. Moreover, the reviewer reports also can serve as a background on the merits of a study, and inform about the discussion between authors and our reviewers […] We have become aware of instances where authors promoted the very positive published reviewer comments of their paper. At the same time, we also had feedback from readers on the peer review discussion of a paper. These instances showcase the potential of transparent peer review—on the one hand to appreciate the value of published papers, and on the other hand to initiate a constructive dialogue between authors and readers on the specifics of a study.”
What are technical best practices concerning publishing peer review?
On October 17. 2018, a group of journal and infrastructure representatives published a workshop report containing recommendations to JATS4R on article and metadata structure for pubilshing peer reviews.