Six essential reads on peer review

In preparation for our meeting on Transparency, Recognition, and Innovation in Peer Review in the Life Sciences on February 7-9 at HHMI Headquarters, we’ve collected some recent (and not-so-recent) literature on journal peer review. A full annotated bibliography can be found at the bottom of this post, and we invite any additions via comments. To make the list more manageable, we’ve highlighted some of the most crucial content here.

1. A multi-disciplinary perspective on emergent and future innovations in peer review

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 2; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.2)

This massive review from Jonathan Tennant et al represents a comprehensive discussion of the origin, evolution, and challenges surrounding modern peer review. Of particular note is Section 1, which offers a history of peer review and an overview of critiques of the system, and Section 3, which discusses how platforms like GitHub, Reddit, Amazon, and Stack Overflow could catalyze innovations in scholarly peer review.

 

2. What is open peer review? A systematic review

Ross-Hellauer T. What is open peer review? A systematic review [version 2; referees: 4 approved]. F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.2)

The very term “open peer review” means many different things to different people. It can refer to revealing reviewer identities or the content of their reviews, allowing reviewers to discuss the article with one another or the author, or a process in which reviewers don’t need to be invited in order to offer feedback. Indeed, Tony Ross-Hellauer has identified 122 definitions for open peer review.

Fortunately for us, he’s also distilled a schema of seven “traits” of open peer review:

• Open identities: Authors and reviewers are aware of each other’s identity
• Open reports: Review reports are published alongside the relevant article.
• Open participation: The wider community are able to contribute to the review process.
• Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
• Open pre-review manuscripts: Manuscripts are made immediately available (e.g., via pre-print servers like arXiv) in advance of any formal peer review procedures.
• Open final-version commenting: Review or commenting on final “version of record” publications.
• Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.

We’ll use this taxonomy throughout our discussions on this blog and at the meeting.

3. Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers.

Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE 12(12): e0189311. https://doi.org/10.1371/journal.pone.0189311

The taxonomy identified above was used in an online survey completed by over 3,000 scholars, 95.5% of whom authored an academic communication and 87.6% of whom had experience as reviewers. While the respondent pool skewed heavily toward earth and environmental sciences (41.6%), 14.6% of respondents were biologists and 14.5% were in the health sciences. Perhaps because this was an opt-in survey conducted by an organization devoted to open access, the respondents sampled were a bit more cynical (but not dramatically so) about the current peer review process than people who had responded to a previous study conducted by Mark Ware for the Publishing Research Consortium.

One highlight of the report is Figure 8, reproduced below, which reports attitudes toward the seven traits of open peer review identified in Ross-Hellauer’s F1000 paper. Respondents had largely positive views toward open interaction, open reports, and open commenting. Interestingly, all of the “traits” save open identity were regarded more favorably than open pre-review manuscripts (ie preprints)!

Given the growth of preprints in the life sciences and other disciplines over the last few years, these data suggest that peer review might be ripe for even more radical change.

4. Peer reviews are open for registering at Crossref

Lin, Jennifer (2017)  Peer reviews are open for registering at Crossref. https://www.crossref.org/blog/peer-reviews-are-open-for-registering-at-crossref/

During Peer Review Week this fall, Crossref—the non-profit organization that issues DOIs for scholarly journal articles (and preprints)—announced that it will also offer a type of DOI specifically for preprints. Registration opened in late October, and the accompanying documentation suggests interesting and helpful use cases. For example, “contributors” to the review can include (in addition to regular “reviewers”) assistant reviewers, stats reviewers, or translators, and all may be anonymous. The metadata contains fields to keep track of whether the review occured pre- or post- publication, the reviewing round, and the license under which the review is released. And the review can be linked to the object being reviewed (which doesn’t need to be a journal article).

5. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial

van Rooyen Susan, Delamothe Tony, Evans Stephen J W (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial BMJ; 341 :c5729  https://doi.org/10.1136/bmj.c5729

One barrier to driving change in peer review is that comparisons between different systems are often clouded by confounders such as different author and reviewer pools across journals or across time (in cases where policy has changed).

The BMJ set a high standard for rigor in studies of peer review by conducting a series of randomized controlled trials in the early 2000’s. In this one, members of the intervention group had their signed reviews posted online, while members of the control group did not. The authors found that:

“Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.”

In 2014, the BMJ announced that all research papers would be published with signed peer review reports, arguing that

“Such open peer review should increase the accountability of reviewers and editors, at least to some extent. Importantly, it will also give due credit and prominence to the vital work of peer reviewers. At present, peer review activities are under-recognised in the academic community. We hope that reviewers will find this increased visibility helpful when demonstrating the extent and impact of their academic work and that they and others will cite and share their reviews as a learning resource.”

They came to this conclusion in spite of their acknowledgement that it might “provide “more scope for power relationships to favour ‘the great and the good,’”” as pointed out by as Karim Khan.

6. Why we don’t sign our peer reviews

Yoder, Jeremy (2014). Why we don’t sign our peer reviews. http://www.molecularecologist.com/2014/04/why-we-dont-sign/

As the authors of the BMJ editorial have acknowledged, making peer review (and especially the names of peer reviewers) transparent can complicate the human relationships that make up science, especially for reviewers who are junior or otherwise marginalized or vulnerable.

To get at some of these issues, Jeremy Yoder has compiled short pieces written by members of his community to explain why they don’t sign peer review.

One anonymous postdoc writes,

“I chose to be anonymous because I am relatively junior, although I am not sure at what point I would consider myself sufficiently senior to change my approach. I expect that my comments might be not taken as seriously if my position (and gender) were known. I also expect that my own comments will be slightly less critical if I reveal my name. Science, and peer review, are political processes and to claim otherwise (i.e., that authors and reviewers will act exactly the same without the protection of reviewer anonymity) seems out of step with the reality.”

Personal connections are also important. Tony Gamble adds that it is especially difficult to reject manuscripts written by friends and colleagues, and Will Pearse has the opposite concern:

“My concern is appearing sycophantic when I enjoy a paper, I’m actually not worried about more negative reviews because I’m always polite and constructive.”

Given these concerns, what benefits are gained by removing or preserving anonymity in peer review? And is posting review reports (without reviewer names) similarly complicated by concerns for protecting authors?

We’ll discuss this (and more) on February 7-9. Tune in to the webcast and join the discussion on Twitter with #bioPeerReview.

 

Further reading

Compiled with Caitlin Schrein, HHMI

Download BibTex

Studies

Bornmann, Lutz, Rüdiger Mutz, and Hans-Dieter Daniel. “A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants.” PLOS ONE 5, no. 12 (December 14, 2010): e14331. https://doi.org/10.1371/journal.pone.0014331

Meta-analysis suggesting that there is low inter-rater reliability between reviewers (Cohen’s Kappa: 0.17).

Glonti, Ketevan, Daniel Cauchi, Erik Cobo, Isabelle Boutron, David Moher, and Darko Hren. “A Scoping Review Protocol on the Roles and Tasks of Peer Reviewers in the Manuscript Review Process in Biomedical Journals.” BMJ Open 7, no. 10 (October 1, 2017): e017468. https://doi.org/10.1136/bmjopen-2017-017468

Helmer, Markus, Manuel Schottdorf, Andreas Neef, and Demian Battaglia. “Research: Gender Bias in Scholarly Peer Review.” eLife 6 (March 21, 2017): e21718. https://doi.org/10.7554/eLife.21718

One of two prominent 2017 demonstrations that editors select more male reviewers.

Lerback, Jory, and Brooks Hanson. “Journals Invite Too Few Women to Referee.” Nature News 541, no. 7638 (January 26, 2017): 455. https://doi.org/10.1038/541455a

One of two prominent 2017 demonstrations that editors select more male reviewers.

Ortega, José Luis. “Are Peer-Review Activities Related to Reviewer Bibliometric Performance? A Scientometric Analysis of Publons.” Scientometrics 112, no. 2 (August 1, 2017): 947–62. https://doi.org/10.1007/s11192-017-2399-6

Rooyen, Susan van, Tony Delamothe, and Stephen J. W. Evans. “Effect on Peer Review of Telling Reviewers That Their Signed Reviews Might Be Posted on the Web: Randomised Controlled Trial.” BMJ 341 (November 16, 2010): c5729. https://doi.org/10.1136/bmj.c5729

Rooyen, Susan van, Fiona Godlee, Stephen Evans, Nick Black, and Richard Smith. “Effect of Open Peer Review on Quality of Reviews and on Reviewers’ recommendations: A Randomised Trial.” BMJ 318, no. 7175 (January 2, 1999): 23–27. https://doi.org/10.1136/bmj.318.7175.23

Ross-Hellauer, Tony. “What Is Open Peer Review? A Systematic Review.” F1000Research 6 (August 31, 2017): 588. https://doi.org/10.12688/f1000research.11369.2

This literature review distills over 100 partially conflicting definitions of “open peer review” into a taxonomy of seven traits.

Ross-Hellauer, Tony, Arvid Deppe, and Birgit Schmidt. “Survey on Open Peer Review: Attitudes and Experience amongst Editors, Authors and Reviewers.” PLOS ONE 12, no. 12 (December 13, 2017): e0189311. https://doi.org/10.1371/journal.pone.0189311

This survey of over 3,000 respondents (almost 30% coming from the life sciences) demonstrates attitudes toward various open peer review traits such as open identities, open reports, etc.

Tomkins, Andrew, Min Zhang, and William D. Heavlin. “Reviewer Bias in Single- versus Double-Blind Peer Review.” Proceedings of the National Academy of Sciences 114, no. 48 (November 28, 2017): 12708–13. https://doi.org/10.1073/pnas.1707323114

Ware, Mark. 2015. “Peer Review Survey 2015: Key Findings.” 2015. http://publishingresearchconsortium.com/index.php/134-news-main-menu/prc-peer-review-survey-2015-key-findings/172-peer-review-survey-2015-key-findings

This 2015 survey conducted by a publishers’ consortium offers a more conservative perspective than Ross-Hellauer’s 2017 work.

Wilkinson, Jo. “Tracking Global Trends in Open Peer Review.” Publons, October 27, 2017. https://publons.com/blog/who-is-using-open-peer-review/

Commentary and reviews

Bastian, Hilda. 2017. “The Fractured Logic of Blinded Peer Review in Journals.” Absolutely Maybe (blog). October 31, 2017. http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/

Bloch, Daniel. n.d. “Expertise in Sciences and the Decision of What Is Publishable: A Noble yet Endangered Task.” The Conversation. Accessed December 30, 2017. http://theconversation.com/expertise-in-sciences-and-the-decision-of-what-is-publishable-a-noble-yet-endangered-task-86339

Cohen. 2017. “The next Stage of SocArXiv’s Development: Bringing Greater Transparency and Efficiency to the Peer Review Process.” Impact of Social Sciences (blog). October 16, 2017. http://blogs.lse.ac.uk/impactofsocialsciences/2017/10/16/the-next-stage-of-socarxivs-development-bringing-greater-transparency-and-efficiency-to-the-peer-review-process/

Commons Select Committee. n.d. “Peer Review.” UK Parliament. Accessed December 30, 2017. http://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technology-committee/inquiries/peer-review/

Epstein, Diana, Virginia Wiseman, Natasha Salaria, and Sandra Mounier-Jack. 2017. “The Need for Speed: The Peer-Review Process and What Are We Doing about It?” Health Policy and Planning 32 (10):1345–46. https://doi.org/10.1093/heapol/czx129

Falavigna, Asdrubal, Michael Blauth, and Stephen L. Kates. 2017. “Critical Review of a Scientific Manuscript: A Practical Guide for Reviewers.” Journal of Neurosurgery, October, 1–10. https://doi.org/10.3171/2017.5.JNS17809

Flier, Jeffrey. 2016. “It’s Time to Overhaul the Secretive Peer Review Process.” STAT. December 5, 2016. https://www.statnews.com/2016/12/05/peer-review-process-science/

Golumbeanu, Silvia. 2017. “In Fermat’s Library, No Margin Is Too Narrow.” Nautilus. October 16, 2017. http://nautil.us/blog/in-fermats-library-no-margin-is-too-narrow

Fermat’s Library is a platform and online community for discussing scientific papers.

Gowers, Timothy. n.d. “Peer Review: The End of an Error?” TheTLS. Accessed December 30, 2017. https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

Groves, Trish. 2010. “Is Open Peer Review the Fairest System? Yes.” BMJ 341 (November):c6424. https://doi.org/10.1136/bmj.c6424

Hames, Irene. 2014. “The Peer Review Process: Challenges and Progress.” Editage Insights (30-12-2017), June. https://www.editage.com/insights/the-peer-review-process-challenges-and-progress

Hopfgartner, Gérard. 2017. “What Makes a Good Review from an Editor’s Perspective?” Analytical and Bioanalytical Chemistry 409 (29):6721–22. https://doi.org/10.1007/s00216-017-0648-4

Khan, Karim. 2010. “Is Open Peer Review the Fairest System? No.” BMJ 341 (November):c6425. https://doi.org/10.1136/bmj.c6425

King, Stuart RF. 2017. “Peer Review: Consultative Review Is Worth the Wait.” eLife 6 (September):e32012. https://doi.org/10.7554/eLife.32012

Kuehn, Bridget M. 2017. “Peer Review: Rooting out Bias.” eLife 6 (September):e32014. https://doi.org/10.7554/eLife.32014

Mayden, Kelley D. 2012. “Peer Review: Publication’s Gold Standard.” Journal of the Advanced Practitioner in Oncology 3 (2):117–22

Panter, Paige. n.d. “Peer Review for Early Career Researchers: Your Basic Questions Answered | Wiley.” Accessed December 30, 2017. https://hub.wiley.com/community/exchanges/discover/blog/2017/12/06/peer-review-for-early-career-researchers-your-basic-questions-answered

Rodgers, Peter. 2017. “Peer Review: Decisions, Decisions.” eLife 6 (September):e32011. https://doi.org/10.7554/eLife.32011

Ross-Hellauer, Tony. 2017. “Open Peer Review: Bringing Transparency, Accountability, and Inclusivity to the Peer Review Process.” Impact of Social Sciences (blog). September 13, 2017. http://blogs.lse.ac.uk/impactofsocialsciences/2017/09/13/open-peer-review-bringing-transparency-accountability-and-inclusivity-to-the-peer-review-process/

Schekman, Randy. 2017. “Scientific Publishing: Room at the Top.” eLife 6 (October):e31697. https://doi.org/10.7554/eLife.31697

Sipido, Karin R., Diane Gal, Aernout Luttun, Stefan Janssens, Maurilio Sampaolesi, and Paul Holvoet. 2017. “Peer Review: (R)evolution Needed.” Cardiovascular Research 113 (13):e54–56. https://doi.org/10.1093/cvr/cvx191

Slavov, Nikolai. 2015. “Point of View: Making the Most of Peer Review.” eLife 4 (November):e12708. https://doi.org/10.7554/eLife.12708

Staniland, Mark. n.d. “Increasing Transparency in Peer Review : Of Schemes and Memes Blog.” Accessed December 30, 2017. http://blogs.nature.com/ofschemesandmemes/2017/10/12/increasing-transparency-in-peer-review

Tennant, Jon. 2016. “What If You Could Peer Review the arXiv?” ScienceOpen Blog (blog). April 6, 2016. http://blog.scienceopen.com/2016/04/what-if-you-could-peer-review-the-arxiv/

Tennant, Jon. 2017. “Peer Review Does Not Have a ‘Gold Standard’, but Does It Need One?” Green Tea and Velociraptors (blog). May 30, 2017. http://fossilsandshit.com/peer-review-not-gold-standard-need-one/

A response to Mayden (2012), this blog post contains a helpful list of existing guidelines to promote best practicesin peer review.

Tomkins, Andrew. n.d. “Understanding Bias in Peer Review.” Research Blog (blog). Accessed December 30, 2017. https://research.googleblog.com/2017/11/understanding-bias-in-peer-review.html

Wilkinson. 2017. “Writing a Peer Review Is a Structured Process That Can Be Learned and Improved – 12 Steps to Follow.” Impact of Social Sciences (blog). May 17, 2017. http://blogs.lse.ac.uk/impactofsocialsciences/2017/05/17/writing-a-peer-review-is-a-structured-process-that-can-be-learned-and-improved-12-steps-to-follow/

Wingfield, Brenda. n.d. “The Peer Review System Has Flaws. But It’s Still a Barrier to Bad Science.” The Conversation. Accessed December 30, 2017. http://theconversation.com/the-peer-review-system-has-flaws-but-its-still-a-barrier-to-bad-science-84223

Yoder, Jeremy. 2014a. “Why We Don’t Sign Our Peer Reviews.” April 9, 2014. http://www.molecularecologist.com/2014/04/why-we-dont-sign/

Yoder, Jeremy. 2014b. “Why We Sign Our Peer Reviews.” April 9, 2014. http://www.molecularecologist.com/2014/04/why-we-sign/

Educational material

“Focus on Peer Review – Nature Masterclasses.” Nature Masterclasses. Accessed December 30, 2017. https://masterclasses.nature.com/courses/205

Glonti, Ketevan, Daniel Cauchi, Erik Cobo, Isabelle Boutron, David Moher, and Darko Hren. “A Scoping Review Protocol on the Roles and Tasks of Peer Reviewers in the Manuscript Review Process in Biomedical Journals.” BMJ Open 7, no. 10 (October 1, 2017): e017468. https://doi.org/10.1136/bmjopen-2017-017468

Panter, Paige. “Peer Review for Early Career Researchers: Your Basic Questions Answered | Wiley.” Accessed December 30, 2017. https://hub.wiley.com/community/exchanges/discover/blog/2017/12/06/peer-review-for-early-career-researchers-your-basic-questions-answered

“Peer Review Training: ACS Reviewer Lab.” Accessed December 30, 2017. https://www.acsreviewerlab.org/

“Publons Academy Supervisors.” Publons. Accessed December 30, 2017. http://publons.com/community/academy-supervisors/