Organizer
Joseph Wade, Wadsworth Center, New York State Department of Health
Website or social media links
Current stage of development
Ready to be implemented
Project duration
One year initially, but potentially indefinitely.
Update
How has your project changed?
We introduced an additional option to incentivize authors to initiate peer review. As an alternative to requesting traditional formal evaluation of their preprint that could potentially be used by a journal, authors can request informal guidance on how to further develop and improve the impact of their study before journal submission, to support a key reason that authors post preprints. We expect that this opportunity will encourage both preprint deposition and peer review as it provides a novel service not formally available elsewhere.
We all know that low quality peer review has damaging consequences on science and scientists. High quality peer review requires a panel of experts with relevant experience covering both technical and conceptual aspects of the paper, as well as oversight to ensure thoroughness and professionalism of each review. For submissions to journals, an editor selects appropriate reviewers and oversees the quality of the reviews. Our proposal creates this valuable peer review process for preprints by having an author-selected Liaison Team assume these responsibilities. To ensure accountability and high quality, the Liaison Team is named, and we added that they meet key criteria (see below). Ultimately, ensuring high quality peer review will effectively curate preprints by segregating strong preprints from those that do not garner sufficient interest for review. Instead of providing an overall rating of the preprint, the Liaison Team will write a non- quantitative Impact Summary , given that a single rating cannot capture all aspects. The time taken to complete reviews will be monitored to ensure a timely process and identify reliable reviewers .
Instead of providing an overall rating of the preprint, the Liaison Team will write a non- quantitative Impact Summary, given that a single rating cannot capture all aspects.
The time taken to complete reviews will be monitored to ensure a timely process and identify reliable reviewers .
Have you integrated any feedback received?
- To prevent gaming of the system we will specify criteria to avoid real and perceived conflicts of interest in the selection of Liaison Teams by authors, as is standard at scientific journals. In addition:
- The Liaison Team comprises multiple members, which should minimise individual biases.
- The Liaison Team is identified, which discourages authors from nominating close friends, and gives the team a strong incentive to do a good job – their reputation is at stake.
- Readers can rate the preprints, summaries and reviews. Disconnects could indicate gaming. Liaison Teams consistently rating papers higher than readers could be identified and excluded.
- The stakes of the preprint review process are lower – it’s not an accept or reject decision – so there is less incentive to game the system.
- The scientific peer review system already works on trust.
- To manage power dynamics, the process formalizes the contribution of graduate students and postdocs by having authorship on the Liaison Team (and reviews, if they choose to reveal their identity).
- To minimize implicit bias, we will provide educational materials and a structured review template .
Have you started any collaborations?
Our proposal is a collaboration between Dr. Joseph Wade and LSE, and we are discussing the proposal with biorxiv/medrxiv. Endorsement and collaboration with the NIH, NCBI and Open Access journals would increase value.
Project aims
Background information on current practices
The current system of peer review is highly inefficient. Many papers are rejected from at least one journal, and are re-reviewed at each journal. Reviews and the authors’ responses are rarely made public. A few journals allow authors to transfer reviews from previous submissions, and in the case of eLife, authors can have reviews attached to a preprint. However, there is no widely adopted system to review preprints.
Overview of the challenge to overcome
The goal is to create a widely adopted system for peer review of preprints. The key part is “widely adopted”, since preprint review processes already exist but are rarely used.
The ideal outcome or output of the project
A preprint review portal that is directly linked from preprint servers during the preprint submission process. Authors would suggest up to five possible editors, who would be contacted by email. If an editor agrees to handle a paper, their role would be to recruit three reviewers and to ensure that those reviewers post a review through the review portal. Authors then have the opportunity to respond to the reviews and to submit a revised manuscript to the preprint server.
Description of the intervention
Create a website that authors are directed to (with the option to decline) when uploading a preprint. The authors would then provide a list of possible “editors”, who would be emailed automatically to ask if they would be willing to handle the paper. The first editor to accept would be chosen. The editor would be responsible for identifying peer reviewers, and soliciting 3+ reviews. Reviewers would be selected through the website by the editor, and would submit their reviews through the website. The editor would accept the reviews, or reject them in rare cases where they are inappropriate. The reviews would then be available for the authors to read – the editor’s role is simply to solicit reviews, not assess them, although there could be an option for the editor to write a 2-3 sentence “impact summary”. Reviews would be posted to the preprint server alongside the paper, and the authors would have the option to upload a response letter with a revised version of the paper.
In the initial period (1 year?), there would be no incentive for editors and reviewers. However, after the initial period there would be a credit system. All authors are given a certain number of free review credits, e.g. credit for reviews of three papers. Without credits, a review could not be requested. Editors and reviewers would gain credits for their service. This is a key difference from services like Review Commons that require professional editors, and presumably will require payment.
Plan for monitoring project outcome
- How many authors selected the review process?
- How many editors/reviewers accepted?
- What fraction of papers received three reviews?
- Were the authors satisfied with the process? ( would require a follow-up survey that could be sent along with the reviews)
What’s needed for success
Additional technology development
A web-based portal for preprint peer review is essential, and this needs to be integrated within the existing preprint websites to maximize author buy-in. It could simply be a link to an external site, with a system to post editor-approved reviews on the preprint site, alongside the associated papers. There are currently sites that facilitate preprint peer review, but they are rarely used, and are not linked to the preprint upload process.
Feedback, beta testing, collaboration, endorsement
Ideally, the project would get buy-in from journals, since I anticipate that many authors will want to transfer the reviews to a journal submission. Ultimately, I foresee a world where people only publish on preprint servers, but initially, journals would need to agree to use peer reviews acquired through the preprint peer review pipeline.
Funding
Paying authors, reviewers, and editors, would certainly promote buy-in, and would be fairly cheap. The system would require a website that would need to be designed and maintained.
2 Comments