Chris Baur, Marek Łaska, Chrissy Prater, Michele Avissar-Whiting (Research Square)
Website or social media links
Current stage of development
Since the Kickoff, we have thought through 1) what a minimal viable product looks like for demonstrating proof of concept and 2) how to concretize what started out as a relatively abstract idea: actively encouraging researchers who upload preprints to “give back” by conducting a review on a preprint in their field. On its face, the proposal is simple – request that preprint authors complete reviews on preprints, and make it easy for them to do so; its practical implementation needs to be seamless and intuitive in order to serve as an effective test.
How has your project changed?
For the purposes of a pilot that is technically feasible within a short timeframe and also affordable to implement, we have dropped the “status as credit” portion of the proposal. The rationale is that peer review has functioned as an institution without such a concept for a long time, so it isn’t a critical component for demonstrating proof of concept for this proposal. We would expect further development of the idea to involve the integration of a credit/recognition system, but it is important to first validate that an appetite for preprint review exists in its absence.
Have you integrated any feedback received?
Yes. At the kick-off, it was mentioned that the reviewer-matching technology used by eLife, Peer Scout, is open source. Therefore, we could leverage this tool to query the database of preprints in need of review on the basis of keywords derived from the uploaded preprint. The app would output a list of suitable preprints on PREreview, and the user would be able to click to view and choose one to complete.
Have you started any collaborations?
Yes. Research Square is discussing an API integration with PREreview that would allow for full or rapid review of preprints on our platform. PREreview’s new upcoming platform for author-driven requests for community feedback would make the PREreview platform a suitable aggregator for preprints in need of review — a critical piece of this project.
Background information on current practices
Preprints are often shared with the specific intention of drawing scrutiny and feedback from the scientific community, yet engagement in the form of substantive comments or reviews has been low.
Overview of the challenge to overcome
The institution of academic peer review is dominated by journals that use editorial moderation to perform peer review in the context of journal scope and impact. Reviewer capacity is already heavily strained by the volume of submissions needing review at these journals. Given that a number of incentive structures already exist to encourage reviewers to volunteer their time to traditional journal peer review, how can the prospect of preprint review compete for this limited resource?
The ideal outcome or output of the project
A marked increase in reviews conducted on preprints and commensurate use or consideration of these reviews by journal editors.
Description of the intervention
Leverage the fact that depositing a preprint is free to encourage authors to ‘pay it forward’ by providing a preprint review for someone in their field who
a) is not currently under review at a journal,
b) has specifically indicated interest in having their preprint reviewed and
c) (optionally) has accrued interest in their preprint from others in the community (some simple counter could be implemented)
The angle here is that people who are provided a free service with some inherent value should be eager to espouse the values that the service represents. In this case, preprints represent a shift toward lowered barriers and greater transparency in the sharing of research, and preprint review represents an important development toward post publication peer review. There is a community of researchers who see great value in this proposition and would understand that the shift they long for will not actualize without their participation. Recent data has shown high levels of support for preprints, particularly among younger scientists. Partnering with institutional journal clubs or postdoc groups could be a way to engage an already bought-in cohort who would also benefit from the experience, start building their reviewer profile early, and get plugged into a range of peer review mentorship resources. We should start with them and encourage their early adoption, framed as participation in a grassroots movement.
Further development of this idea would capitalize on the concept of ‘status’, wherein regular contributions of reviews that meet a certain standard (perhaps those defined by PREreview) pay off in the form of recognition, for example a series of badges or visual indicators of contribution. This is, of course, the basis of the Publons project; however, we believe that Publons’ distance from the source has limited its impact. This recognition system would “live” in the same space as the review and its preprint.
Plan for monitoring project outcome
- Tracking engagement from reviewers and journals during a 6-month trial period
- Tracking review-related metrics (TAT, length, quality)
- Collection of qualitative feedback via surveys
What’s needed for success
Additional technology development
- Messaging at the point of preprint submission
- Curation and surfacing of preprints needing review, by subject matter/expertise
- Design and implementation of a platform-agnostic tier system for reviewers (badging/tagging)
- High-profile sponsorship of the initiative and comms
Feedback, beta testing, collaboration, endorsement
This project will require collaboration among preprint platforms that wish to be involved in the trial and at least one organization that has already established a system for preprint review. Mechanisms for feedback and beta testing will be agreed in the course of project planning.
Product management/Design: ~15k
Project management: ~15k