On December 4, 2020, ASAPbio hosted #PreprintSprint presentations, where 16 projects (down from the 21 presented at the kickoff, partially due to mergers and collaborations) updated approximately 100 attendees on what they had accomplished during the sprint. During the 20 days of the sprint, 9 projects started new collaborations, 14 updated their proposal to incorporate feedback, and 2 merged their efforts.
Project overviews
Projects participating in the sprint approached the problem of encouraging more feedback on preprints from a few different angles.
Standards & interoperability
At the level of the preprint ecosystem, preprint servers and overlay review/curation platforms currently lack interoperability and standards. In terms of individual preprints, a history of evaluation is not visible to readers. Further, preprints are associated with content types that vary in format, versions, and location. These challenges present opportunities to build protocols that transparently and robustly link preprints to reviews.
COAR in collaboration with many partners (PREreview, Grassroots Journals, PCI, eLife, etc) aims to develop a platform-agnostic approach to bi-directionally indicate preprint content to overlay services, which will enhance visibility and usability of preprints and their reviews.
Thomas Guillemaud, Denis Bourguet, and Marjolaine Hamelin (INRAE, France) proposed to display evaluation history and metadata, which could provide recognition for community reviewers and editors.
Daniel Mietchen proposed to develop best practice recommendations based on preprint content type. The project begins with documenting preprint contents from a range of preprint servers.
Scite proposed to use smart citation badges to provide an overview of which preprints have been cited and how they have been cited, whether supported or disputed. Potential reviewers could be identified from scite badges for journal and preprint overlay services.
Platforms
Active and new platforms shared their strategies for building preprint commenting incentives, such as enhancing reviewer recognition, increasing review visibility, broadening community participation, and providing flexible curation workflow and review modules.
Unfold Research is a web publishing platform with integrated points-based mechanics that is in active development. Unfold Research proposed that researchers gain recognition by reputation points, and earn a salary based on their contributions.
Crowdpeer is an open preprint peer review platform that allows the community to evaluate the contributions of others. During the sprint, CrowdPeer integrated feedback on moderation and developed a code of conduct, as well as refined the reviewer recognition model.
Peeriodicals provides a flexible platform for researchers to curate their overlay journals. Periodicals aim to increase publicity in building engagement. During the Sprint, the linking of publications to their comments has been implemented in the PubPeer database.
The NCRC is a centralized public resource for curating and reviewing SARS-CoV-2 and COVID-19 research. The NCRC seeks to direct high-quality curation to its target audience. During the PreprintSprint, the NCRC began exploring collaborations with eLife and PubPeer.
Thomas Lemberger consolidated the proposals for Early Evidence Base (EEB), a platform to surface refereed preprints, combined with machine content curation, and automated aggregation, ranking, and filtering. EEB is to launch soon in December 2020.
BioMed News (Bims) combines researcher selection of machine learning ranked abstracts of newly published papers in PubMed to rapidly disseminate research. Bims seeks to increase users, and could potentially incorporate preprint datasets that are interoperable with PubMed.
Stencila proposes to provide a flexible platform for authors to receive and incorporate feedback on working documents in several formats, make it available to review in Google docs or GitHub, and allow the authors to publish new version(s) of the preprint. During the PreprintSprint, Stencila developed a new demo. You can sign up to provide input for user workflow.
Grassroots Journals is an open post-publication peer review system, based on WordPress Multisite, where every review is a blog post. Details of the design can be found a: https://grassroots.is
Training programs and communities
A great barrier in preprint peer review and curation is the lack of engagement. Several teams tackled the challenge with community building and reviewer mentorship programs.
TCC Africa proposes to use best practices and technology to build peer review communities across Africa. TCC is working with several partners (Eider Africa, COAR, Decentralize Science, PREreview, and Sciety) to build reviewer incentives, journal clubs, and reviewer communities.
Outbreak Science Rapid PREreview (OSrPRE) proposes to engage the community by training and supporting volunteer reviewers and to build a seamless integration of community reviews with journal editorial reviews. During the Sprint, OSrPRE began disseminating a survey to understand the editors’ needs in order to build trust in the reviews and to connect to reviewers.
Through PREreview’s Open Reviewers pilot, early career researchers can gain training and mentorship to become expert reviewers. In addition, PREreview will provide training for researchers to facilitate live-streamed journal clubs, to constructively discuss preprints, and reduce the barrier for review participation.
An initiative from the University of Oxford and Mount Sinai proposed to create cross-institutional journal club hubs to collaboratively assess preprints. After the #PreprintSprint Kickoff OxMS ran their first joint live online journal club with 70 attendees. OxMS journal clubs anticipate future training of their members with the PREreview Open Reviewer program.
Incentive schemes and marketplaces
Novel approaches to incentivize peer review have been proposed at the #PreprintSprint, such as engaging community reciprocity and building a new researcher-driven review pipeline.
Michele Avissar-Whiting proposed a new framework in which authors who submit a preprint are encouraged to ‘pay it forward’ by providing a review for another preprint in their field. During the Sprint, the proposal incorporated PeerScout for review/preprint matching and API integration with PREreview.
I Owe the Academy (IOTA) is a minimal peer review exchange service, which aims to align reviewer and editor incentives to foster community reciprocity. Reviewers can submit tokens as pledges to conduct future reviews. Tokens can be obtained by publication services in order to request reviews. During the Sprint, IOTA developed a white paper to seek feedback, funding, and endorsement.
The Life Science Editors and Joseph Wade proposed similar ideas during the Kickoff, and combined forces to put forth the proposal for a researcher-driven open peer review portal. The team has begun discussions with bioRxiv and medRxiv, and seek additional endorsements.
Award-winning projects

Three judges with expertise and direct experience in preprint commenting (Carole J. Lee, Associate Professor of Philosophy at the University of Washington; Christian Gonzalez-Billault, Professor of Cell Biology at the University of Chile and DORA Advisory board member; and Sandra Iborra Franco, Postdoc at Columbia University and ASAPbio Fellow) deliberated to award recognition to outstanding projects. Together with a popular vote among attendees, the prizes awarded were:
- Best in show (USD 1500) – Strongest proposal overall
- Transforming Peer Review through Mentorship and Community Engagement, PREreview
- Increasing representation (USD 1000) – Most likely to increase diverse representation in review or curation
- Building capacity for preprint peer review and curation in Africa, TCC Africa & AfricArXiv
- Promoting constructiveness (USD 1000) – Most likely to promote constructive reviews
- ‘Take a Penny, Leave a Penny’, Research Square
- Project development (USD 1000) – Most developed during the sprint
- Early Evidence Base, EMBO
- People’s choice (USD 500) – Most popular votes
- Building capacity for preprint peer review and curation in Africa, TCC Africa & AfricArXiv
For more details, view the slides and the recording.
Other coverage of the Sprint
Generation Research. “ASAPBio #PreprintSprint Presentations.” Generation Research, 2020. https://doi.org/10.25815/BC5T-1178.