Skip to navigation menu Skip to main content Skip to footer

Welcome to the new ASAPbio website! See what’s on the roadmap for 2025.

New to preprints and open peer review? Explore our resource library.

CrowdPeer

Organizer

Louis Fisher and Huw Williams (CrowdPeer)

Website or social media links

Current stage of development

Ongoing Project

Project duration

Indefinite

Update

How has your project changed?

  • Added a code of conduct to guide intended use.
  • Added reviewer guidance section describing what comments are expected within the comments section.
  • Further developed reviewer recognition model:
    • Broken down how score is awarded/removed.
    • Started to develop more refined metrics of reputation for display on public profile pages.

Have you integrated any feedback received?

Above changes reflect feedback around:

  • How to moderate the platform and ensure a constructive environment.
  • Understanding of what the different comment categories comprise.
  • What metrics reviewers find useful to build their reputation.

Project aims

Overview of the challenge to overcome

Reviewers lack incentives to comment on preprints. While preprints are meant to speed up the research process, without comments and filtering it is impossible to quickly identify the most impactful preprints and thus the preprints fail to bring about their desired goal.

Better engagement with, and utilisation of preprints relies on identifying a meaningful way to reward reviewers for their contributions.

The ideal outcome or output of the project

  • To encourage more potential reviewers to comment on preprints.
  • To improve engagement between reviewers.
  • To deliver the faster and more thorough peer review by opening up the review process and utilising crowd intelligence.

Description of the intervention

A platform for the open review of preprints that utilises a standardised review structure. An upvoting system allows the community to vote on the value of individual contributions. Reviewers can then leverage their CrowdPeer reviewer reputation for their career progression.  The platform, with its standardised review structure, can additionally act as an educational resource for reviewers.

Plan for monitoring project outcome

  • Total number and distribution of comments.
  • Number of reviewers leaving comments.
  • Number of comment replies.
  • Number and distribution of upvotes/downvotes.

What’s needed for success

Additional technology development

We will need to develop mechanisms for the extraction of value from user comments. This will be used for:

  • Better profiling of reviewers – allowing them to utilise their reviewer reputation for career progression.
  • Better profiling of preprints for the filtering and identification of valuable preprints based on comments.
  • Better profiling of comments allowing indexing based on their content.

Feedback, beta testing, collaboration, endorsement

We are looking for feedback on:

  • The universal structure used for reviews on CrowdPeer.
  • Upvoting and comment replies functionality.
  • The willingness of reviewers to put their name to reviews.
  • Whether our position for building reviewer recognition through comments is fit for purpose.

Beta testing:

  • Beta testing by more reviewers on our open platform and journal clubs within our closed sessions.
  • Use of the platform for the education of early stage researchers.

Endorsement:

  • Endorsement and acknowledgment by institutions such that they recognise a reviewer’s score/reputation as a value metric for career progression.

Funding

None

0 Comments

Leave a Comment

Your email address will not be published or shared. Required fields are marked *.