Organizer
Louis Fisher and Huw Williams (CrowdPeer)
Website or social media links
Current stage of development
Ongoing Project
Project duration
Indefinite
Update
How has your project changed?
- Added a code of conduct to guide intended use.
- Added reviewer guidance section describing what comments are expected within the comments section.
- Further developed reviewer recognition model:
- Broken down how score is awarded/removed.
- Started to develop more refined metrics of reputation for display on public profile pages.
Have you integrated any feedback received?
Above changes reflect feedback around:
- How to moderate the platform and ensure a constructive environment.
- Understanding of what the different comment categories comprise.
- What metrics reviewers find useful to build their reputation.
Project aims
Overview of the challenge to overcome
Reviewers lack incentives to comment on preprints. While preprints are meant to speed up the research process, without comments and filtering it is impossible to quickly identify the most impactful preprints and thus the preprints fail to bring about their desired goal.
Better engagement with, and utilisation of preprints relies on identifying a meaningful way to reward reviewers for their contributions.
The ideal outcome or output of the project
- To encourage more potential reviewers to comment on preprints.
- To improve engagement between reviewers.
- To deliver the faster and more thorough peer review by opening up the review process and utilising crowd intelligence.
Description of the intervention
A platform for the open review of preprints that utilises a standardised review structure. An upvoting system allows the community to vote on the value of individual contributions. Reviewers can then leverage their CrowdPeer reviewer reputation for their career progression. The platform, with its standardised review structure, can additionally act as an educational resource for reviewers.
Plan for monitoring project outcome
- Total number and distribution of comments.
- Number of reviewers leaving comments.
- Number of comment replies.
- Number and distribution of upvotes/downvotes.
What’s needed for success
Additional technology development
We will need to develop mechanisms for the extraction of value from user comments. This will be used for:
- Better profiling of reviewers – allowing them to utilise their reviewer reputation for career progression.
- Better profiling of preprints for the filtering and identification of valuable preprints based on comments.
- Better profiling of comments allowing indexing based on their content.
Feedback, beta testing, collaboration, endorsement
We are looking for feedback on:
- The universal structure used for reviews on CrowdPeer.
- Upvoting and comment replies functionality.
- The willingness of reviewers to put their name to reviews.
- Whether our position for building reviewer recognition through comments is fit for purpose.
Beta testing:
- Beta testing by more reviewers on our open platform and journal clubs within our closed sessions.
- Use of the platform for the education of early stage researchers.
Endorsement:
- Endorsement and acknowledgment by institutions such that they recognise a reviewer’s score/reputation as a value metric for career progression.
Funding
None
Organizer
Louis Fisher and Huw Williams (CrowdPeer)
Website or social media links
Current stage of development
Ongoing Project
Project duration
Indefinite
Update
How has your project changed?
Have you integrated any feedback received?
Above changes reflect feedback around:
Project aims
Overview of the challenge to overcome
Reviewers lack incentives to comment on preprints. While preprints are meant to speed up the research process, without comments and filtering it is impossible to quickly identify the most impactful preprints and thus the preprints fail to bring about their desired goal.
Better engagement with, and utilisation of preprints relies on identifying a meaningful way to reward reviewers for their contributions.
The ideal outcome or output of the project
Description of the intervention
A platform for the open review of preprints that utilises a standardised review structure. An upvoting system allows the community to vote on the value of individual contributions. Reviewers can then leverage their CrowdPeer reviewer reputation for their career progression. The platform, with its standardised review structure, can additionally act as an educational resource for reviewers.
Plan for monitoring project outcome
What’s needed for success
Additional technology development
We will need to develop mechanisms for the extraction of value from user comments. This will be used for:
Feedback, beta testing, collaboration, endorsement
We are looking for feedback on:
Beta testing:
Endorsement:
Funding
None