While preprint feedback is beneficial for the authors, reviewers, readers and other stakeholders, public commenting on preprints has so far remained relatively low. Cultural barriers likely influence participation in public preprint feedback. Authors fear that competitors will leave unfair criticism, or that even fair criticism will bias journal editors and evaluators: while nearly every paper will be thoroughly criticized during journal peer review, the rarity of this feedback being out in the open might lead some to believe that the paper receiving it is especially problematic. Potential reviewers, especially those who rely on more senior colleagues for career advancement, are concerned about retribution for public criticism, or simply harming their reputation by leaving uninformed feedback.
In order to overcome these concerns, we convened a Working Group to discuss how to alleviate the social friction associated with public feedback by developing a set of behavioral norms to guide constructive participation in preprint review. The Working Group brought together relevant stakeholders (researchers, editors, preprint review platform representatives, funders) to discuss the challenges around participation in preprint review and explore what cultural norms could enable and foster further participation in public commentary and feedback.
- FAST principles
- Examples of statements compliant with FAST principles
- Resources
- Working Group participants
FAST principles
The Working Group has developed a set of 14 principles for creating, responding to, and interpreting preprint feedback, clustered around four broad themes: Focused, Appropriate, Specific, Transparent (FAST). The principles are outlined below, and a full description of the FAST principles is available in the preprint ‘FAST principles for preprint feedback.’
The principles apply to the different actors in a preprint feedback environment, including:
Author – Any of those listed in the authorship line of the preprint.
Reviewer – Anyone other than the author(s) who provides comments/feedback on the content and/or claims of the preprint; the feedback may relate to the full paper or to specific parts of the work.
Community – Anyone who engages with the preprint and the feedback on that work. This includes readers but also members of the research community or the public who interact with the authors and/or reviewers, for example, to provide praise, to call out inappropriate behavior or to comment on uses of the preprint and/or preprint feedback by non-specialists.
Examples of statements compliant with FAST principles
Focused
Focus on the content of the preprint
‘This study just reports an observation, there is no mechanistic insight. The authors need to explore and report the mechanism behind the results.’
FAST version – ‘I realize it is beyond the scope of this study, but as possible follow-up steps, you could explore whether the observation may be driven by mechanism x.’
Focus on the science and not the journal
‘This study reports interesting results, but as it stands, does not reach the level required for journal x.’
FAST version – ‘This study reports interesting results, I provide below some suggestions to strengthen the methodology, and to increase the clarity in the interpretation of the results.’
Focus on the scientific and not the personal
‘The authors show sloppy methodology and a poor knowledge of the state-of-the-art in this field.’
FAST version – ‘I have some concerns about the methodological approach in the study; in order to ensure that the conclusions are supported and possible artefacts are ruled out, the study should incorporate controls x and y’.
Appropriate
Polish the tone
‘I don’t get why the authors are not reporting the images behind graph x, and the statistical tests employed in the analyses, what are they hiding?’
FAST version – ‘In order to strengthen the results and increase the reproducibility of the work, I recommend that the manuscript includes the images associated with graph x and report (in the figure legend or the Methods section) which statistical tests were applied.
Reflect on possible biases
‘As others have noted, this is a poorly designed paper and the authors show a poor command of the English language. Also, I disagree with the authors’ interpretation of the findings.’
FAST version – ‘‘I recognize that English may not be the authors’ first language, to increase the clarity in the presentation, I suggest polishing the written text slightly, particularly in sections x and y. This will help ensure the contributions of the work are clearly understood by readers. I have expertise in this area and previously published work that reported findings that do not align to the interpretation reported in the paper. From that perspective, I offer comments in relation to the results and interpretation reported here’
Engage in scientific discourse
‘We did not solicit this feedback on our preprint and will not respond to the comments.’
FAST version – ‘We thank the commenter(s) for reading our work and for provisioning comments. We provide a brief response here on some of the most salient points raised, including where we view some of the comments as falling beyond the scope of our paper.’
Behave responsibly and with integrity
‘That comment on the preprint appears over the top and aggressive, but I do not want to get involved’
FAST version – ‘ ‘I have read the preprint and the feedback by commenter A. While I agree there may be some concerns in relation to the study methodology x, it is important for our feedback to remain polite and constructive, and I encourage commenter A to rephrase the comments.’
Specific
Make it useful
‘‘The study design is flawed, so it is unclear what to conclude from this work.’
FAST version – ‘I have some questions about the study design, in particular, a concern about the lack of a control group involving population without disease x. Without this control group, it does not appear possible to conclude whether the outcome reported is due to the intervention, or to other confounding factors.’
Indicate if critical or optional
‘Here are some comments about the paper, in no particular order.’
FAST version – ‘We outline some questions and comments about the study below, summarising which ones are major items related to study design and methodology, followed by minor items aimed at aiding the clarity in reporting and presentation.’
Assess whether claims are supported by the data
‘I do not trust the conclusions in the paper.’
FAST version – ‘There is a question as to whether the results, as currently reported, fully support conclusion y. The study could incorporate experiment x to adequately support that conclusion, or the text could be revised to ensure the conclusions tightly align to the findings currently included’.
Be candid
‘I note that the manuscript does not report experiment x to support conclusion y, but this is a very interesting study with novel conclusions.’
FAST version – ‘I have a major concern about one aspect of the study. I think an important additional experiment x needs to be provided to support conclusion y; I realize this may require non-trivial additional work, but it appears important to provide this data in order to ensure that no questions arise about the validity of the conclusions’.
Transparent
Sign (or not) your feedback
‘Here are some anonymous comments on the paper.’
FAST version – ‘This review is posted on behalf of a colleague who evaluated the preprint, they have expertise in image data analysis and computational methods.’
Acknowledge oversights
‘We did not solicit feedback on our preprint, so we will not address this comment.’
FAST version – ‘‘We thank the commenter for pointing out this omission in our manuscript, which was due to an oversight. We have now updated the preprint to include the relevant information.’
Credit contributors
‘Here are some comments on the paper.’
FAST version – ‘This review was developed as part of the discussions at our journal club; the comments reflect contributions by all members of the journal club.’
Resources
The Working Group reviewed a number of existing resources and materials around journal peer review, preprint review, and public commenting on scholarly work to inform the development of the FAST principles. We provide a summary of the resources reviewed below, and welcome suggestions for additional relevant resources; if you would like to contribute a resource, please submit it via this form.
Preprint commentary & review
- PREreview has developed a range of resources and templates to guide the development and posting of reviews on preprints. PREreview also has a Code of Conduct for anyone who contributes reviews on the platform.
- Peer Community In (PCI) has a Code of Conduct and a guide for reviewers for PCI registered Reports
Journal peer review
- Buzz Baum developed a reviewer oath, described in his article ‘Reviewing papers as you would like your papers to be reviewed‘. Based on this, a group of 2020 ASAPbio Fellows, developed a slightly modified reviewer oath, available here. Jelena Aleksic and colleagues also develop ‘An Open Science Peer Review Oath‘ going beyond how to review to also outline how to do so in a manner that supports transparent, reproducible and responsible research.
- The Committee on Publication Ethics (COPE) has a set of ‘Ethical guidelines for peer reviewers‘.
- Sense about Science has developed the ‘Peer review – the nuts and bolts‘ guide, aimed at early career researchers.
- Several researchers have shared their approach to completing reviews, guided by a goal to be good and fair reviewers for their peers, see ‘A New “Golden Rule” for Peer Review?’ by Alistair Glen, ‘How to review a paper’ at the RajLab blog and ‘My reviewer oath’ at the Averger blog.
- In the editorial in the Journal of Bone and Mineral Research ‘Editorial Peer Reviewers as Shepherds, Rather Than Gatekeepers’, Joel Boerckel, Lilian Plotkin and Natalie Sims discuss the role of reviewers, noting they should be viewed as shepherds rather than gatekeepers.
Public commentary and critiques on online platforms and social media
- Hilda Bastian has written about the value of a strong culture for post-publication discussion in ‘A Stronger Post-Publication Culture Is Needed for Better Science’, and has provided a guide for how scientists can handle and engage with critiques on their work.
- Veronika Cheplygina and colleagues’ ‘Ten simple rules for getting started on Twitter as a scientist’ includes tips on Twitter etiquette and how to engage constructively on Twitter.
Working Group participants
- Sharon Ahmad (The Company of Biologists)
- Tim Behrens (University of Oxford)
- Gautam Dey (EMBL)
- Sandra Franco Iborra (New York Genome Center)
- Maryrose Franko (Health Research Alliance)
- Samantha Hindle (PREreview)
- Shriyaa Mittal (Harvard Medical School/MGH)
- Sara Monaco (Review Commons / EMBO)
- Dyche Mullins (UCSF)
- Jessica Polka (ASAPbio)
- Iratxe Puebla (ASAPbio)