As Ivan Oransky has noted, ‘science is a proposition and a conversation and an argument’ ; feedback and discussion around scientific reports are integral parts of the scientific process. Preprints are a vehicle for this discussion as they allow any member of the community to read and feedback on the latest research.
Despite the potential benefits, a culture of private feedback has created barriers to a public conversation on research papers, leading to uncertainties around public feedback on preprints: Will the authors welcome comments on their paper? What if I get something wrong in my evaluation? Will I lose favor with that senior researcher if I post a query or critique on their preprint?
In order to foster broad and inclusive participation in this conversation on preprints, we need a positive culture around preprint feedback, but what does this look like in practice? This is the question that the ASAPbio preprint review cultural norms Working Group is tackling. The group is discussing what behaviors reflect the preprint feedback culture we would like to see, with the goal of developing a set of norms for all participants in preprint feedback.
The Working Group is pleased to share an initial draft of principles for creating, responding to, and interpreting preprint feedback, clustered around four broad themes: Focused, Appropriate, Specific, Transparent – FAST. The table below provides elaboration for each of the FAST themes, and we invite feedback from the community on these principles.
|Respect the focus of the paper – additional work may be suggested where it is relevant to the study or if it provides an opportunity for collaborations, but requests that go beyond the initial scope or that relate to perceived impact should be avoided or clearly designated as optional suggestions for a future study, it is not a reviewer’s responsibility or remit to impose research priorities on the authors.||X|
|Focus on the science and not the journal – For work posted as a preprint, the feedback should be “journal agnostic” and focused on the scientific work without specific journal criteria in mind.||X|
|Focus on the science and not the people – Feedback must be objective and focused on the paper or any commentary about the work, not on the authors; there should not be statements that imply specific practice or behaviors by the authors (e.g. misconduct, deliberate omission of a citation to another work).||X|
|Focus on the science and not the people – Be analytical about criticisms on your work, irrespective of who is making the critique; the feedback should be about the work, as an author, do not take critiques of your work personally.||X|
|Tone – The tone of the feedback should always be polite, respectful and constructive, toward the authors and to anyone engaging in the public discourse about that work. Commenters should be mindful of the impact language choices might have on the authors or readers. If you would not do or say something in person, reconsider whether it should be included in your comments, or reflect on whether the comments should be revised to ensure feedback remains constructive and polite.||X||X||X|
|Motivation – Commenters should consider their motivation for providing feedback. Commenters should reflect on whether their perception about the work’s interest may influence their comments, and remind themselves that there should be a focus on constructive objective feedback; if that is not possible, reconsider whether you should provide a review.||X|
|Bias – Commenters should reflect on any biases before, during, and after commenting, including aspects related to their own background, professional position, or perception about the work’s interest; the evaluation of the work should be the same irrespective of who the authors are, their background, seniority etc. Reflect on any potential competing interests or other factors that may influence your feedback (e.g. earlier reactions or critiques on the study on social media). If you think any biases or competing interests will influence the evaluation, reconsider whether you should provide the review.||X|
|Scientific discourse – Review and public commentary on papers is a form of scientific discourse and should be approached as such by all parties who engage in it; authors and commenters should be willing to discuss their work/assessment with others (readers, authors, other reviewers, editors), provided the exchange follows expected norms and behaviors.||X||X||X|
|Responsibility – There is a collective responsibility to call out inappropriate behaviors or critiques. We should favor criticism and disfavor bad reactions to it, as a community we have a collective responsibility to protect those subject to unreasonable criticism.||X||X||X|
|Integrity – Those engaging in commentary should behave with integrity, following ethical conduct expected in all research activities. Provide feedback without seeking to gain benefit from it, critiques aimed at gaining citations for the commenter’s own work, or seeking to damage a competitor are not appropriate.||X||X||X|
|Useful – Feedback should be specific, clear, actionable, and useful for the authors to improve the work. Comments should be outlined in a logical manner and substantiated with evidence. The clearer the feedback, the easier it will be for authors and readers to understand it.||X|
|Critical vs optional – Feedback may include a summary of major strengths and weaknesses of the study; to ensure it is as useful as possible for authors and readers, it is important for the feedback to designate which critiques are major and affect the rigor of the study (flaws/controls) and which are optional or minor issues.||X|
|Are claims supported by the data – One important element of reviews is to evaluate whether the study claims are supported by the data, and it is relevant for public reviews to contain comments about this; if the claims are not supported, the review can include requests for the claims to be revised, or for new data to support the conclusions.||X|
|Candid – Be honest and candid, if you feel the paper needs improvement, it is appropriate to indicate so, supported by the reasons for this and possibly examples/suggestions on how to improve the issue.||X|
|Transparency – Those providing feedback should do so transparently; where applicable, comments on the paper should be accompanied by information on any potential competing interests and the limits to the commenter’s expertise.||X|
|Signing (or not) of reviews – Public feedback is valuable to both authors and the community – it increases awareness and transparency about the scientific process; post your feedback publicly; if you are comfortable doing so, you may sign your comments.||X|
|Acknowledge errors – Embrace feedback, it is good practice to acknowledge any errors in your work, either in the paper or in a review/comment.||X||X||X|
|Credit contributors – Disclose and credit colleagues who contributed to the review.||X|
Author – Any of those listed in the authorship line of the preprint.
Reviewer – Anyone other than the author(s) who provides comments/feedback on the content of the preprint, the feedback may relate to the full paper or to specific parts of the work.
Community/bystander – Any member of the research community or the public who witnesses or interacts with the exchange related to the feedback on the work, but does not provide a scientific evaluation of the preprint content.
The principles above build on existing resources and materials around journal peer review, preprint review, and public commenting on scholarly work – a list of the resources the Working Group reviewed during the development of these principles can be found here.
Importantly, we have developed these principles bearing in mind that in an environment of preprint feedback, there is a broader set of actors compared to the journal peer review process. The principles touch on behaviors expected not only by those contributing feedback (via formal reviews, comments or on social media), but also from preprint authors and any member of the community (‘bystander’) who may witness or interact with the exchange.
We invite feedback on the FAST principles. Are there any themes missing? Any items that need clarification or which you disagree with? Please comment on this post or directly on this document.
We will host a session as part of the #FeedbackASAP meeting on July 21 to solicit feedback on these themes and on how we can put the FAST principles into practice. Join us on July 21 to help build a positive preprint feedback culture.
Register for the #FeedbackASAP meeting here.
Cannot attend? You can share your feedback on drivers, barriers and risks toward a positive preprint feedback culture on this Miro board, share your thoughts on Twitter or email your comments to firstname.lastname@example.org.