Our fourth Community Call of 2023 featured a discussion about the use of preprints in research assessment. The adoption of any product or innovation requires the presence of appropriate incentives and preprints are no different. Academia is driven by research and researcher assessment; from funding bodies awarding grants, hiring committees assessing potential new faculty members and wider bodies assessing entire universities such as the REF (Research Excellence Framework). There is increasing recognition of preprints in research assessment which is particularly important and beneficial for early career researchers. Moreover, a significant benefit of preprints is that they enable greater equality in research assessment and increased fairness across the processes.
We were joined by Hannah Hope, who is the open science lead at the Wellcome Trust, and Professor Needhi Bhalla from the University of California Santa Cruz who has previously written about the role of preprints in improving equality, diversity and inclusion (EDI).
Hannah opened the speaker presentations outlining Wellcome’s ambitious plan to spend £16 billion on research between 2022-2023 whilst emphasizing the values that Wellcome place on how they fund research. In discussing preprints, Hannah highlighted that publication bias does not help Wellcome achieve their aims and that they, as a funder, do not want research to be stuck in opaque peer review processes. She then shared some interesting data demonstrating that in 2021, one fifth of known Wellcome funded research was initially shared as a preprint.
Hannah reinforced the great deal of effort that Wellcome have been putting into working with committees to move away from using journal names as a proxy for “quality”, highlighting the benefit of preprints in this effort in addition to their benefits in relation to timely assessments for researchers. As part of moving away from the use of inappropriate proxies, Wellcome requires narrative CVs where applicants must explain how they have generated knowledge which can include policy reviews and datasets. Additionally, Wellcome does not allow “in press” articles, instead requiring preprints which can be read and assessed. Together, Hannah highlighted that these efforts thread diverse outputs throughout the application process.
Needhi discussed the importance of preprints in supporting equity, diversity and inclusion in research assessment. Throughout Needhi’s presentation, she highlighted the importance of an explicit commitment to equity. Needhi began by outlining proven steps that can be taken to improve equity and diversity in recruitment of new faculty. These included rubrics for assessing diversity statements, cluster hiring to reduce feelings of isolation, inviting postdocs to present at seminars and creating mechanisms to avoid relying on flawed assessment proxies. Expanding on flawed assessment proxies, Needhi explained that these included a focus on where a trainee was trained and who they trained with in addition to issues such as relying on where an applicant had published and reliance on impact factors. Echoing the narrative CV approach, Needhi suggested that applicants should identify and contextualize grad/post-grad work in a cover letter as part of efforts to more appropriately assess candidates. Preprints are an “incredibly powerful mechanism” to address some of these issues around assessment and can help address issues relating to delays in review processes and uncouples assessment from some of the flawed proxies such as impact factors and refocus attention to the actual work performed.
One of the most impactful aspects to Needhi’s presentation was the inclusion of data demonstrating gender and historically underrepresented minority status (URM) had significant impacts on career progression. Data from two studies very clearly revealed striking disparity between the representation of women and URM status and proportion of various career stages, reinforcing the notion that academia is not a true meritocracy.
Following these excellent presentations, we had a Q&A session where we addressed what efforts young researchers (bachelors and masters-level students) can take to improve preprint adoption. Needhi highlighted that these are the next generation and that one of the most powerful things they can do is to normalize preprint use, a point that Hannah strongly agreed with. This can include talking about preprints, asking if they are accepted as research outputs and in dissertations. Simply using preprints, such as citing them in essays and dissertations, can also be a small but effective action that can be taken by all career stages. We also discussed where the balance of change sits and if this is primarily with funders, universities or individuals. A resounding “all of them” was the answer. The speakers both emphasized the importance of collective and collaborative efforts in changing the system for the better.
We ended our community call with an interactive whiteboard activity involving all participants. We asked two questions focussed on preprint recognition in research assessment and participants shared their thoughts via sticky notes on pre-defined whiteboard layouts. Our questions were:
Q1. What are the current challenges or issues with preprint recognition in research assessment?
Q2. What actions can researchers, funders or universities take to drive preprint recognition in research assessment?
This interactive element was a flurry of activity from participants. In response to our first question, peer review and quality of preprints emerged as key issues. In the background, issues around training and true understanding of the peer review process were also identified as areas that needed improvement. Our second question focussed on what actions could be taken. Quality and training was again a component and fell into our “high impact, high effort” area. Another recurring element centered on transparency and normalizing preprints in assessment, graduation and hiring.
We thank our two speakers for a great insight into how preprints are currently used (and can be used) in the assessment of research and researchers. We also thank all participants who made our interactive element so lively. We look forward to sharing our efforts in improving preprint use for research and researcher assessment soon and welcome any suggestions for resources or activities that would further these efforts.
