In conversations about preprints in the UK, the question is often raised: ‘are preprints included in REF?’ In brief: yes. This is most likely to be applicable for any research manuscript that is prepared close to the REF2021 submission deadline, and is deemed to be amongst your best work in the current cycle, but which would otherwise not be eligible for REF2021 due to not having time to be published in a journal before the deadline.
The Research Excellence Framework (or REF) is the exercise the UK higher education funding bodies undertake periodically to assess UK research institutions for excellence and impact of research outputs. The REF scores determine allocation of approximately £2bn/year national funding for research, so REF is a major driver of UK institutional policy and researcher behaviour. Learn more about REF at the end of the post.
Below, we describe how preprints can be included in REF submissions, with extracts from the official REF guidance.
Preprints are valid research outputs for REF2021
Research outputs count towards 60% of an institution’s REF score, and each researcher must provide at least one and no more than five outputs to demonstrate the quality of the institution or unit’s work from 2014 to 2020.
Preprints are eligible for submission as outputs to REF2021:
“The funding bodies recognise that many researchers derive value from sharing early versions of papers using a pre-print service. Institutions may submit pre-prints as eligible outputs to REF 2021 (see Annex K).” – Paragraph 238 of the REF Guidance on Submissions, PDF available from https://www.ref.ac.uk/media/1092/ref-2019_01-guidance-on-submissions.pdf
Preprints are referred to as “working papers” in Annex K and described as “Research papers disseminated to encourage discussion and suggestions for revision. This may be through pre-print dissemination, lodging in an institutional repository or self-publication for distribution.” In Annex K, the process of research itself is also defined: “a process of investigation leading to new insights, effectively shared”. Other outputs eligible for REF include book chapters, conference contributions, patent applications, software and journal articles.
Should I submit a preprint as an output to REF2021?
Consider the manuscript(s) you would submit to a journal close to the REF2021 submission deadline. Is this work that you would count amongst your five best outputs in 2014-2020 but it’s not likely to be published in a peer-reviewed journal before the submission deadline? If so, consider preprinting this work in order to be able to cite it for REF.
Outputs can be updated before final REF submission. Using preprints to demonstrate productivity may help relieve the time pressure of being published in any journal before the REF deadline. You may be able to update the version submitted to REF between your institution’s initial and final submissions: “Where a deposited output is later replaced or augmented following its initial deposit, the updated manuscript may be deposited in place of the originally deposited output.” (Paragraph 240, REF Guidance on Submissions).
Preprints do not count towards the REF open access requirement. For a research output posted on a preprint server to count towards the open access requirement, it must be the version of the manuscript accepted for publication (Paragraph 238, REF Guidance on Submissions; see also https://unlockingresearch-blog.lib.cam.ac.uk/?p=2115). Notably, bioRxiv discourages posting the version of a manuscript that has been accepted for publication at a journal (see https://www.biorxiv.org/about-biorxiv), while several community-moderated OSF Preprints servers do accept author-accepted manuscripts, otherwise known as post-prints.
Only outputs which have been ‘accepted for publication’ (such as a journal article or conference contribution with an ISSN) are within the scope of the REF 2021 open access policy. To take into account that the policy intent for ‘open access’ is met where a pre-print version is the same as the author-accepted manuscript, we have introduced additional flexibility into the open access requirement: if the ‘accepted for publication’ text, or near final version, is available on the pre-print service, and the output upload date of the pre-print is prior to the date of output publication, this will be considered as compliant with the open access criteria (deposit, discovery, and access).” – Paragraph 238 of the REF Guidance on Submissions, PDF available from https://www.ref.ac.uk/media/1092/ref-2019_01-guidance-on-submissions.pdf
Any research output may only be included in one REF cycle unless there have been substantial modifications, so any work included in REF2014 may not also be submitted for REF2021. This rule applies regardless of the version of work: any preprint included in REF2014 cannot be included in REF2021, even if the peer-reviewed version (i.e. journal publication) is now available. The same may apply for REF2021 and any future exercise, although the policy details for any future exercise have not been developed. Consider whether any preprint posted before the submission deadline for REF2021 would still be an output that you may wish to submit for a future assessment exercise (for example, in seven years’ time).
“An output first published in its final form during the REF 2021 publication period that was ‘pre-published’ in the previous publication period – whether in full in a different form (for example, as a pre-print), or as a preliminary version or working paper – is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to REF 2014.” — Paragraph 258, REF Guidance on Submissions
As with all research outputs, if the author has moved institution, REF provides guidance on which institution should reference the preprinted output in their REF submission (see Paragraph 214, REF Guidance on Submissions).
How will preprints be evaluated as outputs for REF2021?
According to the REF Panel Criteria and Working Methods (PDF available from https://www.ref.ac.uk/publications/panel-criteria-and-working-methods-201902/), “[the] sub-panels will assess the quality of submitted research outputs in terms of their ‘originality, significance and rigour’, with reference to international research quality standards.”
The general principle of REF is that preprints should be treated fairly in comparison to other outputs:
“Does the REF assessment process distinguish between research outputs on the basis of mode of publication, place [of] publication or publisher? No. The REF is governed by a principle of equity and is committed to the fair and equal assessment of all types of research and forms of research output.” — https://www.ref.ac.uk/about/faqs/
Dr Kim Hackett, REF Director, tells us, “The principle of equity (all types of research and all forms of research output across all disciplines shall be assessed on a fair and equal basis) is one of three principles governing the conduct of the REF. The assessment structures we have in place for the REF support the delivery of these principles, with a key role of the main panels being to ensure the consistent application of the assessment standards and criteria.”
Specifically, the guidance states that outputs will not be judged by their place of publication or form of output:
“The main panels welcome all forms of research output that fulfil the eligibility criteria for the REF (set out in Part 3, Section 2 of ‘Guidance on submissions’). All forms of output, in any language, will be considered equitably, with no distinction being made between the type of research or form of output submitted. The sub-panels will neither advantage nor disadvantage any type of research or form of output. The main panels encourage submitting institutions to refer to the glossary of output types for information on the categories under which outputs may be submitted for assessment (see ‘Guidance on submissions’, Annex K).”
“No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is published or the medium of its publication.” – Paragraphs 206-207, REF Panel Criteria and Working Methods.
The information provided in the survey of intended submissions later this year will be used to inform further appointments of sub-panel members and assessors, in order to “ensure submissions are assessed by panellists with appropriate expertise”.
Finally, all panel members and assessors will receive training on unconscious bias, as part of efforts by REF to promote equality and diversity in careers (see Paragraph 45, REF Guidance on Submissions).
How else could preprints count in REF2021?
While research outputs determine the largest proportion of the REF score, 25% of the score is assigned based on impact of the research. To assess this, institutions prepare impact case studies including up to six references and detailing impact based on:
- Reach, defined as “the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact. (It will not be assessed in geographic terms, nor in terms of absolute numbers of beneficiaries.)”
- Significance, defined as “the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries.”
Has a preprint influenced the impact of your work? Consider whether the preprinted version of your manuscript has contributed towards the reach and significance of a particularly high impact piece of work produced between 1 August 2013 – 31 July 2020. If so, advise the person who prepares your institution’s impact case studies.
This may be less likely to apply when your preprint posting date is close to the REF submission deadline. Nevertheless, you could take note of any major contributions of sharing the preprint of your manuscript to the reach and significance of the impact of your work, as evidence for REF2021 or a later assessment exercise.
About REF2021
The next REF will be in 2021, and institutions are now preparing their submissions: [the survey of submissions intentions opens in September 2019 and is to be completed by December 2019](https://www.ref.ac.uk/publications/institutions-invited-to-complete-the-ref-survey-of-submission-intentions/). Final submissions to REF2021 are due late November 2020.
Research outputs count towards 60% of an institution’s REF score. For REF2021, outputs must have been produced between 1 January 2014 and 31 December 2020. The institution selects the best outputs for inclusion in the institution’s portfolio, with the total number of outputs submitted equal to 2.5 times the number of full-time equivalent staff ‘with significant responsibility for research’ and with 1 to 5 outputs submitted per included staff member. The process of selection may begin with each researcher considering their five best research outputs in the 2014-2020 period, although this depends on how your research institution manages the process.
For life and biomedical sciences (REF Panel group A), outputs will be assessed for degree of excellence (star rating) using criteria including “scientific rigour and excellence, with regard to design, method, execution and analysis”, “significant addition to knowledge and to the conceptual framework of the field”, “the scale, challenge and logistical difficulty posed by the research” and “applicability and significance to the relevant service users and research users” (for full criteria, see Paragraph 198, REF Panel Criteria and Working Methods). Further, reproducible science and other best practices are explicitly welcomed by this panel, including the use of registered reports and sharing data sets, materials and analytic code (Paragraph 200, REF Panel Criteria and Working Methods).
The life and biomedical sciences panels “will use citation information, where appropriate and available, as part of the indication of academic significance to inform their assessment of output quality” (Paragraph 201, REF Panel Criteria and Working Methods), with the guidance that citation data is seen as only one indicator and one that is not always reliable, absence of citations is not an indicator of lack of quality, citation data has implications in terms of equality, and that any panels using citation data will be receive further guidance from the Forum for Responsible Research Metrics (see Paragraphs 246-247, REF Panel Criteria and Working Methods).
After outputs (60%) and impact (25%), the remaining 15% of an institution’s REF score is based on the research environment “in terms of its ‘vitality and sustainability’, including the approach to enabling impact from its research, and its contribution to the vitality and sustainability of the wider discipline or research base.” – REF Panel Criteria and Working Methods
Full guidance documentation on REF2021 is available from https://www.ref.ac.uk/guidance/, including:
- REF Guidance on Submissions, downloadable from https://www.ref.ac.uk/publications/guidance-on-submissions-201901/
- REF Panel Criteria and Working Methods, downloadable from https://www.ref.ac.uk/publications/panel-criteria-and-working-methods-201902/
- The process and output requirements are described in the slide deck available from https://www.ref.ac.uk/media/1110/final-guidance-and-criteria-slide-pack.pdf.
- Who will assess REF2021? See https://www.ref.ac.uk/panels/.
Note that in REF documentation, preprints are referred to as “pre-prints” and “working papers”.
All documentation was accessed and quoted as of July 31 – August 1, 2019. We advise all researchers check the REF guidance directly for confirmation and updates.
—
We thank Dr Stephen Eglen (Reader in Computational Neuroscience, University of Cambridge) and others for discussions, advice and information.
2019-08-09 – This post was updated to:
- Clarify that the requirement for content posted on a preprint server to be the peer-reviewed version relates to REF’s open access policy only, preprints (not peer-reviewed) are eligible as outputs irrespective of this. Many thanks to those who raised this issue for clarification.
- Amend some factual details about the REF exercise and process, and include a quote from the REF Director.
