APPRAISE (A Post-Publication Review and Assessment In Science Experiment)

By Michael B. Eisen

Investigator, Howard Hughes Medical Institute
Professor of Molecular and Cell Biology, UC Berkeley
mbeisen@gmail.com, @mbeisen

With the rapid growth of bioRxiv, biomedical research is entering a new era in which papers describing our ideas, experiments, data and discoveries are made available to our colleagues and the public without having undergone peer review. This inversion of the traditional temporal relationship between peer review and publication is good for science and anyone who cares about or benefits from its accomplishments. But it also has rendered the existing infrastructure we have for carrying out peer review – scientific journals – obsolete.

It is incumbent upon to scientific community to seize this opportunity, reinventing the ways we assess our research outputs and each other to make them more fair, efficient and effective. This will require new ideas and a lot of experimentation. In that spirit I describe here a new project – called Appraise – that is both a model and experimental platform for what peer review can and should look like in a world without journals.

Background

Scientific journals once served a useful purpose. In the age of ink and paper, it made sense to bundle manuscripts based on audience and to screen submissions before incurring the cost of printing dissemination via the postal service. Remarkably, while the logic for organizing science communication this way evaporated with the birth of the Internet, scientific journals have survived their obsolescence, their anachronistic role in distribution and collation sustained by the stranglehold they have on the process of peer review.

The mainstreaming of preprints in biomedicine has brought the distribution of scientific knowledge into the 21st century. Preprints give scientists control over when to publish their work, eliminated needless delays and barriers to access that impede scientific process and the realization of its benefits.

But we are currently in a weird situation where the scientists increasingly use preprints to communicate their ideas and discoveries, but still rely on Gutenberg era journals as the sole vehicle for assessing the validity, import and audience of works of science, and judging the contributions of the researchers whose ideas and effort they represent.

This is bad for science. Journals are expensive – we spend around $10 billion per year to keep them in business. They are slow – it takes an average of nine months to publish a paper. And most importantly, the system of peer review they oversee does not work very well.

When everything can be published with the click of a button, it is nonsensical to structure peer review around deciding what works get into which publications.  It makes no sense to evaluate the impact of a paper only at the time of its initial publication. It makes no sense to limit the assessments that matter to the two or three people a journal was able to coax into serving as reviewers. It makes no sense to reduce something as complex and multidimensional as the contributions and impact of a paper and its likely audience to a binary decision to publish in a particular journal. And it is bad for science to rely on a system whose entire premise is that  the most important thing about a paper is where it was published, and not what it actually says.

Why does a field based on empiricism and rationality use a system that is so manifestly ineffective and illogical? It is not that people disagree with this assessment. One of the few constants in my nearly thirty years as a scientist is that everyone complains about nearly every aspect of peer review. Yet we seem trapped by the Churchillian view that our current system of peer review is the worst possible system, except for all of the others.

I do not believe this is true. The rise of preprints gives us the perfect opportunity to create a new system that takes full advantage of the Internet to more rapidly, effectively and fairly engage the scientific community in assessing the validity, audience and impact of published works.

Axioms of Post Publication Peer Review

I do not want to underestimate the challenges in creating and operating a robust system of peer review that would work robustly for the ~2 million scientific papers published every year. Problems with the journal based, pre-publication peer review are manifest, but the system does have one virtue: it exists and functions.

The devil is ultimately in the details of course. But there are some general principles that should govern the development of a new system for scientific publishing.

  1. Researchers decide when their work is published. When scientists have new ideas, data or conclusions they think will be valuable to their colleagues they should be able to share them without the mediation of a journal. Posting a preprint is not a prelude to publication; it is publication. There should be a basic screen for appropriateness – not every document with scientific content is a work of science – but once a work passes this screen it should be considered published. This act of publishing should generate an identifier and should be permanent and irreversible. Publishing new versions of works should be encouraged.
  2. Peer review after publication. An obvious corollary of author driven publishing is that all peer review will happen after publication. Many of the issues with journal driven peer review stem from it occurring exclusively prior to publication. Post-publication peer review, if done right, can be far more effective. This will require that people no longer equate a work being published with it having gone through peer review.
  3. Peer review is a non-exclusive annotation. One problems with journal based peer review is that it is exclusive the the journal in which it was published, and thereby excludes any and all other inputs. If instead we focus on reviewing works that have already been published, there is no limit on who can review a paper. All reviews are new annotations that add to the body of curation developed around every paper.
  4. Peer review is ongoing. Another absurdity of journal based review is that the assessment of a paper – recorded in the title of the journal in which it was published – never changes, even as people read the paper, use its methods and data, and view it in the ever changing context that is science. It should be possible to review papers at any point during the useful lifetime of a work, and for new reviews to contribute to an evolving understanding of the validity, audience, value and impact of the work.
  5. Peer review is open. Everyone should be able to participate in the process of peer review. There need to be systems to guard against conflicts of interest and various forms of gaming. The default should be for reviewers to be identified, but there also need to be mechanisms to allow reviews where the identity of the reviewer is not made public.

APPRAISE (A Post-Publication Review and Assessment In Science Experiment)

It is easy to rant about the evils of journals and philosophize about a better system in theory (I have been doing it for twenty years). But it has proven to be incredibly difficult to successfully launch alternative peer review systems in the presence of an existing system that dominates hiring, funding and promotion in academic science.

However the increasing number of unreviewed preprints, and the impact they are beginning to have on science communications, has created a unique opportunity to experiment with new systems. Guided by previous experiences with new initiatives (especially PLOS) to avoid trying to do everything all at once, I am launching an experiment in the post-publication review of papers published in bioRxiv called Appraise.

The founding mission of Appraise is to demonstrate the potential for post publication peer review to make the assessment of research publications and scientists more efficient, fair and effective. We will do this by:

  • Recruiting scientists eager to focus their reviewing efforts on post-publication review
  • Creating a review format and implementing software that augments a careful assessment of the validity and value of the published work with tools to help scientists find useful papers, to put them into context and to fairly assess the contributions of the people who did the work
  • Implementing a system to dynamically develop interactive consensus reviews and to highlight dissenting opinions
  • Working with bioRxiv and others to develop a badging system to convey to a wide audience the extent to which a work has been reviewed and the nature of the reviews

Appraise is committed to building a peer review system that facilitates open dialog around published works of science where everyone is empowered to share their views honestly and completely.

It is perhaps easiest to think of Appraise as an editorial board without a journal (and we hope to be a model for how existing editorial boards can transition away from journals). Like journal editorial boards they will curate the scientific literature through the critical process of peer review. However members of Appraise will not be reviewing papers submitted to a journal and deciding whether it should be published. Rather Appraise reviewers are working in service of members of the scientific community, selecting papers they think warrant scrutiny and attention, and reviewing them to help others find, understand and assess published paper.

Each Appraise review will have many tangible products. A written review. An assessment of the likely audience for the paper. A description of the paper’s most important contributions, as well as both quantitative and qualitative measures of reviewers.

Reviews will be done solely at the discretion of members of Appraise. Authors will not submit their manuscripts for consideration, nor will they be able to decline to have their papers reviewed. However once an Appraise member reviews a paper they commit to considering revisions and updating their review as appropriate.

In the spirit of openness we encourage Appraise members to identify themselves, but recognize that the ability to speak freely sometimes requires anonymity. Appraise will allow members to post reviews anonymously provided that there are no conflicts of interest and the reviewer does not use anonymity as a shield for inappropriate behavior. Whether reviewers are publicly identified or not, Appraise will never tolerate personal attacks of any kind.

We are launching Appraise with a small group of scientists. This is for purely practical purposes – to develop our systems and practices without the challenges of managing a large, open community. But the goal is to as quickly as possible open the platform up to everyone.

This is very much an experiment and a work in progress, and many details will evolve over time as we gain more experience. You can read more about this at www.appraisescience.org, which will be active before the meeting in February.

  • Tim Vines

    Fascinating stuff, but what’s the financial model here? Quality peer review needs to be managed to keep it on time and on track, and to catch errors before they become visible and cause serious reputation damage. Volunteer academics can’t provide this oversight for anything but a very small operation, so to scale you’ll need an editorial office of some sort. That means employees, and that means getting revenue. Reflect on this: commercial publishers have strived to cut costs for decades, and almost none of their journals operate without an editorial office.

    • Volunteer academics typically already to this for journals for free. If this is going to work it’s not unreasonable to suggest that funding is sourced at government level for the admin and additional curation involved (a lot less than $10bn, or even the pure governmental contribution towards that). Journals are broken. Time to put them out of our misery.

      • Tim Vines

        Volunteer don’t do editorial office work for journals for free – they act as editors. It’s an important distinction, because the former get paid full time salaries.

        • Unpaid academics provide the peer review. Employees of journals who put the stuff together get paid, but they don’t peer review. Editorial office work is what we’re talking about doing away with altogether and replacing with a different model of dissemination. It’ll need funded of course, but by a different mechanism.

          • Tim Vines

            You’re right, EO’s don’t peer review. They manage peer review, making sure that the articles are in a fit state to review, that reviewers and editors get nudged to return their comments, that major errors are spotted and fixed, that ethical issues are caught, etc etc. Volunteer academics just don’t have the bandwidth to do this on a consistent basis at scale.

  • Mike — very interesting. In many ways this is very similar to the model developed by F1000 (and its sister sites like Wellcome Open Research and Gates Open Research).

    In line with your proposal, F1000, WOR etc operates as follows
    a) researchers decide when (and what to publish)
    b) everything – once it has passed a series of checks for plagiarism, adherence to ethical standards etc – is published
    c) peer review happens post publication (reviewers are suggested by the author)
    d) peer review is fully open (the reviews and the names)
    e) articles can be updated as many times as the author wishes (the site supports versioning)

    What do you see as the main differences between this model — which is already live and scalable – and APPRAISE?

  • J.B. Shaw

    Michael B. Eisen, Preprints can now be annotated using Hypothese.is @ https://www.hypothes.is . Now, it’s time to read and consider signing the San Francisco Declaration on Research Assessment @ https://sfdora.org . Enjoy 🙂