Author Archives: Jessica Polka

Preprint journal clubs

For authors, one of the most exciting potential benefits of preprints is the ability to attract early feedback from broad and diverse sources during the preparation of a scientific manuscript. Preprint journal clubs can provide this input – and a more meaningful review experience for their own members as well. Here are some examples and resources for setting them up.

Prachee Avasthi’s preprint journal club

Prachee Avasthi, an Assistant Professor at the University of Kansas Medical Center, runs a course for graduate students called “Analysis of Scientific Papers.” The class takes the shape of a journal club in which students learn how to critically evaluate scientific manuscripts.

What makes Prachee’s course unique is that the papers under evaluation are drawn exclusively from preprints. As she explains in the video above, this has several benefits:

  • Students’ feedback is actually useful to authors since it’s created while a manuscript is under revision, instead of after it has been published.
  • Since students are expected to share their reviews, they must pay more attention to maintaining high quality commentary and a productive tone.
  • Posting these reviews publicly helps to demonstrate the review process to other students and to scientists interested in the evolution of the paper in question.

Prachee has generously shared her syllabus and introductory slide deck, and the students’ reviews can be found on the Winnower.

Other preprint journal clubs

We’d like to collect a list of similar groups and the tools that facilitate them. If you’re aware of others, let us know at jessica.polka@asapbio.org.

Whether in a course or on a blog, feedback from journal clubs can have a positive impact on authors and their science.

10 ways to support preprints (besides posting one)

Preprinting in biology is gaining steam, but the process is still far from normal: the upload rate to all preprint servers is about 1% that of PubMed. The most obvious way for individual scientists to help turn the tide is, of course, to preprint their own work. But given that it now takes longer to accumulate data for a paper, this opportunity might not come up as often as we’d like.

So, what else can we do to promote the productive use of preprints in biology?

1. Cite preprints

Many biologists, especially early career researchers, are concerned that their preprints won’t be properly acknowledged.

If you’d like to see some anecdata, here’s a word cloud generated by digitally polling the audience at the 2016 EMBO Long Term (postdoctoral) Fellowship retreat in November, 2016 (39 devices responded). The prompt was, “What is your biggest concern about preprints?”

embo-fellows-2

While we have yet to hear an example of a preprint author getting scooped, the concern remains very real. To counter this fear, we need to set an expectation that work disclosed in preprints will be cited fairly when relevant to other preprints and journal articles. A commitment to fairly cite relevant preprints was included in a draft statement from our first meeting, and it was widely endorsed.

2. Comment on a preprint

One of the greatest opportunities preprinting presents is the chance to receive more feedback on a paper. For example, Nikolai Slavov describes how thoughtful, constructive feedback helped his paper improve:

By using this feedback mechanism, we can strengthen one anothers’ science.

3. Set up email alerts

With an increasing number of preprint servers, it can be difficult to manually visit each one to stay on top of the literature. ASAPbio is working to facilitate an aggregation tool that will make this easier, but for now, there are several ways to get automatic email alerts on preprints of interest to you, including PrePubMed’s RSS tool. More details and instructions for different preprint alert options are described here.

prepubmed

4. Review a preprint in your journal club

Reviewing preprints may be even more rewarding that reviewing papers: you have the option to share your opinions with the authors, publicly or privately.

It’s a great educational experience for students, too. Prachee Avasthi at the University of Kansas Medical Center draws material for her “Analysis of Scientific Papers” course exclusively from preprint servers. She’s generously shared her syllabus and introductory slide deck, and the students’ reviews can be found on the Winnower.

See more examples of preprint journal clubs here.

5. Stickers

There are probably researchers in your department who aren’t aware that someone they know has posted a preprint. You can spark conversations around your lab or at conferences by affixing a sticker to your laptop, water bottle, or office door. See examples below for inspiration.

@mcdawg
@pollyp
@sciezgin
@BrianKelch
@clathrin
@aidarodrigo
@JonathanPDrury
@vinjlynch

Fill out this simple form to request some free stickers.

6. Add a message to your email signature

You can raise awareness about preprints with every message sent. Here’s an example:

7. Tell your preprint story

We’re collecting stories about researchers’ experience with preprints. You can tweet them @jessicapolka or using the #ASAPbio hashtag. You can also make a video (similar to Nikolai’s, above), or email jessica.polka@asapbio.org if you’d like to share something in longer written format.

8. Become an ambassador

ASAPbio ambassadors have agreed to act as local points of contact for discussions regarding preprints. They are listed on asapbio.org and have a private discussion group and access to shared presentation materials like slides and posters.

Sign up here.

9. Promote policy change

Journal, funder, and university policies are critical to make preprinting a viable options in biology. If journals with restrictive preprint policies operate in your field, you could write to editors to request that they reconsider. Requests could be made spontaneously or by using an ongoing correspondence to bring attention to the matter.

If you’re on a faculty search committee, consider working to insert a call for preprints into the job ad:

We’re keeping a list of university, funder, and journal policies to provide examples of existing progressive policies that you can reference.

10. Add a slide about preprints to the end of your talkssingle-slide-png

Of course, this works best if customized to fit your own experience (eg, a screenshot of the preprint you’ve been discussing in the talk). You can download a template in pptx here.

Note: the original version of this post encouraged responses to the NIH’s RFI on preprints as item #10.

Please share more ideas below!

Update on development of a Central Service Request for Applications (RFA)

At the ASAPbio Funders’ Workshop in May of 2016, representatives of funding agencies requested that ASAPbio “develop a proposal describing the governance, infrastructure and standards desired for a preprint service that represents the views of the broadest number of stakeholders.” Toward this end, we proposed a model for a “Central Service” (CS) that would aggregate content from multiple preprint servers, facilitating human and machine access to preprints via a search tool and an API.

Three separate processes are now ongoing to define this service:

Continue reading

ASAPbio newsletter vol 5 – Tell the NIH what you think about preprints, Crossref service launches today, new resources

Dear ASAPbio subscriber,

Tell the NIH what you think about preprints

The NIH has recently released a request for information (RFI) on the use of preprints and other interim research products. We encourage all interested parties to respond to the RFI using the submission website by the deadline of December 9th (extended from November 29th).

ASAPbio’s draft response is posted here. Even if you completely agree with our draft, we encourage you to submit your own responses as well. A large number of responses will be critical in conveying a strong message of community interest in preprints and other interim research products to the NIH. Responses from individual scientists at all career stages are encouraged. You do not have to respond to all questions, and the responses can be short. If you would like to share comments or your own response to the RFI, please use the comment section below the post.

Crossref launches preprint service

Today, Crossref, the organization that assigns DOIs for journal articles, launches their preprint service! The service will offer a specialized content type for preprints, enabling them to be linked to their corresponding journal article. This development will make it easier for preprint servers and journals to display links (backwards and forwards) between different versions of the same article, and it will facilitate pooling of metrics, citations, etc between the versions. This is a landmark in the development of preprints as an integral part of the scholarly literature.

New resources at ASAPbio.org

How many life sciences preprints were posted in September 2016? Which journal now has Preprint Editors? Which funder is requiring preprint deposition? And which med school accepts preprints in tenure packages?

We’re now tracking the growth of preprints in the life sciences as well as new developments in funder, university, and journal practices and policies regarding preprints. You also can now view all of these newsletter posts (including this one) on the web. Finally, we printed stickers (below) to help create visibility and spark conversations about preprints. Just fill in the form at asapbio.org/stickers to request some!  

rect13145-6-5-6-2-6-8_desat

Best,
Jessica Polka
Director, ASAPbio

ASAPbio’s response to the NIH RFI on preprints

Note: the RFI is now closed. The NIH has announced a policy that encourages the use of preprints.

The NIH has recently released a request for information (RFI) on the use of preprints and other interim research products. We encourage all interested parties to respond to the RFI using the submission website by the deadline of December 9th 2016 (extended from November 29th).

ASAPbio’s draft response is posted below. Even if you completely agree with our draft, we encourage you to submit your own responses as well. A large number of responses will be critical in conveying a strong message of community interest in preprints and other interim research products to the NIH. Responses from individual scientists at all career stages are encouraged. You do not have to respond to all questions, and the responses can be short. If you would like to share comments or your own response to the RFI, please use the comment section below the post.
Continue reading

ASAPbio newsletter vol 4 – Technical workshop, new website features, ambassadors

Dear ASAPbio subscriber,

Here’s what’s new:

  • We held a successful Technical Workshop to discuss the feasibility of creating a central preprint service. All the notes are online, and you can also view the archived video stream.
    • We’re working on a request for information to identify potential suppliers, their implementation strategies, and their predicted costs and development timescales. We will present all reasonable responses to a group of funders as part of our response to a request that emerged from the ASAPbio Funders’ Workshop. More details about our planned process can be found here.
  • We’ve added some new features to the website.
  • The ambassador program kicked off in earnest.
    • Check out the map to see who’s near you.
    • It’s also not too late to sign up – we’re looking for people willing to act as local points of contact about preprints. We’re also providing resources to help ambassadors give talks about preprints at their home institutions or while traveling to conferences and other meetings.

Please let us know if you hear of any exciting developments in preprints in life sciences!

Jessica

Jessica Polka, PhD

Director, ASAPbio

ASAPbio newsletter vol 3 – REQUESTING FEEDBACK on a central preprint service for biology

Dear ASAPbio subscriber,

It’s been an exciting few months at ASAPbio! Here’s what’s happened:

  • The report of our February meeting at HHMI was published in Science, and Ron Vale and Tony Hyman recently published an article about priority of discovery & preprints in eLife.
  • ASAPbio was awarded grants totalling $400,000 in provisional funding from the Arnold, Sloan, Simons, and Moore foundations for a period of 18 months.
  • We held a Funders’ Workshop at the NIH on May 24th.
  • As an output of this, representatives from funding agencies called for ASAPbio to develop a proposal for a preprint service for biology.
  • In response to this request, we’re now seeking feedback from the community on a draft proposal for a central preprint service that could aggregate content from multiple servers. Please consider leaving a public comment on the web and sharing the link with your networks. After future iterations, we will present several variations to funders in the fall.
  • To develop the technical aspects of this proposal, we’re hosting a Technical Workshop in Cambridge, MA on 8/30. We’re aiming to provide a video stream so that anyone can follow along.

Finally, effective 8/1, I’m now serving as full-time director of ASAPbio! Please don’t hesitate to contact me with any comments, questions, or ideas on how we can work together to advance the productive use of preprints in biology.

Best,

Jessica Polka, PhD

Director, ASAPbio

Appendix 2: Current feedback on Central Service features

Central Service model documents

Current discussions with the community on proposed features of the Central Service

Surveys and information from scientists

We will continue to engage the scientific community on what services they want to see in a next generation of preprints. However, based upon a survey that ASAPbio conducted in May 2016 (Results summary (pdf) and Anonymized responses (xls)) and other resources (e.g. Preprint user stories compiled by Jennifer Lin at Crossref and ASAPbio survey #1 (early 2016)), we believe that biologists want:

  • High visibility and discoverability of preprints
    • A single recognized website
    • Good search tool
    • Email notifications
  • Web-readable xml format
    • Click on link to figures to display them
    • Ability to click on links to references
    • Export to more readable and compact pdfs
  • A system for cross-referencing versions of the same work
    • Linking the final journal publication to preprint versions (and vice versa), so that the history of the work is transparent and preserved

as a reader servers responsible submission process

Input from servers, publishers, funders and data management experts

In July-August 2016, ASAPbio conducted informal interviews with preprint servers, funders, scientists and developers. We originally presented a variety of Central Preprint Service models of increasing complexity and centralization, ranging from a PubMed-like metadata search tool (Model 1) to a PubMed Central-like database that hosts well-formed XML content (JATS) and makes it available through a web display tool and an API (Model 4). One version also included a central submission tool (Model 5).

5models

While responses to the creation of a central tool were generally very positive, opinions on the best implementation varied. Below is a summary of some of the critical feedback we received.

  • Models 1 & 2 provide little benefit over the current state of affairs. These models generated less interest among funding agencies.There are already multiple ways to search preprints (search.bioPreprint, PrePubMed, Google Scholar) and existing preprint servers already preserve their own content.
  • Models without an open API and common licensing will stifle innovation. Without free access to content, 3rd parties will have difficulty in implementing new services (such as peer review, data mining, or aggregation)
  • Providing central submission and full-text display would be undesirable for some existing servers. These tools would directly compete with existing servers for traffic and recognition in the community. Also, display in multiple locations could disrupt download/view metrics and commenting systems. However, some funders felt that the CS should have the ability for full display as well as drive traffic to server sites. Some funders have expressed an interest in allowing submission directly to the CS (Model 5), but most favor a practical solution that embraces the needs of the ecosystem.
  • Many of the original models are complicated and development of any system with many moving parts will take a long time. Therefore, “perfection must not be the enemy of the good.” There will be a need to generate a CS that will work “out of the box” and improve on it over time. The CS needs to take into account realistic development of technologies.
  • Technological limitations make the use of JATS impractical. No good unsupervised .doc -> JATS converters currently exist. Thus, the conversion process requires human intervention.
  • Document conversion is costly. Server-side conversion to a structured format (such as JATS) is expensive (on the order of ~$20+); therefore, it doesn’t make sense for preprint servers to provide this, especially when preprints generate no revenue. The CS should be close to cost-neutral to servers and other publishing entities.
  • Licensing has generated a diversity of opinion. Some parties favor author or publisher choice in licensing, arguing that scientists will have concerns of the re-use of their material. Our own surveys and interactions with scientists suggest that most do not understand licensing options and their associated benefits/disadvantages. Most funders favor a uniform licensing policy for the material in the CS in order to allow re-use in innovative ways and avoid complicated restrictions for data mining. The license most favored at the moment is CC-BY, although this may require research and engagement with the scientific community.
  • Servers, Platforms, Publishers consulted were generally interested in working with the CS. However, alignment and preference for models varied between model 2 and model 4.  
  • A major topic in which opinion varies is ‘display’. Some funders and publishers the CS should have capability of displaying its archived content.  Others feel that the CS should not display content to readers (other than abstracts) and that display should reside with servers and publishers.
  • Use existing technologies whenever possible. Don’t reinvent the wheel, and carefully evaluate existing software/infrastructure.
  • Balances immediate concerns against opportunities for future development. Expressed by many, this sentiment emphasizes the need for a governance body that can continuously weigh these issues over time and make adjustments. In addition, inter-operability between preprints systems in biology, physics and other disciplines may need to be considered in the future.

We have drafted a provisional model of the Central Service (Summary) that takes into account the various input received above. The model emphasizes the development of document conversion services and the provision of web-ready full-text outputs to the input server. Providing full-text display via intake server/publishers will deliver to scientists many of the benefits they want while providing intake servers with incentives to participate in the program.

Possible benefits of the proposed service

To scientists

  • Preservation
  • Ease of use and readability (through web display at intake server)
  • Adherence to standards of author identity and ethical guidelines for research and disclosure
  • Potential for innovative reuse (with appropriate attribution)

To intake servers/platforms

  • No-cost document conversion into web-readable format
  • No-cost preservation
  • Improved exposure through a search portal that links exclusively to the intake server for display
  • “Accreditation” of servers or individual preprints through central screening process

To funders

  • Uniform standards of quality
  • Access to entire corpus via API
  • Ability to search/filter by funding source

Desired technical features for discussion

We welcome comments on the list of desired features below, which could become an agenda for discussion at the Technical Workshop (August 30, 2016).

Input (collected from the author)

  • Original manuscript file (.doc so that reference metadata can be extracted)
  • Supplementary files
  • ORCID
  • License (note- the Governance Body task force will also address this issue)
  • User authentication
  • Metadata (if extracted from .doc, get the user to check)
  • Grant support
  • Ethical statements (note- a separate task will also address this issue)
    • Self-ID COI
    • All authors agree on submission
    • Methods needed to reproduce this work are contained within the work
    • The work has been conducted in agreement with human & animal research guidelines

Document conversion

  • Extraction of text from source file
  • Extraction of metadata (such as title, authors, affiliations, keywords, and abstract)
  • Extraction of references
  • Insertion of figures, or recognition of existing in-line figures

Screening and moderation

  • Automated plagiarism detection
  • Automated detection of non-scientific content (via arXiv-like algorithm)
  • Interface for human-supervised screening/curation/moderation

Versions and identifiers

  • Unique, persistent ID for each version
  • All versions linked to one another (and to published journal article)
  • Linked to datasets
  • Tombstone pages for retracted content

Archiving

  • Stable archiving of source file (.doc) and also derivatives
  • Permission to display content if intake server reaches end of life

API

  • Bulk download of all content (.doc) and also derivatives
  • Filtering by metadata

Discovery tool

  • Full-text indexing of all content in the central database
  • Advanced search (boolean operators, search fields such as author, keyword, funding support)
  • Alerts (RSS/email)
  • Display of abstracts, etc, but exclusive link to intake server for full-text display

Proposal development process

The output of the Technical Workshop will be an announced in a Request For Information (RFI), in response to which any interested party can provide information on the development and approximate costs of developing a CS. The responses to the RFI will be shared by ASAPbio with major international funding agencies for potential consortium support. Pending their collective interest in financially supporting a plan for a CS and refining its method of operation and governance, a formal RFA may follow the RFI to which interested parties could apply for funding.

decision process

Appendix 1: Rationale for a Central Service

Central Service model documents

Do we need new infrastructure and governance for preprints?

Because the preprint server arXiv was born very early in the history of the internet and served its community well, it has become the de facto repository for preprints in the physical, mathematics and computer sciences without any major competitors. During the past two decades, various scientific disciplines decided to join arXiv rather than start their own servers. Thus, arXiv has become a “central server” for the physical science community and has achieved high visibility.

The success of preprints in physics was aided by the coalescence of a large body of content in one highly visible site (arXiv) that had a high standard for quality, attracted outstanding work, and had a scientist-led governance model. Biologists could attempt to replicate this single server model. However, biologists already deposit work in several existing preprint servers, notably bioRxiv (established 2013), PeerJ Preprints (established 2013), and the q-bio section of arXiv (established 2003). In addition, PLOS has had a long-standing interest in posting pre-peer review manuscripts as an option for submitted papers, which could add considerable content in the near future. F1000 has developed a publishing platform that provides access to manuscripts before formal peer review (effectively a preprint; see definition below) and the Wellcome Trust has adopted the F1000 platform to launch Wellcome Open Research. Other journals or funding agencies may also decide to develop similar dissemination mechanisms for pre-peer review content. Thus, the concept of disseminating “pre-peer review” manuscripts is broadening beyond a traditional “arXiv”-like server.

The future of preprints in biology is now poised at both an exciting as well as fragile moment. If organized and thought-through properly, preprints could accelerate scientific communication, serve the public good, clarify priority of discovery, and help career transitions of young scientists (see Preprints for the Life Sciences, Science). The development of preprints could be governed by the scientific community, in partnership with publishers and other service providers, leading to exciting and innovative possibilities for science communication.

However, the future of preprints could be less bright. Preprints could become fragmented among the efforts of multiple competing parties, lack overall visibility and critical mass, fail to harness modern possibilities for dissemination and use, and lack clear governance. Preprints may fail to achieve the level of respectability needed to convince scientists, funders, and universities that the disclosure of work by scientists plays a valuable role in the ecosystem of science communication, along with post-peer review journal publications. If preprints fail to grow substantially in submissions and readership in the next five years (e.g. to the level of arXiv), scientists and funders will view them a failed experiment. Because the future of preprints is poised in critical time window, it is important to think through issues of execution that will maximize the chance of preprint adoption by the community.

If the scientific community does not act, continued fragmentation of preprint sites could undermine the potential of this communication system by generating:

  • Ambiguity about what qualifies as an acceptable preprint and a recognized content provider. Currently, funding agencies and universities are considering whether preprints or other “pre-peer review” publications should be included in applications. However, what is defined as a “recognized preprint server” is ambiguous at the moment. Every server or publisher may define their own screening protocol, causing uncertainty about whether a preprint has been screened for plagiarism or adheres to ethical standards. In this current system, each journal, funding agency, and hiring or promoting committee must define a list of approved preprint sources based on their own assessments of preprint servers. This practice, which is already occurring at certain journals, will create a situation that is confusing and discouraging for researchers.
  • Lack of visibility and difficulty of discovery. If preprints are spread across multiple sites, they will become more difficult to find. Maximizing discoverability, visibility, and respectability are key to adoption and widespread use by scientists, as is suggested by the success of arXiv.
  • Variable and potentially limited access to data. In the current system, each server sets its own licensing policies and is responsible for archiving its own content. This puts content in danger of being held under restrictive licenses or lost altogether.

Limited potential for technology development. If each server must create or outsource IT infrastructure, overall costs of the preprint system will be high, many servers will not have funds for more advanced IT development, and the potential for using and disseminating information may be limited.

Value of a Central Service

To overcome the deficiencies described above, we believe it would be in the best interest of the scientific community to create a Central Service (name subject to change at a later date) that will aggregate “pre-peer review” manuscripts from several sources, maintain standards of quality for its intake, preserve content for posterity, and disseminate information in a manner that advances scientific progress. The Central Preprint Service would, in essence, function as a database that serves the public good, analogous to the Protein Data Bank or Pubmed Central. We envision that a Central Preprint Service will be supported by a consortium of funding agencies for a minimum five year term of operation. It wil be overseen by governance body that will be 1) international, 2) led by highly respected members of the scientific community, and 3) transparent in all of its proceedings, actions, and recommendations.

Partnerships with journals and servers

The Central Service will host manuscripts that contain 1) data, 2) the methods needed by other scientists to replicate that data, and 3) an interpretation of that data. The governing body will determine how manuscripts are screened for entry into the Service (for example, to exclude content that is plagiarized, non-scientific, or in violation of ethical guidelines). However, the Service will not engage in validation or judgment of the work as is performed by traditional peer review. Thus, the Central Service will work as a partner, and not a competitor, with existing journals.

The Service also seeks to act as a partner with preprint servers and publishers that ingest manuscripts from authors. Partners who can deposit their content into the Central Preprint Service will benefit from additional infrastructure support (e.g. plagiarism detection, conversion tools, etc) and most importantly will have greater appeal to scientists who will want their preprint broadly viewed and recognized by grant and promotion committees.