The Past

The scientific publication system is simple. Once scientists have written a paper about their science, they send their work to a company called a publisher. The publisher then gives other scientists (called reviewers) internet points to determine whether it is worth their time to publish. If the reviewers decide the paper is good enough, the publisher will generously allow the authors to pay the journal to put the paper on their website. There is, of course, another fee if the authors want people to be able to read their paper without paying a further toll.

It wasn’t always this way, though.

Historically, scientists communicated to the public through books and communicated to each other with letters. Then, around the 17th century, scientific societies were formed to “share their results in person, and presumably consume alcohol.” Over time, these societies began to publish periodicals, which were the precursors to modern academic journals. However, the publication system we have today is largely a result of the expansion and codification of academia in the decades after WWII¹.

The present-day academic publication system is inefficient, and the costs of this inefficiency are borne both by scientists and society as a whole. Fortunately, with the advent of the internet, several alternatives have arisen. This post discusses issues in the current publication system and what the future of academic publishing might look like.

The Present

Paper Ownership
When a paper is published in a journals, the publisher typically ends up owning it² . In the past, this meant that only people who paid the journal’s access fees were able to read papers. After pushback (and an NIH mandate), it is now possible to read some publicly funded research for free (after an embargo) through PubMed Central³. Of course, the publisher still keeps the copyright, so only a subset of published articles are available for research via text mining.

Even paying an institutional licensing fee doesn’t give you free rein. Chris Albon once tried to scrape JSTOR articles to a tablet so he could spend a summer backpacking and preparing for his comprehensive exams. As those familiar with the publishing system would predict, JSTOR threatened sanctions against his university and made him delete all the articles. Possession of large numbers of academic articles will cause people to show up at your office. Possession with the intent to distribute invariably incurs legal action.

Publication Delays
If all goes well, scientists write a paper, submit it, wait, get reviews, address them, wait, then publish. In biology, the median time between having a paper complete enough to preprint and having it published is about 150 days. During this time, the research is done, and the paper is written. However, unless the paper is preprinted, no one other than the authors, reviewers, and the publisher can read it⁴. As a result, the rate of scientific progress is slowed down by months.

Peer Review
Modern peer review is a little strange. It involves for-profit companies getting consulting hours from scientists with advanced degrees for free. I’m of the opinion that this is bad, and not just because I’m a grad student who likes having money. When making a resource free (or very cheap), any system attempting to maximize profits will use it for anything they can draw value from. While peer review itself isn’t a broadly useful resource, academic labor is. If we’re not careful about giving away scientist-hours for free, we might end up with universities posting unpaid professor positions or something. Oh, wait.

In addition to flaws in pricing, there are also issues with the current framing of peer review. Publishers aren’t trying to determine how good a paper is, they’re trying to determine whether they should publish it. In an ideal world, these would be the same thing, but in practice they diverge due to perverse incentives.

Authors want their paper to be published after one round of review so that they can spend their time doing science instead of corresponding with reviewers. Accordingly, the acceptable response to reviews takes the form “We thank the reviewer for their suggestion that we cite their marginally related articles, and have done so in the revised manuscript.” Disagreements are heavily penalized as they increase the probability of a negative assessment from the reviewer and a second round of “suggestions.”

Reviewers, knowing this, can get away with misbehavior, like increasing their citation count by recommending that the authors cite their papers. Alternatively, if the author is a competitor, a reviewer can emphasize the flaws in the paper in an attempt to keep it from getting published.

Because reviewers have no skin in the game, it’s challenging to put checks on this kind of malfeasance. The worst-case scenario is that they get reported and removed from their unpaid job as a reviewer for that journal.

Ultimately, the current system of peer review infrequently changes the results of a paper but frequently causes delays.

Externalized Costs
One reason that the inefficiencies in the modern publishing system aren’t more visible is that many of the costs are externalized. For example, the cost of paper access is paid for at least in part by taxpayers/nonprofits through universities’ indirect costs on grants ⁵.

Academics generally aren’t presented with the cost to access a journal unless their university system or country decides to fight a publisher. They simply log in from a university IP address or use a VPN and get automatic access. Meanwhile, taxpayers funding the research have to hope the article they want is open access, wait for the article to appear on PubMed Central, pay hundreds of dollars for a subscription to the journal, or pirate the paper from a site like SciHub.

Likewise, the costs of peer review are borne by academics. As things stand now, peer review is a system where world-class experts provide billions of dollars of consulting to for-profit publishing companies for free.

The Future

Flaws aside, modern academic journals play several vital roles.

For authors, journals act as platforms, allowing scientists a place to disseminate their ideas. Even if the authors of a paper aren’t widely known, if they can write a paper good⁶ enough to get into, e.g., Nature or Science, it will get readership. While increasing the readership of specific papers isn’t the goal of science per se, citations are a unit of academic currency, and journals can help mitigate the extent to which academic success is driven by networking ability.

For readers, journals act as filters, separating “can’t miss” papers from the pack. Thousands of biological papers are posted each month, making it challenging for scientists to keep track of which papers are relevant. To alleviate that issue, journals coordinate peer review as an additional filtering mechanism to augment the editor’s discernment with that of experts in the field.

In general, I don’t believe in providing criticism without alternatives, so how do we fill the roles played by academic journals while reducing the drawbacks?

Platforms

While in the past the only way for scientists to publicize their work was through journals, letters, or conferences, the internet has made distribution much more accessible. The number of papers being submitted to preprint servers is growing rapidly, allowing scientists to share their work with the world for free (!). While preprint servers don’t necessarily have the same prestige as major journals, removing the barrier to actually reading the work in question opens the door for alternative platforms.

For example, deep learning research uses a hybrid model of preprints and conferences⁷. Deep learning papers tend to be published as preprints when they are completed,⁸ then submitted to conferences instead of journals (if they are submitted at all). Conferences like NeurIPS and ICML are the equivalents of Cell or Nature in biology and are paid close attention to by the field as a whole. While the conference system has its own problems, at least it’s an order of magnitude cheaper for scientists.

Another platform of note is Science Twitter. Though I don’t think the format is healthy for communicating science (or anything for that matter), it is effective at transmitting ideas to a large number of people⁹.

Twitter isn’t designed to be a platform for discussing and circulating scientific research, but some websites take on parts of that goal. Publons, for example, aims to give more credit to peer reviewers. Due to anonymity, peer reviews historically haven’t been rewarded. Publons gives scientists the ability to point to a trusted source and say, “I reviewed X articles in Y journals.”

Filters

Given infinite time, scientists would try to read all papers in existence, but we don’t live in that world. Instead, we try to read as many papers on our research topic as we can while reading only the essential articles for less related topics¹⁰. As things stand, journals are organized to fit this system well. There are journals dedicated solely to (in their opinion) papers of high general importance and other journals that focus on the minutiae of a specific topic.

Requiring scientists to skim all the preprinted papers to stay up to date in their field is likely an impractical amount of work. To avoid this fate, we need to replicate two key functions: aggregation (“what is relevant?”), and peer review (“what is good?”).

Alternate aggregation
Speaking of impractical amounts of work, let’s talk about deep learning research as a field. Because of its conference-based system and the fast pace of progress, deep learning scientists rely on preprints in a field that posts hundreds per day. To survive such a dynamic environment, deep learning communities have responded by building tools to make the firehose of information more manageable. As a result, several of the best methods for aggregation I’ve seen so far come from the deep learning community.

Ironically, given the field’s drive to automate tasks done by humans, one of the current solutions is to throw scientist-hours at the problem. Freelance paper aggregators such as AK and Aran Komatsuzaki¹¹ select the preprints that they think are the most important each day and post about them on Twitter.

A more formal implementation of this same idea is something called an “overlay journal.” Overlay journals are called such because they build on top of the existing publication infrastructure. Instead of handling peer review, hosting, and so on, overlay journals simply aggregate papers on a particular topic as they are preprinted or published elsewhere. In the overlay paradigm, individuals can create a collection of papers relevant to their field without the costs of starting their own publication system.

Further afield, creators like 3Blue1Brown show that you can learn science by seeing it instead of by reading it. Embracing that thesis, channels like OMGenomics and Yannic Kilcher have sprung up to do walkthroughs of scientific projects and papers. Instead of just selecting interesting papers, such channels add their own interpretation and understanding to the science.

While the methods I’ve presented so far have all involved manual curation, there are other ways of approaching the aggregation problem. Several natural language processing methods exist for finding and aggregating papers. Such methods can take in a larger quantity of papers than a human and may be less swayed by a publication’s venue or the status of the authors writing the paper. The hard part, as always, lies in getting the algorithm’s results to align with the intent of the human driving it.

Since paper aggregation is largely a matter of taste, I don’t think there is a single answer for the best alternative aggregation system. I believe future scientists will employ several different manual and automatic systems, driven by popularity and suitability for particular uses.

Peer review
Although peer review is a helpful signal about a paper’s quality¹², in its current form, it is inefficient and misaligned with the goals of science.

One potential method for aligning the incentives of peer reviewers, authors, and publishers would be to pay peer reviewers. Paying reviewers encourages publishers to value reviewers’ time by avoiding sending papers for peer review that are unlikely to be accepted. It also encourages reviewers to behave well to avoid risking their payment. You could even imagine a system where grad students looking to gain domain knowledge and augment their stipends could review papers rather than teaching classes.

Another promising way to improve the review process would be to separate it from publishers’ decisions to accept or reject the paper. One group leading the way on this is ELife, who announced that their new publishing model will leave the accept/reject framework behind in favor of public peer review. The implications of this move are exciting, and I look forward to seeing how things play out over the coming years.

Instead of overhauling the journal publication process, others are attempting to make the existing processes more efficient. Peer review process takes months, and having to rereview a paper due to journal fit or expected impact exacerbates the issue. Review Commons is developing a way to prevent rereviews (at least for the life sciences). They coordinate peer review, then send the resulting reviews to their affiliate journals. While they still reject papers based on topic scope and perceived importance, Review Commons is still moving the field towards journal-agnostic peer review, which is a win in my book.

While Review Commons makes the peer review process more efficient by coordinating journals, OpenReview has gone a step further by building an entire platform. Currently, this platform is tied to individual conferences; works are submitted to a conference and reviewed by people affiliated with the conference. The idea of a peer review platform promises more, though.

With very similar technology, it would be possible to coordinate freelance paid peer review (thankfully, OpenReview is open source). The majority of peer review today is anonymous and fragmented. Each journal knows the quality of individuals’ reviews, but there are so many journals that it is challenging to get an idea of reviewers’ calibration¹³. Meanwhile, Publons knows who has reviewed for which journals but not their scores. By centralizing reviews and reviewers, a platform could simultaneously shine light on bad behavior, disentangle the peer review and journal acceptance process, and coordinate reviewer payment. As a bonus, by making the system pseudonymous, such a platform could achieve all these goals while maintaining single- or double-blind peer review.

The article of the future

As the world continues to grow and change, the systems around academic publication will change with it. But there is more to publication than its attendant systems. The basic units of scientific communication are also beginning to change.

Null results and lab notebooks
Currently, scientists communicate results in the form of discrete units like papers, posters, and conference talks. Such research artifacts are post hoc methods of distilling a series of questions and analyses into a cohesive story. This distillation is helpful; a paper is easier to read and understand than someone’s lab notebook. However, if the table stakes for official scientific communication are weeks or months of work on the submission, then more minor findings will fall through the cracks. Few scientists are willing to spend that kind of time to compile and submit a null result or a result that is low impact but raises new questions. The effort required to do so would be disproportionate to the benefits.

Fortunately, we live in the 21st century. The internet gives us the ability to make notes available to the entire world with little friction. Now that distribution costs are negligible, it is no longer infeasible to maintain a repository of null results. Simple signposts of the form “I tried X, but it didn’t work. Here’s my code and experimental description” could save vast amounts of scientist labor by steering people away from known pitfalls.

We could even go further and normalize keeping public electronic lab notebooks. Having a record of the original work would help people trying to extend a paper’s results, as they would be able to see the logic behind what is included or left out. It would also be a valuable educational tool. Students are currently told to read papers and speculate about why authors did what they did. In a world of public lab notebooks, a class could teach the entire process that led to a finding instead of solely the final product.

Bug bounties and code review
Right now, the quality control process for scientific publications is some combination of “three scientists peer reviewed this, so it’s probably good,” “no one I’ve seen roasted this paper on Twitter yet,” and “getting a paper retracted is embarrassing, so there probably aren’t any retraction-worthy errors in this paper.” How much do you trust the average paper in your field? How much more would you trust it if the authors said they would give $200 to anyone who can find an error?

If you don’t like getting emails from random people on the internet about how you’re wrong, maybe you’d feel more comfortable with someone you know doing it. Code review is a software engineering practice common in industry but infrequent in academia. The idea behind code review is that the author of an analysis might miss problems due to their familiarity with the code, while a second person might not¹⁴.

On a similar tack, in my recent paper we¹⁵ proposed having a collaborator or coauthor whose role is to make sure the work is reproducible. We describe the role more eloquently there: “We envision a reproducibility collaborator as someone outside the primary authors’ research groups who certifies that they were able to reproduce the results of the paper from only the data, models, code and accompanying documentation.”

Interactive papers
Papers now live primarily on websites, which in turn live on computers. Things that once made sense, like limiting the amount of color in figures or having a strict length limit, make less sense today. However, we can do much better than “Wow, this paper would have cost a lot to print on paper. That figure has five colors!”

We can make papers interactive. Why explain an acronym once when you could have optional expansion though the entire paper? Why not have the corresponding regions in a genome browser light up whenever you hover your cursor over a Hi-C heatmap? Why use only 2D plots when you could make a 3D figure that you can rotate in your browser?

For computational analyses, we can go a step further and give readers runnable code within the paper or attached as a notebook. Reading about an analysis is great, but being able to ask questions and see what changes when you tweak the parameters is even better.

Conclusion

Where does that leave us? The probability that the world will decide one day to reject the status quo and adopt all of the systems above is vanishingly small. Institutions like academia and governments large enough to contend with publishers are too large to move that quickly.

Instead, I think the situation is more “the future is already here but not evenly distributed.” In biology, preprints have gone from a novelty to an appreciable fraction of the total literature. Likewise, having a Twitter account dedicated to tweeting about papers from a field is uncommon, but some individuals already have tens of thousands of followers from doing precisely that. In fact, many of the ideas listed in this post are already being implemented on some level but aren’t the predominant mode of operation. All of these ideas have the potential to grow; it’s just a matter of seeing what people will decide to use.

Personally, the future that I hope for is a preprint → pseudonymous paid peer review → overlay journal system. I think it’s close enough to our current institutions to avoid breaking anything while alleviating many pain points. But the beauty of living on the edge of the future is that regardless of what I think, you can choose your own ideal scientific system. So start an overlay journal, or submit your paper to ELife once they become peer review coordinators, or come up with something better than anything I’ve written about. In a world that is changing every day, go make your own choice for the future happen.

Further Reading

• Arcadia, a recently founded research organization, implements several of the ideas above in their publishing platform.
• This page has more visions of a potential futures (though not all of their ideas are publication related).
• “This has … been a time of growing divergence between the different roles of academic publishing: as a means of disseminating validated knowledge, as a form of symbolic capital for academic career progression, and as a profitable business enterprise.”
• “Whatever happens to Sci-Hub or Elbakyan, the fact that such a site exists is something of a tragedy.”
• “…if peer review is so central to the process by which scientific knowledge becomes canonised, it is ironic that science has little to say about whether it works.”

Footnotes