The future of peer review

Joel Kingsman.

The peer review system has been the dominant method of moderating scientific research for the last half century or so. Much major work and innovative progress has been made during this time yet now a point has been reached where the peer review system itself is becoming an obstruction to the work of scientists. Modifying the peer review process from its 20th century origins to suit 21st century science is certainly achievable. The research itself is not and indeed never has been the issue. It is the system of publishing and disseminating it that needs fresh consideration.

The current situation sees a scientific paper written, submitted to a journal, scrutinized by several other scientists in the same field, then either rejected to be modified and resubmitted or published. On the surface it certainly appears to be a reasonable process for ensuring the quality and accuracy of published science. Indeed it has been practised for many years with success. But digging a little deeper reveals several troubling issues.

The first problem is to do with the reputations of the publishers themselves – not all journals are created equal. Big name publishers such as Science and Nature with their reputations for high profile research and their high impact factors give a paper and its author a high level of prestige. Whether these impact factors are truly a good indication of the value of the research is something worthy of debate in itself. Safe to say they are not the be all and end all and perhaps not even that. Regardless of content however, if a scientist has work published in Science, Nature or certain other journals it will likely be seen as worth more than work published elsewhere. These journals have higher rejection rates from 80 to 98 percent which of course means that they reject plenty of important work which later surfaces in supposedly less prestigious journals. It is a great shame that scientists are often judged not on the basis of their actual research but rather on where it was published. Moreover these judgements can make or break a career. The difference between someone remaining in a somewhat precarious academic position or heading towards university tenure may simply be the name of a publisher on their resume. Given that science is a sector underpinned by the testing of theories based on evidence it is both surprising and somewhat shocking to discover how much influence such a loose method of judgement can have.

Despite all this papers that get published are generally of good quality. However if science performed for the progression of human knowledge then this may not be good enough. New discoveries replacing and updating previous theories is one thing. In fact that is good and exciting science. Still there have been cases when peer reviewed papers have been accepted by journals and then later shown to be clearly erroneous. One example of this is case of Jan Hendrik Schön who rose to prominence for publishing fraudulent results whilst researching semiconductors at Bell Labs. In the year 2000 alone, Schön published eight papers in Science and Nature and received multiple awards too. Graphs showing identical noise for some experiments on single molecule semiconductors he conducted at different temperatures were actually noticed only to be dismissed by the editors after Schön claimed he had simply submitted the same graphs by mistake. He was exposed in the end but the fact that it was not the biggest name journals but rather individuals who brought this work to account is worrying. Schön’s works were later revoked by the journals. This was just one case that demonstrated the flaws of the peer review process and raised many questions about the responsibilities of both reviewers and editors in permitting the publication of bad science.

For the editor of a journal reviewers can be hard to find. Scientific fields can be very specialized and niche so the pool of willing reviewers can be small. The reviewers are in a tough position too. They are scientists in their own right most likely bogged down with administrative and teaching duties let alone their own research. They are not paid to conduct peer review of the papers sent to them and the reward awaiting a good reviewer is often even more requests to perform reviews and so even less time for their other duties. Doubtless their when reviewing instincts are to maintain the integrity of their field. However one can easily imagine how reviewing slips down an academic’s priorities and how papers may not be given the time their thorough examination demands. Furthermore reviewing is difficult work. It depends on the reviewers being up to date on the latest developments in their field and it is not uncommon for reviewers to disagree especially considering that they are often scrutinizing novel studies. A cautious editor will probably reject a paper in such cases and therefore what may turn out to be excellent research is forced to await publication elsewhere.

In this modern interconnected age there is in principle no reason why the peer review process cannot be updated. A promising idea is using an open source framework alongside of traditional journals. Currently editors are solely accountable for the work they publish even though many simply do not have the time to double check the comments of their anonymous reviewers. Open source peer review allows other scientists to read and critique the comments of the reviewers. Removing anonymity may make it harder than ever to attract reviewers but this is not strictly necessary. If just their comments are shown then the thinking which led to the paper being published is clear. This all works to aid the reader in making informed decisions concerning the article. Although peer reviewed, it need not be accepted so readily by an individual. The reader now also shares responsibility for how they view the paper.

Several open access repositories already exist. A notable example is Academia.edu which is a social networking platform for academics. In 2013 the site received a notice from Elsevier to remove thousands of papers published by the journal. This was understandable as essentially the journal wanted to defend its profits. After all why would anyone pay a subscription fee to Elsevier in order to access a resource freely available elsewhere? However this led to a backlash and boycott of Elsevier by many scientists who see open access as the future and Elsevier as somewhat behind the times. A great example demonstrating the potential for open access is the Atmospheric Chemistry and Physics journal. This uses a multi stage open peer review process and is successful. It has rejection rates of just 15% yet also remains one of the highest ranked journals in its field. A further advantage of open access is that the long embargoes from submission to publication are removed. These embargoes have been shown to lead to fewer people accessing a paper and of course can slow the progress of research.

Mistakes will always happen. However in science the mistakes permitted by the current peer review system can be costly. It can drastically affect career prospects and establish bad science such that it is difficult to overturn. To only accept work that has been published in journals is an outdated format which is simply no longer necessary. Open source publishing is the future and can help to redress the balance from the often unmerited dominance of high prestige journals. Peer review will always be necessary as the vital unseen ingredient in ensuring good quality science. However now that better methods are available the traditional peer review process must be updated. Otherwise scientific progress will continue to be hampered. There will remain a place for the traditional journals so that certain research is correctly recognised and promoted. However there is nothing to prevent scientists from simultaneously sharing their research with the wider scientific community. Trained scientists are intelligent people. By opening up peer review we can allow them to weigh up the work of their peers for themselves.

 

References:

https://newrepublic.com/article/135921/science-suffering-peer-reviews-big-problems

http://www.science20.com/science_20/jan_hendrik_sch%C3%B6n_world_class_physics_fraud_gets_last_laugh_whole_book_about_himself

http://sciencemadeeasy.kinja.com/peer-review-fixing-the-problems-1721409266

https://blogs.scientificamerican.com/information-culture/an-introduction-to-open-peer-review/

http://occamstypewriter.org/scurry/2013/12/09/why-elsevier-is-completely-in-the-right-and-totally-wrong/

http://journal.frontiersin.org/article/10.3389/fncom.2012.00033/full

 

 

The future of peer review

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s