The Fool’s Gold Standard: Is Peer Review Working?

James Barr

The peer review process is the mechanism which drives scientific discovery. It is far more than just quality assurance of the scientific academic world: it can make and break people’s careers, result in awards, and allocate grants. Despite its prevalence, over the last few decades the infamous peer review system has come under heavy scrutiny.

The peer review process is difficult to define, as it is not standardised across journals, let alone disciplines. However, the general process, summarised in the figure below (from M. Ware, ‘Peer Review: An Introduction and Guide’), involves the review of a submitted paper by experts within the chosen field prior to publication.

peerreview

Despite the controversy, the process has its benefits. There are two methods in which peer reviewing improves the quality of work. The author has acceptance motives, working to receive publication. The second comes in the form of the feedback obtained after peer reviewing. Repeated surveys have shown that over 90% of the surveyed researchers said that the process had not only improved scientific papers in general, but also improved their own publications. [1] Crucially, the peer review process filters the growing number of scientific papers, by removing poorly designed, low quality or unoriginal work. Currently, if a researcher believes they have a high quality paper with wide implications to the scientific community, the researcher submits to a more prestigious journal, for example ‘Nature’. If they are unsuccessful, they submit to a less prestigious journal, and the cycle continues. Consequently the journal’s name has become an indication of the research quality. This allows the academics to concentrate on a manageable number of papers. Finally, the peer review process is a tried and tested method which has clearly resulted in scientific developments.

On the other hand, one of the contentious issues surrounding the peer review process is the peer selection. Who is a peer? Do they have vested interests in negatively reviewing the work? It’s fairly likely that a peer reviewer is in the same field of research and could well be a competitor. This leads to the first of the pitfalls. Historically, there have been cases where reviewers have stolen ideas and submitted them under their own name. Drummond Rennie tells his story where exactly this happened [2]. Rennie submitted his paper to the New England Journal of Medicine where he received a critical review from Vijay Soman. Soman was then caught submitting a paper where he had copied paragraphs of Rennie’s original work. Fortunately he was caught and subsequently left the country. [2] Obviously this was an extreme case; however, there are other methods in which reviewers could abuse the system. They might submit a harsh review to block or just slow the publication of the work. Peer reviewing is certainly open to abuse.

The peer review process is considered slow and expensive. It is not unusual for a paper to take more than a year to be published once submitted. Although reviewers are not paid, there is a substantial opportunity cost, when considering the time spent by the reviewers. There is also a hefty financial cost to access these papers once they are published.  They currently believe that the academic community pays approximately £3000 for access to a peer reviewed paper. Additionally, there is also an author cost (equivalent to approximately £300-£1500). There are obviously also overheads to pay. However, a substantial fraction of these costs become large profits for the publisher. If the review system were to change there could be significant financial gains for the academic community. [3]

Often, mistakenly, the peer review process is considered consistent, reliable and objective. However, there are numerous examples where this is simply not true.  The reviews are often completely contradictory. Here’s one example where two reviewers are commenting on the same paper:

Reviewer 1: `I found this paper an extremely muddled paper with a large number of deficits’

Reviewer 2: `It is written in a clear style and would be understood by any reader’. [3]

Not only are the reviewer comments inconsistent, but the peer review process is becoming less reliable, indicated by the increasing proportion of retracted papers. PubMed reported 52.9 retractions per 100 thousand publications in 2013, compared to 9.8 in 2003. This could be a consequence of over saturation caused by the growing number of paper submissions. The retracted publications cause many problems for the academic community. I believe three of the most damaging are the papers are misinforming the researchers and students; another paper has been rejected in place for a retracted paper, and in the past, these retracted papers have had a devastating effect on the scientific-public relationship. The latter has huge consequences, in the form of subsequent policy-making and funding reforms.

The peer review process has been shown to be biased against the author. One of the most famous studies was conducted by D. Peters and S. Ceci in 1982 [4]. They resubmitted twelve papers from so-called prestigious institutions which had already been published. “They made minor alterations, changing the authors’ names and, in particular, replacing the institution name with a less prestigious made-up name. They resubmitted the papers to the original publishing journals. A quarter of the papers were rejected on the grounds of originality. Shockingly, eight out of the remaining papers were rejected on the grounds of poor quality. Peters and Ceci concluded that this was evidence of bias against institutions. [4] Unfortunately, this isn’t the only bias. Evidence from Sweden analysed by C. Wennerâs, has suggested that the peer review process is nepotistic and sexist. [5]

So what’s the solution? Do we scrap peer reviewing in its current form? Do we use an open review system as seen in the Amazon Marketplace? Or do we improve what we already have? There’s no clear answer.

Some have suggested that signed, double-blind reviewing or training for reviewers would improve the current system. The majority of journals use the single blind review process whereby the reviewers remain anonymous, but the author’s name is known. The hope was that applying a double-blind system, where the author is anonymous to the reviewers, would make the process more impartial. The signed process, opposing the double-blind model, looks to make the reviewer accountable for their comments by enforcing the reviewer to sign the paper, once again hoping to improve the quality of the reviewing. In studies, looking at both the signed and double-blind process, no real improvement to the quality of review has been reported. One possible improvement could potentially lie in training the reviewers. Currently, reviewers undergo no formal training. Small studies have presented small but potentially significant improvements after training. [3]

The future of our peer review system is uncertain, but the one thing we can do is to make the review system open, following Sweden’s example. This would have huge benefits. Future generations would be able to analyse the peer review system in ways in which are currently not possible. This could lead to further insights into how to improve the process.

To summarise, we are using an expensive, slow, inconsistent and biased system. But does the peer review process work? Yes. Could we do better? Certainly.

References

[1] M. Ware, Peer Review: An Introduction and Guide, Publishing Research Consortium, 2013.

[2] D. Rennie, Misconduct and Journal Peer Review, In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences, 2nd edn. London: BMJ Books, 2003: 118-29.

[3] R. Smith, Peer Review: A Flawed Process at the Heart of Science and Journal, J R Soc. Med. 2006 April; 99(4): 178–182.

[4] D. Peters & S. Ceci, Peer-Review Practices of Psychological Journals: the Fate of Submitted Articles, Submitted Again, Behav Brain Sci., 1982, 5: 187-255

[5] Wennerâs C, Nepotism and Sexism in Peer-Review, Nature, 1997, 387: 341-3

The Fool’s Gold Standard: Is Peer Review Working?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s