Who reviews the reviews?

The seemingly innocuous phrase ‘peer review’ can evoke some strong emotional responses. For its proponents, it is one of the cornerstones of science, acting as a quality assurance system that helps maintain scientific standards and integrity. But its opponents claim it fails to achieve these objectives and wastes vast amounts of time and money for no obvious benefit. What is indisputable is how widely used it is: most journal papers are reviewed before publication, most grant applicants are peer reviewed, and many applications to present at conferences are peer reviewed too. But why did peer review come into existence and why is the scientific community so reliant on it today?

Peering into the past

It may surprise people to hear that 50 years ago not every journal used peer review, according to Aileen Fyfe, a historian at the University of St Andrews in the UK. She is currently leading a project looking at the history of Philosophical Transactions, the world’s oldest science journal published by the Royal Society. It was in the decades after the second world war that peer review really took off, she says, but it’s been around longer than that. ‘Peer review in the sense that we now know it is a 19th century invention, prior to that it was a very distinctive thing to do with society publications.’ The Royal Society became one of the first learned societies to introduce written peer review in 1832. Philosophical Transactions had already been published for over 150 years by this point, and for the previous 80 years had been using a committee to make decisions on submitted articles.

One of the questions Fyfe’s team has been probing is why it was deemed necessary to introduce it at this time. ‘It is to do with collective responsibility,’ she explains. Sending the manuscripts to a couple of members of the society who were experts in the field for comment meant that the decision whether to publish became the collective judgement of the society, not that of just a few people on a committee. The desire for collective responsibility also explains the desire for anonymity of the reviewers, although some of the early reviewers happily outed themselves in their referee reports.

Peer review’s appearance in society publishing can therefore be explained, but understanding why it then started to be taken up by commercial publishers is still work in progress for historians. Commercial journals, such as Nature, were originally run by editors who made decisions alone on whether or not to publish submitted work. However, in the second half of the 20th century this started to change. In the mid-1970s, for example, Nature introduced peer review as a standard part of its editorial process (it was only used occasionally before then). ‘People seemed to jump on a peer review bandwagon for reasons that have yet to be fully understood,’ says Fyfe.

A sacred cow

Not only do we not know why peer review came to be a system for determining the worth of manuscripts submitted to almost all journals, but we also have surprisingly little proof of its effectiveness. Few robust studies to determine its worth in journal publishing have been conducted, and those that have been were largely negative. Richard Smith, a former editor-in-chief of the British Medical Journal, is one of peer review’s harshest critics. He has described journal peer review as a sacred cow that is ready to be slain. Smith says that peer review fails to detect errors or fraud in papers and its use in journal publishing should be abolished. Instead, he claims, the articles should all be published and the community can decide for themselves what matters and what doesn’t.    

Currently, however, only a small minority of the scientific community appear to agree with Smith’s views. Jan Velterop, a retired scientific publisher who campaigns widely on journal publishing issues, says the reasons for that are not very scientific. ‘It’s very much bound up with the whole culture of science that things should be peer reviewed,’ he explains. ‘Smith presents a very good argument, but I wouldn’t go so far as saying we should forget peer review altogether. It may be logically the right thing to do but psychologically it is not.’

And although there is a lack of robust studies that support the use of peer review in journal publishing, a recent study found that the peer review of grant applications is working well. The researchers examined 130,000 projects funded by the US National Institutes of Health between 1980 and 2008, finding that those with better peer review scores consistently generated more publications, citations and patents.

Increasing secrecy levels

While support for peer review remains high, there is also strong support for its development. In science, the peer review process is typically single-blind and has been ever since peer review was first introduced. This means that the scientists reviewing the papers (the referees) are given the names of the authors, but the authors are not told the names of their referees.

Opinions are changing and we keep an open mind.

In March this year, Nature Publishing Group (NPG) began offering their authors the option of double-blind peer review across all of their journals. This means that the authors’ names and affiliations are hidden from the referees. ‘The reviewers don’t know the name of the authors until the paper is accepted,’ explains Veronique Kiermer, an executive editor at NPG. This option was introduced after a nearly two-year-long trial on two of the publisher’s journals: Nature Geoscience andNature Climate Change. ‘We had about 20% of authors opting in for double-blind peer review during the trials,’ she says.

The introduction of a double-blind peer review option is something that NPG did in response to demand from the scientific community, Kiermer adds. ‘In particular, early career scientists around the world really show a desire for having the option of double-blind peer review due to a perception that there is bias in the [single-blind peer review] system associated with authorship.’ Whilst Kiermer says that this isn’t the case, the need to address these concerns was the reason for this option being introduced.

Double-blind peer review is widely used in the humanities, but by few scientific journals to date. ‘We think that we are the first publisher to deploy double-blind peer review at any scale in the natural sciences,’ says Kiermer. NPG intends to be very open regarding the success or failure of the experiment. ‘I think the peer review system is strained to a point by volume, and authors are expressing frustrations, and therefore it is time for all publishers to experiment and share the results of these experiments. We ought to work together to make the peer review system better.’           

Being more open

The Royal Society, meanwhile, is offering the option to do away with anonymity in the peer review process for its new journal Royal Society Open Science. This journal – which published its first papers in September 2014 – offers open peer review: meaning that both the authors and the referees know who each other are. The journal also publishes the referees’ names, along with their reports and all correspondence, alongside the paper. ‘The open peer review option has been taken up by about 70% of authors so far,’ says Stuart Taylor, who heads the Royal Society publishing department.

Open peer review is offered on a number of other journals including BMJ Open and those published by BioMed Central. ‘There are a lot of discussions about peer review at the moment and a lot of people calling for more openness and transparency in general in science,’ says Taylor. ‘As this is a new journal, we felt that it will be worth trying an experiment and rather than moving in the opposite direction which is towards double-blind peer review, we felt that it would be better to move towards a more open system.’

As the journal is less than a year old, any effect this openness is having on the quality of the referee reports has not yet been analysed. For four months in 2006, Nature trialled a similar set-up where they invited authors of newly submitted papers to have them posted on a website for anybody to comment on. For this trial, the papers were also subjected to the standard single-blind peer review process. All the comments provided were open and names were required to be included. Only 5% of the authors took up the open peer review option during the trial and nearly half of these received no comments. Most of the comments that were posted were deemed not technically substantive by the journal’s editorial staff. ‘We therefore didn’t pursue open peer review at that time,’ says Kiermer. ‘But opinions are changing and we keep an open mind. I think open peer review might be revisited [on our journals] at some point.’

Registered reports

Another accusation against peer review is that it is biased towards studies with positive or more eye-catching results. ‘There’s a lot of concern about the fact that certain types of studies get published in preference to others and that people find it difficult to publish negative or replicated findings,’ says Taylor. To overcome this, the selection process for Royal Society Open Science does not take into consideration the likely impact or importance of the work, just whether or not it is scientifically sound, when deciding whether or not to publish an article.

Publishers charge a lot of money for organising the peer review process

This summer, the journal will take a further step towards eliminating publication bias with the introduction of registered reports. Pioneered by the Elsevier journal Cortex, these split the peer review process into two distinct stages. The authors first submit their research methods, protocols and hypotheses for pre-approval for publication before data is collected. The decision to accept or reject a paper will then be made before the results are known. ‘The decision [on whether or not to publish] is separated from what the results actually are,’ explains Taylor. When the research has been done the paper will be checked by referees again and – provided the authors have followed the protocol they said they were going to follow – the work will be published regardless of whether the results are positive or negative. ‘We are hoping that this will remove publication bias effectively.’

Speeding things up

Another of peer review’s issues, regardless of which of the above guises it takes, is speed – or lack of it. It is notoriously slow for numerous reasons, including editors struggling to find suitable referees and then some referees taking longer than is reasonable to file their reports. And that’s without taking into account extra work required by reviewers. One approach to chivvying things along is to pay reviewers a nominal fee for their work. Earlier this year, NPG briefly trialled the use of an external fast-track peer review company called Rubriq on its large open access journal Scientific Reports. Rubriq charges authors $600 (£380) for its services and claims to obtain peer reviews for articles within two weeks.

‘What we wanted to do was see if paying reviewers a small amount of money, not enough to influence their decision-making process, but enough so that they put the peer review at the top of their to-do list, you could expedite the process,’ explains Nandita Quaderi, publishing director for open research at NPG. Only about 1.5% of authors submitting articles during the four-week period opted to have their article peer reviewed through Rubriq. Quaderi says the results of the trial are still being analysed to determine its success, but at this point they don’t have any plans to roll it out on Scientific Reports or any of the other NPG journals. Some of the reason for this reluctance may well be the negative reaction of some of the Scientific Reports editors to this trial. This journal has academic editors to manage its peer review process, unlike other NPG journals that have in-house professional editors, and some of academics threatened to resign when they found out about the trial, generating a flurry of media stories.

Rubriq is however being used by other publishers, but on a very small scale so far. Since it started out in 2012, it has reviewed 1125 papers. Peerage of Science, which offers a different model of external rapid peer review, has reviewed just 199 papers since 2011.   

Putting control back into academics’ hands

Publishing campaigner Velterop is pioneering an even more radical approach to improving the peer review process: taking the control out of publishers’ hands and into that of academia. From this summer, with Velterop’s support, the journals published by Science Open will offer a ‘peer review by endorsement’ option. When participating authors submit their articles to the journals, they will also need to submit reports from two peers. There are rules about who can and cannot act as a referee, such as they can’t be based at the same institution as the authors or have recently co-authored a paper with them. The signed referee reports will be published alongside the articles.

One of the big advantages of this approach, according to Velterop, is cost. ‘Publishers charge a lot of money for organising the peer review process,’ he says. He estimates that article processing fees could be slashed from around $3000 to nearer $500 for papers that use peer review by endorsement. ‘What I don’t expect is that everybody all of a sudden will start to do this. I think there may be a certain category of articles – such as those with negative results or those that are lengthier than average – for which it is interesting at first. And then once it is a more established way of doing things then other people will come on-board as well I’m sure,’ Velterop explains. ‘Not everybody buys into open access either yet even though it’s been around for around 15 years,’ he adds.

Post-publication peer review

The journals published by Science Open also host post-publication peer review on their publishing platform (as do a number of other journals). This means that comments can be added by anyone who wishes to do so underneath the articles online, provided they use their real name. There are also websites not affiliated with any publisher that have been set up exclusively for the purpose of post-publication peer review, such as PubPeer. Velterop isn’t however yet convinced by the value of this form of peer review and wonders whether it will see out the weathers of time. ‘Early experiences show that the take-up is very low and very often it’s just a comment of one line. And that’s not what I call peer review,’ he explains. ‘It is also largely very negative comments that you get and sometimes even very rude ones.’

Post-publication peer review may be a new term but its concept has been around for as long as journals themselves. ‘That is the way that science has worked for a very long time,’ Velterop says. ‘If you don’t agree with what’s been written then you write your own article refuting it or whatever and that’s a kind of post-publication peer review. So, it’s nothing new.’         

And complaining about peer review is nothing new either, as shown by Albert Einstein’s 1936 response to Physical Review after receiving his first ever referee report. ‘Dear Sir, we (Mr Rosen and I) had sent you our manuscript for publication and had not authorised you to show it to specialists before it is printed. I see no reason to address the – in any case erroneous – comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.’ And while peer review may be currently undergoing a transformational phase, it appears – that for the immediate future at least – both it and all the grumbles about its effectiveness are here to stay.

Nina Notman is a science writer based in Salisbury, UK