Post publication peer review comes of age

It is rare for scientific controversies to make it into the mainstream media. But the high profile retraction of a paper on stimulus-triggered acquisition of pluripotency (STAP) by Nature in July – just five months after its publication – made national news.

The lead author of the paper – which described a new way of producing stem cells by exposing mature cells to stress – was found guilty of misconduct, after figures and data in the paper were found to be mislabelled. But suspicions were raised long before official investigations began on PubPeer, a ‘post publication peer review’ website set up by an anonymous group of scientists. On PubPeer, unnamed researchers can comment on others’ research papers and the authors are then automatically notified and invited to respond. Attention was first drawn to inaccuracies in the STAP paper by researchers unable to replicate the study.

In the relatively short time since the site’s conception, comments posted by reviewers both named and anonymous have fuelled controversies across various disciplines, such as over the ‘striped nanoparticles’ papers published 10 years ago that still rumbles  on to this day. And last year’s striking images of hydrogen bonds have also recently come under scrutiny.

Social commentary

This kind of online debate is not limited to PubPeer. Many journals offer comment facilities under papers uploaded on their own websites. Social networks such as Twitter and personal blogs have also enabled researchers to share and comment on papers. This trend towards trial-by-internet seems to be gathering pace in many areas, but does it threaten to topple the traditional peer review process?

The mechanism has changed because the community has grown and grown and grown

For David Fernig, a biochemist at the University of Liverpool in the UK, systems that allow post publication comments have always been an essential part of the scientific process. ‘Without being able to have those discussions, those arguments, science actually does not exist,’ he says. He is keen to point out that post publication peer review is not a new practice. But now, thanks to online forums, discussions that might once have been restricted to scientific meetings can take place more publically.

‘There was a time when you could fit all the molecular scientists on Earth into a meeting at Cold Spring Harbour [Laboratory, US] – today the largest stadium wouldn’t accommodate them all,’ Fernig says. ‘The mechanism has changed because the community has grown and grown and grown.’

Sites like PubPeer, he adds, essentially act like a ‘giant journal club’, allowing researchers to critically analyse published work. Often – due to the sheer numbers of people involved – this is the only way that flaws and inconsistencies that have gone unnoticed in the original peer review process can be revealed.

Flawed process

This was the aim of PubPeer’s moderators when the site was originally set up in 2012. ‘Like all scientists, we were frustrated at the difficulty of discussing flaws in published work,’ one the site’s moderators tells Chemistry World. The site offers an alternative to the lengthy and often frustrating process of trying to raise criticisms through more official channels.

Philip Moriarty from the University of Nottingham in the UK has been involved in the striped nanoparticles debate for years, and has participated in discussions on PubPeer, as well as in the literature. He is all too familiar with the difficulties of trying to publish work that criticises existing papers.

‘We contacted a number of the journals who had published stripy nanoparticle work in the past, and basically they weren’t interested in publishing corrections,’ he says. Even when they found a publisher willing to take their work, it took almost a year for it to be published. ‘And all the time the paper’s linked on PubPeer picking up loads of comments,’ he says. It is little wonder that the site has become a first port of call for those who want to follow or join debates on published work.

Up close and personal

For all its successes, PubPeer’s acceptance of anonymous comments has attracted criticism. As with any internet forum, there is a fine line between constructive criticism and ‘trolling’ – posting disruptive or malicious comments.

‘There are cases where the authors have got back and very comprehensively addressed comments, and that’s improved the situation,’ says Moriarty, ‘but of course when you’re critiquing somebody in a public forum like that it can turn nasty very, very quickly.’

When you’re critiquing somebody in a public forum it can turn nasty very, very quickly

In the striped nanoparticles saga, heated discussions on PubPeer have caused the papers’ original author, Franceso Stellacci, to accuse critics of personal attacks. And in another recent development, one researcher has brought a legal case against the moderators of PubPeer. Fazlul Sarkar, a professor in the department of pathology at Wayne State University in the US, is seeking to force PubPeer to disclose the identities of people who commented on his work, as he claims that a job offer from the University of Mississippi was withdrawn as a direct result of what was said on the website.

But the moderators intend to fight this action, and are convinced that their policy on anonymity has been one of the biggest drivers of the site’s success. ‘Initially we only accepted comments from registered users,’ one says. ‘Once we started accepting anonymous comments, it became clear that many of the most significant comments had been held back by fear of reprisals.’ Fernig agrees that anonymity is crucial, particularly for younger researchers who are worried about their career prospects. ‘They wouldn’t dare use it otherwise – it’s that simple.’ he says.

Anonymity optional

Other sites aim to serve a similar function to PubPeer but without anonymity. PubMed Commons allows authors with papers indexed in PubMed to comment on others’ work. And Open Review, which was launched by networking site ResearchGate earlier this year, allows registered members to upload lengthier, more formal reviews of published papers. Both require commenters to be named, but so far neither have been associated with as many high-profile misconduct cases as PubPeer. Some publishers, such as F1000 research who handle life sciences papers, have even adopted post publication peer review as a way of vetting manuscripts, with pre-publication treatment limited to a quick ‘sanity check’ by editorial staff.

Moriarty agrees that anonymity is important, but says some of the more vitriolic exchanges on the stripy nanoparticle work has made him realise the existing setup isn’t perfect. For him, a happy medium would be for PubPeer to allow anonymous comments, but to first accredit users to make sure they were appropriately qualified to comment on research. ‘Assuming the legal challenge doesn’t go ahead and that you can’t just subpoena the email addresses, then that to me seems to be the best of both worlds.’ he says.

Another issue entirely is where the responsibility for maintaining these forums should lie. Many agree they are best kept independent from publishers.

Chemistry under the spotlight

But maintaining systems can take a great deal of time, effort and resources for little or no return. This is a problem that See Arr Oh – a US-based chemist who blogs anonymously – has experienced. In 2013, he was part of a group who set upBlogSyn – a project which aimed to scrutinise published chemical syntheses in the same way as the journal Organic Syntheses. Each member of the group would try to replicate the synthesis in a published paper, reporting back on whether they were able to make the reactions work and achieve similar yields to the original authors, and sharing details of their set up, pictures and results on their blog.

The pursuit of knowledge and science isn’t the goal. Capitulation of one combatant to another is

The project received mixed reviews from the community. ‘Graduate students and postdocs everywhere said, “Wow, this is really cool, how can I participate?”’ See Arr Oh tells Chemistry World. But he adds the reception from more senior researchers was largely negative. ’There was a back-channel of communication that went on behind the blog … let’s just say they were not always friendly.’

After a handful of posts the blog was discontinued, due partly to negative responses and threats, but also because the amount of time and effort needed to keep it going was just too much for such a small group.

However, some tangible benefits came out of the project. Synthetic chemist Phil Baran, whose work was the subject of one of the BlogSyn posts, responded by setting up a similar blog scrutinising his own experiments, complete with pictures and insider tips on how to run the reactions. And the bloggers did successfully engage with other authors to identify where their written methods could have been clearer. ‘We uncovered things that the original authors had not looked into that were critical for the reactions’ success,’ says See Arr Oh. ‘That was the best validation of what we had done.’ The main problem, he explains, is finding a way to continue this labour-intensive but valuable work, while retaining the independence and impartiality of a small group.

Collegial thinking

‘Even though we only did six experiments, it was six experiments across a bunch of different continents and time zones, and we relied on the good nature of our respective employers,’ he says. ‘But throughout the process people would contact us to say “Is this a viable business model? What if I were to invest in this?” And the answer to that was just sort of no, because once you institutionalise it you lose some of the feeling that you’re doing the right thing.’

PubPeer’s moderators echo this line of thinking, saying they attribute their site’s success to its independence from journals or publishers, as well as the acceptance of anonymous comments.

One relatively new review site that has attempted to apply the PubPeer model to synthetic chemistry is SynValuate, which has similar aims to BlogSyn, but ‘crowdsources’ reviews from a community of anonymous users like PubPeer. Users can post their own syntheses, or review others’ procedures from the literature, attaching a star rating and comments. Similarly, the Royal Society of Chemistry’s ChemSpider Synthetic Pages – while not strictly speaking post publication peer review – allow users to submit details of reaction procedures they have tried, which are given a DOI and can then be searched, cited, viewed and commented on by community members.

In a perfect world, See Arr Oh would like to see a dedicated group set up to check the reproducibility of as many syntheses as possible – but warns that such a thing would have to be well funded. ‘If you wanted to do that for even 5% of the current organic chemistry literature, you would need to put together a small staff of probably 20 or 30 people,’ he says.

While See Arr Oh’s vision remains, for the time being, a pipe dream, it is clear that online post publication peer review is here to stay. A recent report from the UK government’s science and technology committee recognised its growth, but stressed the continued importance of the traditional review process for assessing the technical soundness of research, saying: ‘We encourage the prudent use of online tools for post publication review and commentary as a means of supplementing pre-publication review.’ For PubPeer specifically the short term future is likely to be dominated by the pending lawsuit. Assuming this goes well, the moderators say they are optimistic that the site, and its influence, will continue to grow.

But for many, post-publication review remains a thorny issue. The American Association for the Advancement of Science’s Kent Anderson has criticised online commenting in the past. In a post on the Society for Scholarly Publishing’s Scholarly Kitchen blog, he said: ‘Far too often, commentary forums degrade into polemical attacks with win or lose dynamics at their heart. The pursuit of knowledge and science isn’t the goal. Capitulation of one combatant to another is.’ It remains to be seen whether critics will embrace the idea more fully in future. ‘I think publishers mainly see it as a threat,’ says Moriarty. ‘But maybe they should start to see it as more of an opportunity.’