The illusion of self-correction

As retracted articles accumulate, and concerns are voiced more and more strongly about the reproducibility of research findings, many scientists continue to find consolation in the idea that science is inherently self-correcting. It is not immune to fraud and error, but invalid findings are exposed and discarded when other researchers identify flaws or obtain better data. Through this winnowing process, science continues to advance toward the truth despite encountering false leads along the way. While there is certainly some truth to this notion, the process of self-correction is far from perfect. Consider, for example, the case of the reactome array.

In October 2009, Science published a paper claiming to have synthesised 1676 quenched fluorescent dye-metabolite compounds capable of screening for enzymatic activity.¹ The publication was met with scepticism, with several prominent figures expressing doubts about the stability of vital chemical linkages, despite the subsequent release of a revised figure and additional supporting information.

In January 2010, Science issued an ‘Editorial expression of concern’,² noting that ‘serious questions have been raised about the methods and the data’ and that ‘new descriptions of the synthetic methods that differ substantially from those in the published article’ had been provided. Six months later, an ethics committee from the Spanish National Research Council’s Institute of Catalysis (CSIC) – the institution of the lead author and several of her co-authors – recommended that Science retract the article, citing ‘clear indications of deviation (from) good scientific practices in the lack of controls… in the treatment of data… [and] in the responses to the journal’. The authors received some support from Nobel laureate Richard Roberts, who visited the Madrid laboratory of the corresponding author, Manuel Ferrer, and confirmed that the array was able to accurately identify a number of blinded protein samples. Nevertheless, the paper was retracted in November 2010, owing to ‘errors and omissions in the information provided on the chemistry underlying array compound synthesis’.³

Deeper concerns

At first glance, the system for self-correction appears to have worked. A published paper was found to be flawed, the community responded, the institution investigated and the journal promptly withdrew the paper from the literature. However, there is more to this story than meets the eye.

Last year, Pere Puigdomènech, chair of the CSIC ethics committee, wrote a commentary on the Spanish experience with research integrity issues that included a brief discussion of the reactome array paper. This caught the attention of Thomas Hettinger, a chemist at the University of Connecticut, US, who after reading the initial Science paper had performed a forensic analysis of the mass spectrometry data. To Hettinger’s dismay, he found that the authors had mistakenly used molecular weight instead of molecular mass in predicting the mass values of the metabolites.⁴ The former is simply the sum of the atomic weights of the component elements; the latter is calculated from the atomic masses of the dominant isotopes. He was also perturbed by the highly consistent relationship between the values calculated and those found by the array, and the listing of found values for some ‘impossible’ compounds. He concluded that the ‘lack of variation of the found values proves that they have been fabricated’. Hettinger contacted Science with his findings, but the journal told him that investigation of possible misconduct is the responsibility of the institution, not the journal.

In his reply, Puigdomènech said that, acting on behalf of the committee, experts from outside CSIC ‘analyzed the article and reached conclusions very similar to those indicated by Dr Hettinger’. But he added that ‘actions such as the identification of individual responsibilities or to apply disciplinary measures were outside the mandate of the committee’.⁵

Ferrer initially stated that ‘his team continues to use the technology and plans to submit new papers with thorough chemical characterizations’.  However, five years have elapsed since the retraction of the paper, and no additional articles have been published. The first and corresponding authors of the reactome paper were contacted but did not respond to our requests for a response to Hettinger’s analysis. The first author, Ana Beloqui, has previously denied any wrongdoing and stated that ‘the array generally worked as anticipated’.

A lack of answers

What can now be concluded about the ‘reactome array’? The ability of Ferrer’s lab to rapidly and accurately identify blinded enzyme samples provided by Roberts suggests that the array was in fact not a hoax, although a more rigorous test would have been to characterise complex cellular or tissue extracts. It is likely that the senior authors believed in the validity of the array, otherwise they would not have filed a patent application for the technology and solicited external collaborators. Nevertheless, the errors in the initial description of the chemical strategy used to construct the array suggest that the investigators did not have an adequate understanding of the chemistry involved. Moreover, Hettinger’s analysis suggests that at least some of the supporting mass spectrometry data were, in his words, fabricated.

This is the current unsatisfying state of affairs with regard to self-correction. Science can be self-correcting, but this requires the concerted efforts of scientists, journals, institutions and governments. The self-correction process lacks transparency and consistency, and many potential conflicts of interest may interfere along the way. The reactome array paper retraction notice cites only error on the part of the authors and not the wider concerns.

In Spain, as in many other countries, there is no independent agency with the authority to oversee investigations of alleged research misconduct. It is time for countries, journals, scientific societies and funding agencies to forge a consensus for a uniform approach to the problem of questionable papers in the literature. As a community, we can and must do better. The credibility of science is at stake.

Ferric Fang is a professor of laboratory medicine, microbiology, medicine and pathobiology at the University of Washington, US. Arturo Casadevall is a professor of molecular microbiology and immunology at Johns Hopkins University, US

Editor: An earlier version of this article noted that Pere Puigdomènech, chair of the CSIC ethics committee, had declined to comment. This was not accurate. Manuel Ferrer and Peter Golyshin, corresponding authors of the reactome array paper, were contacted by Chemistry World but did not reply. Pere Puigdomènech was also approached for comment prior to this article’s publication and has responded to say that he provided Ferric Fang with an interview in 2014 to discuss the matter.

Searching for the truth

When questionable science is corrected, it is often pointed out – illogically – that the system works. Yet we have no way of knowing how much faulty science exists or whether it will ever be corrected. The system of self-correction does not work because authors, institutions and journals can produce different versions of the same story. We are left in a quandary as to what is science fact and what is science fiction.

My analysis of the reactome array paper concluded it contained thousands of fabricated mass spectrometry data values, yet fabrication was not even mentioned in the retraction notice. The paper was retracted – not for fabrication or fraud – but for reasons of ‘skepticism’ and ‘errors’ in the chemistry. When a paper is retracted, readers might expect that the authors, their institutions and the journal provide a verifiable narrative of events that led to the publication and retraction of the paper. That did not happen. Instead we have a vague, ambiguous story that leaves readers in the dark.

The reactome array story demonstrates the difficulty of resolving disputes about errors and misconduct allegations. It is one of the most elaborate cases of alleged research fraud ever seen in chemistry, yet it has still not been fully adjudicated five years after the paper was retracted.

International standards should be developed to deal with allegations of research misconduct, but this will be very difficult to achieve – for many of the same reasons that authors, institutions and journals do not agree on what is valuable science. For legal reasons, it is practically impossible to arrive at a real closure for cases of research misconduct. The US Office of Research Integrity (ORI) recently reported findings of research misconduct against Anil Potti, who, despite an extensive list of retractions and clear evidence of research fraud in multiple journal articles, reached a legal settlement whereby he ‘neither admits nor denies ORI’s findings of research misconduct’.

Perhaps a policy of reconciliation could be used to motivate fraudsters to admit what they did without penalty and so achieve a true ’self-correction’. That would, ultimately, help to make science more credible.

Thomas Hettinger is a research associate at UConn Health, US