Getting repetitive

‘Replication is a normal and essential part of science’. That uncontroversial statement is one of the conclusions of a recent report by the Royal Netherlands Academy of Arts and Sciences. Replication and reproducibility are defining characteristics of science – so you might wonder that it needs to be said at all. Yet it seems that taking reproducibility for granted might actually be the problem. The report makes a number of recommendations aimed at making replication an explicit component of research, even going so far as to propose that funding should be made available specifically for repeat studies.

The report is the latest response to the concerns about science’s ‘reproducibility crisis’ – a growing concern that the results of published studies cannot be replicated. The problem is particularly severe in psychology, but it affects all sciences to some degree. In 2016, Nature surveyed 1500 scientists, 90% of whom said that science has a reproducibility problem. And a higher proportion of chemists said they’d failed to replicate someone else’s work than scientists from any other field.

We’ve covered this topic several times over the last few months. Most recently, Mark Peplow discussed an example from materials science. This time its David Sholl’s findings that gas adsorption measurements of MOFs could be unreliable. As Peplow notes, Sholl’s findings underline the point that the problem isn’t just that some results can’t be reproduced – it’s the absence of any repetition at all.

That’s disappointing, but perhaps not surprising. In the culture of professional science, certain aspects of the scientific process are emphasised over others – in particular, those efforts that yield publishable outputs. That often means satisfying criteria such as novelty and impact; there is little reward for confirmation, replication or review. This is hardly a modern phenomenon, however, and science manages to muddle on – science’s reliance on prior work and its incremental nature means it can tolerate a few faults in the structure. The extraordinary claims receive extraordinary scrutiny and if something unremarkable goes unchecked for a while, well what’s the harm?

Yet the potential for wasted effort is too great to ignore. And perhaps worse is the potential for wasted opportunities. For example, machine learning and artificial intelligence approaches that can analyse reaction sequences or predict material properties could be immensely powerful, but these systems will only be as healthy as the data they are fed. And then there’s the erosion of trust, both in terms of scientists’ confidence in the corpus and the public’s faith in science.

It’s 10 years since the financial industry imploded, as structures built on bubbles and bad debt collapsed with devastating effect. One of the biggest changes that happened in the aftermath was the emergence of bitcoin – a currency that did not rely on trusting banks. Bitcoin is creating financial concerns of its own these days, but its underlying blockchain technology is attracting interest from other sectors because of its inherently open and transparent nature – a shared record of events where every interaction is known to everyone.

Science would seem to be very far from the catastrophic failures that brought down those financial institutions but there are parallels – short-termism, incentives skewing behaviour, and closed or opaque systems of trust. And in bitcoin and blockchain, we see the same principles of transparency that are championed by advocates of open research. Some anlaysts are predicting that blockchain itself could provide answers to science’s trust issues.

Hopefully these efforts mean that we’ll soon see questions about reproducibility in science attain the same status as queries about Popes and bears. Do chemists reproduce in the lab? You bet they do.