Let's see that again

A few years ago, the topic of whether scientific papers are reproducible or not would have been an odd thing to see in a newspaper. But not any more: both the popular media and the journals themselves have been trying to deal with the topic, amid reports that far too many results can’t be replicated. Large scale efforts have begun to examine key papers in experimental psychology, among other areas. Reports from the biopharma industry about the numbers of interesting biology papers that don’t hold up have stirred alarm as well. But as far as I can tell, chemistry has largely escaped the current rounds of criticism.

 Do we deserve to? We do have a few natural advantages that protect us. Our experiments are (for the most part) easier to check than some other fields of research. The reagents are more readily available, the products are easier to thoroughly characterise, and our experiments are less affected by the subtle sources of bias in behavioural psychology and by the equally subtle variabilities in cell and molecular biology. And we don’t have the gigantic scale of some of the cutting-edge physics experiments to deal with, either. All in all, a motivated experimentalist could, in theory, take a crack at the majority of the papers that show up in chemistry journals. 

But we still have our own smaller-scale reproducibility problems, as anyone who’s tried following literature procedures will tell you. There are sins of omission all over the place, with people glossing over tricky parts in the experimental procedures and not citing relevant literature along the way. The excuse for that sort of thing used to be space constraints in the journals, but with the advent of electronic supplementary information, only the less honorable reasons would seem to be left. 

A motivated experimentalist could, in theory, take a crack at most chemistry papers

These are the problems most related to reproducibility: the reported chemistry does work, more or less, but it turns out that the solvent has to be degassed, and reagent Y can’t be used out of the bottle, and that the high quoted yield is the one outlier when the reaction really worked well, and so on. 

All this still doesn’t rise to the level of the problems that some other sciences are dealing with. If you report that showing people pictures of iguanas makes them (the people, not the lizards) perform better on vocabulary problems, then you and your LVE (Lizard Verbal Enhancement) theory will have a problem if no one else can demonstrate any such thing. 

I can think of similar instances in chemistry where the entire paper was wrong, top to bottom, but because our variables are more easily characterised, most of those have been outright frauds. (That’s as opposed to the situation in other fields, where there are indeed frauds, but there are also a lot more reasons for people to run with shaky results that they truly believe). Complete fakery in chemistry is relatively infrequent – one hopes. 

In my more optimistic moments, I tend to think that chemistry is a bit more protected even from that, because those lower barriers to replication make it harder to get away with if the paper is on a subject that anyone cares about. I’m sure that we have just as many pathological people as the other branches of science do, but they just have to find other ways to demonstrate their talents.

This all makes chemistry a good counterexample to those people (and there are some) who argue that the reproducibility crisis shows that there’s something rotten at the centre of the whole scientific enterprise. But before we get too full of ourselves, it’s worth remembering what may be our biggest shame: no one would bother reproducing many of the papers that appear, much less faking them, because they fall into Wolfgang Pauli’s category of ‘not even wrong’. We have too many poor-quality journals, full of papers that no one reads and no one cites. They exist because there are too many people carving their results into tiny slices to pad their publication records, which in turn means that there are too many people, and committees, impressed by long publication lists. 

So here’s a rallying cry for you: get out there and produce some results that someone else would have wanted to fake! Lack of reproducibility is a sin, but there may be even greater ones.

Derek Lowe (@DerekLowe) is a medicinal chemist working on preclinical drug discovery in the US