A question of reproducibility

People can get a little defensive when David Sholl takes the stage at conferences. It’s understandable – he’s often about to tell them there could be a one in five chance their results are wrong.

Sholl works on metal–organic frameworks (MOFs), one of the fastest-growing families of materials with about 70,000 structures logged in the Cambridge Structural Database over the past 20 years. MOFs are lattices of metal-based nodes connected by organic linkers, and their pores can be used to store or separate gases such as carbon dioxide, methane or hydrogen.

For that reason, reports of new MOFs are often accompanied by CO₂ adsorption isotherm measurements, which show how much gas a MOF sucks up at various pressures and constant temperature (give or take 5°). One might therefore expect that this property is well established for many MOFs. Not so, according to Sholl.

One in five CO₂ adsorption results in MOFs appear to be unreliable

Last year, he and his colleagues from the Georgia Institute of Technology in Atlanta set out to assess the reproducibility of these measurements by comparing different values obtained for the same material. Having trawled a database of thousands of MOFs curated by the US National Institute for Standards and Technology, they found that only 27 MOFs had more than one measurement listed, and just nine MOFs boasted five or more replications. Worse, statistical tests showed that about 20% of the results for those nine MOFs were ‘outliers’, completely at odds with the other measurements for the material.¹

There’s no suggestion of misconduct, but the implications are stark. One in five CO₂ adsorption results in MOFs appear to be unreliable — but we cannot be sure how widespread the problem is, because only a tiny fraction of MOFs have had this key measurement reliably reproduced.

Robust results

This is not the first time that gas adsorption measurements have raised concerns. Back in 1997, for example, carbon nanotubes looked to be an ideal material to hold H₂ as a transportation fuel, after reports that they could store 5–10% of their own weight in hydrogen at ambient temperature. Yet the results could not be reproduced, largely because the samples also contained a lot of soot and metallic nanoparticles.²

H₂ adsorption measurements are tricky, because the amount of gas being adsorbed is tiny. CO₂, in contrast, adsorbs much more strongly in MOFs, and the measurement should be very replicable because it uses a commercial instrument. So the data gaps and discrepancies found by Sholl are more likely to be caused by disparities in the materials themselves, rather than errors in the measurement.

No two syntheses are ever identical, but a material must be robust to minor variations if it is to be practically useful. It’s also important to have reliable numbers for theoretical simulations, which are increasingly being used to guide materials research. Fortunately, there are clear steps that the research community can take to deal with this predicament.

Plugging the gap

The explosion of interest in MOFs brought lots of people to the field, and many of them focused on making new MOFs. That perhaps led to less attention being devoted to characterising and improving existing materials, work that almost inevitably involves replicating key measurements such as CO₂ adsorption. And even when researchers made the measurements, they might not have published them. Both of these issues need the research community — including funders and journals — to decide that CO₂ adsorption is a key property for characterising MOFs that must always be reported.

Greater transparency could also help to resolve the questions hanging over the mysterious ‘outliers’. We can only speculate about the reasons for these rogue results because the raw data is not available; making those data public could reveal the cause.

The Protein Data Bank (PDB) uses this strategy, requiring researchers to supply raw data when they submit new structures. Various validation tools are then used to spot any problems before the structure is accepted, and journals increasingly expect researchers to submit an official validation report from the PDB along with their paper.

Independent verification can also help. Researchers who make photovoltaic cells always report their power conversion efficiencies, for example, but the gold-standard measurements come from recognised test centres, such as the US National Renewable Energy Laboratory. Even without a formal service like this, Sholl suggests that a concerted effort by a relatively small number of research groups to check CO₂ adsorption measurements could make the MOF literature much more reliable.

And probing well-established MOFs can still yield significant breakthroughs. Take HKUST-1, one of the materials that kicked the field into high gear when it was made in 1999. In December, David Fairen-Jimenez at the University of Cambridge showed that simply by using a different preparation method, the methane adsorption of HKUST-1 could be increased dramatically, giving it the highest reported capacity for any porous solid.³

There’s no doubt that replication takes effort, but it is always worth building stronger foundations for a burgeoning field. Just consider how much more time might be wasted if the literature is plagued by phantom results.