Diet approach to DFT benchmarking

Preliminary benchmarking of density functional theory (DFT) methods no longer needs to take as long. So says Tim Gould, of Griffith University in Australia, who has developed three diet subsets of a well-known molecular database.¹ These systems can accurately rank DFT methods with fewer computational resources than comprehensive benchmarking. Gould hopes they will help efforts towards benchmarking solid-state methods, by using datasets that are more comparable to solid-state systems.

There are hundreds of different DFT approximations, better known as functionals, out there, and method developers rely on benchmarking to test new ones. They are often ranked according to their accuracy: a more accurate functional is able to reproduce key chemical properties that can be calculated by a higher level of theory for comparison. One state-of-the-art benchmarking system is the General main group thermochemistry, kinetics, and non-covalent interactions database – generally abbreviated to GMTKN55. It is designed for gas-phase molecular functionals, containing 1500 energy differences, and is able to analyse DFT functionals to find an overall quality measure. ‘The GMTKN55 database has quickly become a new standard to analyse the vast zoo of DFT methods,’ explains database creator Lars Goerigk from the University of Melbourne, Australia. His work in this area has provided insight into hundreds of DFT approximations.^2–4

Evolve and optimise

‘I’m a bit grumpy about chemists always looking at molecules, and ignoring materials in the solid state,’ says Gould. ‘We needed a way to test on both types of systems simultaneously. The problem is, making tests for solids is much harder than for molecules, so even if we had a comprehensive test for solids it would be so much smaller than what was out there for molecules. To cross this gap we needed something smaller than the state-of-art for molecules, which ideally was just as accurate.’ Gould got in touch with Goerigk to ask if they had any plans for a mini version of their GMTKN55 work, but it wasn’t a direction that Goerigk’s research was heading. ‘If I wanted a smaller set I’d have to do it myself,’ says Gould.

Fortunately, all of the work for GMTKN55 was available online so Gould could easily gather the information he needed. He condensed the data down to a representative subset using two optimisation phases that were loosely based on evolution: the optimisation could evolve to give a best-fit system. This only took a few minutes on a laptop and the resulting diet subsets of 100 and 150 systems were able to reproduce the mean absolute deviation of each functional to good accuracy compared to the full fat GMTKN55 database. Fewer than 400 single point calculations are needed to benchmark with the 150 system subset, compared to comprehensive benchmarking which can require nearly 2500 calculations. An additional starvation subset of only 30 systems wasn’t quite as accurate but it was still able to reproduce key trends.

‘A smaller dataset is a boon for functional developers,’ says Torbjörn Björkman from Abo Akademi University in Finland, who works on benchmarking systems in the solid state. He thinks this diet set will help developers who ‘often need to make quick tests to see whether ideas seem to lead in the right direction at all.’ He adds that Gould’s work ‘also raises some interesting questions concerning the tests themselves.’ Björkman suggests that it could be possible to find a subset that, instead of reproducing averages like Gould’s work does, perhaps maximises the differences between methods. Then developers could investigate which methods are the most alike and dig a bit deeper into the benchmarking methods themselves.

I’m a bit grumpy about chemists ignoring materials in the solid state

However, Gould notes ‘you can never get the same sort of statistical depth out of a diet set as it simply doesn’t cover as much chemistry as its full fat parent.’ Goerigk agrees adding, ‘while it does not replace GMTKN55, it is a very useful addition and can be used for preliminary testing of new methods.’

‘This work would never have happened without Goerigk’s team making their work available, which let me do new science on existing data,’ says Gould.

In the future, Gould plans to try to do for solids what GMTKN55 and others have done for molecules, and create the biggest and best benchmark possible. ‘I’ll be begging people I know – and maybe don’t know – who’ve made steps in that direction. That would finally help me get less grumpy about the focus on molecules. But what I really hope is that others can take my data or the method I introduced and do something new with it, like I did.’