Meaningless metrics limit researchers

No comments

It’s time to end the misuse of metrics

I’m a sucker for gamification. Give me points or badges to reward my efforts and I’ll obsessively rearrange my life to get more – for a couple of weeks at least.

But numbers without context can be misleading. The Duolingo app recently congratulated me on spending 223 days in a row learning Japanese. Sounds impressive: time spent learning a language should be a rough proxy of linguistic ability. But after all that time the only sentence I can reliably translate is ‘I eat vegetables’. The problem is that only my activity within the app is rewarded; the hard work that I’d need to do outside the app to consolidate my learning is not.

There are parallels here with how we reward and assess academic researchers. Publication metrics – which are much easier to measure than it is to objectively assess the quality of a researcher’s work – have traditionally been used as a proxy for research ability. The number of papers a researcher produces supposedly indicates productivity; and citation counts apparently represent the quality of that work.

But, just like with my language-learning streak, high numbers do not necessarily indicate high quality. On first glance, ‘salami slicing’ a big study into five different papers looks as productive as publishing the results of five independent studies. And as a database published last year shows, some researchers are citing themselves at rates that could suggest they’re gaming the reputation system.

That’s not to say self-citation is always bad – after all, it would be strange if researchers never built on their previous work. What’s damaging is that we have a system where researchers might limit how often they cite their own results for fear of having their professional standards called into question.

On p58, Rachel Brazil investigates some of the initiatives that are aiming to improve how we reward and promote researchers. I’m pleased to see that many of these efforts are moving away from metrics, and placing more weight on free-text summaries that allow researchers to summarise their contributions in their own words. Even better are the moves to show that researchers are more than what they publish. Teaching, mentoring and serving on committees are valuable activities that benefit the community and deserve incentives to encourage more people to take them up.

This month in Chemistry World, Rachel Brazil has investigated some of the initiatives that are aiming to improve how we reward and promote researchers. I’m pleased to see that many of these efforts are moving away from metrics, and placing more weight on free-text summaries that allow researchers to summarise their contributions in their own words. Even better are the moves to show that researchers are more than what they publish. Teaching, mentoring and serving on committees are valuable activities that benefit the community and deserve incentives to encourage more people to take them up.

Of course, the reason why we use metrics is to save time. But important decisions like funding and promotion deserve to have time spent on them. Reduced to a series of numbers, an academic career becomes as limited as my Japanese. Yasai o tabemasu.