Quality by numbers

The UK government has announced a shake-up in the way university research is assessed and funded. In its pre-Budget report on 6 December, the Treasury set out a new framework for the Research Assessment Exercise (RAE), which rates the quality of university research in preparation for receiving government funds (so-called QR, or quality weighted research funding). 

The new system will measure research quality with a collection of numerical measures (metrics), such as the amount of research income a department earns, its number of PhDs, or its researchers’ citation rates. The aim is to streamline the current RAE, widely criticised as onerous and time-consuming, in which expert panels have to rate research submitted by academics. Even non-scientific disciplines, less suited to a quality-by-numbers assessment, will face a ’reduced, light-touch peer review process.’ 

Metrics will not be introduced in a hurry, though. The Higher Education Funding Council for England (Hefce) is taking a further year of consultation to decide precisely how packages of metrics can be introduced for different subjects. This leaves the 2008 RAE - itself modified from the previous 2001 RAE - to go ahead under the old system of panel peer review.

After shadow trials alongside the 2008 RAE, the new system will be tried out in science-related subjects in 2009, and the ratings it produces will affect QR funding in 2010-11. Other subjects will have to wait until 2013 for their first assessment. 

Aside from saving time and energy, a metrics-based system would also allow fast, up-to-date assessments of a department’s research health. As Bristol vice chancellor Eric Thomas points out, Hefce’s ?1.3 billion QR funding this year is still apportioned according to the 2001 RAE ratings, which in turn were based on research surveys as far back as 1996. 

Chemistry should fit well into a metrics-based exercise, according to Jeremy Sanders, chair of the 2008 RAE Chemistry panel. He points out that statistics such as grant income and spending on facilities are already considered under the current peer review system. Rather more troublesome, says Sean McWhinnie, science policy manager for the Royal Society of Chemistry, is how a metrics approach would measure esteem and scholarship. 

High citation rates, for example, could suggest that researchers had published scholarly work. But it could also mean that they had the good fortune to publish in a highly cited area, or worse, that they merely published controversial work, or reviews of the literature. 

Perhaps with this in mind, Ole Petersen, Chair of the Royal Society working group on the RAE, has judged the government’s apparent rejection of a peer review process ’extremely disappointing.’ ’There is no substitute for expert judgement in assessing research being undertaken by a university department,’ he said. 

Phillip Diamond, assistant director of science and education at the Institute of Physics, was equally disappointed. ’The only system that will have the respect and support of the science community is peer review of research,’ he said. 

But McWhinnie has cautiously welcomed the new system, so long as a basket of metrics is used, rather than a reliance on one statistic like research income. Some element of peer review could happily lie behind a metrics-based system in the sciences, he pointed out. For example, experts could decide which metrics would best summarise a subject, and how their relative importance might be weighted; problematic metrics such as citation rates could be regularly reviewed. 

And many contacted by Chemistry World thought the RAE itself caused so many problems for UK research that squabbling over whether metrics would produce a fairer rating rather missed the bigger picture. ’People now talk about planning research for the RAE, not for knowledge generation,’ lamented Thomas, while Michael Sterling, vice chancellor of Birmingham University, said that the RAE inhibited interdisciplinary research and wasn’t supportive of applied research. 

The 2006 pre-Budget has also tried to address the latter problem, with the announcement of an extra ?60 million QR funding from 2007-08 for applied research. But Peter Cotgreave, director of the Campaign for Science and Engineering, was not impressed by the hand-out: ’In terms of universities, ?60 million for applied research is not going to solve their financial problems,’ he said. ’It won’t save financially struggling science departments from closure.’ 

Richard Van Noorden