Comment

Karen Gurney and Jonathan Adams of Evidence Ltd investigate how the numbers stack up

From 2009, there will be a step change to the way the UK government funds research in universities and colleges. The Research Assessment Exercise (RAE), with its dozens of panels of academic peer reviewers, will be swept away.¹ Instead, there will be the Research Excellence Framework (REF) - a new assessment process that will use a metrics-based formula to determine universities’ research funding allocations. And rather than assessing research, a slimmed down peer review process will ’moderate’ the outcome. 

The funding body for English universities and colleges, the Higher Education Funding Council for England, has said that the key measure of research quality to be used in the REF will be bibliometrics. More specifically, it will look at the citation rate for journal articles catalogued on the databases of Thomson Scientific, which covers about 8500 leading international journals. On 22 November, Hefce launched a consultation on its proposals.² 

As more nations try to ensure their universities are among the best in the world, there is also growing interest internationally in measuring and comparing research impact.^3,4 

So, in the light of this increasing emphasis on metrics, where does UK chemistry stand?  

The global picture 

The UK performs strongly in international comparisons, as shown in our reports to government. That pattern is seen for chemistry, and for the research base generally. 

We took the broad Thomson journal category for chemistry and looked at two five-year periods, using these ’windows’ to avoid annual perturbations. The UK improved between the two windows and remains ahead of most other major research economies, second only to the USA in our analysis (see table 1). 

The US actually fell back over the same period because of the growing influence of new competitors such as China and the Asian tiger economies.⁵ 

The UK picture 

Within the UK, in 2006, chemistry articles made up about 9.5 per cent of the total of Thomson Scientific’s catalogued publications with at least one UK address. This was a decrease from a share of over 12 per cent in the early 1980s. That reflects two things: a relatively faster rise in outputs from other fields, particularly biological and environmental sciences; and the effect of management of the database, which has introduced more social science journals. 

In the 1980s, UK chemistry declined relative to the competition (see figure 1). Since the late 1980s, however, the UK has shown progressive improvement compared to the world average and has done so despite the growth of new competitors in Asia. This may be partly attributable to the process of research selectivity and concentration applied to higher education through the RAE. The improvement is underpinned by growth, so the current research base is not only producing excellent research but more publications of the highest quality. However, there are signs that output has now plateaued and the improvement in impact may also be levelling out. 

Figure 1

These measures of impact are averages, taken across more than 5000 articles per year: a diverse range of material of different quality. All research data are skewed, with many occurrences of low value and a few high value outliers, so we transform the data to get an informative picture of what that distribution looks like. We build impact profiles not of cites per paper but of citation categories relative to world average.⁶ 

Figure 2 reveals the impact profile of UK chemistry research, compared to the UK research base in all subject areas, for the most recent 10 years (1997-2006). The distribution of excellence across chemistry research is similar to that of the research base as a whole. About 17 per cent of journal articles remain uncited in any 10-year period. This is less than for other subjects and more than half were recent publications, from the last two years.  

Figure 2

What may surprise many is that, although the average impact of UK research is well above world average (see table 1 and figure 1), the mid-point of the impact distribution is below world average. In fact, the share of journal articles uncited or with impact below world average is actually over 60 per cent of the total. This indicates the extent to which the average outcome of bibliometric analyses is dependent on a relatively small proportion of publications which attract exceptionally high numbers of citations. 

About 300 substantive research papers per year - 6 per cent of UK articles, which is somewhat less than in other subject areas - are in the two relatively highly-cited categories with 4-8 times and at least 8 times world average impact. That suggests a continuing relative paucity of really top-quality chemistry papers. In a recent interview (see Chemistry World, November 2007, p13), David Delpy, the Engineering and Physical Sciences Research Council’s new chief executive, said that the UK chemistry community ’has largely focused on relatively less ambitious goals’. Our analysis suggests he may be right. 

Chemistry in context 

Research metrics show that UK chemistry is pretty healthy and has improved since a low period in the 1980s. But that growth of excellence still leaves a deficit compared to some subject areas. 

The impact of physics articles has improved markedly compared to world baselines since the early 1990s, as has the impact of molecular biology. By contrast, mathematics - a rather volatile area so far as indicators go - appears to have slipped (figure 3). Against those subjects, chemistry’s improvement seems less substantial. Even in 2002-06 it remains behind all three on this measure. 

We need to refer between the international comparisons in table 1 and the intra-national subject comparisons in figure 3 to build a proper picture of a subject’s standing. UK chemistry performs strongly compared to other countries, but UK physics gets cited more frequently compared to all other physics research around the world. This is not a simple picture to interpret. 

Figure 3

Looking ahead 

It is tempting to speculate that some of the research effort of UK chemists has shifted out of core chemistry into better funded areas such as environmental sciences or structural biology, enabling progress in those areas. While this would be of undoubted intellectual and economic benefit to the UK, could this lead to a relative stagnation of ’traditional’ chemistry research? Or, in due course, will it create feedback and point to new challenges? 

Managers have relied for many years on metrics that produced averages. New methods, such as our impact profile, reveal new features and change our view. They raise new questions about the distribution of excellence, and how it might be sustained and expanded. We need to understand what factors affect the patterns: does peak depend on platform, and are outputs by good performers also distributed across the lower categories? 

Metrics can be useful research management tools, providing support and challenge for expert committees seeking to arrive at decisions about grant funding and corporate investment. But will the shift from expert peer review to metrics still respond to the assessment of applied research, where impact is measured not by citations but by relevance and utility? Are indicators responsive to innovative research areas which, living at the margin away from the subject core, tend to attract lower citation rates? 

Most critically, how will chemistry perform against areas with different citation dynamics? How can we interpret metrics which seem to show chemistry ahead of other countries but trailing other subjects? There is a risk that if chemistry is still growing its share of the more highly-cited papers - recovering from a low in the 1980s - then applying metrics could draw resources away to other areas at a time when they are needed most. This is not just an academic issue. If the economy needs further investment and improvement in basic research to underpin key economic areas such as new materials, novel energy sources and climate change in the face of growing competition in the East, then this may not be the time to ’drop the pilot’ of expert and informed review. 

Evidence Ltd is a knowledge based company specialising in research performance analysis and interpretation. The firm produces the annual target indicators report on UK research for the Department of Innovation, University and Skills and the UK Higher Education Research Yearbook.