Brace for research impact exercise

University research funding will be distributed partly according to measures of ’research impact’, the Higher Education Funding Council for England (Hefce) has revealed.

On 23 September, Hefce released a full draft of the Research Excellence Framework (REF) for consultation. The new framework will replace the Research Assessment Exercise (RAE) and take place in 2013 to inform funding from 2014 onwards. Hefce distributes about ?1.6 billion of quality-related research funding each year - and the inclusion of impact assessment as a formalised element in the REF heralds a shift in the way that academic excellence is assessed.

The consultation reveals that the REF will measure research excellence by focusing on three distinct elements. Output quality (60 per cent) and Environment (15 per cent) will remain as assessment categories, with expert review continuing to play a major role. Impact will account for remaining 25 per cent, and will require the academic community to describe their contribution in narrative statements and case studies supported by impact metrics. Much of the detail of impact assessment has been postponed until completion of a pilot process, to be conducted in 2010. 

All three elements of the REF will be assessed by expert panels. To attempt to reduce academic burden, Hefce have reduced the number of units of assessment from 67 to 30 - though chemistry remains its own distinct unit - and suggest that the number of overarching main panels should be as few as four. Hefce believe that they can retain the granular information while increasing consistency with this reduced number. However, roughly the same number of academics will be involved in panels due to a ’sub-panel’ structure. For example, it is proposed there is an impact ’sub-panel’ which will include academic and user members from the main panel but also co-opt extra users. David Sweeney, director of research, innovation and skills at Hefce said: ’The 

panel that judge the credibility of impact will be largely populated by research users. Users will play a significant part in this exercise.’

Impact issues

Neville Richardson from the University of St Andrews, a member of the RAE 2008 Chemistry panel, welcomes the reduction of the number of units of assessment and Chemistry retaining its own panel. However he is worried about the assessment of impact in the REF: ’The greatest concern for all disciplines, not least chemistry, is how impact will be assessed - and this consultation document provides little reassurance that a fair, robust and meaningful methodology can be developed. While purporting to recognise the problems of time lag, complexity and non-linearity in the "research to impact" process, the proposals clearly fail to adequately take these into account in any realistic way.

’For example, for any impact for which third parties, rather than members of the unit themselves, are responsible for the intermediate steps to achieving impact, will be disregarded. This is a nonsensical limitation - science relies on researchers being able to "stand on the shoulders of giants". The giants should not be ignored as contributors to impact.’

The Russell Group of research-intensive universities share this apprehension. Wendy Piatt, director general of the Russell Group, said: ’The links between research and its wider impact are not necessarily straightforward, as Hefce admits. This is why the pilot studies are so important to ensure methods of measuring impact are consistent and robust. Both Hefce and the universities should be confident in the new system of measuring impact before it is implemented.’

The numbers game

Hefce’s previous attempt at a new framework to replace the reviewer-intensive RAE process was firmly based on bibliometrics, but academic feedback warned that although this reduced the burden on reviewers, it seriously risked obscuring the full picture of academic excellence. The REF will use increased metrics compared to the RAE but Sweeney says that bibiometrics are considerably less central that in the previous pilot. ’We are not layering a full bibliometric exercise on top of the REF. It will be left to panel judgement to what extent they use bibliometrics.’

The consultation, released today, gives higher education institutions and individuals the opportunity to comment and feedback to Hefce. ’We have ideas, but we are in listening mode -  although we think we have done what the sector has asked,’ Sweeney said.

However, some academics are questioning the fundamentals of what the REF is trying to achieve. Lee Cronin from the University of Glasgow thinks the REF could potentially be harmful to UK science: ’The UK has an obsession with measuring and by measuring our economic impact and industry interaction our long term economic competitiveness may be damaged. The REF should be used to judge our international long term competitiveness not perceived short term impact.’

Leila Sattary