In SAgE we are about to begin the next internal quality exercise (IQA) to update the REF returnability of academic staff. At my previous institution, similar exercises for the RAE were always tortuous, complex, and caused great angst for a number of staff. Given the rules of REF, it is difficult to see how we can avoid these difficulties. Nonetheless, we must take this exercise very seriously. For those unfamiliar with the process, we are in effect endeavouring to balance two outcomes – the best possible position in the ubiquitous league tables, and the optimum financial return to the institution. The latter is a very measurable quantity known as ‘QR’ income, which is in effect an annual block grant awarded to the institution to support research activity – SAgE received ~£11M of QR income this year, so it is clearly critically important for us. The former is more difficult to measure, but there is adequate evidence to suggest that league table position impacts on the ability to attract high quality students and to win research grants. Moreover, to know that one’s institution is in, say, the ‘top 10’, is a significant boost to morale.
So why the angst? In these difficult financial times, only the best research gets funded, so it is pointless submitting work that is below par. But the bar is very high – as part of the REF process, each eligible member of academic staff needs to submit four ‘outputs’ over the ‘REF period’, and each of these is given a star rating – 4*, 3*,2* and 1*. Importantly, only 4* and 3* work will attract any funding, and 4* activity is worth three times more than 3*. Thus, as a faculty (and indeed institution) we need to judge whether a piece of work (typically a research paper or peer-reviewed conference proceeding) is 4* or 3* and thus worthy of inclusion. Now here’s the rub – how do we measure quality across such a broad spectrum of activity? In these times of sound bites and limited time for anything, there is a tendency to use quantitative metrics. These might include ‘journal impact factor’ or ‘number of citations’. The danger of using such measures in isolation is so obvious that it need not be stated.
The SAgE Faculty Executive Board has discussed how to measure quality for the next exercise. The consensus view is not particularly imaginitive – rather than using quantitative metrics as the primary tool, we feel that the panel criteria published in January (which were not available for our first IQA) will be a much more rigorous guide. For example, summarising some of the criteria for main panel B (where the majority of our work will likely be returned), one can find the following statements (page 46 of the REF 01.2012 document):
4*: world leading in terms of originality, significance and rigour-
- research that is leading or at the forefront of the research area
- great novelty in developing new thinking, new techniques or novel techniques
- developing new paradigms or fundamental new concepts for research
3*: internationally excellent in terms of originality…falls short of highest standards of excellence-
- makes important contributions to the field at an international standard
- contributes important knowledge…..likely to have a lasting influence, but not necessarily leading to fundamental new concepts
2*: recognised internationally in terms of originality, significance and rigour-
- provides useful knowledge and influences the field
- involves incremental advances, which might include new knowledge which conforms with existing ideas and paradigms…..
With these definitions, I suggest it should be straighforward to assess star quality in the majority of cases. At the risk of sounding glib, I would chacterise them as ‘paradigm shifting’, ‘influential’ and ‘derivative’ respectively. However, quantitative metrics might have a secondary role in cases where there might be some doubt. Let me give an example – imagine a journal article that was published in 2008 and that is judged to be 4* – is it thinkable that such an article would have received no citations?
As a final comment, note that the definition of 2* does not give the impression of poor quality work. Indeed, it is recognised internationally. I suspect we have all regularly published 2* work, and the fact that this does not qualify for REF submission is a sign of the very high standards to which we must aspire in these challenging times.