Patrick raised a couple of interesting observations in response to last week’s blog, which I’d like to expand on this week. Firstly, I’m a strong believer that KPIs should be generated ‘bottom-up’, so that those who are tasked with meeting the associated targets have ownership of them. However, since Patrick has laid down the gauntlet, and I’ve never been able to resist a challenge, here are my thoughts on the KPIs that he has highlighted.
Taking the first issue – “to grow areas of strength and address areas of weakness to achieve international excellence in all we do” – let’s look at this from the research perspective. We can hardly set a target for this objective without consideration of REF, or rather RAE2008, given that this (arguably) represents the most comprehensive and rigorous dataset avalable to us (the other option being HESA statistics, which are not as useful for reasons that I might blog about at a later date). Given these data, what is the most appropriate measure to construct our target? We could say that we’d like to be in the top ten (or better) for quality in all subjects. However, if there are only ten submissions in the relevant Unit of Assessment (UoA) then this might not be a great accolade. Moreover, we could probably achieve first place by returning only one person – although this would likely be catastrophic financially! What we need is a measure of both quality and volume, and in this regard the diagram below is helpful. This plots the rank quartile of 4* quality versus 4* volume for each UoA submitted by Newcastle in RAE2008. The numbered dots are SAgE UoAs. I’ll leave you to figure out who the stars performers are, but suffice to say most of our activity is mid-ranking on this plot. I could show you a similar plot for a highly ranked Russell Group institution (not in the Golden Triangle by the way) whose UoAs are tightly bunched around the first quartile. As an aside, if a UoA plots above the diagonal then volume is emphasised at the expense of quality, and vice-versa below the diagonal.
So should our target be first quartile performance for all UoAs for both quality and volume? (ie dark green box in the figure). Yes indeed. But here’s the difficulty – the data in this plot are about 5 years old since they derive from the 2007 census date for RAE2008. We won’t be able to update this plot until the results of the REF are known, unless we regularly assess the outputs of all our competitors by trawling through Web of Science and making our own judgement on star quality. Any volunteers? Clearly, ‘first quartile’ is not a useful target since we can’t easily create a KPI that we can monitor regularly.
Here’s a simpler approach. If we look at the fraction of 4* outputs for UoAs of relevance to SAgE, even the very best performers typically manage about 25% or so 4* activity. So how about this for a target – every member of staff who is REF eligible has at least one 4* output for REF. If we can achieve this I dare bet that we will be well within the upper ranks in every UoA (taken as a given that we won’t typically include any 2* outputs). The KPI is then simply the fraction of staff that meet this criterion. This of course begs the question how we might define 4* activity, but I’ve already written about that here.
Enough rambling for one week – Patrick’s second challenge, the target for ‘innovation’, is more difficult. I’ll need to lie down and think about that one for a while!