KPIs and Targets (continued)

Patrick raised a couple of interesting observations in response to last week’s blog, which I’d like to expand on this week. Firstly, I’m a strong believer that KPIs should be generated ‘bottom-up’, so that those who are tasked with meeting the associated targets have ownership of them. However, since Patrick has laid down the gauntlet, and I’ve never been able to resist a challenge, here are my thoughts on the KPIs that he has highlighted.

Taking the first issue – “to grow areas of strength and address areas of weakness to achieve international excellence in all we do” – let’s look at this from the research perspective. We can hardly set a target for this objective without consideration of REF, or rather RAE2008, given that this (arguably) represents the most comprehensive and rigorous dataset avalable to us (the other option being HESA statistics, which are not as useful for reasons that I might blog about at a later date). Given these data, what is the most appropriate measure to construct our target? We could say that we’d like to be in the top ten (or better) for quality in all subjects. However, if there are only ten submissions in the relevant Unit of Assessment (UoA) then this might not be a great accolade. Moreover, we could probably achieve first place by returning only one person – although this would likely be catastrophic financially! What we need is a measure of both quality and volume, and in this regard the diagram below is helpful. This plots the rank quartile of 4* quality versus 4* volume for each UoA submitted by Newcastle in RAE2008. The numbered dots are SAgE UoAs. I’ll leave you to figure out who the stars performers are, but suffice to say most of our activity is mid-ranking on this plot. I could show you a similar plot for a highly ranked Russell Group institution (not in the Golden Triangle by the way) whose UoAs are tightly bunched around the first quartile. As an aside, if a UoA plots above the diagonal then volume is emphasised at the expense of quality, and vice-versa below the diagonal.

So should our target be first quartile performance for all UoAs for both quality and volume? (ie dark green box in the figure). Yes indeed. But  here’s the difficulty – the data in this plot are about 5 years old since they derive from the 2007 census date for RAE2008. We won’t be able to update this plot until the results of the REF are known, unless we regularly assess the outputs of all our competitors by trawling through Web of Science and making our own judgement on star quality. Any volunteers? Clearly, ‘first quartile’ is not a useful target since we can’t easily create a KPI that we can monitor regularly.  

Here’s a simpler approach. If we look at the fraction of 4* outputs for UoAs of relevance to SAgE, even the very best performers typically manage about 25% or so 4* activity. So how about this for a target – every member of staff who is REF eligible has at least one 4* output for REF. If we can achieve this I dare bet that we will be well within the upper ranks in every UoA (taken as a given that we won’t typically include any 2* outputs). The KPI is then simply the fraction of staff that meet this criterion. This of course begs the question how we might define 4* activity, but I’ve already written about that here.

Enough rambling for one week – Patrick’s second challenge, the target for ‘innovation’, is more difficult. I’ll need to lie down and think about that one for a while!

Advertisements

About stevehomans

Professor Steve Homans is a structural biologist with an international reputation in the study of biomolecular interactions. He obtained his first degree and DPhil in Biochemistry at Oxford University, and secured his first academic position as Lecturer at the University of Dundee. In 1998 he received the Zeneca award from the Biochemical Society and was elected Fellow of the Royal Society of Edinburgh. Prior to his current appointment he was Dean of the Faculty of Biological Sciences at the University of Leeds. Professor Homans brings extensive expertise of academic leadership and management, with a particular emphasis on organisational change.
This entry was posted in Uncategorized. Bookmark the permalink.

One Response to KPIs and Targets (continued)

  1. As I see it the problem with using a KPI based on the REF description of “excellence” (e.g. 4*, 3*, etc) is that REF definition it isn’t clear enough to be used as a KPI (and the process of determining whether a publication is 4* is too time consuming). Alternatively, while for the REF papers that are “internationally leading” is to be determined by processes that are apparent to only on the panel, we can probably all agree that an internationally leading program of research should be publishing regularly in top tier publication outlets. The Australian Research Council (ARC) did a reasonable job of defining such outlets in its 2010 ERA exercise (see: http://www.arc.gov.au/era/era_2010/archive/default.htm) by working with leading academics to define such outlets as A*, A, B or C-rated. Of course, this ranking can be refined by Schools themselves (but maybe taking the ARC list as a starting point), but a KPI based on numbers of A and A* rated outputs would be the clearest metric and easiest to administer (an important factor). Note that the ARC discontinued the use of these rankings as it was felt that Universities were using it (too much) to direct academics to publish in the higher ranking outlets; although (for science and engineering anyway) this is what we all do with our own PhDs and RAs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s