To say that the assessment of research quality keeps me awake at night would be an exaggeration. But a lot of my waking hours are taken up thinking about and discussing what we mean about quality in research, and how it can and should be assessed. So I was pleased to hear about a new project, The Qualities of Quality, that is taking a multidisciplinary approach to addressing these important questions.
A key point is made in the blog post introducing the project: research quality is not a simple linear scale, but depends on a number of axes of quality:
However quality can be re-imagined as a multi-variate construct that can be deployed to address different priorities. This shift from “quality” to “qualities” has potentially valuable practical outcomes in focussing our attention on different aspects of communicated research outputs. It also, importantly, should give cause for pause when the term is used across disciplinary boundaries; quality and its evaluation must be tied to the purpose of the research which, itself, must be situated within specific disciplinary practices.
Looking back on the last UK national research assessment, the Research Excellence Framework (REF), I would argue that we took significant steps to explore and assess the qualities, plural, of excellent research. In this post I want to expand on this point.
In the REF, at the highest level, there are two axes along which quality is assessed, an assessment of research outputs from an academic perspective, and an assessment of the broader societal impact of research. One of the interesting questions raised by, and potentially answerable from, the structure of the REF is the extent to which these dimensions of quality are related to one another. Does high performance on an academic scale co-occur with high performance in terms of societal impact? Looking at the level of the individual submissions made to REF2014, it is clear that there is a relationship:
In general higher performance on outputs and impact occur in the same submissions, but there is a reasonable spread of performance, with some examples of submissions that perform well on only one aspect. In considering this relationship it is important to remember the level of the analysis. We can conclude that high performance on both axes occurs in the same places and involves the same researchers, but the research may be different.
These two axes are reminiscent of Pasteur’s Quadrant that characterises research on two axes, fundamental understanding and considerations of use. But there is an important difference here. The axes of Pasteur’s quadrant reflect the motivations behind the research, but we know that impact, assessed after the fact as in the REF, can derive from research that was either motivated by considerations of use or purely by questions of understanding.
The REF is not just a two-dimensional view of research quality, but both outputs and impact are split into further axes of quality: rigour, originality and (academic) significance for outputs and reach and (societal) significance for impact. Although individual scores aren’t allocated to each of these axes in the process, research quality can be thought of as being specified along these five dimensions.
Of course it would be possible to break each of these aspects of quality down to lower levels of abstraction. But whereas the 5 dimensions are reasonably generic, at lower levels the dimensions become more discipline specific. Notions of rigour are very different in, say, history and physics, and there are even differences between STEM disciplines in standards of rigour.
While all five of these dimensions of quality are important aspects of research excellence and can vary considerably, some have minimum thresholds, below which it is questionable whether the activity counts as research at all. In particular, there are minimum standards of rigour. Activity that does not meet these standards – bad research – cannot be considered excellent however great its originality, reach and significance, for society or academia.
In other aspects, these dimensions of quality are independent of one another. Work may not be significant for an academic audience while being incredibly important for other parts of society. But, equally, there may be work of limited interest beyond the academy that is nonetheless important. A key challenge is to make sure that all research that excels across some or all of the dimensions of quality is valued, and those that carry it out are appropriately rewarded.
