This post was originally published on the HEFCE blog on 22 March 2018 (archived copy).
Over the last 40 years, leading orchestras have been through a remarkable transformation. Moving from what was a largely male-dominated profession, there is now significant representation of women in professional orchestras. While there are no doubt many factors behind the change, much is attributed to a single intervention – the use of screened auditions. Since selection panels do not know the gender of candidates, who perform in auditions from behind screens, the bias against female musicians has been significantly reduced.
In the context of research, it is generally felt that peer review, while a key part of the processes of validation and assessment, also has biases – both conscious and unconscious. These might be biases against female authors, in favour of research from well-known researchers or prestigious institutions, or relating to nationality or race. And as with orchestras, the use of anonymous review is often held up as a – or even the – solution. It is suggested for all types of peer review – the assessment of research proposals, the evaluation of research outputs before publication, and the post-publication review of research.
Looking at post-publication review in particular, it was recently suggested that the next Research Excellence Framework (REF) should use anonymous review.
Would this be wise? Let’s look at some evidence. Most of the tests of anonymous or 'double-blind' review have been carried out on pre-publication reviews of journal articles, with mixed results. Analysis of an economics journal shows some benefits of anonymous review in terms of types of institution, but not on gender bias. Two studies in computer science demonstrate less bias against new researchers or less well-known institutions.
However, some evidence reveals a significant issue. For example, analysis of anonymous review in a neurobiology journal suggests that reviewers are able to identify a significant proportion of manuscripts, and show a bias towards accepting those that they recognise.
This is an important issue. While names and institutional affiliations can be removed from items to be reviewed, it is hard to remove all clues as to the author's identity. The use of citations and the way they are referred to will potentially reveal the author, and a 30-second google search of the title of a paper or book is likely to allow identification. Of course, this requires the article to have been published, but this will be the case for post-publication review, and increasingly so for 'unpublished' journal articles with the increasing use of preprints and working papers.
In post-publication review, it is also important to note that reviewers will be specialists in the field concerned and may well recognise the work they are reviewing – even if it is anonymised and no steps are taken to break the anonymization.
So, anonymous review has the potential to introduce rather than eliminate bias. It is possible that reviewers’ assessments will be influenced by the familiarity with the field, coming to different views about outputs they recall seeing before. And other biases will resurface for those articles that are recognised, so will apply to some, but not all the assessments.
With all due respect to orchestras, the world of research evaluation is not so simple. While it may sound like an appealing idea to introduce anonymous review into the REF, when everything is considered it does not seem a sensible step.
