I was recently asked to comment on a new paper looking at the relationship between REF scores, funding outcomes and venue of publication. I have made some annotations onto the paper using hypothes.is.
Overall, I don't think that the paper adds very much. There are two issues covered: an attempt to calculate the funding allocated on the basis of 4-star and 3-star outputs, and an investigation of the relationship between REF scores and the venue of publication of journal articles.
The analysis of monetary value is essentially predictable from the details of the funding formula published on the HEFCE website. There is a relative weighting of 4-star to 3-star research activity of 4:1, and different disciplines are funded at different rates, to reflect the relative costs of research. These factors are responsible for a lot of the differences observed in the new paper. The rest is due to the relatively small differences in performance between different disciplines in REF2014.
The paper's findings on outputs are similar to those in a blog post published by Mark Reed and Simon Kerridge earlier in the year. They looked at the funding outcome associated with both impact case studies and outputs.
The second part of the paper looks at the relationship, in selected UOAs, between REF scores and citation performance at the level of the journals in which outputs were published. For some UOAs they find a positive correlation, which is not especially surprising. A similar effect was also found in more detailed analysis [pdf] carried out as part of the Metric Tide report. The correlation is relatively weak, absent in some UOAs and there are significant outliers.
And it is important to remember that this is only a correlation, and it would be unwise for a university to select their outputs for submission on the basis of journal-level metrics. There are well established limitations with journal-level metrics, and the REF panels were instructed to avoid using venue of publication or proxies such as Journal Impact Factor in their assessment, but to judge each output on its merits. With HEFCE being a signatory of the San Francisco Declaration on Research Assessment, there is no chance of the approach being any different next time.
