The benefits of basic research

The debate about the importance of basic research in innovation was recently reignited by a piece in the Wall Street Journal by Matt Ridley. Much of Ridley’s argument has been ably dismantled by Jack Stilgoe, but there are legitimate questions about how the balance of investment should be distributed between different types of research. There are opportunity costs, both in terms of the investment of financial resources and researchers time. Time and money spent on one type of research can’t be used on another, so there is a need to optimise allocation.

Before thinking about how this balance might work, it is important to be clear about what we actually mean by the term ‘basic research’. As was pointed out some time ago, there are two issues at play here – the nature of the understanding that is being sought in research, and the motivations of the researchers. This gives rise to the idea of the well-known 2×2 matrix:

When people talk about ‘basic research’ they are often conflating motivation and the nature of the research; they mean research that both seeks to answer fundamental questions and is motivated by researchers’ intrinsic interest and curiosity. But it is equally possible that research to enhance fundamental understanding is needed to address specific problems or challenges. Research in three of the quadrants highlighted in the matrix above can deliver benefits to society, and the motivations for carrying out the work are not always obvious after the event. And, of course, the motivations can be mixed.

There is relatively little evidence that seeks to understand what types of research (and motivations) bring about the greatest benefit. One example is an interesting study by Jonathan Grant and colleagues which traced the citation history of clinical guidelines, using these documents as a proxy for broader benefit. By looking at the papers cited in the guidelines, and then the papers cited in those and so on, a type of family tree can be built up. This ‘family tree’ gives a history of the research that has resulted in a clinical intervention.

The analysis revealed that much more of the research cited, even going back as many as four ‘generations’, is published in journals that focus on applied rather than basic research. The result suggests that research with a specific end in mind has contributed more to clinical practice. It would be interesting to see if this finding generalizes to other fields. And it is also worth considering the extent to which the chains of citations used fully describe the influence that research has had on clinical guidelines. However, the analysis does begin to challenge the idea that research that derives from curiosity necessarily brings the biggest benefit.

Another, and somewhat contrasting, source of evidence on the relationship between research and benefits comes from the results of the UK’s Research Excellence Framework. In this assessment, separate scores were given for academic value and broader societal impact for each unit assessed. As I have discussed in a previous post, units that have high scores for academic performance also tended to score well on impact. If academic value is likely to be higher for research that produces fundamental insights, then this would tend to suggest a big role for research of that type in delivering benefits.

There is a different interpretation of the REF results that is also compatible with Jonathan Grant’s work. Perhaps it is the same researchers who deliver research that has high academic value and brings broader societal benefits. There isn’t necessarily a direct link between the research of academic impact and broader benefit, but it is the same talented people who deliver both the advance of knowledge and societal impact.

This idea could be tested using an extension of the citation-based approach of Grant. If I am correct then the researchers who are publishing in applied journals and influencing clinical guidelines should also be publishing more fundamental research at the same time.

So, the question of resource allocation with which I started this post needs to be reframed. It is not about the balance between research that brings fundamental understanding and an orientation towards other goals. Much more important is the extent to which researchers are free to follow their interests as opposed to being directed to address specific questions. Giving researchers at least some scope to follow their interests is important, not just because that research itself may turn out to bring benefits. This freedom also ensures that the very best researchers are motivated to work in our universities, and, while there, they can also turn their talents to addressing broader questions of societal relevance. Both directly and indirectly funding for basic research delivers the benefits we seek.


Posted in research policy | Tagged , | 3 Comments

Dimensions of quality research

To say that the assessment of research quality keeps me awake at night would be an exaggeration. But a lot of my waking hours are taken up thinking about and discussing what we mean about quality in research, and how it can and should be assessed. So I was pleased to hear about a new project, The Qualities of Quality, that is taking a multidisciplinary approach to addressing these important questions.

A key point is made in the blog post introducing the project: research quality is not a simple linear scale, but depends on a number of axes of quality:

However quality can be re-imagined as a multi-variate construct that can be deployed to address different priorities. This shift from “quality” to “qualities” has potentially valuable practical outcomes in focussing our attention on different aspects of communicated research outputs. It also, importantly, should give cause for pause when the term is used across disciplinary boundaries; quality and its evaluation must be tied to the purpose of the research which, itself, must be situated within specific disciplinary practices.

Looking back on the last UK national research assessment, the Research Excellence Framework (REF), I would argue that we took significant steps to explore and assess the qualities, plural, of excellent research. In this post I want to expand on this point.

In the REF, at the highest level, there are two axes along which quality is assessed, an assessment of research outputs from an academic perspective, and an assessment of the broader societal impact of research. One of the interesting questions raised by, and potentially answerable from, the structure of the REF is the extent to which these dimensions of quality are related to one another. Does high performance on an academic scale co-occur with high performance in terms of societal impact? Looking at the level of the individual submissions made to REF2014, it is clear that there is a relationship:


In general higher performance on outputs and impact occur in the same submissions, but there is a reasonable spread of performance, with some examples of submissions that perform well on only one aspect. In considering this relationship it is important to remember the level of the analysis. We can conclude that high performance on both axes occurs in the same places and involves the same researchers, but the research may be different.

These two axes are reminiscent of Pasteur’s Quadrant that characterises research on two axes, fundamental understanding and considerations of use. But there is an important difference here. The axes of Pasteur’s quadrant reflect the motivations behind the research, but we know that impact, assessed after the fact as in the REF, can derive from research that was either motivated by considerations of use or purely by questions of understanding.

The REF is not just a two-dimensional view of research quality, but both outputs and impact are split into further axes of quality: rigour, originality and (academic) significance for outputs and reach and (societal) significance for impact. Although individual scores aren’t allocated to each of these axes in the process, research quality can be thought of as being specified along these five dimensions.

Of course it would be possible to break each of these aspects of quality down to lower levels of abstraction. But whereas the 5 dimensions are reasonably generic, at lower levels the dimensions become more discipline specific. Notions of rigour are very different in, say, history and physics, and there are even differences between STEM disciplines in standards of rigour.

While all five of these dimensions of quality are important aspects of research excellence and can vary considerably, some have minimum thresholds, below which it is questionable whether the activity counts as research at all. In particular, there are minimum standards of rigour. Activity that does not meet these standards – bad research – cannot be considered excellent however great its originality, reach and significance, for society or academia.

In other aspects, these dimensions of quality are independent of one another. Work may not be significant for an academic audience while being incredibly important for other parts of society. But, equally, there may be work of limited interest beyond the academy that is nonetheless important. A key challenge is to make sure that all research that excels across some or all of the dimensions of quality is valued, and those that carry it out are appropriately rewarded.

Posted in research policy | Tagged , , | 4 Comments

Evidencing the REF

I am speaking at a conference next week, Research Impact: Evidencing the REF, and in advance wrote a blog post the Open Forum Events website, which was also cross-posted on the HEFCE blog. It is reproduced in full below.

In December of 2014 the results of the most recent national assessment of university research in the UK, the Research Excellence Framework (REF), were published. Representing the culmination of a huge amount of work for universities and the members of the assessment panels, the REF provides a comprehensive assessment of research performance, including, for the first time, an assessment of the wider impact of research on and for society. And the picture painted is a positive one; world-leading research, across a broad discipline base, that delivers huge benefits to society in the UK and beyond.

It’s important to celebrate this success, and to use the wealth of data contained in the REF results and submissions to enhance our understanding of the research base. Analysis of the impact case studies is a particularly rich source of information. At the same time we have also been carrying out detailed evaluations of the process in order to learn lessons for the development of future exercises. Though the ink is barely dry on REF 2014, we are already thinking about how we can further enhance the process and its effectiveness.

There are many issues to explore. How can we reduce the burden? Should the selective nature of the exercise remain? How can peer review judgements be supported by meaningful quantitative data? And many more besides. Two questions are particularly challenging and important.

As we consider how to frame the second national assessment of the broader impact of research we need to decide how to handle impact case studies that were submitted to the previous exercise that are continuing to deliver additional benefits. On the one hand if we were to exclude such cases we would provide an incentive for universities to turn away from long-term benefit generation in favour of developing new areas. We want balance in these choices, so shouldn’t apply pressure in one direction. But if developing impacts are eligible, how will we decide how much additional benefit merits resubmission of an impact case study? Do we risk applying an incentive that discourages the delivery of new areas of impact?

A second important area for consideration is the handling of multi- and interdisciplinary research (MIR) in a future exercise. Evidence from the analysis of the impact case studies from REF2014 demonstrates that the combination of knowledge and expertise from different disciplines plays a central role in the delivery of impact from research. At the same time there remain concerns within the research community that MIR is not always assessed fairly, or that the exercise encourages researchers to focus on the ‘core’ of disciplines rather than the interfaces between them. We are seeking to enhance the evidence base around these questions, and are also exploring ways of further reducing perceived or actual barriers to MIR. For example, should researchers be allowed to submit their research outputs to multiple assessment panels? Or is there a potential to have explicitly identified assessment panel members who bring experience of working across disciplinary boundaries who would act as ‘champions’ of interdisciplinary research?

None of these questions have easy answers, and the Higher Education Funding Bodies won’t develop solutions on our own. We need everyone to contribute ideas, to challenge the ideas of others, to engage in a debate about future research assessment exercises. Over the coming months there will be workshops, informal discussions and a written consultation. The best future process will be shaped with input from the whole research community.

Posted in research policy | Tagged , | Leave a comment

On scholarly monographs and the sciences

One of the most enjoyable aspects of working in research policy is that it forces you to explore and understand the research cultures of disciplines other than your own. I am a life scientist who specialised in plant biology by background, but in my current role I need to understand the processes, challenges and norms of the whole range of research disciplines. And it is fascinating to see both the differences and, importantly, the similarities across diverse subjects.

Over the last twelve months I have been privileged to have played a small part in the review of monographs and open access carried out by Professor Geoff Crossick, and through this have developed a much deeper understanding of the research culture of those disciplines that communicate their research through long-form writing, and the place writing occupies in the research process. The report has now been published, and it makes an important contribution, not only to debates about open access, but also more generally to our understanding of the role of the book.

What emerges from the evidence and analysis in the report is a very special role for the sustained writing required for longer works:

In many cases, the most effective way of communicating several years of sustained research on a single topic is to present it as a monograph. This does not preclude the publication of articles en route to the book itself, but the book has a special place in the culture of research publication. It provides the length and space needed to allow a full examination of a topic, with the objective of presenting complex and rich ideas, arguments and insights supported by carefully contextualised analysis and evidence. […] Writing a monograph allows the author to weave a complex and reflective narrative, tying together a body of research in a way that is not possible with journal articles or other shorter outputs.

But it’s not just about the space that a book affords to set out arguments and analysis, the process of constructing the narrative does itself influence the outcome:

The term ‘thinking through the book’ emerged through the consultations, and it is a powerful concept, effectively reintegrating the research into the writing process itself. Monographs should not be seen simply as the way in which research findings are communicated, because the act of constructing and writing a book is often a core way to shape the ideas, structure the argument, and work out the relationship between these and the evidence that has emerged from the research process.

Reflecting on these passages, I was struck by how these points could be applied to disciplines in the natural sciences, where the scholarly monograph is much less common. When scientists write books they are much more likely to be aimed at communicating research to a non-specialist audience, rather than presenting, synthesising and developing research ideas for fellow specialists.

It wasn’t always this way. What is On the Origin of Species if not a monograph? And there are plenty of other examples from the past. For my own PhD research I read Cybernetics: Or Control and Communication in the Animal and the Machine by Norbert Wiener, an important book published in 1948, that has influenced many fields of research. Perhaps some of the most recent examples are some of the works of Stephen Jay Gould which were important vehicles for setting out his ideas about evolution, but they were still written very much with the non-specialist in mind.

Hearing about the role of long-form writing in the humanities and social sciences, I am left with a feeling that the natural sciences are missing out by not embracing the monograph. Of course, there is a tradition of review articles in the sciences; while some of these are summaries of current knowledge and evidence on a particular topic, others are more synthetic in nature, advancing new ideas based on a thorough consideration of current understanding. However, these are more limited in size than the scholarly monograph, and don’t afford the opportunity to develop arguments at length. The opportunity to ‘think through the book’ is much less evident.

How scientists would find the space in the fast-paced competitive world of scientific research is hard to imagine, when being the first to new findings is so important. Perhaps it is this feature of the natural sciences that results in less focus on communicating in long-form works, rather than any structural reason why the monograph couldn’t exist or be important. It is worth reflecting, though, that the norms of scholarly communication in the natural sciences may be incomplete, and that we lose something in the development of research because this is so.

Posted in research policy | Tagged , , | 1 Comment

Venture research and diversity

Earlier this month, I attended a discussion event organised by UCL's public policy unit on the idea of 'Venture Research'. This notion has been developed and operated by Professor Don Braben. With sponsorship from BP, a scheme was run in the 1980s, and now there is a similar, smaller scale, initiative operated within UCL.

The aim of the venture research approach is to tackle a perceived research policy issue: the idea that peer review, especially of research proposals, stiffles creativity and the solution is to give able scientists resources and let them follow their curiosity with limited or no constraints. If we don't do this, argue Braben and his supporters, we are unlikely to see a continuation of the major scientific breakthroughs of the 20th Century.

I have written about Braben's ideas before. I think they are based on a particular reading of history, and ignore the fact that there are mechanisms that allow all sorts of researchers to follow all sorts of ideas. I suspect there is a place for something like the venture research fund in the range of funding approaches, although it would be problematic for anything more than a fraction of funding to be distributed in this way. I wasn't clear from the event whether Braben and his supporters would like to see all or just a small proportion of public funding allocated by a venture-research mechanism.

While the principles behind the venture research fund are sensible as long as the scale is kept appropriate, there are considerable challenges in implementation. Whatever the scale, a fund like this will need to be selective, so how are researchers selected to benefit? In the case of the original venture research fund, selection was made on the basis of a very short proposal, and an interview. The selection was made by Braben and two other people. This approach raises considerable concerns from a diversity perspective. Selection by small groups of people from a non-anonymised applicant pool is extremely likely to introduce selection biases against certain groups. These biases can be conscious or unconscious.

I asked a question about this at the event. While this point applies to all under-represented groups, to make my question simple I asked about the gender make-up of the selection panel of three, and of the 26 successful applicants to the original venture fund. Rather shockingly, Braben would not answer the question! So I have done a little digging… I can't find the identity of the other two members of the selection panel, or a full list of the successful applicants. But there is a list of those funded by the venture fund who have made (in Braben's view) significant breakthroughs. The 13 'transformational discoveries' listed refer to 26 researchers. I can only find two female names on the list, both involved with the same discovery.

In the absence of any positive evidence, and much to be concerned about, I conclude that the selection approach for the original venture fund is likely to accentuate the already significant biases against women, people from non-white ethnic group and people with disabilities. The detailed process for the more recent UCL version of the fund isn't clear. As far as I can see only one award has been made under the scheme so far, so it is a little early to draw any conclusions.

Interestingly, Iain Foulkes from Cancer Research UK, who also spoke at the event, mentioned their plans to introduce a 'Pioneer Research' scheme not dissimilar to venture research. CRUK plan to anonymise applications to this scheme, which will go some way to addressing these issues. With this plan they will not be able to have the interview stage, which Braben suggests is an essential part of his process.

UK research already performs exceptionally well, and I don't see evidence of a lack of creativity among our researchers. The problem that UK research does have is its lack of diversity. Imagine how much better we could be if we were able to exploit the full talent base of the groups that are currently under-represented within research? And to benefit from the different approaches and ways of thinking that these groups would bring? Solving the diversity issue in research is the real priority.


Posted in research policy | Tagged , | Leave a comment

Questioning research culture

Earlier this week the Nuffield Council on Bioethics launched the report of its enquiry into scientific research culture. I was invited to respond to the report at the launch event, and this post is a summary of my comments.

The Nuffield Council’s report on research culture is an important piece of work. It tackles important questions for research policy-makers, in general, and for the development and shaping of proposals for the next Research Excellence Framework (REF). As the report acknowledges the REF is one of the influences on research culture, and, of course, these effects are part of the considerations that need to be taken into account.

The question of research culture is also a fascinating one for me personally. As a researcher-turned-policy-maker I have thought about these questions from a range of perspectives. It is an issue of complexity and, for me, one of considerable academic interest, but also an area where we need to be able to find pragmatic solutions. I am currently investigating some these questions as an Associate Fellow of the Centre for Science and Policy(CSaP) at the University of Cambridge, and my reflections on the report are in part informed by the thinking that has resulted from my own research for the fellowship.

The report will be a key contribution to the debate about research culture. The evidence collected as part of the inquiry, and summarised in the report, provides a robust assessment of the perceptions of the scientific community about the culture within which they work. It tells us how it feels to be a scientist in the UK today, how the policy, institutional and individual imperatives combine to make our research culture.

And there is a lot of positives to take away from the report. People see collaboration and working across disciplines as important, and many aspects of the funding and policy environment as supportive of these aspects of research. The open research agenda – open access to publications, open data, and working with people and groups from outside the academy – are valued, and are seen to be moving in the right direction. There is also support for the value and necessity of ensuring the benefits of research are delivered to society.

Of course there are perceived challenges too. Career progression and development, and the related issue of ensuring the diversity of the research workforce are familiar areas of focus. There is also evidence of a disconnect between researchers and the policy environment in which they sit, with both misconceptions of policy requirements and mistrust of their implementation.

In addressing these and other challenges it is welcome that the report acknowledges that the drivers are both external, from the research and institutional policy environment, and inherent in some of the behaviours and practices of researchers themselves. So it follows that the solutions are also shared by all the actors. To coin a phrase, we really are all in it together.

As I read the report, and in wider discussions I have been having as part of my CSaP fellowship, I was struck by a central theme. Within the research environment there are a number of tensions or oppositions that need to be worked out for a successful research culture. These include:

  • collaboration and competition;
  • whether research is inspired by researchers or inspired by potential uses and the wider needs of society;
  • a focus on single discipline as opposed to multi- or inter-disciplinary research;
  • whether objectives should be short-term or long-term.

All of these tensions are characterised by the need to find the appropriate balance. They represent a spectrum of options, and I would argue that for all of them the extremes are not desirable. Take competition and collaboration. The report presents important positive aspects of both attributes. Collaboration is valuable, but the complete absence of competition would lead to lower quality research.

If we accept that the research culture challenge is finding the correct balance between these tensions, I think this raises three questions:

  • What is the current balance point?
  • Who decides what the correct balance should be?
  • What steps should we take to alter the balance if necessary?

The first – What is the current balance point? – is one where the Nuffield report makes an important contribution, providing evidence on the perceptions of researchers on these important issues. But this evidence is not complete. While I would agree, as it states in the forward to the report, that:

It is the people engaged in scientific research who are in the best position to tell us what it is like to be a researcher

There is another side to the question. We need an external view, provided by social science, of how these tensions are being resolved within the academy. The Department for Business, Innovation and Skills (BIS) published an assessment of the features that lead to a productive research culture earlier this year. And at HEFCE we are currently planning a study to examine the factors that have resulted in strong performance in the current REF. However, I think further work is needed to improve our understanding of the current position.

My second question – Who decides what the correct balance point should be? – is an equally important one. Of course the academy itself will have a view, and again the Nuffield report provides evidence to illuminate the views of researchers on what they see as the optimum culture. Wider society may take a different view, which we also need to understand and balance with the views of researchers. For example, people outside of the academy may place a greater emphasis on research aimed at application. In response, researchers might point out that history tells us that some undirected research has delivered huge benefits to society. We need a dialogue about these matters, and a resolution that balances all the viewpoints.

Part of the answer in resolving the tensions I have described may depend upon thinking about the best level to attempt resolution. For example, should every researcher attempt to balance fundamental and applied research in their work, or is this balance best addressed by groupings of researchers seeking to achieve the appropriate balance across their portfolio of work?

Once we have resolved my first two questions, and have a sense of where we are, and where we want to be, we can naturally turn to the third question – What steps should we take to alter the balance? As a policy-maker this is the question that falls squarely into my domain, and is a challenging one. The Nuffield report demonstrates the complexity. There are many factors that determine the research culture, and individual policy-makers have limited influences on them. While I can help determine the shape of the next REF, I can’t control how many ‘mock’ exercises a Vice-Chancellor may decide to run, or the performance metrics a university might use to evaluate its staff.

Added to this complexity, is the diverse policy environment with many organisations, within and outside the public sector, making interventions. Undoubtedly, part of the answer is for research funders to work together more, and consider the holistic impact of what we do. But I do worry that this will inevitably lead to more alignment, something we are often exhorted to do. Maybe it would be better sometimes for different funders to have distinct policies that act in different directions and so balance one another? Either way, we should not underestimate the challenge of balancing the inherent (and healthy) tensions that exist within the research culture. In any considerations of changing research policy in the UK we also need to be mindful that we are dealing with a highly successful system by many measures, something we need to protect and nurture.

So, in conclusion, I strongly welcome the Nuffield Councils report. This is a conversation we all need to be having, and this report is the solid foundation on which to build the future analysis and debate.


Posted in research policy | Tagged , , | Leave a comment

Open access: because it’s worth it

The cost of implementing open access policies in the UK remains a hot topic. It is part of the deliberations of the review of RCUK policy of which I am a member, and the question of costs continues to be raised in the context of the open access requirements for the next Research Excellence Framework.

As we transition to a fully open access world, by whatever route, there will undoubtedly be additional costs. Even in the long run, if the costs of subscription access to research don't fall there may be additional costs. And either way the transition period may be a long one. If the UK makes rapid progress, it may take the rest of the world time to catch up.

We can debate the costs involved, and should seek to reduce them if possible, but for me this misses the point. The goal we are trying to achieve is worth the extra cost.

The move to open access is about more than efficiency in an overly narrow sense, or achieiving instrumental benefits. The move to open access is about doing the right thing; making the findings of research available to as wide an audience as possible. Research that is locked away where many people can't access it is less valuable, so it is worth doing a bit less research in total, if the research we do fund is widely accessible. If the UK is at the forefront of the move to open access, and so incurs some additional cost, then that's fine. We are doing the right thing, and that is something to be proud of.


Posted in research policy | Tagged , , | 1 Comment

Understanding academic culture

Next week I am going back to school. I will start a research project as an Associate Fellow1 of the Centre for Science and Policy (CSaP) at the University of Cambridge. The aim of my research is to investigate how decisions of policy-makers that shape the research policy environment are translated into actual outcomes and responses within the academy; how policy and research culture interact. You can read a slightly more expansive summary of the project here[pdf].

Although not part of the scheme, I will be using the tried-and-tested methodology of the CSaP Policy Fellowships programme. Next week, and during a second visit in November, I will meet with a range of experts from within Cambridge and beyond. They will be drawn from a range of different disciplines, and will bring diverse perspectives on the research questions I am interested in. I am hoping they will navigate me into research literatures, and help me develop and refine the questions I am addressing. It promises to be an illuminating, exciting and exhausting few days.

The central question – what determines research culture – is an important one. Many of the debates around research policy centre on issues related to it. Do we have the right research culture? Is it changing for the worse? Do research policy interventions foster a positive research culture? The question forces us to examine the balance between academic freedom, and the imperative of the academy to be responsive to society’s needs. The current review by the Nuffield Council on Bioethics is exploring many of these issues, and I am hopeful that my research will illuminate them further. The mechanisms by which policy interventions lead to changes in culture are important to understand, and I think we need to learn not just from how things work in academia, but also from other sectors.

As well as the research questions themselves, I am also keen to reflect on the process of investigation. Seeking insight to how the culture in academia works from experts who are operating within that culture themselves brings front and centre questions of the independence of experts. I am firmly of the view that whenever expertise is brought to bear on a policy question it is important to recognise that the experts will bring biases of one sort or another, some of them unconscious. When we are dealing with ‘policy for science’ (as opposed to ‘science for policy’) the need to keep this in mind becomes all the more apparent.

I will be blogging about this project as I go, writing about the experience, what I am learning, and interesting research sources that I come across. I am also keen to hear your views. What should I be exploring more? Are there research papers or reports that I ought to read? I have no idea where this research will take me, but I am sure it will be an interesting journey.

  1. As I write the website has yet to be updated to include my name on the list…^
Posted in research policy | Tagged , , , | 4 Comments