Questioning research culture

Earlier this week the Nuffield Council on Bioethics launched the report of its enquiry into scientific research culture. I was invited to respond to the report at the launch event, and this post is a summary of my comments.

The Nuffield Council’s report on research culture is an important piece of work. It tackles important questions for research policy-makers, in general, and for the development and shaping of proposals for the next Research Excellence Framework (REF). As the report acknowledges the REF is one of the influences on research culture, and, of course, these effects are part of the considerations that need to be taken into account.

The question of research culture is also a fascinating one for me personally. As a researcher-turned-policy-maker I have thought about these questions from a range of perspectives. It is an issue of complexity and, for me, one of considerable academic interest, but also an area where we need to be able to find pragmatic solutions. I am currently investigating some these questions as an Associate Fellow of the Centre for Science and Policy(CSaP) at the University of Cambridge, and my reflections on the report are in part informed by the thinking that has resulted from my own research for the fellowship.

The report will be a key contribution to the debate about research culture. The evidence collected as part of the inquiry, and summarised in the report, provides a robust assessment of the perceptions of the scientific community about the culture within which they work. It tells us how it feels to be a scientist in the UK today, how the policy, institutional and individual imperatives combine to make our research culture.

And there is a lot of positives to take away from the report. People see collaboration and working across disciplines as important, and many aspects of the funding and policy environment as supportive of these aspects of research. The open research agenda – open access to publications, open data, and working with people and groups from outside the academy – are valued, and are seen to be moving in the right direction. There is also support for the value and necessity of ensuring the benefits of research are delivered to society.

Of course there are perceived challenges too. Career progression and development, and the related issue of ensuring the diversity of the research workforce are familiar areas of focus. There is also evidence of a disconnect between researchers and the policy environment in which they sit, with both misconceptions of policy requirements and mistrust of their implementation.

In addressing these and other challenges it is welcome that the report acknowledges that the drivers are both external, from the research and institutional policy environment, and inherent in some of the behaviours and practices of researchers themselves. So it follows that the solutions are also shared by all the actors. To coin a phrase, we really are all in it together.

As I read the report, and in wider discussions I have been having as part of my CSaP fellowship, I was struck by a central theme. Within the research environment there are a number of tensions or oppositions that need to be worked out for a successful research culture. These include:

  • collaboration and competition;
  • whether research is inspired by researchers or inspired by potential uses and the wider needs of society;
  • a focus on single discipline as opposed to multi- or inter-disciplinary research;
  • whether objectives should be short-term or long-term.

All of these tensions are characterised by the need to find the appropriate balance. They represent a spectrum of options, and I would argue that for all of them the extremes are not desirable. Take competition and collaboration. The report presents important positive aspects of both attributes. Collaboration is valuable, but the complete absence of competition would lead to lower quality research.

If we accept that the research culture challenge is finding the correct balance between these tensions, I think this raises three questions:

  • What is the current balance point?
  • Who decides what the correct balance should be?
  • What steps should we take to alter the balance if necessary?

The first – What is the current balance point? – is one where the Nuffield report makes an important contribution, providing evidence on the perceptions of researchers on these important issues. But this evidence is not complete. While I would agree, as it states in the forward to the report, that:

It is the people engaged in scientific research who are in the best position to tell us what it is like to be a researcher

There is another side to the question. We need an external view, provided by social science, of how these tensions are being resolved within the academy. The Department for Business, Innovation and Skills (BIS) published an assessment of the features that lead to a productive research culture earlier this year. And at HEFCE we are currently planning a study to examine the factors that have resulted in strong performance in the current REF. However, I think further work is needed to improve our understanding of the current position.

My second question – Who decides what the correct balance point should be? – is an equally important one. Of course the academy itself will have a view, and again the Nuffield report provides evidence to illuminate the views of researchers on what they see as the optimum culture. Wider society may take a different view, which we also need to understand and balance with the views of researchers. For example, people outside of the academy may place a greater emphasis on research aimed at application. In response, researchers might point out that history tells us that some undirected research has delivered huge benefits to society. We need a dialogue about these matters, and a resolution that balances all the viewpoints.

Part of the answer in resolving the tensions I have described may depend upon thinking about the best level to attempt resolution. For example, should every researcher attempt to balance fundamental and applied research in their work, or is this balance best addressed by groupings of researchers seeking to achieve the appropriate balance across their portfolio of work?

Once we have resolved my first two questions, and have a sense of where we are, and where we want to be, we can naturally turn to the third question – What steps should we take to alter the balance? As a policy-maker this is the question that falls squarely into my domain, and is a challenging one. The Nuffield report demonstrates the complexity. There are many factors that determine the research culture, and individual policy-makers have limited influences on them. While I can help determine the shape of the next REF, I can’t control how many ‘mock’ exercises a Vice-Chancellor may decide to run, or the performance metrics a university might use to evaluate its staff.

Added to this complexity, is the diverse policy environment with many organisations, within and outside the public sector, making interventions. Undoubtedly, part of the answer is for research funders to work together more, and consider the holistic impact of what we do. But I do worry that this will inevitably lead to more alignment, something we are often exhorted to do. Maybe it would be better sometimes for different funders to have distinct policies that act in different directions and so balance one another? Either way, we should not underestimate the challenge of balancing the inherent (and healthy) tensions that exist within the research culture. In any considerations of changing research policy in the UK we also need to be mindful that we are dealing with a highly successful system by many measures, something we need to protect and nurture.

So, in conclusion, I strongly welcome the Nuffield Councils report. This is a conversation we all need to be having, and this report is the solid foundation on which to build the future analysis and debate.

 

Posted in research policy | Tagged , , | Leave a comment

Open access: because it’s worth it

The cost of implementing open access policies in the UK remains a hot topic. It is part of the deliberations of the review of RCUK policy of which I am a member, and the question of costs continues to be raised in the context of the open access requirements for the next Research Excellence Framework.

As we transition to a fully open access world, by whatever route, there will undoubtedly be additional costs. Even in the long run, if the costs of subscription access to research don't fall there may be additional costs. And either way the transition period may be a long one. If the UK makes rapid progress, it may take the rest of the world time to catch up.

We can debate the costs involved, and should seek to reduce them if possible, but for me this misses the point. The goal we are trying to achieve is worth the extra cost.

The move to open access is about more than efficiency in an overly narrow sense, or achieiving instrumental benefits. The move to open access is about doing the right thing; making the findings of research available to as wide an audience as possible. Research that is locked away where many people can't access it is less valuable, so it is worth doing a bit less research in total, if the research we do fund is widely accessible. If the UK is at the forefront of the move to open access, and so incurs some additional cost, then that's fine. We are doing the right thing, and that is something to be proud of.

 

Posted in research policy | Tagged , , | 1 Comment

Understanding academic culture

Next week I am going back to school. I will start a research project as an Associate Fellow1 of the Centre for Science and Policy (CSaP) at the University of Cambridge. The aim of my research is to investigate how decisions of policy-makers that shape the research policy environment are translated into actual outcomes and responses within the academy; how policy and research culture interact. You can read a slightly more expansive summary of the project here[pdf].

Although not part of the scheme, I will be using the tried-and-tested methodology of the CSaP Policy Fellowships programme. Next week, and during a second visit in November, I will meet with a range of experts from within Cambridge and beyond. They will be drawn from a range of different disciplines, and will bring diverse perspectives on the research questions I am interested in. I am hoping they will navigate me into research literatures, and help me develop and refine the questions I am addressing. It promises to be an illuminating, exciting and exhausting few days.

The central question – what determines research culture – is an important one. Many of the debates around research policy centre on issues related to it. Do we have the right research culture? Is it changing for the worse? Do research policy interventions foster a positive research culture? The question forces us to examine the balance between academic freedom, and the imperative of the academy to be responsive to society’s needs. The current review by the Nuffield Council on Bioethics is exploring many of these issues, and I am hopeful that my research will illuminate them further. The mechanisms by which policy interventions lead to changes in culture are important to understand, and I think we need to learn not just from how things work in academia, but also from other sectors.

As well as the research questions themselves, I am also keen to reflect on the process of investigation. Seeking insight to how the culture in academia works from experts who are operating within that culture themselves brings front and centre questions of the independence of experts. I am firmly of the view that whenever expertise is brought to bear on a policy question it is important to recognise that the experts will bring biases of one sort or another, some of them unconscious. When we are dealing with ‘policy for science’ (as opposed to ‘science for policy’) the need to keep this in mind becomes all the more apparent.

I will be blogging about this project as I go, writing about the experience, what I am learning, and interesting research sources that I come across. I am also keen to hear your views. What should I be exploring more? Are there research papers or reports that I ought to read? I have no idea where this research will take me, but I am sure it will be an interesting journey.


  1. As I write the website has yet to be updated to include my name on the list…^
Posted in research policy | Tagged , , , | 4 Comments

His master’s voice?

There are many different styles of blog covering an incredible range of content. Many people write about their personal life, topics unconnected with their professional roles, but equally a lot of blogs, like this one, cover topics where the author has a direct professional interest. This raises interesting questions for the author, and by extension the readers: whose voice is it, the author or the corporate? And if the author is writing their personal views, how does this relate to the views of their employer.

Since I have been writing here I have felt these questions acutely. I have also had feedback from readers that reflect these concerns. During one of the all-to-frequent gaps in my blogging, a reader ask whether I had been 'gagged' by my then employer following this post that strayed into the politics of research funding. I should stress that neither of my employers during the period I have been blogging, Research Councils UK and Higher Education Funding Council for England (HEFCE), have ever even suggested I shouldn't blog or tried to remove or alter anything I have written here or posted on Twitter. And the views I express here are very much my own. This isn't a corporate blog in disguise.

This tension, between the personal and the corporate view, is one that seems to be surfacing increasingly. Phil Ward, the author of the excellent Research Fundermentals blog, has some interesting reflections on his experience. And the pages of Times Higher Education have had a number of stories recently about universities attempts to silence academics, whose views aren't aligned with those of the institution.

The debate is also live within HEFCE as we are exploring new ways to present our policy and analysis work, including through more use of social media. The debate centres around the processes we might use, and the extent to which the individual voice that such communications will involve should necessarily align with the corporate view. This is making us all think about how we should describe ourselves on twitter bios for personal accounts where we tweet professionally-related material. I have also been giving some thought to how a post I write here might differ, if at all, from a post on a corporate blog.

These questions go well beyond social media behaviousr, and are fundamentally about the relationship between individuals and the organisations they work for, and in some senses represent. Do organisations and their positions appear weaker because individuals who work for them hold and express contrary views? I think not. Anyone who has ever worked for an organisation knows that not everyone agrees with everything the organisation says or does. The development of organisational positions is a process of discussion, debate, negotiation and challenge. That organisations draw on, and work with a range of different view points and values is a strength not a weakness, and to surface those differences provides evidence of that strength. And if an organisation's position can't withstand the debate generated by expression of a contrary view, then maybe the position isn't the right one after all.

Some organizations demand total fealty, and often that means never questioning those in authority.

Those organizations are ultimately doomed.

Respectfully challenging the status quo, combined with relentlessly iterating new ideas is the hallmark of the vibrant tribe.

Seth Godin: Confusing Loyalty with Silence

 

Posted in writing | Tagged | 3 Comments

The many paths to PhD study

I presented at the Vitae Researcher Development International Conference today. My slides have been published and this post is a summary of the issues covered in my talk.

Every year more than 20,000 students embark on a journey towards qualifying with a PhD. As part of that journey they make a significant contribution to the research endeavour. And when they graduate they progress into all sorts of careers, inside and outside of research, here and overseas, bringing the benefits of doctoral training in a myriad of ways.

It is hugely important that we set a policy environment that ensures that talented people with the commitment and aptitude for PhD study make the successful transition, with the minimum of barriers, and certainly no barriers that lock people out of this course for reasons outside of their control. To set this policy environment we need to understand how and when the transition happens, who makes the transition and what motivates them.

New data and new analysis is illuminating the transition process, and it turns out to be more complex than many might think. HEFCE analysis published last year demonstrates that there are some students who transfer directly from their undergraduate courses within a year or two, and without acquiring any additional postgraduate qualifications. But for others there is a much longer gap between their first degree and PhD study, often including a period studying a taught postgraduate course.

The students who transfer directly have some particular characteristics. They are mostly male, mostly study STEM subjects, and mostly graduate from more selective institutions in terms of undergraduate entry.

New analysis has shown that the group who begin their PhD studies after a period of taught postgraduate training are more likely to be in the social sciences or humanities, but this is not exclusively the case. For example, around a third of the students starting a PhD in the Life Sciences have a prior postgraduate qualification. Those with prior qualifications also tend to have longer gaps between their last period of study and starting on PhD. In contrast to the direct transferrers, these students are moving from study to the workplace and back again over an extended period before PhD study.

Understanding the many and varied pathways to PhD is important to the policy goal of minimising barriers to the pursuit of doctoral study. The challenges facing students taking the longer, more complex routes involving taught postgraduate study will be different from those moving directly from undergraduate courses. There will be different financial challenges, for example. Even navigating the processes of recruitment and selection for PhD places may be more complex for those students doing so from a position outside of the Higher Education sector. These issues are strongly confirmed by a qualitative study of the recruitment and selection of PhD students carried out for HEFCE by CRAC.

Further insight into the transition into PhD study is also coming from the new Intentions After Graduation Survey (IAGS). This survey, part of the National Student Survey, is providing important information on the proportion of final year undergraduates who are planning postgraduate study, and their motivations for doing so.

According to these data, some 44% of final year students are either 'certain' or 'likely' to progress to postgraduate study, and around a fifth of these express an intention to study for a doctoral degree. A further 14% are planning a postgraduate taught course as a step towards a PhD. This information will be an important baseline for discerning any effects of the undergraduate fee regime on intentions for further study. But the survey is also revealing important information about the transition. Some ethnic groups, for example, have a very much higher expressed intention to progress to PhD study than is achieved in reality, helping to pinpoint potential barriers to progression.

So there are many paths involved in progression to PhD study, each with their own issues and possible barriers. Does the policy environment appropriately cope with this diversity? At one level the answer is yes – the evidence published today suggests that there is no shortage of well-qualified candidates applying for places. But in other respects we are less successful. There are still issues with the diversity of the PhD student population; in terms of gender, ethnicity, disability and socio-economic background the population does not reflect the representation in the undergraduate population from which they are drawn. To address these issues a better, more nuanced understanding of the pathways of transition is essential.

 

Posted in research policy | Tagged , | Leave a comment

On commonplace books

If you are looking to kill some time try Googling 'commonplace book'. You will find an endless stream of fascinating material, both examples of commonplace books themselves, and writing about commonplace books, including, eventually this post.

The commonplace book is a seventeenth century innovation, and the idea is a simple one: A notebook for capturing interesting quotes from reading, ideas, snippets of text for writings, diagrams, sketches, anything that comes to mind. Over time these notebooks developed into personal anthologies of thought and reflection, and were often accompanied by elaborate schemes of indexing, so that the entries could be located and themes extracted.

The age of the internet has the potential to be the golden age of the commonplace book. First we have an unprecedented opportunity to read and access texts of all sorts, and secondly it is simple – no more complicated than 'copy and paste' – to bring elements of text together into places where search tools allow the rapid compilation of themes.

I have been using Evernote as a commonplace book for a number of years. All sorts of things get saved into my Evernote notebooks, some of them automatically, and then the search function allows later retrieval. For example, a quick search for 'commonplace book' reveals that, rather spookily I was contemplating drafting a blog post on the topic exactly a year ago today. I was also able to identify previous reading I had done about commonplace books, and a quote from 'Where Good Ideas Come From: The Natural History of Innovation' by Steven Johnson:

The great minds of the period—Milton, Bacon, Locke—were zealous believers in the memory-enhancing powers of the commonplace book. In its most customary form, “commonplacing,” as it was called, involved transcribing interesting or inspirational passages from one’s reading, assembling a personalized encyclopedia of quotations.

The search also provided me with a link to a related piece I had read by James Gleick on digitising books.

The commonplace book was a powerful idea in the seventeenth century but digitised text takes it to a new level. This idea is explored and developed further by Johnson in a blog post. In this post Johnson points out that searching for text can, in an instant, assemble a type of commonplace book using an algorithm. The google search I linked to at the beginning of this post is an example. The search results are presented in a particular order, and to an extent that order is customised to the individual. A new association of words and ideas is being created, specific for the reader, and in no way predictable by the authors of the original texts:

What you see on [a Google search results] page is, in a very real sense, textual play: the recombining of words into new forms and associations that their original creators never dreamed of.

Johnson goes on to consider the value that is created through these new combinations of text:

When text is free to combine in new, surprising ways, new forms of value are created. Value for consumers searching for information, value for advertisers trying to share their messages with consumers searching for related topics, value for content creators who want an audience. And of course, value to the entity that serves as the middleman between all those different groups.

And there is a following discussion on paywalls and technologies that prevent text to be mined and combined in new ways. The whole, long post is well worth a read. His conclusion is that access to text and reasonable re-use rights are central to ensuring that the potential benefits of the internet-enabled commonplace book. In Johnson's words we need text to be in a commonplace book, not a glass box.

This is one of the reason that open access to the scholarly literature is so important. At the moment much of the scholarly literature is, at best, in a glassbox and at worst in a locked chest for which only a select few hold the key. Not only does the scholarly literature need to be made more available, but also licensed in such a way that re-use and re-purposing is possible. As Cameron Neylon has recently argued permissive licensing is essential. Access through glass boxes, like the Access to Research initiative is also deeply limited in its value.

I wonder what those seventeenth century 'commonplacers' would make of all this. I think they would be amazed by the potential of the digital commonplace book, but shocked to see how we have locked away some of the most valuable text, preventing real value to be obtained.

 

Posted in research policy | Tagged , | 1 Comment

Openness and research integrity

Last week I spoke at the annual conference of the UK Research Integrity Office. The slides I used have been published, and this post sets out the argument that I made.

Research misconduct happens. There is less certainty about how much research misconduct, but there is no doubt that we need a policy response to address the issue. In formulating this response it is important to acknowledge that the term ‘misconduct’ covers a range of activities, with different motivations, and levels of seriousness. And that this variation will influence the policy response.

At one end of the spectrum are deliberate acts by researchers to falsify or misrepresent research. I believe this is rare, although it often the type of misconduct that gets the most publicity. Further down the range are small acts of misconduct that, while deliberate, aren’t motivated by a desire to falsify the conclusions of the research. In this category I would include leaving out inconvenient data points, or perhaps even inconvenient experiments, that challenge conclusions that the researcher strongly believes, often on the basis of other evidence, to be true. At the other end of the spectrum is less deliberate misconduct, that stems from lack of knowledge or expertise on the part of the researcher, or possibly the cutting of corners to speed up the research process. This is accidental misconduct where the researcher doesn’t realise that anything is wrong.

Regulation – rules, processes to enforce them, and sanctions applied when they are broken – clearly has an important part to play across this spectrum, and is probably the only policy option for tackling deliberate and large-scale fraud. However, in tackling most misconduct there is an essential and important role in ensuring that we have the right culture within our universities and research organizations. We need to focus on stopping misconduct from happening; catching misconduct after the event is both a very costly approach, but it is also less effective to allow poorly conducted or fabricated research to be disseminated.

If we accept that tackling misconduct is about changes to the whole culture within which research is conducted, then rather than being a specific ‘policy issue’ it becomes a cross-cutting theme that underpins the whole research policy agenda. We need to consider the implications for the research culture of all the policy interventions we make: from the design of funding processes, to the mechanisms of research assessment. The current investigation by the Nuffield Council on Bioethics will, no doubt, provide some useful insights into the relationship between policy interventions and the research culture that emerges.

There is one aspect of the research culture that stands out for me – the extent to which research is conducted openly, and especially the extent to which research data is made available for scrutiny and further analysis. As has been eloquently argued by Andrew Rawnsley there is, or ought to be, an intimate link between research integrity and research openness.

This is not just because the sharing of data allows other researcher to check the findings of research. There is also reason to believe that a presumption of openness will influence research cultures in a positive way. For example, a study carried out in 2011 by Wicherts, Bakker and Molenaar concluded that:

the reluctance to share data [was] associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

As these authors rightly point out the study describes a correlation, and it is not possible to draw a causal inference. They propose a number of possible explanations, but one is particularly compelling:

statistically rigorous researchers may archive their data better […and,so…] will more promptly share their data

There could be a deep cultural linkage between the rigour with which data is analysed, the care with which data is curated and the willingness to share. Or maybe the act of sharing itself encourages a further review of the data and its analysis, a point recently well made by Dorothy Bishop. Either way, data openness should be part of the research culture that supports and fosters research integrity.

As I have written previously, opening up research data is not without its challenges. But I would argue that the benefits that more openness will bring for research integrity are significant, and provide part of the case for universities, funders and journals intervening to foster the culture of open research.

Posted in research policy | Tagged , , | 3 Comments

How many research mavericks do we need?

The topic of ‘scientific mavericks’ is receiving some attention at the moment. There was a letter, signed by a number of Nobel laureates and FRSs among others, published this week in the Guardian suggesting that the current structures inhibit mavericks, and that this is a problem. And this follows on the heels of comments by Peter Higgs last year. The arguments in the Guardian letter are flawed in my view (there is an excellent post by Jenny Rohn that sets out why), and based on a very particular reading of history.

But the question of providing space within the research policy landscape for the exploration of radical ideas and approaches is an important one.

Of course, in the policy world we more often talk about ‘high risk/high reward’ research rather than mavericks, but the basic concept is the same. The question that is regularly debated by research policy people is how to ensure that the structures and environment we create around research don’t produce barriers that prevent researchers from trying out ‘crazy ideas’. In thinking about how to tackle this question there are a number of challenges. A big one is that we don’t actually know how much ‘high risk/high reward’ research is going on. As Jenny indicates in her post, researchers find ways to do this type of work, and, by definition, most of it will fail, so we will never hear about it. The successful projects – the extraordinary findings supported by extraordinary evidence – can potentially look very mainstream quite quickly, so the visibility of successful ‘high risk/high reward’ research can be low too. In the absence of good evidence on the volume of activity it’s hard to know whether the current policy environment is really encouraging or inhibiting.

By far the biggest challenge in formulating policy for ‘high risk/high reward’ research, however, is deciding how much research of this type is needed. I think there definitely should be space for some, but it is also self-evident that we don’t want all research to be like this. That would result in too much public money being spent on research that leads nowhere, not even advancing its discipline, let alone bringing wider benefits to society. As in many areas of research policy, we need to find the appropriate balance point.

Part of the responsibility for addressing this issue lies with researchers and universities themselves. Through the UK’s dual funding system, universities have access to funding, provided by HEFCE and the other UK funding bodies, that could be used to support ‘high risk/high reward’ research. Whether the block grant funding is used in this way, and, if it isn’t, why not, are interesting questions worth exploring. It is the case that researchers and universities are themselves well placed to decide how much ‘high risk/high reward’ research should take place. Perhaps increasing the proportion of research funding that is distributed through this route might shift the balance, and a allow a few more mavericks to be nurtured?

Posted in research policy | Tagged , , | Leave a comment