Earlier this month Times Higher Education published their annual collection of data on Research Council grant application success rates. They report a drop in success rates, and this has, rightly, triggered some debate on the effort expended on unsuccessful applications, and potential alternative approaches to allocating funding.
Central to any discussion on the amount of effort expended on unsuccessful applications is an estimate of the time it takes to write a grant application. The Times Higher piece reports an average figure of 171 hours per application, which is based on a study published last year in PLOS-ONE, and authored by Ted and Courtney von Hippel .
My initial reaction was that this estimate seemed rather high, certainly based on my own, albeit somewhat out-of-date, experience of writing grant applications. I certainly can't recall spending over 4 weeks of 8-hour days on a single application. And the estimate is nearly double the average of 12 days reported in work carried out for RCUK in 2006.
Taking a closer look at the PLOS-ONE paper, reveals that the analysis is beset by the problems of analysing the time people spend on activities. The study is based on a sample size of just 195, drawn from only 2 disciplines (astronomy and psychology). Most significantly, the time estimates are based on self-reported time spent, recorded after the fact, and without prior warning that this time estimate would be requested. There is a large literature that demonstrates how unreliable people are in recalling the time they spend on activities. In this study it is made worse by the fact that the 171 hours estimate is made up of two components - the time spent by the Principal Investigator (PI) on the grant (116 hours) and the time spent by other researchers (55 hours). But both of these figures are estimated by the PI, so the second figure is even more unreliable than the first.
As might be expected, there is big variation in the time estimates provided. The paper reports standard deviations on these estimates of 97 hours and 79 hours, respectively. The data is available for the PLOS-ONE paper, so I delved a bit deeper. The ranges of time estimates are large, 10-1000 hours for PI time and 0-6000 hours for other contributors, and the distributions are skewed with medians being quite a bit lower than the mean for both estimates.
My conclusion from this is that the paper tells us little about how much time it takes to write a grant application. At best it gives some sense of the wide range of times, but even then the survey method is likely to represent a biased estimate. I certainly think reporting the 171 hours figure, with its spurious accuracy and without the context is a little misleading.
Where does this leave us in terms of understanding the time investment in writing grant applications? The study described above has a median PI time of about 100 hours (around 12 days) which isn't far from the median value of 10 days from the RCUK study. Both of these are much lower than the 38 day estimate from a third study. But all three of these estimates suffer from the same limitation - they are based on self reported survey data, and show there is huge variability in the reported times. The variability will be due to a combination of actual variation in the time taken, and differences in reporting. We really don't have a systematic answer to this important question.
Before we can make sensible policy choices about what is an acceptable success rate we need better evidence. A study needs to include a large enough sample to account for differences between disciplines, career stage, and types of grant application. And there needs to be a less subjective approach to time estimation. Technologies that monitor activity directly, or that gather data from random, what-are-you-working-at-now prompts. As success rates fall a robust and reliable estimate is needed more than even.
