Over recent months many of us have become accustomed to conducting all manner of business through video calls. Tasks that, we would have argued, had an absolute requirement to be conducted face-to-face have transferred almost seamlessly to an online or virtual context. From a personal perspective, I have found this transition to be an effective one, with business being conducted as effectively, and sometimes more so.
But it is important to challenge perceptions and seek evidence about the effectiveness of virtual meetings, especially when face-to-face interactions are central to a process. An example that demands particular scrutiny in the context of research policy is the use of panel meetings for peer review. Panel meetings are at the heart of funding decisions for research projects, and play a part in some research evaluation processes. As we get used to living with an dangerous and endemic virus, the conduct of peer review panels remotely is set become the norm. We need to understand the effectiveness of this new setting so that we can mitigate any negative effects and ensure the robustness of processes.
All peer review panels for grant proposal review operate in a similar manner. Panel members read the proposals in advance of the meeting and assign a provisional score or rating to them. A subset of the panel focus on particular proposals as the lead reviewer or reader, in order to manage and distribute the volume of work. At the panel meeting the panel as a whole review each proposal, listening to the views of the lead reviewer(s) and coming to a collective and agreed score or rating. While not always stated explicitly, the purpose of the panel discussion is to bring a greater diversity of views to bear and to mitigate against any implicit or unconscious biases that the lead reviewer(s) might have.
The extent to which panels are effective in delivering their objectives is an important question. However, in this post I want to focus on the performance of virtual panel meetings compared to the more usual face-to-face setting. From the evidence I have looked at (see the annotated bibliography below) I draw four conclusions.
There is limited evidence on the effectiveness of virtual panel meetings. There are only a handful of studies that have examined this question systematically, and they are all concerned with the assessment of medical research funding applications. The disciplinary focus is perhaps not a big problem, but the small number of studies does limit the extent to which the findings can be confidently generalised. It is also worth noting that none of the studies examined the effectiveness of the panel process directly; they did not ask whether the correct funding decisions were made. Instead two types of information have been collected: the perceptions of panel members of the effectiveness of the process, and data relating to the panel operation, like the extent to which scores were adjusted or the time spent discussing each application. The studies were exclusively in the North American context.
In general, panel members prefer face-to-face meetings to virtual meetings. Surveys of panel members reveal a preference for face-to-face meetings. Panel members suggested that better communication, both during the formal parts of the meeting and in the spaces around the formal sessions, is driving their preference. The minority of panel members who preferred virtual meetings did so because of the logistical convenience.
Despite their preferences, panel members do not report major differences in effectiveness between face-to-face and virtual meetings. Survey evidence did not reveal statistically significant differences in panel members perception of the effectiveness of the process using tele- or videoconference methods. Effectiveness was assessed through questions relating to use of expertise, quality of discussion and its facilitation, and the perceived robustness of the outcomes. There were some perceived reductions in quality for 'web-based meetings'. A precise definition of 'web-based meetings' wasn't provided, but the implication is that this was a text-only format not involving an audio or video link.
Virtual panel meetings have similar outcomes, but there is less time spent on discussing proposals Measures of panel process are similar for face-to-face and virtual meetings. For example, the extent to which scores were adjusted from the reviewers personal scores after discussion was similar in different settings. The only notable difference identified is that discussions tended to be shorter in virtual settings. This may be important, as panel members perceive that shorter discussions are more likely to result in biased outcomes. However, there is no evidence that this is the case from the scoring distributions, and further work would be needed to explore this matter fully.
There does not appear to be significant evidence to suggest that remote peer review panels are less effective than those conducted face-to-face. However, the evidence base is small and limited in scope, and is focussed on perceptions of panel members, and measures of the mechanics of panel processes. The current shift to more virtual panel meetings presents an opportunity to improve the evidence base on this important question.
Finally, the evidence available is limited to panels assessing grant proposals. I was not able to find any studies looking at the less common use of panels for the ex post evaluation of research outputs and impacts, such as used in national research evaluations like the UK's Research Excellence Framework. With the likelihood that the assessment phase of the 2021 REF will be conducted at least partially via remote meetings, there is an opportunity to gather new evidence for this different context.
If you are aware of relevant evidence that I have missed, please add a comment below.
Annotated bibliography
Primary research
Gallo, S. A., Carpenter, A. S., & Glisson, S. R. (2013). Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes. PLoS ONE, 8(8), e71693. https://doi.org/10.1371/journal.pone.0071693
This paper reports on a difference-in-difference approach to examine the effect of switching from face-to-face grant panel meetings to meetings conducted by teleconference.
The study examines a number of metrics of the peer review process and finds no significant differences,
with the exception of a reduction in discussion times for meetings conducted by teleconference.
Surveys of panel members did not reveal any differences in their perceptions of the robustness of the process.
Carpenter, A. S., Sullivan, J. H., Deshmukh, A., Glisson, S. R., & Gallo, S. A. (2015). A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels. BMJ Open, 5(9), e009138. https://doi.org/10.1136/bmjopen-2015-009138
This study uses the same dataset and approach as the authors previous work, above.
In addition to confirming the findings of the previous study,
this analysis shows that there is a small reduction in the changes between pre-meeting and post-discussion scoring in the teleconference environment.
Gallo, S. A., Schmaling, K. B., Thompson, L. A., & Glisson, S. R. (2020). Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Research Integrity and Peer Review, 5(1). https://doi.org/10.1186/s41073-020-00093-0
This paper reports on a survey of reviewer perceptions of the role of panel discussion in grant peer review outcomes.
It reports that many panel members think that the discussion is valuable,
but there are concerns about poor and/or short discussions not addressing or even accentuating bias.
The central role of the panel chair in facilitating discussions is a key factor for panellists,
accompanied with evidence that the role is sometimes perceived to have been carried out poorly.
Training for panel chairs is recommended as an outcome from the research.
There is only passing mention of remote panel meetings,
with evidence that shifts in scoring and the length of discussions tend to be shorter in virtual settings.
Gallo, S. A., Schmaling, K. B., Thompson, L. A., & Glisson, S. R. (2019). Grant reviewer perceptions of panel discussion in face-to-face and virtual formats: lessons from team science? bioRxiv https://www.biorxiv.org/content/10.1101/586685v2?versioned=true
Note: This is a preprint version of Gallo et al. (2020) that is substantially different and contains more information on virtual panels.
Based on a survey of panel members,
this paper reports a preference among panel members for face-to-face meetings,
but no evidence to suggest that panellists felt that the quality of discussion was significantly impacted when face-to-face is compared to tele- or video-conferencing.
Pier, E. L., Raclaw, J., Nathan, M. J., Kaatz, A., Carnes, M., & Ford, C. E., (2015). Studying the study section: How group decision making in person and via videoconferencing affects the grant peer review process (WCER Working Paper No. 2015-6). https://wcer.wisc.edu/docs/working-papers/Working_Paper_No_201506.pdf
_This study examines the effects of discussion on score adjustment,
compared different panels' assessment of the same proposals,
and examined whether there were any differences between face-to-face and video conference panels
The panel meetings adopted NIH procedures,
but used already reviewed proposals,
and did not lead to any changes in outcome for those proposals.
There were three face-to-face panels and one by videoconference.
There was no evidence of any difference between the face-to-face and virtual panels,
except less time was spent on each proposal in the virtual panel.
Some panel members perceived that the face-to-face meetings were more effective,
although the evidence does not support the existence of material differences.
Vo, N. M. & Trocki, R. (2015). Virtual and Peer Reviews of Grant Applications at the Agency for Healthcare Research and Quality. Southern Medical Journal, 108(10) 622-626
This paper reports on six review panel meetings that were conducted via video conference because of hurricanes in 2012.
The virtual panels are compared to five face-to-face panels.
The report concludes that there was no evidence of reduced effectiveness in the virtual panels.
Review articles
Guthrie, S., Ghiga, I., & Wooding, S. (2018). What do we know about grant peer review in the health sciences? F1000Research, 6, 1335. https://doi.org/10.12688/f1000research.11917.2
This review article summarises evidence available on peer review, with a focus on the life and health sciences.
A range of areas of evidence are covered including the use of remote meeting formats for peer review.
Shepherd, J., Frampton, G. K., Pickett, K., & Wyatt, J. C. (2018). Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PLOS ONE, 13(5), e0196914. https://doi.org/10.1371/journal.pone.0196914
This paper reports a systematic review of literature examining the effectiveness of innovations in peer review.
It includes a review of evidence on using remote meetings for peer review.
